Incident: Servicedesk Offline (Resolved)

Summary of Issue: Servicedesk Offline
Impacted Audience: All Users
Impact of Issue: Effects Support
Event Start: 22:00 BST on 08/09/2019
Incident Last Updated: 13:21 BST  on 11/09/2019
Estimated Event End: 13:21 BST  on 11/09/2019
Affected Systems: Helpdesk

Update #1: We have had to restore the system from a backup, which has resolved this issue. The Servicedesk is now back online and is running normally again.

Sorry for the inconvenience this issue has caused.

**************************

We’re currently investigating why our Helpdesk System has gone down. At the moment, this means that users are unable to open support tickets via the portal. However, emails sent to the Helpdesk are not affected by this, nut will not be processed until the system is back up and running.

Sorry for the inconvenience and please watch this page for further updates.

 

Continue reading »

Incident: Problems with Airsuite Legacy Platform (Resolved)

Summary of Issue: Problem with Airsuite Legacy Broadcast Platform
Impacted Audience: Presentation and Production Team
Impact of Issue: Effects Leagacy Broadcast Platform Users
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 7-09-2019 @ 14:30 BST
Estimated Event End: 17-09-2019 @ 14:30 BST
Affected Systems: Airsuite Legacy Broadcast Platform

#Update 1

We now know why users are unable to encode to the server. This is because of legacy applications, which are not compatible with Icecast2 and therefore, cannot encode to the server.

We suggest that users download and install the freeware application: EddCast. (Please see here for instructions on how to do this and how to setup your encoder.

Sorry for the confusion with this issue.

******************

We are currently investigating a problem with the Airsuite Legacy Broadcast Platform. It seems that presenters who are set to use the Legacy Platform are unable to connect to the server. This issue seems to be affecting multiple users and means their shows are not being broadcast at present.

Our host has been made aware of this and we will update this page with further information soon.

Sorry for the inconvenience.

Continue reading »

Email System Currently At Risk (Resolved)

Summary of Issue: Email \ Security Issue
Impacted Audience: Presentation and Production Team
Impact of Issue: Effects All Staff \ Visitors
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated:17-07-2019 @ 13:18 BST
Estimated Event End: 26-07-2019 @ 13:00 BST
Affected Systems: Email Systems

Update #2

This issue has seemed to of resolved itself. We are still unsure why this happened and what has caused this issue. Our email system has been fine for over a week and we are satisfied that this issue will not cause our email system to go down again.

However, please raise a ticket if you encounter any additional issues with our email services.

Sorry for the inconvenience this issue has caused.

******************

Update #1

Our host has said that this issue is possibly being caused by a malicious script on the server, which has caused this to happen. However, we have not found evidence of such a script and we are continuing to investigate this issue.

Our host has restored a backup file, which has bought the email system back online again. However, at this time, Email services should be treated as “at risk”, until further notice.

Sorry for the inconvenience, this ongoing issue is causing.

******************

We are currently investigating a major problem with our email system. Last Sunday, someone gained unauthorized access to our server and deleted all mailboxes from the server.  Our Host ran scans on the system, to check for spyware; but that came back clear. Our Host was able to restore the server from a backup and I had changed the Server Root Password and other system passwords.

However, the password change had little effect (even though we used a strong password) and once again the server has been attacked! Once again, our email system has been compromised again. As far as we know, there has not been a data breach, as the hackers simply have removed everyone’s mailboxes from the server. Each mailbox is password protected, so who ever it was that has broken into the server; would not of had access to individual email inboxes. However, we know that the unauthorised access to our server was performed using SSH. (Secure Socket Layer, which is similar to the DOS interface on Windows Operating Systems). Even though we have changed the Root Password, (which is the server master password)  it seems that who ever has attacked the server has managed to somehow break into the server. We have checked the userlist, by running the following command.

cut -d: -f1 /etc/passwd

However, this did not throw up anything unusual and we at this moment in time, we have no idea who is accessing the sever and how they are getting into the system.

Our host is now aware of the problem and we hope to have this issue resolved as soon as we can. However,  as this has happened for the 2nd Sunday in a row, IT Services have decided to keep this status report open, so that if there is a repeat of this issue, we do not need to post a duplicate status report.

This issue means that our email system is currently at risk for all users and that means that our email system is currently at risk of further outages.

We’ll keep you posted on this issue.

Sorry for the inconvenience this causes.

Continue reading »

Incident: Problems with Live Shows Using Airsuite (Resolved)

Summary of Issue: Problems with Airsuite kicking Presenters of air
Impacted Audience: Presentation and Production Team
Impact of Issue: Affects Department
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 14-07-2019
Estimated Event End: 14-07-2019 @ 15:47
Affected Systems: Airsuite Users

Update: #1

Our host has come back with the following information about this issue:

Unfortunately, as the AirSuite Broadcaster is web based and due to the limitations of some devices we are unable to guarantee uninterrupted streaming when using the Web Based AirSuite Broadcaster, therefore this is provided as an ‘as is’ service with the Third Party option as the recommended option for presenters that experience disconnection issues.

So due to this, we advise Presenters using Laptops and slower devices, to use th 3rd party encoder option, such as Butt or another 3rd party encoder. To do this, you will need us to enable Legacy Broadcasting on your Airsuite Account. This means that you will need to raise a support ticket, so we can enable this for you.

Sorry for the inconvinience.

*************************

There are some issues with Airsuite, which causes Presenters to be disconnected from their shows at random. We have our host investigating this issue and will post an update, as soon as possible.

Please note: This issue does not affect users using the Legacy Broadcast Platform.

Sorry for the inconvenience.

Continue reading »

[Resolved] Incident: Issues with Remote Relays

Summary of Event: Issues with remote relay streams not connecting as they are scheduled to
Impacted Audience: Presenters relaying live syndicated streams, listeners
Impact of Event: Affects Live Relays
Event Start: 01:52 on November 6th 2018
Estimated Resolution: 13:00 on November 10th 2018
Event Duration: 13 Hours 52 Minutes
Affected Systems/Services: Radio.co
Incident Last Updated: 13:06 on 10th November 2018

********************************

Update #2: Our host think that they have found the problem and have reset the service that controls the remote streams. We are now monitoring this issue and we hope that tommrow afternoon, we may have this issue fully resolved.

Thank you

********************************

Update #3: The reset of the remote relay service seems to have resolved the issues with remote relay streams. This means that shows they relay their shows to us, now are up and running again.

This incident has now been closed.

Thank you.

********************************

Update #1: Our host has escalated this issue, so that they can find the cause and get the problem fixed, by their Development Team. We will post further updates as we receive them. However, it only seems to be affecting a couple of the remote streams and not all remote streams. This issue also does not affect the station output as a whole.

Thank you.

********************************

There are some issues with remote streams this morning, which means that remote relays cannot be aired at this time. We have alerted our host and are awaiting them to find a resolution to this issue. Please be aware that this issue does not impact the station output as a whole and is impacting a small group of users that relay shows. Our backup  playlist will kick in the meantime, while these issues are resolved.

We’ll update this page, when we have any further information on this on this issue. If you have any questions on this issue, please email: helpdesk@shoutradio.org.uk quoting #INC-6051118

Thank you.

 

 

Continue reading »