Live Shows From HQ Currently Suspended

Summary of Issue:  Network Security Breach
Incident Status: Under Investigation
Incident Priority: Priority 1 (Critical)
Impacted Audience: All Network Users at Shout HQ
Impact of Issue: Affects Organization:
Event Start: 1700 BST on 05-04-2021
Incident Last Updated: 
Estimated Event End: –
Affected Systems: Central File Server, HQ IT Systems

We are aware of a serious breach of Network Security at Shout Radio HQ. There has been a unauthorized  attempt to purge data off of one of the Server Hard Drives, resulting in data loss. This data does not include any personal information, but we are still treating this as a serious security incident and we we have a duty to investigate.

While the IT Department conduct these investigations, All Shows broadcast from Shout HQ have been temporary suspended. with immediate affect. This only affects a handful of shows and the bulk of programming will be unaffected by this suspension. In the meantime, the Central File Server has been disconnected from the network, so that evidence can be preserved, which will aid the investigation.

We will update you on this incident with more information as we get it.

Sorry for the inconvenience.

Incident: Slow Network Speeds at HQ

Summary of Issue: Office Network Partial Outage
Incident Status: Closed
Incident Priority: Priority 2 (Urgent)
Impacted Audience: All Network Users at Shout HQ
Impact of Issue: Affects Organization:
Event Start: 23:00 GMT on 17-02-2021
Incident Last Updated: 18/02/2021 at 16:20
Estimated Event End: 18/02/2021 at 16:0
Affected Systems: Office Network: Wired Network, Office Network: Print Services


Update #2 (18th February 2020 @16:20 GMT)

We are still investigating the reasons for slow network access at Shout HQ. We have determined that the issue is NOT being caused by a faulty adaptor, but is actually caused by poor network speeds. The other parts of the building now have network access, but it is very slow. Network Printing is also now back up and running, so users should be able to print, to the network printer again.

In the meantime, the Wired Network will now be classed as “degraded”, as there is still a risk that the wired network may partially go down, due to the issues we are experiencing.

We’ll keep investigating these issues and will update this page, when we have more information.

Apologies for the prolonged inconvenience this issue may have caused you.


Update #1: (17th February 2021 @11:32 GMT) The adapter has been shipped and will be with us tomorrow.

Sorry for the inconvenience this issue has caused.


We are aware of some issues with the Office Network, causing a partial outage. This has been caused by a faulty In-line network Adapter. This has caused very slow Network and Internet speeds, in parts of the building. (On the Wired Network) This incident also effects Network Storage Services, from the affected areas of the building. Additionally, users are unable to print to the network printer at present; as the printer is in the part of the building; which affected by the outage on the wired network.

We have ordered a new adapter, which should be with us before the weekend.  However, this issue means that there is no access to the network, in some parts of the building. You can use Wi-Fi to access the network,  while this issue is resolved.

Please keep an eye on this page for further updates, on this issue.

Sorry for the inconvenience, this causes.

Incident: Airsuite Server Failure

Summary of Issue: Airsuite Server Faliure
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Station Staff \ General Public
Impact of Issue: Affects Multiple Departments
Event Start: 20:15 GMT on 27/12/2019
Incident Last Updated: 23:23 GMT on 27/12/2019
Estimated Event End: 23:23 GMT on 27/12/2019
Affected Systems: All Airsuite Services


Update #1: This issue has now been resolved and was caused by an issue with the DNS on the server.

Sorry for the inconvenience this issue has caused.


We’re currently investigating an issue which has taken our Airsuite Server offline. We are currently not sure why this has happened,  but we are working on getting the station back on air as soon as possible.

Sorry for the inconvenience and please watch this page for further updates.

Incident: Servicedesk Offline

Summary of Issue: Servicedesk Offline
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Station Staff \ General Public
Impact of Issue: Affects Organization
Event Start: 22:00 BST on 08/09/2019
Incident Last Updated: 13:21 BST  on 11/09/2019
Estimated Event End: 13:21 BST  on 11/09/2019
Affected Systems: IT Servicedesk


Update #1: We have had to restore the system from a backup, which has resolved this issue. The Servicedesk is now back online and is running normally again.

Sorry for the inconvenience this issue has caused.


We’re currently investigating why our Helpdesk System has gone down. At the moment, this means that users are unable to open support tickets via the portal. However, emails sent to the Helpdesk are not affected by this, nut will not be processed until the system is back up and running.

Sorry for the inconvenience and please watch this page for further updates.

 

Incident: Problems with Airsuite Legacy Platform

Summary of Issue: Problem with Airsuite Legacy Broadcast Platform
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Presentation and Production Team
Impact of Issue:  Affects Department
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 7-09-2019 @ 14:30 BST
Estimated Event End: 17-09-2019 @ 14:30 BST
Affected Systems: Airsuite Legacy Broadcast Platform

#Update 1

We now know why users are unable to encode to the server. This is because of legacy applications, which are not compatible with Icecast2 and therefore, cannot encode to the server.

We suggest that users download and install the freeware application: EddCast. (Please see here for instructions on how to do this and how to setup your encoder.

Sorry for the confusion with this issue.


We are currently investigating a problem with the Airsuite Legacy Broadcast Platform. It seems that presenters who are set to use the Legacy Platform are unable to connect to the server. This issue seems to be affecting multiple users and means their shows are not being broadcast at present.

Our host has been made aware of this and we will update this page with further information soon.

Sorry for the inconvenience.

Incident: Email System Currently At Risk

Summary of Issue: Email \ Security Issue
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: All Staff
Impact of Issue:  Affects Organization
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated:17-07-2019 @ 13:18 BST
Estimated Event End: 26-07-2019 @ 13:00 BST
Affected Systems: Email Systems


Update #2

This issue has seemed to of resolved itself. We are still unsure why this happened and what has caused this issue. Our email system has been fine for over a week and we are satisfied that this issue will not cause our email system to go down again.

However, please raise a ticket if you encounter any additional issues with our email services.

Sorry for the inconvenience this issue has caused.


Update #1

Our host has said that this issue is possibly being caused by a malicious script on the server, which has caused this to happen. However, we have not found evidence of such a script and we are continuing to investigate this issue.

Our host has restored a backup file, which has bought the email system back online again. However, at this time, Email services should be treated as “at risk”, until further notice.

Sorry for the inconvenience, this ongoing issue is causing.


We are currently investigating a major problem with our email system. Last Sunday, someone gained unauthorized access to our server and deleted all mailboxes from the server.  Our Host ran scans on the system, to check for spyware; but that came back clear. Our Host was able to restore the server from a backup and I had changed the Server Root Password and other system passwords.

However, the password change had little effect (even though we used a strong password) and once again the server has been attacked! Once again, our email system has been compromised again. As far as we know, there has not been a data breach, as the hackers simply have removed everyone’s mailboxes from the server. Each mailbox is password protected, so who ever it was that has broken into the server; would not of had access to individual email inboxes. However, we know that the unauthorized access to our server was performed using SSH. (Secure Socket Layer, which is similar to the DOS interface on Windows Operating Systems). Even though we have changed the Root Password, (which is the server master password)  it seems that who ever has attacked the server has managed to somehow break into the server. We have checked the userlist, by running the following command.

cut -d: -f1 /etc/passwd

However, this did not throw up anything unusual and we at this moment in time, we have no idea who is accessing the sever and how they are getting into the system.

Our host is now aware of the problem and we hope to have this issue resolved as soon as we can. However,  as this has happened for the 2nd Sunday in a row, IT Services have decided to keep this status report open, so that if there is a repeat of this issue, we do not need to post a duplicate status report.

This issue means that our email system is currently at risk for all users and that means that our email system is currently at risk of further outages.

We’ll keep you posted on this issue.

Sorry for the inconvenience this causes.

Incident: Problems with Live Shows Using Airsuite

Summary of Issue: Problems with Airsuite kicking Presenters of air
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Presentation and Production Department
Impact of Issue: Affects Organization
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 14-07-2019
Estimated Event End: 14-07-2019 @ 15:47
Affected Systems: Airsuite Users


Update: #1

Our host has come back with the following information about this issue:

Unfortunately, as the AirSuite Broadcaster is web based and due to the limitations of some devices we are unable to guarantee uninterrupted streaming when using the Web Based AirSuite Broadcaster, therefore this is provided as an ‘as is’ service with the Third Party option as the recommended option for presenters that experience disconnection issues.

So due to this, we advise Presenters using Laptops and slower devices, to use th 3rd party encoder option, such as Butt or another 3rd party encoder. To do this, you will need us to enable Legacy Broadcasting on your Airsuite Account. This means that you will need to raise a support ticket, so we can enable this for you.

Sorry for the inconvenience.


There are some issues with Airsuite, which causes Presenters to be disconnected from their shows at random. We have our host investigating this issue and will post an update, as soon as possible.

Please note: This issue does not affect users using the Legacy Broadcast Platform.

Sorry for the inconvenience.

Incident: Issues with Remote Relays

Summary of Event: Issues with remote relay streams not connecting as they are scheduled to.
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Presentation and Production Department
Impact of Event: Affects Single Department
Event Start: 01:52 on November 6th 2018
Estimated Resolution: 13:00 on November 10th 2018
Event Duration: 13 Hours 52 Minutes
Affected Systems/Services: Radio.co
Incident Last Updated: 13:06 on 10th November 2018


Update #2: Our host think that they have found the problem and have reset the service that controls the remote streams. We are now monitoring this issue and we hope that tommrow afternoon, we may have this issue fully resolved.

Thank you


Update #3: The reset of the remote relay service seems to have resolved the issues with remote relay streams. This means that shows they relay their shows to us, now are up and running again.

This incident has now been closed.

Thank you.


Update #1: Our host has escalated this issue, so that they can find the cause and get the problem fixed, by their Development Team. We will post further updates as we receive them. However, it only seems to be affecting a couple of the remote streams and not all remote streams. This issue also does not affect the station output as a whole.

Thank you.


There are some issues with remote streams this morning, which means that remote relays cannot be aired at this time. We have alerted our host and are awaiting them to find a resolution to this issue. Please be aware that this issue does not impact the station output as a whole and is impacting a small group of users that relay shows. Our backup  playlist will kick in the meantime, while these issues are resolved.

We’ll update this page, when we have any further information on this on this issue. If you have any questions on this issue, please email: helpdesk@shoutradio.org.uk quoting #INC-6051118

Thank you.