Incident: Airsuite Server Failure

Summary of Issue: Airsuite Server Faliure
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Station Staff \ General Public
Impact of Issue: Affects Multiple Departments
Event Start: 20:15 GMT on 27/12/2019
Incident Last Updated: 23:23 GMT on 27/12/2019
Estimated Event End: 23:23 GMT on 27/12/2019
Affected Systems: All Airsuite Services


Update #1: This issue has now been resolved and was caused by an issue with the DNS on the server.

Sorry for the inconvenience this issue has caused.


We’re currently investigating an issue which has taken our Airsuite Server offline. We are currently not sure why this has happened,  but we are working on getting the station back on air as soon as possible.

Sorry for the inconvenience and please watch this page for further updates.

Advisory: Christmas Office Hours (2019)

Summary of Work:
Advisory Status: Expired
Impacted Audience: General Public, Staff
Impact of Work:  Affects Organization
Event Start: 23/12/2019 @17:00 GMT
Estimated Event End: 05/01/2019 @ 09:00 GMT
Duration of Event: N\A
Affected Systems: – 


Please read this post carefully, regarding the arrangements for contacting us over Christmas. 

Please be aware: The below information does not apply to station output. The station output will not be affected during the holiday period. Also, you will still be able to contact the studio, as normal and interact with shows and their presenters. This closure only applies, if you are contacting the station office.

The Shout Radio Office will be closed over the Christmas Holidays. The Office will be closed from 5pm on Thursday 19th December and the office will re-open at 9am on Monday 6th January 2020.  During this time, our phone lines will be closed and emails may not be actioned. (Except for the shows inbox and studio inbox) Please note, shows will continue to be broadcast, even though our office is closed.

Please note: Normal Office Hours will continue, up until 5pm on Thursday 19th December and will resume after Monday 6th January 2020. (Normal office hours: Mon – Fri 9am – 5pm)

There will be a Duty Manager On Call while the office is closed for Christmas. However, this service is for EMERGENCIES ONLY. If you need to speak to someone in an emergency, while the office is closed, please call the office as usual on: 01606 339915 as normal, as you will be put through to the Duty Officer on Call.

(IE: If the station goes off air, or another emergency situation, which requires urgent intervention, before the office opens again in January.)

For all Non urgent enquries, please email: management@shoutradio.org.uk. For IT related problems, please email: helpdesk@shoutradio.org.uk. If your IT related problem is urgent, please contact On Call, (once you have opened a ticket by email) via the office phone and explain this to the on call Duty Manager.

For non urgent issues, please contact us in the normal way and we will contact you once the office opens again.

From all of us here at Shout Radio, we wish you a pleasant and safe Christmas and a very Happy New Year.

Shout Radio Management

 

Advisory: Remembrance Sunday 2019

Summary of Work: 2 Minute Silence on Remembrance Sunday
Advisory Status: Expired
Impacted Audience: General Public, Station Staff
Impact of Work: Affects Organization
Event Start: 10\11\2019 :11:00
Estimated Event End: 10\11\2019 :11:00
Duration of Event: Approx: 2 minutes
Affected Systems: – 


Shout Radio will be taking part in the Remembrance Sunday 2 Minute Silence, at 11:00 on Sunday 11th November 2019. Dave Mcfarlaine’s Positive Spin will be aired, but there will be a 2 minute silence, which means the 2nd hour of Dave’s show will air 2 minutes later than usual.

Maintenance: Cisco Switch Firmware Update

Summary of Work: Update of Cisco Switch Firmware
Maintenance Status: Completed
Impacted Audience: General Public, Staff
Impact of Work: Affects Organization
Event Start: 20\10\2019 :0200
Estimated Event End: 20\10\2019 :05:00
Duration of Event: Approx: 5 minutes
Affected Systems: All Web Services


Our Host has notified us that they will be performing a Firmware update on a network switch, attached to our Server Raq, on Sunday 20th October 2019; between 02:00 and 05:00.

We anticipate a loss of service for between 5 – 15 minutes shortly after 02:00 BST, however service should be considered to be at risk through the duration of the maintenance window. This means there is a chance our website and email system may be disrupted during this time.

If you have any questions about this planned maintenance window, please open a support ticket, by emailing: helpdesk@shoutradio.org.uk

Sorry for any inconvenience this causes you.

Incident: Servicedesk Offline

Summary of Issue: Servicedesk Offline
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Station Staff \ General Public
Impact of Issue: Affects Organization
Event Start: 22:00 BST on 08/09/2019
Incident Last Updated: 13:21 BST  on 11/09/2019
Estimated Event End: 13:21 BST  on 11/09/2019
Affected Systems: IT Servicedesk


Update #1: We have had to restore the system from a backup, which has resolved this issue. The Servicedesk is now back online and is running normally again.

Sorry for the inconvenience this issue has caused.


We’re currently investigating why our Helpdesk System has gone down. At the moment, this means that users are unable to open support tickets via the portal. However, emails sent to the Helpdesk are not affected by this, nut will not be processed until the system is back up and running.

Sorry for the inconvenience and please watch this page for further updates.

 

Advisory: Updated Airsuite Client Released!

Summary of Work: N\A
Advisory Status: N\A
Impacted Audience: Presentation and Production Department
Impact of Work: Affects Department
Event Start:
Estimated Event End: –
Duration of Event:
Affected Systems: Airsuite


Our Host, (Creative Dork) have informed us that they have retired the Browser Broadcaster, which was accessed via the Airsuite Panel. They have now released a desktop client, which acts the same way the browser based client worked. However, at this moment in time, you are still required to login to the panel for the countdown. (Creative Dork have said they are releasing an update to the client soon, which will contain a countdown)

The Client is only available to users using Windows 7,8,8.1 & Windows 10 PCs or iMac users. It is possible that users using Windows Vista and Windows XP SP3, may be able to use our client, but this is not fully supported by us. The new client itself, can downloaded from the Airsuite Panel, by clicking the spanner on the left  menu and choosing “desktop tools”

This advisory only applies to users who use the Airsuite Client on Windows and not users using 3rd party tools. If you are a Windows 10 \ Mac user and would like to use the broadcast client, please open a support ticket, so we can update the settings on your account, which will allow you to use the broadcast client, instead of using a “3rd Party Encoder.”

Please open a Support Ticket, if you need any assistance with this.

Incident: Problems with Airsuite Legacy Platform

Summary of Issue: Problem with Airsuite Legacy Broadcast Platform
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Presentation and Production Team
Impact of Issue:  Affects Department
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 7-09-2019 @ 14:30 BST
Estimated Event End: 17-09-2019 @ 14:30 BST
Affected Systems: Airsuite Legacy Broadcast Platform

#Update 1

We now know why users are unable to encode to the server. This is because of legacy applications, which are not compatible with Icecast2 and therefore, cannot encode to the server.

We suggest that users download and install the freeware application: EddCast. (Please see here for instructions on how to do this and how to setup your encoder.

Sorry for the confusion with this issue.


We are currently investigating a problem with the Airsuite Legacy Broadcast Platform. It seems that presenters who are set to use the Legacy Platform are unable to connect to the server. This issue seems to be affecting multiple users and means their shows are not being broadcast at present.

Our host has been made aware of this and we will update this page with further information soon.

Sorry for the inconvenience.

Incident: Email System Currently At Risk

Summary of Issue: Email \ Security Issue
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: All Staff
Impact of Issue:  Affects Organization
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated:17-07-2019 @ 13:18 BST
Estimated Event End: 26-07-2019 @ 13:00 BST
Affected Systems: Email Systems


Update #2

This issue has seemed to of resolved itself. We are still unsure why this happened and what has caused this issue. Our email system has been fine for over a week and we are satisfied that this issue will not cause our email system to go down again.

However, please raise a ticket if you encounter any additional issues with our email services.

Sorry for the inconvenience this issue has caused.


Update #1

Our host has said that this issue is possibly being caused by a malicious script on the server, which has caused this to happen. However, we have not found evidence of such a script and we are continuing to investigate this issue.

Our host has restored a backup file, which has bought the email system back online again. However, at this time, Email services should be treated as “at risk”, until further notice.

Sorry for the inconvenience, this ongoing issue is causing.


We are currently investigating a major problem with our email system. Last Sunday, someone gained unauthorized access to our server and deleted all mailboxes from the server.  Our Host ran scans on the system, to check for spyware; but that came back clear. Our Host was able to restore the server from a backup and I had changed the Server Root Password and other system passwords.

However, the password change had little effect (even though we used a strong password) and once again the server has been attacked! Once again, our email system has been compromised again. As far as we know, there has not been a data breach, as the hackers simply have removed everyone’s mailboxes from the server. Each mailbox is password protected, so who ever it was that has broken into the server; would not of had access to individual email inboxes. However, we know that the unauthorized access to our server was performed using SSH. (Secure Socket Layer, which is similar to the DOS interface on Windows Operating Systems). Even though we have changed the Root Password, (which is the server master password)  it seems that who ever has attacked the server has managed to somehow break into the server. We have checked the userlist, by running the following command.

cut -d: -f1 /etc/passwd

However, this did not throw up anything unusual and we at this moment in time, we have no idea who is accessing the sever and how they are getting into the system.

Our host is now aware of the problem and we hope to have this issue resolved as soon as we can. However,  as this has happened for the 2nd Sunday in a row, IT Services have decided to keep this status report open, so that if there is a repeat of this issue, we do not need to post a duplicate status report.

This issue means that our email system is currently at risk for all users and that means that our email system is currently at risk of further outages.

We’ll keep you posted on this issue.

Sorry for the inconvenience this causes.

Incident: Problems with Live Shows Using Airsuite

Summary of Issue: Problems with Airsuite kicking Presenters of air
Incident Status: Resolved
Incident Priority: Priority 1 (Critical)
Impacted Audience: Presentation and Production Department
Impact of Issue: Affects Organization
Event Start: 03/7/2019 @ 13.00 BST
Incident Last Updated: 14-07-2019
Estimated Event End: 14-07-2019 @ 15:47
Affected Systems: Airsuite Users


Update: #1

Our host has come back with the following information about this issue:

Unfortunately, as the AirSuite Broadcaster is web based and due to the limitations of some devices we are unable to guarantee uninterrupted streaming when using the Web Based AirSuite Broadcaster, therefore this is provided as an ‘as is’ service with the Third Party option as the recommended option for presenters that experience disconnection issues.

So due to this, we advise Presenters using Laptops and slower devices, to use th 3rd party encoder option, such as Butt or another 3rd party encoder. To do this, you will need us to enable Legacy Broadcasting on your Airsuite Account. This means that you will need to raise a support ticket, so we can enable this for you.

Sorry for the inconvenience.


There are some issues with Airsuite, which causes Presenters to be disconnected from their shows at random. We have our host investigating this issue and will post an update, as soon as possible.

Please note: This issue does not affect users using the Legacy Broadcast Platform.

Sorry for the inconvenience.

Maintenance: Web Server Reboot Scheduled

Summary of Work: Server Reboot
Maintenance Status: Completed
Impacted Audience: General Public, Station Staff
Maintenance Status: Completed
Impact of Work: Affects Organization
Event Start: 27/5/2019 @ 00:00 BST
Estimated Event End: 27/5/2019 @ 00:05 BST
Duration of Event: 5 Minutes
Affected Systems:Primary Web Server, Mail Server, FTP Server, MySql Services


We will be performing a planned reboot on our primary server: mollie.shoutradio.org.uk on Saturday morning at 00:00. This planned maintenance will result in our website, email system and databases being unavailable for approx: 5 minutes, while the server reboots. We are performing this reboot, as we have updated the system software that runs on the server and a reboot is needed to apply these updates to the server Operating System.

There is a 20% chance the server may crash, while applying updates to the Operating System.  Due to this, we’ve made our host aware of this maintenance window, should the server crash and require a hard reboot.

The station will remain on air and this reboot does not affect our Broadcast Server.

If you have any questions about this maintenance window, please open a Support Ticket.

Sorry for the inconvenience.