Affected Services: Request Server and Desktop Web App for PLEX.
Update 13, 10:03 PM CDT (Chicago): We continue to monitor the software reinstallation across our servers. We'll have another update in 60 minutes.
Update 12, 9:33 PM CDT (Chicago): Software installation continues to progress on the servers. During this time, certain applications including sshfs, vim, and Sonarr will be unavailable until their reinstallation has completed. We'll have another update in 30 minutes.
Update 11, 9:03 PM CDT (Chicago): Further updates have been pushed to resolve some software installation errors. We'll have another update in 30 minutes.
Update 10, 8:33 PM CDT (Chicago): We are continuing to monitor the software reinstallations as updates are applied. We'll have another update in 30 minutes.
Update 9, 8:03 PM CDT (Chicago): Software packages continue to be reinstalled on the servers. This has nearly completed on some servers, and is still in progress for others. We'll have another update in 30 minutes.
Update 8, 7:49 PM CDT (Chicago): Please be aware that there is no risk of customer data loss as a result of this outage. Some system applications were gracefully uninstalled by mistake, however, no customer data was touched during this process. A full post mortem will follow when all services are restored. We'll have another update in 15 minutes.
Update 7, 7:43 PM CDT (Chicago): Updates are being pushed to the servers to finish reinstalling system applications. We'll have another update in 15 minutes.
Update 6, 7:27 PM CDT (Chicago): We are seeing favorable results in resolving on-going problems with Deluge. We'll have another update in 15 minutes.
Update 5, 6:58 PM CDT (Chicago): Progress continues to be made to fix the affected servers. Most applications that were previously missing have now been restored. We'll have another update in 30 minutes.
Update 4, 6:28 PM CDT (Chicago): Progress is ongoing to fix the affected servers. A credit will be issued for the downtime in accordance with our SLA. We'll have another update in 30 minutes.
Update 3, 6:20 PM CDT (Chicago): We continue to make progress on fixing the servers. We'll have another update in 30 minutes.
Update 2: The fix is still being applied to all servers.
Update 1: We have tested a fix and are applying it to all servers. Some programs will take longer to come back online than others.
There is an outage affecting all servers and slots. We are actively working on the issue and are aiming to have it resolved as soon as humanly possible. We will update this news post with more deta