Jobs are executing again and we are monitoring for any repeat issues.
Posted Aug 20, 2021 - 18:35 UTC
Update
Access to user data has been restored, but job execution is still paused as engineers continue to work on a full resolution to the issue.
Posted Aug 20, 2021 - 16:37 UTC
Identified
Engineers at our host believe they have identified some bad network hardware responsible for the performance problems that were occurring, and are working on a solution.
We do not yet have a time estimate on how long the service will be affected, but will continue to post updates when they are available.
Posted Aug 20, 2021 - 16:17 UTC
Investigating
User data is currently not accessible and jobs are not executing on usegalaxy.org. The filesystem housing datasets is currently not accessible to Galaxy systems due to performance problems. This is believed to be due to issues with the networking infrastructure. Engineers at our host (the Texas Advanced Computing Center, TACC) are investigating the problem. We do have an estimate on how long the service will be affected, but will post updates when they are available.
Galaxy Europe (usegalaxy.eu) and Galaxy Australia (usegalaxy.org.au) are unaffected and can be used in the interim. These servers use separate accounts and do not share data, so you will need to create accounts and upload data as needed.
Posted Aug 20, 2021 - 03:59 UTC
This incident affected: Galaxy Main (Webserver 1, Webserver 2).