tag:status.galaxyproject.org,2005:/historyThe Galaxy Project Status - Incident History2024-03-29T05:24:04ZThe Galaxy Projecttag:status.galaxyproject.org,2005:Incident/203458562024-03-28T01:00:02Z2024-03-28T01:00:02ZUseGalaxy.org (Main) Database Maintenance<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>23:22</var> UTC</small><br><strong>Verifying</strong> - Database maintenance is complete and UseGalaxy.org is available.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>23:29</var> UTC</small><br><strong>Scheduled</strong> - UseGalaxy.org (Galaxy Main) will undergo database maintenance and upgrades. The site will be unusable during this time. Other UseGalaxy.* sites will be unaffected and still available for use.</p>tag:status.galaxyproject.org,2005:Incident/198915972024-02-05T14:00:49Z2024-02-05T14:00:49ZUseGalaxy.FR Downtime<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>09:09</var> UTC</small><br><strong>Scheduled</strong> - Due to an unscheduled electrical maintenance at IDRIS, we are forced to shut down UseGalaxy.fr from Friday, February 2 (14h) to Monday, February 5, 2024 (17h). Sorry for the inconvenience and the short delay.</p>tag:status.galaxyproject.org,2005:Incident/198673202024-01-30T21:34:51Z2024-01-30T21:34:51Zusegalaxy.org is offline<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>21:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Monitoring</strong> - usegalaxy.org is reachable again</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Investigating</strong> - Due to unknown issues at our host, usegalaxy.org is currently unreachable.</p>tag:status.galaxyproject.org,2005:Incident/196879652024-01-12T13:20:41Z2024-01-12T13:20:41Zusegalaxy.org is down<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>13:20</var> UTC</small><br><strong>Resolved</strong> - The database server is now operational.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>Identified</strong> - The database server for usegalaxy.org is down, and so Galaxy is unavailable. We are working to return the database server to service as soon as possible.</p>tag:status.galaxyproject.org,2005:Incident/186888962023-10-04T18:44:35Z2023-10-04T18:44:35Zusegalaxy.org is down<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>18:44</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>15:25</var> UTC</small><br><strong>Monitoring</strong> - The database server has been returned to service. We are monitoring performance and may need to restart services for configuration changes.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>14:14</var> UTC</small><br><strong>Investigating</strong> - The database server for usegalaxy.org is currently unreachable. We are investigating the problem and will provide updates when available.</p>tag:status.galaxyproject.org,2005:Incident/185452302023-09-19T13:55:54Z2023-09-19T13:55:54Zusegalaxy.org is offline<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>13:55</var> UTC</small><br><strong>Resolved</strong> - The database server is operational again and usegalaxy.org is functioning normally.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>12:28</var> UTC</small><br><strong>Investigating</strong> - The database server for usegalaxy.org is currently unreachable, causing a complete outage of the service. We are investigating and will provide an update when more details are available.</p>tag:status.galaxyproject.org,2005:Incident/171559292023-05-05T17:07:49Z2023-05-05T17:07:49Zusegalaxy.org is down<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>17:07</var> UTC</small><br><strong>Resolved</strong> - The database server has been returned to service</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Investigating</strong> - The database server for usegalaxy.org is currently down. We are working on returning it to service.</p>tag:status.galaxyproject.org,2005:Incident/168030092023-04-10T18:13:17Z2023-04-10T18:13:17Zusegalaxy.org is down<p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>18:13</var> UTC</small><br><strong>Resolved</strong> - Both webserver VMs have been returned to service</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a temporary workaround. usegalaxy.org is operational, but you may experience decreased performance and other issues until service is fully restored.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>22:18</var> UTC</small><br><strong>Identified</strong> - Both Webserver VMs are currently offline. We do not know how long it will take for them to return to service, but will update as soon as we know.</p>tag:status.galaxyproject.org,2005:Incident/167717472023-04-06T19:37:13Z2023-04-06T19:37:13Zusegalaxy.org Webserver 1 is offline<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Resolved</strong> - Webserver 1 is functioning normally again.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>16:35</var> UTC</small><br><strong>Investigating</strong> - One of the two usegalaxy.org webservers is currently offline. Traffic has been diverted to the second webserver until this issue is resolved. You may experience degraded performance until this issue is resolved.</p>tag:status.galaxyproject.org,2005:Incident/110419212022-09-02T19:18:41Z2022-09-02T19:18:41Zusegalaxy.org network instability<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>19:18</var> UTC</small><br><strong>Resolved</strong> - The network appears to have stabilized and Galaxy is accessible.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>16:21</var> UTC</small><br><strong>Investigating</strong> - UseGalaxy.org is currently experiencing intermittent availability due to unexpected network instability at our host. We are not aware of the cause at this time, but will update the incident when we know more or the issue is resolved.</p>tag:status.galaxyproject.org,2005:Incident/104715272022-07-05T16:42:14Z2022-07-05T16:42:14ZUsegalaxy.org Webserver 1 is down<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>13:32</var> UTC</small><br><strong>Investigating</strong> - Traffic is being routed to Webserver 2, but this increased load may cause delays and other problems until Webserver 1 is restored.</p>tag:status.galaxyproject.org,2005:Incident/102221952022-07-05T13:31:08Z2022-07-05T13:31:09ZUseGalaxy.org (Galaxy Main) Upgrade<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>13:31</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:08</var> UTC</small><br><strong>Verifying</strong> - We are verifying proper functionality of the service after upgrading.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Scheduled</strong> - UseGalaxy.org will be down for maintenance as we perform various upgrades.</p>tag:status.galaxyproject.org,2005:Incident/101420502022-06-03T21:15:45Z2022-06-03T21:15:45ZUS Galaxy Test Server Maintenance<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>17:15</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>17:06</var> UTC</small><br><strong>Scheduled</strong> - The US Galaxy Test server (test.galaxyproject.org) will undergo configuration changes necessitating a full downtime.</p>tag:status.galaxyproject.org,2005:Incident/93182532022-02-15T01:50:25Z2022-02-15T01:50:25Zusegalaxy.org is down due to a filesystem space issue<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>01:50</var> UTC</small><br><strong>Resolved</strong> - usegalaxy.org is now back in service.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>01:16</var> UTC</small><br><strong>Identified</strong> - We are moving some data around in order to free some space, and should be back online shortly.</p>tag:status.galaxyproject.org,2005:Incident/90490912022-01-11T00:48:10Z2022-01-11T00:48:10ZNetwork services are offline at TACC<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>00:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>00:31</var> UTC</small><br><strong>Monitoring</strong> - Network services appear to have been restored.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>20:52</var> UTC</small><br><strong>Investigating</strong> - All network services at TACC (where usegalaxy.org is hosted) appear to be offline. We do not have any additional information at this time but will update when more information becomes available.</p>tag:status.galaxyproject.org,2005:Incident/87099412021-11-28T22:00:00Z2021-11-29T15:42:32Zusegalaxy.org is unavailable for scheduled network maintenance<p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Identified</strong> - Network services for UseGalaxy.org are down for scheduled maintenance. We apologize for the lack of advance notice.</p>tag:status.galaxyproject.org,2005:Incident/86696622021-11-25T13:52:59Z2021-11-25T13:52:59Zusegalaxy.eu DB server and Galaxy code upgrades<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>13:52</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>09:24</var> UTC</small><br><strong>Scheduled</strong> - From November 24th, 2021 at 17:45 pm CET and for the next 24 hours, UseGalaxy.eu services will be shut down.<br /><br />During this time we will be upgrading Galaxy to the latest version (21.09) and migrating/upgrading the DB server.</p>tag:status.galaxyproject.org,2005:Incident/83656022021-11-03T01:30:43Z2021-11-03T01:30:43ZPower works<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>01:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:48</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:48</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:31</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:29</var> UTC</small><br><strong>Scheduled</strong> - Galaxy Australia will be offline until 5:00pm AEST 2021-11-03. Our service provider is undergoing power works at their data centre and our VMs are all affected.</p>tag:status.galaxyproject.org,2005:Incident/79912642021-09-16T20:28:53Z2021-09-16T20:28:53Zusegalaxy.org filesystem upgrades<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>20:28</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. Cluster capacity is still slightly reduced, so you may encounter longer job waiting times than usual, but Galaxy is otherwise operating normally. We will update the masthead banner at usegalaxy.org when the cluster capacity issue is resolved.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Verifying</strong> - Filesystem maintenance is complete and we are monitoring for any related problems. Cluster capacity is temporarily slightly reduced so job wait times might be higher than normal.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>20:43</var> UTC</small><br><strong>Update</strong> - Filesystem maintenance is scheduled until 14:00 (2:00 PM) UTC on September 16, not for the 7 hours originally reported. We apologize for this error.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:23</var> UTC</small><br><strong>Scheduled</strong> - Filesystem upgrades will be taking place at our host. usegalaxy.org services will be unavailable during this time.</p>tag:status.galaxyproject.org,2005:Incident/78042102021-08-20T20:54:07Z2021-08-20T20:54:07Zusegalaxy.org datasets and jobs are currently not available<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>20:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>18:35</var> UTC</small><br><strong>Monitoring</strong> - Jobs are executing again and we are monitoring for any repeat issues.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Update</strong> - Access to user data has been restored, but job execution is still paused as engineers continue to work on a full resolution to the issue.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:17</var> UTC</small><br><strong>Identified</strong> - Engineers at our host believe they have identified some bad network hardware responsible for the performance problems that were occurring, and are working on a solution.<br /><br />We do not yet have a time estimate on how long the service will be affected, but will continue to post updates when they are available.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>03:59</var> UTC</small><br><strong>Investigating</strong> - User data is currently not accessible and jobs are not executing on usegalaxy.org. The filesystem housing datasets is currently not accessible to Galaxy systems due to performance problems. This is believed to be due to issues with the networking infrastructure. Engineers at our host (the Texas Advanced Computing Center, TACC) are investigating the problem. We do have an estimate on how long the service will be affected, but will post updates when they are available.<br /><br />Galaxy Europe (usegalaxy.eu) and Galaxy Australia (usegalaxy.org.au) are unaffected and can be used in the interim. These servers use separate accounts and do not share data, so you will need to create accounts and upload data as needed.</p>tag:status.galaxyproject.org,2005:Incident/64361712021-03-03T17:20:06Z2021-03-03T18:33:30Zusegalaxy.org database server upgrades<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>14:43</var> UTC</small><br><strong>Scheduled</strong> - We will be performing database server upgrades at this time</p>tag:status.galaxyproject.org,2005:Incident/64004102021-02-25T21:26:48Z2021-02-25T21:26:48Zusegalaxy.org is down due to database issues<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>21:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>21:16</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>21:07</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>20:52</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.galaxyproject.org,2005:Incident/61359262021-01-28T17:54:18Z2021-01-28T17:54:18Zusegalaxy.org system updates<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>17:54</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Verifying</strong> - Maintenance is complete and we are monitoring to ensure everything is working properly</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>15:12</var> UTC</small><br><strong>Scheduled</strong> - usegalaxy.org will be down briefly to apply critical system updates</p>tag:status.galaxyproject.org,2005:Incident/59526942021-01-05T22:36:02Z2021-01-05T22:36:02Zusegalaxy.org is unreachable<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>22:36</var> UTC</small><br><strong>Resolved</strong> - TACC network services have remained stable throughout the day, and usegalaxy.org is fully available.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>17:31</var> UTC</small><br><strong>Monitoring</strong> - Network services at TACC have been restored, but we are awaiting official word that services are expected to remain stable.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Identified</strong> - TACC is experiencing firewall issues that are causing an outage to all services inside their network. No ETA for service restoration is available at this time, we will update if one becomes available.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>14:01</var> UTC</small><br><strong>Investigating</strong> - usegalaxy.org is currently unreachable due to network problems at our host, the Texas Advanced Computing Center (TACC). More details will be provided when they become available.</p>tag:status.galaxyproject.org,2005:Incident/53159982020-10-15T15:47:21Z2020-10-15T15:47:21ZScheduled filesystem maintenance on usegalaxy.org<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>13:29</var> UTC</small><br><strong>Verifying</strong> - All services are now restored and we are verifying normal operation.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>01:08</var> UTC</small><br><strong>Update</strong> - Services have been partially restored and we have started up the Galaxy UI, but job execution is currently paused because not all necessary services have been restored yet.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>15:03</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance has been extended through the end of October 14. Further updates on the estimated return time may be available later in the day.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>21:53</var> UTC</small><br><strong>Update</strong> - Maintenance is taking longer than planned and is now expected to be completed by 10:00 AM CDT on October 14.<br /><br />Additional details and progress can be found at https://portal.tacc.utexas.edu/user-news/-/news/106272</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>14:58</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>14:57</var> UTC</small><br><strong>Scheduled</strong> - usegalaxy.org (Galaxy Main) will be down for filesystem maintenance.</p>