Live Status\Incident History
July 31, 2024
Site Unreachable
Outage
Opened: Jul 31, 2024, 9:34 PM UTC
Duration: 3m 9s
Affected services:License Manager
- OpenedJul 31, 2024, 9:34 PM UTC
Issue opened on monitor failure.
- ResolvedJul 31, 2024, 9:37 PM UTC
Issue resolved on monitor recovery.
July 17, 2024
Seqera Cloud Temporarily Unavailable
Outage
Opened: Jul 17, 2024, 7:43 PM UTC
Duration: 9m 59s
Affected services:Platform Cloud app
- IdentifiedJul 17, 2024, 7:43 PM UTC
It has been detected that the Seqera Platform Cloud is unavailable. The cause has been immediately determined and a fix is underway.
- ResolvedJul 17, 2024, 7:53 PM UTC
The issue has been identified and resolved.
July 11, 2024
Seqera Cloud Unavailable
Outage
Opened: Jul 11, 2024, 6:26 PM UTC
Duration: 7m
Affected services:Platform Cloud app
- MonitoringJul 11, 2024, 6:26 PM UTC
We have identified an issue with Seqera Cloud that was preventing it from loading. We have potentially resolved the issue and are confirming the fix before marking this issue as closed.
- ResolvedJul 11, 2024, 6:33 PM UTC
The issue was identified to be an infrastructure problem that has been rectified. No further issues should remain.
July 8, 2024
Site Unreachable
Outage
Opened: Jul 8, 2024, 10:07 PM UTC
Duration: 27m 19s
Affected services:Cloudinfo
- OpenedJul 8, 2024, 10:07 PM UTC
Issue opened on monitor failure.
- ResolvedJul 8, 2024, 10:35 PM UTC
Issue resolved on monitor recovery.
July 3, 2024
Pipeline Execution Issues Due to "Missing Parameters"
Degraded Performance
Opened: Jul 3, 2024, 2:21 PM UTC
Duration: 1h 22m 59s
- InvestigatingJul 3, 2024, 2:21 PM UTC
An issue has been reporting in which some users are getting errors reporting "missing required parameters" on pipelines that were previously operating correctly. We are currently investigating the issue.
- ResolvedJul 3, 2024, 3:44 PM UTC
This issue has been identified and a workaround resolution has been discovered. Multiple customers reported 'Missing required parameter:' errors like: * Missing required parameter: --input * Missing required parameter: --outdir while executing pipelines that previously ran without incident. The error was replicated on Seqera Platform by leaving the 'pipeline parameters' field empty in the Launchpad pipeline configuration empty and supplying parameters via the schema-defined form only. The error is bypassed if any parameter is supplied via 'pipeline parameters' in the Launchpad configuration. Affected customers can use this as a workaround, adding to pipeline Launchpad configuration any parameter common to workflow executions.
Certain New Pipelines Unable to be Created
Degraded Performance
Opened: Jul 3, 2024, 11:30 AM UTC
Duration: 2h 21m 59s
- IdentifiedJul 3, 2024, 11:30 AM UTC
We have identified a degraded service event affecting certain pipeline conditions for cloud users and those on enterprise version 24.1.0. The issue impacts users of nf-core pipelines or pipelines based on the nf-core template, specifically those relying on shared configs from https://nf-co.re/configs/ or those custom configs where an external includeConfig is utilized. Under these circumstances, new pipelines within these parameters cannot be created or launched. Our team is currently investigating the issue, and a fix will be released shortly.
- ResolvedJul 3, 2024, 1:52 PM UTC
A patch for Seqera Platform Cloud has been released and this should no longer be impacting Cloud users. A patch for enterprise is imminent and this incident will be closed for now as no existing enterprise users are affected by this.
Some Pipeline Execution Impacted by Serialization Issues
Degraded Performance
Opened: Jul 3, 2024, 10:40 AM UTC
Duration: 3h 20m
- IdentifiedJul 3, 2024, 10:40 AM UTC
An issue has been identified in which the execution of pipelines may have been affected due to a bug introduced by the upgrade of a library used by the platform to serialize/deserialize some data stored in our caching layer.
- ResolvedJul 3, 2024, 2:00 PM UTC
The problem has been solved and pipelines executions suffering from this issue should be working properly now.
May 11, 2024
Wave Endpoint Issues
Minor Outage
Opened: May 11, 2024, 12:48 PM UTC
Duration: 2h 39m 59s
- InvestigatingMay 11, 2024, 12:48 PM UTC
Wave is reporting errors causing it to be unavailable for a subset of nextflow/platform users.
- IdentifiedMay 11, 2024, 3:15 PM UTC
The issue was determined to be caused by a Redis cache problem, affecting only customers using the old platform endpoint https://api.tower.nf .
- ResolvedMay 11, 2024, 3:28 PM UTC
No further issues should exist.
April 30, 2024
Wave Container Issues
Minor Outage
Opened: Apr 30, 2024, 2:00 PM UTC
Duration: 6h
Affected services:Wave
- IdentifiedApr 30, 2024, 2:00 PM UTC
A recent update to Wave caused inconsistency in the composition of container names which may have resulted in some users being unable to run pipelines relying on Wave.
- ResolvedApr 30, 2024, 8:00 PM UTC
This change has been rolled back and should no longer cause issues.
April 25, 2024
Site Unreachable
Outage
Opened: Apr 25, 2024, 8:23 AM UTC
Duration: 33s
Affected services:License Manager
- OpenedApr 25, 2024, 8:23 AM UTC
Issue opened on monitor failure.
- ResolvedApr 25, 2024, 8:24 AM UTC
Issue resolved on monitor recovery.
Site Unreachable
Outage
Opened: Apr 25, 2024, 8:20 AM UTC
Duration: 1m 39s
Affected services:Cloudinfo
- OpenedApr 25, 2024, 8:20 AM UTC
Issue opened on monitor failure.
- ResolvedApr 25, 2024, 8:22 AM UTC
Issue resolved on monitor recovery.
March 22, 2024
Wave Connection Issues
Degraded Performance
Opened: Mar 22, 2024, 11:20 PM UTC
Duration: 43h 10m
Affected services:Wave
- OpenedMar 22, 2024, 11:20 PM UTC
Start of Wave errors Retroactively, it was discovered that transient spikes in Wave container errors had occurred causing temporary degradation of service and impacting a limited number of customers.
- InvestigatingMar 24, 2024, 7:37 AM UTC
A potential issue with the Wave service has been discovered impacting a limited number of clients.
- IdentifiedMar 24, 2024, 11:56 AM UTC
Wave issues have been confirmed and a potential cause has been pinpointed.
- ResolvedMar 24, 2024, 6:30 PM UTC
A change has been pushed to the Seqera platform that has remedied the problem. The initial issue was caused by a misconfiguration that was exhausting the available connection pool.
January 11, 2024
Job Submissions Stalling
Minor Outage
Opened: Jan 11, 2024, 3:00 PM UTC
Duration: 1h 30m
- IdentifiedJan 11, 2024, 3:00 PM UTC
An issue has been identified that is causing jobs in Platform Cloud to stall in the "Submitted" state.
- ResolvedJan 11, 2024, 4:30 PM UTC
The problem was caused by a stuck thread in our job scheduler component that was preventing job submissions from being performed correctly. We restarted the service and implemented a code fix to prevent the scenario from happening again. All stalled jobs should now be proceeding as normal.
November 27, 2023
Wave HTTP Errors
Minor Outage
Opened: Nov 27, 2023, 1:55 AM UTC
Duration: 6h 33m 59s
Affected services:Wave
- InvestigatingNov 27, 2023, 1:55 AM UTC
A timeout issue has been identified that is impacting users of Wave containers [HTTP status=408].
- ResolvedNov 27, 2023, 8:29 AM UTC
The issue has been identified and resolved.
November 14, 2023
Issue with quay.io container registry
Minor Outage
Opened: Nov 14, 2023, 9:30 PM UTC
Duration: 2h 19m 59s
- OpenedNov 14, 2023, 9:30 PM UTC
An incident with quay.io container registry is impacting the launch of data pipelines on Seqera Enterprise on-prem installations. We are monitoring the problem. Please contact Seqera support for more details.
- ResolvedNov 14, 2023, 11:50 PM UTC
The incident was solved by quay.io team.
November 1, 2023
AWS Batch users: "Invalid Certificate Error"
Degraded Performance
Opened: Nov 1, 2023, 4:10 AM UTC
Duration: 5h 54m
- InvestigatingNov 1, 2023, 4:10 AM UTC
- ResolvedNov 1, 2023, 10:04 AM UTC
An "Expired License Error" prevented a small number of users running pipelines on AWS Batch from launching pipelines between midnight and 09:00 CET on November 1st. We have now fixed the issue and we will be implementing solutions to prevent this from happening in the future. We apologize for any inconvenience this may have caused.
October 11, 2023
Tower system temporary connection failure
Minor Outage
Opened: Oct 11, 2023, 8:50 PM UTC
Duration: 40m
- InvestigatingOct 11, 2023, 8:50 PM UTC
A temporary connection failure has been reported by some users. We are investigating the possible problem.
- ResolvedOct 11, 2023, 9:30 PM UTC
A failure during the rollout of a system upgrade caused some backend replicas not to reach the ready status, resulting in a temporary system unreliability.
October 2, 2023
Incident with Wave service
Minor Outage
Opened: Oct 2, 2023, 12:20 PM UTC
Duration: 0s
Affected services:Wave
- ResolvedOct 2, 2023, 12:20 PM UTC
An issue has been identified in the Wave service that caused a partial system outage from Fri, September 30 15:30 until Mon, October 2 12:20. The problem is related to a bug in Java HTTP client used by Wave backend to access remote repositories, which caused the system to consume the Java VM heap memory in an uncontrolled manner and exhaust the underlying threads pool handling the incoming client requests. https://bugs.openjdk.org/browse/JDK-8308144 A solution has been delivered to prevent this problem.
October 1, 2023
Site Unreachable
Outage
Opened: Oct 1, 2023, 8:35 AM UTC
Duration: 8m 16s
Affected services:Wave
- OpenedOct 1, 2023, 8:35 AM UTC
Issue opened on monitor failure.
- ResolvedOct 1, 2023, 8:44 AM UTC
Issue resolved on monitor recovery.
Site Unreachable
Outage
Opened: Oct 1, 2023, 5:27 AM UTC
Duration: 7m 49s
Affected services:Wave
- OpenedOct 1, 2023, 5:27 AM UTC
Issue opened on monitor failure.
- ResolvedOct 1, 2023, 5:35 AM UTC
Issue resolved on monitor recovery.
Site Unreachable
Outage
Opened: Oct 1, 2023, 1:33 AM UTC
Duration: 9m 4s
Affected services:Wave
- OpenedOct 1, 2023, 1:33 AM UTC
Issue opened on monitor failure.
- ResolvedOct 1, 2023, 1:42 AM UTC
Issue resolved on monitor recovery.
September 30, 2023
Site Unreachable
Outage
Opened: Sep 30, 2023, 2:30 PM UTC
Duration: 53s
Affected services:Wave
- OpenedSep 30, 2023, 2:30 PM UTC
Issue opened on monitor failure.
- ResolvedSep 30, 2023, 2:30 PM UTC
Issue resolved on monitor recovery.
September 29, 2023
Site Unreachable
Outage
Opened: Sep 29, 2023, 1:58 PM UTC
Duration: 15m 28s
Affected services:Wave
- OpenedSep 29, 2023, 1:58 PM UTC
Issue opened on monitor failure.
- ResolvedSep 29, 2023, 2:14 PM UTC
Issue resolved on monitor recovery.