Resolved -
This incident has been resolved.
Jun 12, 02:53 UTC
Update -
Looks like a broader Quarkus issue.
Downgraded to last known stable version: 3.35.4
Jun 11, 18:29 UTC
Monitoring -
A fix has been implemented by temporarily downgrading Agroal to 3.0.1 (quarkus-agroal:3.35.4), while keeping Quarkus at 3.36.2.
Sustained load tests have not created any zombie workers so far.
We will continue monitoring the issue and provide updates.
Jun 11, 16:50 UTC
Update -
Further investigation points to an Agroal 3.x issue. Agroal 3.x was introduced in Quarkus 3.36.x. While Agroal 3.2 seems to have fixed the issue, there may be unresolved regressions.
Jun 11, 16:36 UTC
Update -
Further investigation shows Quarkus 3.36.x versions are causing zombie workers in low resource containers. Based on the tests done, a fix is being formulated.
Jun 11, 15:49 UTC
Identified -
The issue has been pinpointed to having too many zombie transactions under high load. A fix has not been devised yet.
Jun 11, 14:43 UTC
Monitoring -
A temporary workaround has been implemented and we are monitoring the issue
Jun 11, 08:29 UTC
Update -
We are continuing to work on a fix for this issue.
Jun 11, 07:26 UTC
Update -
Connection issues to the database service is causing workers to be marked as zombies and the transaction to be marked for rollback, but the owning thread is blocked for too long.
Jun 11, 07:25 UTC
Identified -
The issue appears to be tied to an ongoing incident affecting our cloud service providers.
Jun 11, 06:33 UTC
Investigating -
API Service 1 - Northflank - Europe West Instance may have degraded performance due to an unidentified issue. We are actively investigating the cause behind it and will keep providing updates.
Jun 11, 05:16 UTC