API Service Degradation

Incident Report for PaperTrailBot Status

Resolved

This incident has been resolved.
Posted Jun 12, 2026 - 02:53 UTC

Update

Looks like a broader Quarkus issue.
Downgraded to last known stable version: 3.35.4
Posted Jun 11, 2026 - 18:29 UTC

Monitoring

A fix has been implemented by temporarily downgrading Agroal to 3.0.1 (quarkus-agroal:3.35.4), while keeping Quarkus at 3.36.2.
Sustained load tests have not created any zombie workers so far.

We will continue monitoring the issue and provide updates.
Posted Jun 11, 2026 - 16:50 UTC

Update

Further investigation points to an Agroal 3.x issue. Agroal 3.x was introduced in Quarkus 3.36.x. While Agroal 3.2 seems to have fixed the issue, there may be unresolved regressions.
Posted Jun 11, 2026 - 16:36 UTC

Update

Further investigation shows Quarkus 3.36.x versions are causing zombie workers in low resource containers. Based on the tests done, a fix is being formulated.
Posted Jun 11, 2026 - 15:49 UTC

Identified

The issue has been pinpointed to having too many zombie transactions under high load. A fix has not been devised yet.
Posted Jun 11, 2026 - 14:43 UTC

Monitoring

A temporary workaround has been implemented and we are monitoring the issue
Posted Jun 11, 2026 - 08:29 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jun 11, 2026 - 07:26 UTC

Update

Connection issues to the database service is causing workers to be marked as zombies and the transaction to be marked for rollback, but the owning thread is blocked for too long.
Posted Jun 11, 2026 - 07:25 UTC

Identified

The issue appears to be tied to an ongoing incident affecting our cloud service providers.
Posted Jun 11, 2026 - 06:33 UTC

Investigating

API Service 1 - Northflank - Europe West Instance may have degraded performance due to an unidentified issue. We are actively investigating the cause behind it and will keep providing updates.
Posted Jun 11, 2026 - 05:16 UTC
This incident affected: API Services (API Service 1 - Northflank - Europe West Instance) and Database Services (Database Service 1 - Northflank - Europe West).