Ookla Status Page

Nov 18, 2025 · resolved Nov 19

This incident has been resolved.

Oct 1, 2025 · resolved Oct 1

## Incident Summary **Date:** October 1, 2025 **Impact:** Full outage for all European customers **Status:** Resolved --- ### Incident Overview On **October 1st at 11:10 BST**, our internal service discovery system in the European region experienced a failure following a low-risk certificate update. This unexpectedly caused dependent systems—including databases and backend services—to become unavailable. As a result, **all customers using our European platform were impacted** and experienced downtime. --- ### Timeline \(BST\) | Time | Event | | --- | --- | | 10:58 | Internal service certificates were renewed and deployed \(low-risk change\) | | 11:00 | Backend services began receiving updated certificates | | 11:10 | Core coordination service failed to re-establish leadership; investigation began | | 11:15 | First customer reports of service errors received | | 11:36 | Incident publicly posted on status page | | 12:40 | Root cause identified and recovery process initiated | | 12:45 | Status page updated | | 13:40 | Coordination service quorum restored and systems recovered | | 13:50 | Application deployments restarted to refresh service connectivity | | 14:07 | Status page updated | | 14:35 | Incident marked as resolved | --- ### What Went Wrong An internal certificate update—considered low risk—led to a failure in one of our backend coordination systems. This caused a loss of service discovery across our European platform, impacting all services that rely on this mechanism to communicate, including core databases. The issue required manual intervention to restore control and quorum within the system. --- ### How We Resolved It * Launched a new set of coordination nodes using a known working configuration * Temporarily bypassed strict internal security controls to allow recovery * Manually brought the core cluster back online and ensured stability * Restarted dependent services to refresh their service discovery state * Closely monitored system health during and after recovery --- ### What We're Doing Next To prevent similar incidents in the future, we are: * **Improving certificate update validation and rollback procedures** * **Automating recovery workflows** for coordination service failures * Beginning to **evaluate alternative service discovery strategies** with reduced operational risk and complexity --- ### Final Note We sincerely apologise for the disruption caused. This incident affected all customers using our European platform, and we take this extremely seriously. We are committed to improving our systems to reduce the risk of similar outages in the future.

Nov 8, 2024 · resolved Nov 11

This incident has been resolved.

Jan 7, 2024 · resolved Jan 8

All systems are operating as expected.

May 17, 2023 · resolved May 17

This incident has been resolved and all services are accessible as expected.