Testsigma Status Status Page

Oct 1, 2026 · resolved Oct 1

## Summary On 1 October 2026, between 13:27 and 13:31 IST \(07:57 to 08:01 UTC\), the Testsigma web application in the US region was unavailable for approximately four minutes. A burst of requests from a single API integration exhausted the database connection capacity on our application servers, and the automatic health checks then restarted all servers at the same time. The database itself remained healthy, and no customer data was lost, altered or exposed. Service was fully restored by 13:31 IST. ## Impact All accounts hosted in the US region were affected for approximately four minutes. The EU and India regions were not affected. During the window, users of the web application and of the public API may have seen: * Pages loading slowly, timing out, or returning an error. * Actions such as saving, starting a run, or fetching results failing at the moment of the restart. These actions were not partially applied; they either completed or did not, and can be retried. * API requests returning errors or timing out. Integrations that retry with backoff recovered automatically once service was restored. Test runs already executing on agents continued; results submitted during the window were accepted once the servers were back. ## Timeline All times are on 1 October 2026 in IST \(UTC\+5:30\). * **13:26:55** — An integration belonging to one account begins a bulk data export through the public API at a very high request rate. * **13:27:00** — Every application server has run out of database connections. New requests wait for a connection instead of being served. * **13:27:54** — The automatic health check, which also needs a database connection, starts timing out on every server. * **13:28:06** — Requests waiting for a database connection begin failing with errors. * **13:28:54** — After repeated failed health checks, the platform restarts all application servers at once. * **13:29:00** — The integration retries its export while the servers are restarting. * **13:29:00 to 13:30:30** — New application servers start up. * **13:31:35** — Service is confirmed fully restored and stable. * **13:31 onwards** — Engineering investigates application logs, database metrics and API traffic, and identifies the cause within the day. ## Root cause The outage was triggered by a sudden, sustained burst of bulk data-export calls from a single customer integration, which saturated the application tier's database connection capacity within seconds. Four contributing factors, each benign in isolation, compounded into a brief region-wide interruption. 1. **Unthrottled ingress from a single tenant.** The integration drove request volume several orders of magnitude above its established baseline. At the time, the public API enforced no per-account rate controls, so the surge competed on equal terms with interactive traffic from every other account. 2. **Write amplification on a read path.** The list operation involved carried a latent inefficiency that caused it to perform persistence work for every record it returned. At normal volumes this was imperceptible; under bulk retrieval it extended connection hold times from milliseconds to seconds. 3. **Undifferentiated connection pooling.** Application servers draw on a fixed, shared pool of database connections with no partitioning between bulk, background and interactive workloads. Once the burst occupied the pool, requests from all accounts queued behind it and ultimately timed out. 4. **Health-check coupling.** Server liveness checks exercised the same connection pool and therefore failed alongside application traffic. The orchestration layer responded as designed by restarting the affected servers, but because every server was affected simultaneously, a degraded-but-serving state became a short, complete interruption. The database tier itself remained healthy throughout: load, locking and response times stayed within normal ranges, no failover occurred, and no customer data was lost, altered or exposed. No hardware fault or security event was involved. ## Resolution Service recovered within four minutes once the servers restarted. Full recovery was confirmed at 13:31 IST and the platform has been monitored for stability since. The investigation across application logs, database metrics and API traffic is complete, the root cause has been identified, and the integration that generated the burst has been identified. **Short-term measures.** We have started a set of configuration and monitoring changes that require no code release and will be completed over the coming days. These changes allow the platform to tolerate a temporary database slowdown without restarting application servers, alert our on-call team as soon as database connection usage runs high, and address the specific integration traffic pattern that triggered this incident. **Permanent measures.** In parallel, our product and engineering teams are implementing changying causes: per-account limits on API request rates so that no single integration can affect capacity shared by other customers, improvements to the affected API so that reading data does not create additional database work, bounded page sizes on list APIs, dedicated connection capacity for interactive use, and regular load testing against traffic surges. Most accounts will see no difference from the API request limits, which will be set well above normal usage; the limits will be published, and accounts with high-volume integrations notified, before they take effect.

May 22, 2026 · resolved May 22

### **Incident Summary** On **May 22nd**, **some customers** faced issues with their local **Testsigma Terminal** connections. The incident was caused by a configuration problem that affected the terminal’s ability to establish or maintain connections correctly for a limited set of customers. The issue was detected at around **4:30 AM UTC**, and the engineering team immediately started investigating the problem. After identifying the configuration gap, the team applied the necessary fix and validated the terminal connectivity. The issue was fully resolved by approximately **6:00 AM UTC**. ### **Impact** Only **some customers** using the local Testsigma Terminal experienced connection issues during the incident window. The issue did not impact all customers. **Incident window:** **May 22nd, 4:30 AM UTC to 6:00 AM UTC** **Affected component:** Testsigma Terminal connections **Customer impact:** A limited set of customers were unable to use terminal-based connectivity features during the affected period. ### **Root Cause** The incident was caused by a configuration problem that had previously gone unnoticed. This configuration issue impacted the ability of **some local Testsigma Terminal instances** to connect successfully. ### **Resolution** The team identified the configuration issue and applied the required fix. After the fix was implemented, the team validated the terminal connection flow and confirmed that the affected functionality was restored for the impacted customers. Customers who were impacted may be required to update their local Testsigma Terminal to ensure they are using the latest corrected version. ### **Preventive Actions** To reduce the likelihood of similar incidents in the future, we are taking the following actions: 1. Automating the management and validation of configuration-related changes. 2. Improving internal checks to detect configuration issues earlier. 3. Updating the local Testsigma Terminal to better handle configuration changes on customer machines. 4. Strengthening release validation around terminal connectivity scenarios. ### **Current Status** The issue has been resolved, and terminal connectivity has been restored for the impacted customers. We will continue improving our configuration management and terminal handling mechanisms to prevent similar incidents in the future.

Mar 19, 2026 · resolved Mar 19

We experienced an issue affecting application availability in the US region, where some users may have faced difficulty accessing the platform or experienced degraded performance. Initial investigation indicated that the issue was caused by a hardware-related problem at the server level. Due to this, the auto-scaling mechanism took some time to respond and provision additional capacity, which led to a temporary delay in service recovery. All services are now fully operational, and system performance has returned to expected levels. We are actively monitoring our systems to ensure everything remains stable and fully operational.

Dec 16, 2025 · resolved Dec 16

# **Root Cause Analysis \(RCA\) – Database Service Disruption** ## **Incident Summary** On **16th December, 2025**, the database service experienced an unexpected outage caused by **heavy load generated from an unintended, resource-intensive operation**. The surge in load exceeded the database’s handling capacity, leading to service instability and eventual crash. ## **Impact** * **Affected Services:** Database-dependent application functionalities * **Customer Impact:** Intermittent application unavailability and degraded performance * **Regions Affected:** EU Region * **Duration:** 10:28 UTC to 16:37 UTC ## **Root Cause** The root cause of the incident was an **unintended execution of a heavy, non-optimized database operation**, which generated an abnormal spike in load. This operation was not adequately throttled or isolated and therefore consumed excessive database resources, leading to system exhaustion and crash. ## **Contributing Factors** * Lack of safeguards to prevent or limit execution of high-cost database operations * Insufficient load-aware throttling and query governance for resource-intensive tasks * Limited early warning signals for sudden abnormal load patterns ## **Resolution** * The offending operation was **immediately stopped** * Database services were **restarted and stabilized** * System performance was closely monitored to ensure full recovery * No data loss was observed ## **Preventive & Corrective Actions** To prevent recurrence and improve system resilience, the following actions will be implemented: 1. **Operational Safeguards** * Introduce strict controls and validation for high-load database operations * Enforce execution limits and safety checks for resource-intensive tasks 2. **Load Management & Scalability** * Implement throttling, rate-limiting, and workload isolation mechanisms * Improve query optimization and execution governance 3. **Monitoring & Alerting** * Enhance monitoring to detect abnormal load patterns earlier * Add proactive alerts for sudden spikes in database resource usage 4. **Architecture Improvements** * Design heavily loaded operations to execute asynchronously and in a scalable manner * Ensure graceful degradation instead of service disruption under extreme load ## **Current Status** * **Incident Status:** Resolved * **System Health:** Stable and operating normally * **Ongoing Monitoring:** Active

Dec 11, 2025 · resolved Dec 11

We have resolved the database server incident that impacted customers in the Europe region. Services have been restored, and we are closely monitoring the system for any further issues.

Is Testsigma Status down right now?

No — Testsigma Status is up. All systems operational as of Oct 12, 4:51 AM UTC.

What is Testsigma Status's current status?

Testsigma Status: All Systems Operational. Alert24 checks Testsigma Status's status page continuously and can notify you the moment it changes.

How do I get alerted when Testsigma Status goes down?

Alert24 monitors Testsigma Status and 3,700+ other cloud and SaaS providers. When an outage is detected it updates your status page automatically and pages your on-call team — no manual checks. Start free at alert24.net.