WattTime API Status Page

Apr 29, 2026 · resolved Apr 29

**Start:** 2026-04-29 22:24 UTC **End:** 2026-04-29 22:48 UTC **Endpoints:** All \(due to /login outage\) **Impact:** API requests failed ~100% due to authentication outage. **Cause:** Data processing jobs caused out-of-memory events, which in turn blocked a distributed cache that serves authentication. All logins were unsuccessful due to the blocked cache.  **Fix:** Internal, automatic notifications informed the team at the time of the incident. We recovered by downscaling the authentication service to avoid the distributed cache. We then removed the use of a distributed cache. **Prevention:** We configured our data jobs to prevent memory issues by both right-sizing them and setting default memory requests.

Sep 23, 2025 · resolved Sep 23

**What**: Forecast API endpoints experienced missing data & increased errors **Start**: 2025-09-23 09:35 UTC **End**: 2025-09-23 14:56 UTC **Time To Recovery:** 321 minutes **Outage Duration**: 46 minutes **Endpoints**: _/forecast, /forecast/historical, /signal-index_ **User impact:** API requests to /forecast, /forecast/historical, and /signal-index were taking longer than usual to query for data due to missing or delayed data at the time of request. In most cases requests were successful, though recent data were missing. For some of this time, intermittently, these endpoints also had an increased error rate \(partial outage totaling roughly 46 minutes; this extent varied by user\). **What went wrong**: A low-level cluster planning tool inadvertently filled a critical data store, which interrupted the forecast data generation. Missing data slowed the API response time, and the API did not scale to meet more requests in parallel. **How we fixed it**: Several adjustments were made to recover data generation. We identified a number of improvements to our recovery process to limit the impact of similar events going forward. We apologize for any disruption the data delays and errors caused. We are committed to maintaining a secure and reliable API, and though we're proud of how well we perform, we still look for ways to improve.

Mar 26, 2025 · resolved Mar 26

**What**: An API outage during unplanned maintenance **Outage Start**: 2025-03-26 08:43 UTC **Outage End**: 2025-03-26 09:29 UTC **Duration**: 46 minutes **Endpoints**: All **What went wrong**: In response to news the day before about a K8s ingress-nginx vulnerability, WattTime readied and deployed a patch during an unplanned maintenance window to address the security vulnerability. The upgrade affected some configurations in an unexpected way which resulted in errors to all traffic, including many HTTP 302s. **How we fixed it**: The patch was rolled back to restore traffic. Then, we deployed an upgrade that was compatible with our configurations. The second upgrade was successful at fixing the security vulnerability and caused no downtime. We apologize for any disruption the outage from this unplanned maintenance caused. We are committed to maintaining a secure and reliable API, and though we're proud of how well we perform, we still look for ways to improve.

Oct 31, 2024 · resolved Oct 31

Due to routine database maintenance, our authorization system experienced an outage and some users were unable to query data. This outage affected most users starting at 13:58 UTC, and most users' access was restored by 14:23 UTC. No data generation or storage was affected, and there should be no ongoing issues now that this is resolved. As always, we will add new checks to avoid this issue in the future.

May 18, 2024 · resolved May 18

This statuspage was reporting a Major Outage for about 12 hours from 1430 UTC Saturday until 0215 Sunday. This was a false alarm caused by an issue in the alerting pipeline. No actual outages were observed or reported during this period.