PowerOutage.us Status Page

Jul 18, 2024 · resolved Jul 19

Azure has fully recovered US Central; no other issues have been found on our end. If anyone runs into anything reach out to us at [email protected] We will be making improvements to our redundancy, we were multi-zone redundant before this incident but will be expanding to multi-region redundancy, we now have a backup stack available in Azure US West and will be fleshing that out and testing failover in the coming weeks.

Jul 20, 2023 · resolved Jul 20

From 2:42 EDT to 3:47 EDT the GeoJson Outages API endpoint returned 500 errors in some cases. A utility had unexpected characters in their geometry data. These characters were ingested and included in our dataset, but when that data was attempted to be returned by the GeoJson API our JSON formatter caught the bad characters and errored out, resulting in 500 errors for any GeoJson Outage API requests that included the effected utility. This has been resolved, and we will be adding additional checks to our ingestion code to prevent this from happening again. We will also be adding a Malformed JSON check to our alerts, instead of waiting for this to get caught by an elevated 500 errors alert. If any clients have any questions, please reach out to [email protected]!

Sep 26, 2022 · resolved Sep 27

DNS changes have been successfully propagated to core DNS servers. No additional issues have been found. Issue is being closed, but we will continue to monitor server health in detail. Feel free to reach out to [email protected] or join our discord discord.gg/PRrqwe7dpD if you have any questions!

Sep 26, 2022 · resolved Sep 26

We experienced a brief increase in web request errors for about two minutes. This was caused by a sudden increase in traffic which was seemingly designed to circumvent our first line of page caching. In total our servers served back 165k 503 errors, and successfully handled 302k requests, all in the two-minute span. Before this surge we were serving about 61k web requests an hour, about 1k a minute. Our server auto-scaling for handling increase loads cannot respond that fast. In this case there is not much we can do in the short term to prevent this type of abuse, how they circumvented the cache forcing our web servers to have to build pages for each request can't be fixed quickly, but it is now on our radar for future development. In the meantime, we have left our web cluster at a higher than normal scaling to help handle other similar request storms if that occur.

Jul 21, 2022 · resolved Jul 22

Azure has confirmed the issue and they are investigating on their end. We have seen limited impact to map load times from our work around so will mark this as resolved. You can check our discord for more follow ups: discord.gg/PRrqwe7dpD