Redox Status Page

Sep 9, 2026 · resolved Sep 9

## Summary Following an emergency failover carried out in response to a third-party outage on September 8, a configuration change made to stabilize that failover infrastructure reduced capacity on the system powering dashboard and log visibility. Message processing itself was not affected — data continued to be sent and received normally throughout this incident. Customers began experiencing delayed visibility into message logs and payload details overnight; we identified the issue by 7:45 AM Central on September 9 and fully resolved it by 1:51 PM Central. No data was lost at any point. ## What Happened As part of stabilizing the backup infrastructure used to restore service during a separate third-party outage earlier that day, our team identified an opportunity to make its scaling configuration more resilient to future failover deployments, and moved quickly to put that improvement in place. That change was deployed around 4:34 PM Central on September 8. It did not have the intended effect, and capacity on the backup system supporting dashboard and log visibility was reduced as a result. Message processing itself was not affected. Monitoring coverage on that temporary infrastructure was still being extended at the time, so the reduced capacity was not immediately flagged, and a visibility backlog accumulated overnight. ## Impact Customers experienced delayed visibility into message logs and payload details in the Customer Dashboard, beginning overnight and continuing into the morning of September 9. Message processing was not affected — data continued to be sent and received normally. The impact was limited to the dashboard and log visibility layer. ## How We Resolved It Customer reports and internal alerts led our team to begin investigating at approximately 7:30 AM Central on September 9. Within 15 minutes, we identified that the failover infrastructure was underscaled and began remediation. Between 8:00 AM and 12:00 PM Central, we: * Increased infrastructure capacity to add processing headroom * Scaled the services powering dashboard and log visibility * Distributed traffic across infrastructure to accelerate recovery Visibility into message logs returned to fully caught-up status by 12:51 PM Central. We returned to normal operating capacity by 1:51 PM Central and confirmed resolution on our status page. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * Require monitoring and alerting to be fully wired up on any temporary or backup infrastructure before a team stands down from an incident * Add automated deployment tests to catch configuration resets before they reach production, including on temporary or failover infrastructure * Consider implementing absolute minimum scale limits so services cannot drop below production-safe thresholds, while preserving a standard, well-documented method for intentionally adjusting capacity when needed We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _This issue arose while our team was stabilizing infrastructure used in an emergency failover for a separate third-party outage. See \[Incident 1: Third-Party Service Outage & Processing Delay\] for details._

Sep 8, 2026 · resolved Sep 8

## Summary On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost. ## What Happened In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing. This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved. ## Impact Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours. ## How We Resolved It Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day. ## What We're Doing About This * Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine * We are actively migrating away from this third-party provider We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team. ‌ _The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See \[Incident 2: Dashboard & Log Visibility Delay\] for details._

Jul 9, 2026 · resolved Jul 9

## Summary Between Wednesday, July 8, 2026 at approximately 1:00 PM CT and Thursday, July 9, 2026 at 12:47 PM CT, customers using Carequality DocumentGet received empty document data in responses. Requests appeared to complete successfully on both ends, but document content was missing from what was returned. PatientSearch and PatientQuery were not affected. Service has been fully restored, and no data was lost. All documents were successfully retrieved during this window but not correctly included in outbound responses. ## What Happened The incident was triggered by a routine software update deployed on Wednesday afternoon. * The update introduced an incompatibility in how we format and map document data into outbound responses. Rather than producing an error, the affected step silently returned empty content. * Because the system reported no errors and appeared healthy, our automated monitoring did not alert. The issue was identified through customer reports on Thursday morning. * Once our engineering team identified the root cause, we rolled back the update, which restored correct behavior. Service was confirmed recovered at 12:47 PM CT on July 9. ## What We Are Doing About This To prevent this from happening again and to catch similar issues faster, we are taking the following actions: * **Fixing the Underlying Incompatibility:** We are updating the specific configuration affected by this change so the original update can be safely re-applied without reintroducing the issue. * **Broader Configuration Audit:** We are reviewing other configurations that could be affected by the same underlying change to ensure no similar failures exist elsewhere in the platform. * **Improved Monitoring:** We are adding monitoring that validates response content — not just whether a request completed — so that data-omission failures trigger alerts rather than going undetected. * **Expanded Test Coverage:** We are adding tests that verify the actual content of mapped responses, so this class of failure is caught during development before it reaches production.

Jun 22, 2026 · resolved Jun 23

This incident has been resolved.

May 6, 2026 · resolved May 6

### Summary Between **4:20 PM CT and 5:38 PM CT** on May 6, 2026, some customers experienced an interruption in outbound traffic for VPN-based connections. During this window, data egressing via the network proxy service failed to authorize, resulting in unsuccessful transmissions for MLLP traffic. Service was fully restored by **6:10 PM CT** after configuration settings were corrected and gateway services were restarted. ### What Happened The service interruption was triggered during a planned migration of our network configuration management system. * **Summary of Causes:** During the transition to a new configuration framework, an authentication credential used by the network proxy service was incorrectly identified as unused and subsequently removed. While pre-deployment testing appeared successful, the removal of this credential caused authorization failures once applied across the broader production environment. * **Summary of Impact:** VPN-based connections for outbound data experienced a loss of egress connectivity. They would get an error on the first outbound message on a subscription to a VPN destination. Transmissions were queued during the issue and sent after the problem was resolved; no transmissions were dropped. ### What we are doing about this: * **Enhanced Testing Protocols:** We are developing new end-to-end tests specifically for outbound MLLP traffic to ensure that connectivity is verified across all gateways during configuration changes. * **Improved Service Observability:** We are updating our alerting logic for our proxy services to trigger more immediate notifications if egress traffic volume drops below expected thresholds, ensuring faster detection.

Is Redox down right now?

No — Redox is up. All systems operational as of Oct 12, 4:56 AM UTC.

What is Redox's current status?

Redox: All Systems Operational. Alert24 checks Redox's status page continuously and can notify you the moment it changes.

How do I get alerted when Redox goes down?

Alert24 monitors Redox and 3,700+ other cloud and SaaS providers. When an outage is detected it updates your status page automatically and pages your on-call team — no manual checks. Start free at alert24.net.