Socure Status Page

Jun 26, 2026 · resolved Jun 26

Socure services have now fully recovered and are operating normally. Our engineering and operations teams will continue to work to conduct a comprehensive root cause analysis (RCA) of the incident. We will share the RCA publicly within the next few days once this review is complete.

Jun 26, 2026 · resolved Jun 26

## **Root Cause Analysis** **Incident:** RiskOS Delayed Webhook Deliveries **Date Range:** June 25, 2026 2:06 PM ET to June 26, 2026 6:03 AM ET **Impact:** Customers subscribed to webhook events experienced delayed webhook deliveries during the incident window. Approximately half of webhook events were delayed; no webhook events were lost. The webhook processing pipeline auto recovered without manual intervention, and no customer action was required. **Status:** Resolved **1.Summary** Between approximately 2:06 PM ET on June 25, 2026 and 6:03 AM ET on June 26, 2026, customers subscribed to RiskOS webhook events experienced delayed webhook deliveries due to an unexpected degradation in the webhook processing pipeline. During the impact window, approximately half of webhook events were delayed. No webhook events were lost. The processing backlog did not surface within the webhook workers until 5:06 AM ET on June26. Once it became available, the queued webhook events were processed, and webhook delivery latency returned to normal at approximately 6:03 AM ET. The pipeline auto-recovered without any manual intervention. Subsequent log and metric analysis identified an issue in the upstream message consumption layer as the primary contributor: following an infrastructure scale-up, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. As a result, webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition self-resolved when consumer group membership stabilized, at which point the queued events were processed quickly and deliveries returned to normal. Standard Kafka topic-lag monitoring did not surface the condition in a timely manner because aggregate topic lag remained within thresholds while the affected partitions accumulated work in the background. Socure engineering has identified a set of corrective and preventive actions — described below — to harden the webhook processing pipeline against partition rebalance failures and to improve observability of partition-level consumption health. As an immediate preventive action, an additional alert that detects this class of failure has already been deployed. **2. Timeline** | **Time \(ET\)** | **Event** | | --- | --- | | **Thu, Jun 25 — 2:06 PM**  | Webhook delivery for a subset of events starts getting impacted. This was following an infrastructure scale-up performed, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. Webhook events on those partitions begin to accumulate without being delivered. Standard Kafka topic-lag monitoring does not surface because aggregate topic lag remains within thresholds. | | **Thu, Jun 25 2:06 PM – Fri, Jun 26 – 5:06 AM** | Webhook delivery latency remains elevated for approximately half of webhook events. Events on unaffected partitions continue to be delivered normally. No webhook events are lost. | | **Fri, Jun 26 — 5:06 AM**  | The processing backlog surfaces within the webhook workers. Consumer group membership stabilizes, the affected partitions begin to be consumed, and the webhook workers report a steep increase in lag as they begin processing the queued events. | | **Fri, Jun 26 — 5:06 AM –6:03 AM** | Queued webhook events are processed rapidly. Webhook delivery latency returns to normal at approximately 6:03 AM ET. No manual mitigation is required. | | **Fri, Jun 26 — 10:00 AM** | On-call engineering reviews telemetry from the overnight processing spike and correlates it with customer reports of delayed webhook deliveries. | | **Fri, Jun 26 — 11:57 AM**  | Incident formally declared in Socure’s incident management system as a retrospective incident. Incident response team assembled and investigation bridge convened. Telemetry for the webhook processing pipeline reviewed in parallel. | | **Post Incident**  | Detailed log and metric analysis of the webhook processing pipeline. Primary contributor identified as a Confluent Kafka | **3. Root Cause** **Primary Root Cause:** Following an infrastructure scale-up performed Thu, Jun 25 02:06 PM, the Kafka consumer group that feeds Socure’s webhook processing pipeline did not rebalance correctly. As a result, a subset of Kafka partitions was effectively unassigned for consumption, and webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition was transient: when consumer group membership stabilized at approximately 5:06 AM ET on June 26, the affected partitions resumed consumption, the queued events were processed rapidly, and webhook delivery latency returned to normal at approximately 6:03 AM ET. No webhook events were lost. **Contributing Factors:** * **Consumer group rebalance failure:** The configured Kafka consumer rebalance protocol did not produce a complete partition assignment for the consumer group following the scale-up.Investigation revealed an edge-case consumer group rebalance failure during the infrastructure scale-up. We have since implemented application-level validation to actively monitor partition health.. * **Observability gap on partition-level consumption:** Existing Kafka topic-lag monitoring did not surface the condition in a timely manner. Topic-aggregate lag remained within thresholds during the impact window because lag on actively consumed partitions stayed at zero, masking the accumulating lag on the affected partitions until consumption resumed and the workers reported a step increase. **Non-contributing Factors:** * **Webhook worker fleet capacity:** The webhook worker fleet was operating within nominal capacity throughout the incident window. No worker-level resource saturation, restart, or scaling failure was associated with the incident. * **Customer-side delivery endpoints:** Customer-owned webhook endpoints were not associated with the incident. Once the backlog began processing, deliveries to customer endpoints completed at expected rates. * **Recent application code changes:** No Socure webhook application code release during the incident window was associated with the incident. **4. Resolution** **Service restoration:** The condition was transient and self-resolved when the affected Confluent Kafka consumer group’s partition assignment stabilized at approximately 5:06 AM ET on June 26, 2026. At that point, the previously unassigned partitions resumed consumption, and the webhook workers processed the queued events rapidly. Webhook delivery latency returned to normal at approximately 6:03 AM ET. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window. **Validation and verification:** A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the Confluent Kafka consumer group rebalance failure as the primary contributor, occurring in the context of an infrastructure scale-up performed the prior day. As an immediate preventive action, Socure engineering deployed an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period — designed specifically to detect scenarios where partitions are not being consumed even when topic-aggregate lag does not surface the delay. Additionally, we have now added a safety net for watching partition updates for ensuring the partition assignments. **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Partition-level consumption monitoring** | Deploy an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period, complementing existing Kafka topic-lag monitoring to detect partition-level consumption gaps that aggregate lag alone does not surface. | **Completed** | | **Consumer group handling partitional updates** | We have implemented application-level event listeners to dynamically monitor Kafka partition state changes. In the event of a recurring Kafka broker or partition failure, the application will now automatically detect the disruption and execute a self-healing recovery sequence. | **June 26th, 2026** | | **Scale-up runbook hardening** | Update the infrastructure scale-up runbook for the webhook processing pipeline to include explicit verification of Kafka consumer group rebalance completion and partition assignment health before the scale-up is declared complete. | **mid-Q3, 2026** | **6. Lessons Learned** This incident exposed two specific gaps in the webhook processing pipeline. First, infrastructure scale-up operations can leave the Confluent Kafka consumer group in an incomplete rebalance state, with a subset of partitions effectively unassigned, without that condition surfacing to operators in real time. Second, Kafka topic-aggregate lag is not a sufficient indicator of partition-level consumption health when only a subset of partitions is affected, because lag on actively consumed partitions can mask accumulating lag on unassigned ones. Follow-up work is focused on detecting and preventing partition-level consumption gaps, hardening the scale-up runbook for the webhook processing pipeline, and strengthening observability so that rebalance anomalies are surfaced as they occur rather than after they have already produced customer-visible impact. **7. Next Steps & Ongoing Commitment** Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers. The corrective actions underway focus on three areas: strengthening partition-level consumption monitoring for the webhook processing pipeline so that rebalance-related consumption gaps are detected as they occur, identifying and remediating the underlying cause of the Confluent Kafka consumer group rebalance failure, and hardening the infrastructure scale-up runbook to verify consumer group health and partition assignment before any scale-upis declared complete. We remain committed to improving the resilience and maintaining a reliable experience for our customers.

Jun 23, 2026 · resolved Jun 23

**Root Cause Analysis** **Incident:** RiskOS Evaluation API Elevated 502 Errors **Date Range:** June 22, 2026, 10:23 PM EDT – 10:59 PM EDT **Impact:** Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 \(Bad Gateway\) responses for approximately 36 minutes. **Status:** Resolved ## **1. Summary** On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors. The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 \(Bad Gateway\) responses. Socure's automated monitoring detected the elevated error rate at **10:24 PM EDT** and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations. Evaluation APIs returned to normal operation by **10:59 PM EDT**. Aggregations were re-enabled after validation at approximately **11:13 PM EDT**. Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary. ## **2. Timeline** | **June 22, 2026 / Time** | **Event** | | --- | --- | | **10:22 PM EDT** | A configuration change in Entity Aggregations introduced an invalid value. | | **10:23 PM EDT** | Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. | | **10:24 PM EDT** | Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. | | **10:25 PM to 10:40 PM EDT** | Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. | | **~10:40 PM EDT** | Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. | | **~10:45 PM EDT** | Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. | | **10:55 PM EDT** | Aggregations processing was fully disabled. Evaluation API traffic stabilized. | | **10:59 PM EDT** | Evaluation APIs returned to normal operation. Elevated 502 errors were resolved.  | | **10:59 PM to 11:13 PM EDT** | Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. | | **11:13 PM EDT** | Aggregations re-enabled following successful validation of the corrected configuration. | ## **3. Root Cause** #### **Primary Root Cause:** An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle. While the affected services were cycling, the RiskOS Evaluation APIs' available capacity to serve traffic dropped, and a portion of incoming requests reached the gateway without a healthy backend service available to respond. The gateway returned HTTP 502 \(Bad Gateway\) responses for those requests. #### **Contributing Factors:** * **Insufficient configuration validation:** The server did not gracefully reject the invalid Aggregation configuration at the load time. Instead of failing safely, the invalid spec propagated into a runtime failure path that crashed the affected pod, producing the customer-visible 502 responses. * **Shared processing impact:** Because the issue affected shared Evaluation API processing, a configuration-specific issue resulted in elevated API failures for customers during the incident window. #### **Non-contributing Factors:** * **Infrastructure health:** No underlying cluster, network, storage, or platform infrastructure issue contributed to the incident. Compute, network, and storage layers supporting RiskOS remained healthy throughout the incident window. Once the offending configuration was mitigated, the backend service stabilized with Evaluation processing restored back to normal. ## **4. Resolution** Socure restored service by disabling aggregations processing entirely, which stabilized Evaluation API traffic and returned normal operation by 10:59 PM EDT. The team then identified and removed the specific invalid aggregation configuration, validated the correction, and re-enabled aggregations at 11:13 PM EDT. Runtime safeguards have been implemented to handle invalid aggregation configurations gracefully without affecting service availability. Pre-activation validation checks currently in development will prevent invalid configurations from reaching production. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Runtime safeguards for Aggregations Configurations** | Updated Evaluation API processing to handle incomplete or missing aggregation configuration values gracefully across all similar configuration processing paths, preventing service crashes. The specific condition that triggered this incident is now handled safely without impacting service stability. | **Completed** | | **Aggregations configuration validation** | Added stricter validation so invalid, incomplete, or unsupported configurations - are rejected before they can become active in production. | **In Progress** | | **Pre-activation configuration checks** | Added checks before aggregation configurations are used in Evaluation API processing to identify invalid or conflicting states earlier in the pipeline, before they can impact live traffic. | **In Progress** | | **Blast radius control**  | Strengthen isolation so an issue in a specific workflow path does not affect broader shared Evaluation API processing. | **Q3, 2026** | ## **6. Lessons Learned** This incident highlighted a specific gap between Aggregations configuration validation and Evaluation API runtime handling. Follow-up work is focused on preventing invalid configurations from becoming active, improving runtime safeguards, and strengthening isolation for workflow-specific issues. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS Evaluation APIs for reliable, real-time decisioning. We take ownership of the disruption caused by this incident and are implementing changes to reduce the likelihood and impact of similar issues. The corrective actions underway focus on three areas: strengthening aggregations configuration validation, improving Evaluation API runtime safeguards, and limiting the blast radius of any future configuration-related issues.  We remain committed to improving the resilience and maintaining a reliable experience for our customers.

Jun 16, 2026 · resolved Jun 16

**Root Cause Analysis** ### Incident: Unable to generate DocV links via dashboard **Date Range:** Tue, Jun 16, 2026 8:52 am – 10:12 am ET **Impact:** Document Verification link generation via Admin Dashboard **Status:** Resolved ### **1. Summary** On Jun 16, 2026, a scheduled deployment introducing stricterOn June 16, 2026, a scheduled deployment introducing stricter access control logic inadvertently broke the Document Verification \(DocV\) link generation workflow for a subset of client configurations. The issue persisted for approximately 1 hour and 20 minutes after being reported by a client. The incident was mitigated via a full rollback. We are actively overhauling our release validation protocols and implementing shadow-mode deployment capabilities to ensure enterprise permission configurations are thoroughly tested before enforcing hard access restrictions in the future. ### **2. Timeline** | **Time \(ET\)**  | **Event** | | --- | --- | | **June 16, 5:16 AM ET**  | A production deployment was completed, introducing updates to the system's permission logic. This logic contained a defect rendering the DocV request links feature unusable. | | **8:52 am ET** | Clients started reporting Admin Dashboard DocV link generation stuck in loading for some users. | | **9:04 am–9:24 am ET**  | Upon determining that the problem stemmed from recent updates to role and permission provisioning within the admin dashboard, the team resolved to execute a revert. | | **10:00 am–10:08 am ET**  | The rollback was verified in a staging environment before being deployed to production, followed by thorough regression and sanity testing. | | **10:10 am ET**  | The client confirmed that the Document Verification link generation workflow was fully restored. | | **10:11 am ET** | The incident was resolved. | ### **3. Root Cause** * **Primary Root Cause:** * Stricter permission validation logic was deployed in the morning that was meant to make our permission model more consistent across pages. However, this update caused unintended disruptions for specific role setups, preventing authorized users from accessing the document request link generation process. ### **4. Resolution** * To restore functionality, the team implemented a rollback of the stricter permission enforcement logic. Following the deployment, we performed validation of the admin dashboard's link generation flow and verified that the service was operating correctly. ### **5. Corrective and Preventive Actions** | **Action**  | **Description** | **ETA / Status** | | --- | --- | --- | | Revisit stricter access control with a safer rollout approach | Reintroduce the permission enforcement in shadow mode first using a safer rollout approach \(e.g., staged rollout and additional validation\) to prevent unintended impact to existing customer role configurations. Communicate with impacted customers to guide them in updating their permission settings | **07/03** | | Add monitoring for errors/issues in document link generation | Add monitoring for errors in document link generation to be notified of and address issues in the future proactively. | **07/03** | | Expand validation of access control changes to cover all user configurations | Update our release validation process so access-control and permission changes are tested against a broader range of role setups \(including environment-scoped scenarios\) prior to production release. | **07/03** | ### **6. Lessons Learned** * Permission/environment enforcement needs testing across representative customer role setups. Dark-launch/shadow-mode logging to measure who would be blocked, communicating to customers, and establishing a rollout plan is a safer path than hard enforcement.  ### **7. Next Steps & Ongoing Commitment** We are committed to _Corrective and Preventive Actions_ noted above.

Jun 16, 2026 · resolved Jun 16

**Root Cause Analysis** **Incident:** RiskOS API Services Degradation During NAT Gateway Remediation **Date Range:** 06-16-2026 **Impact:** Intermittent degradation for a subset of API requests **Status:** Recovered ## **1. Summary** On June 16th, 2026, RiskOS API services experienced intermittent degradation for a subset of API requests between **3:46 AM EDT and 4:02 AM EDT**. During this period, some requests failed while network connectivity was being transitioned as part of a controlled remediation. Earlier that day, Socure observed elevated connection error alerts and identified critically high Cloud NAT port utilization in the RiskOS commercial environment. The condition was not impacting existing customer transactions at the time of detection, but it created a significant capacity risk for outbound connectivity if additional workloads restarted, scaled, or deployed. To reduce the risk of broader impact, Socure performed a controlled network remediation during a lower-workload window. The remediation improved capacity handling and reduced shared NAT resource contention across origin IP blocks. During the transition, some existing connections were briefly interrupted and had to be re-established, resulting in intermittent request failures. The remediation activity completed in approximately 5 minutes, and overall recovery and stabilization completed within approximately 15 minutes. No customer action was required after connectivity was restored. ## **2. Timeline** | **June 16th,** **2026 / Time** | **Event** | | --- | --- | | 12:00 AM EDT | The Socure Engineering team observed elevated connection error alerts and began an investigation. No customer transaction impact was identified at this time. | | 12:30 AM EDT | Investigation identified critically high Cloud NAT port utilization, creating a capacity risk for outbound connectivity. SRE confirmed that existing customer transactions were not being impacted at that time. | | 12:30–3:45 AM EDT | The Socure Engineering team evaluated remediation options, assessed the risk of leaving the configuration unchanged, and planned remediation during a lower-workload window to reduce potential customer impact. | | 3:45 AM EDT | The engineering team proceeded with the planned network capacity remediation. | | 3:46 AM EDT | Intermittent request failures began as some existing network connections were briefly interrupted during the transition to the updated configuration. | | 3:52–4:00 AM EDT | Network configuration updates were applied to improve capacity handling and reduce shared NAT resource contention across origin IP blocks. | | 4:02 AM EDT | Intermittent request failures ended after traffic completed transition to the updated configuration and connections were re-established. | | ~4:02 AM EDT | Platform connectivity stabilized. | | ~4:15 AM EDT | Incident considered resolved after validation and monitoring confirmed recovery. | ## **3. Root Cause** **Primary Root Cause:**The incident was caused by critically high Cloud NAT port utilization in the RiskOS commercial environment. The NAT configuration had insufficient available port capacity to provide safe headroom for normal connection churn, workload restarts, scaling activity, or deployments. **Contributing Factors:** * The residual NAT capacity risk was a consequence of emergency recovery activities undertaken after the June 9 RiskOS service outage, where the primary focus was restoring service availability. * NAT port utilization reached approximately 99–100%, creating a significant risk of outbound connectivity degradation. * NAT capacity was shared across origin IP blocks, which increased the likelihood of port exhaustion under workload pressure. * During remediation, some existing connections were reset while traffic transitioned to the updated network configuration, resulting in temporary intermittent request failures. ## **4. Resolution** Socure updated the NAT configuration to improve capacity handling and reduce shared-resource contention across origin IP blocks. The remediation was performed because the existing NAT utilization level presented a significant reliability risk, and delaying the change could have increased the chance of broader service degradation. During the transition, some existing connections were reset and had to be re-established. Once the updated configuration took effect, intermittent request failures stopped and platform connectivity stabilized. No customer action was required after connectivity was restored. Requests retried after recovery were expected to process normally. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | Reduce NAT capacity contention | NAT usage was separated to reduce shared capacity risk and limit the blast radius of future NAT capacity issues. | Completed | | Enable dynamic port allocation | Dynamic port allocation was enabled to improve NAT port capacity management and reduce exhaustion risk. | Completed | | Improve NAT utilization monitoring | Add or tune monitoring for NAT port utilization, dropped packets, and connection errors with actionable alert thresholds before utilization reaches critical levels. | Completed | | Move network configuration to Infrastructure as Code | Continue moving RiskOS network configuration into version-controlled Infrastructure as Code to improve reviewability, repeatability, and recovery as a part of AWS Migration. | Planned | ## **6. Lessons Learned** * NAT port utilization should be treated as a critical reliability signal for workloads that depend on outbound connectivity. * NAT capacity should have sufficient headroom for normal connection churn, workload restarts, scaling events, and deployments. * Remediation of high-risk network conditions can still create temporary impact when existing connections are reset, so change planning should include clear validation and communication steps. * Network changes should include pre-change checks for current utilization, available NAT capacity, expected connection reset behavior, and rollback options. * Continued investment in environment isolation, Infrastructure as Code, and stronger monitoring will reduce the likelihood and impact of similar incidents. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS to be available and reliable. We take accountability for the disruption caused during this remediation and are continuing to improve our network architecture and operational controls. To reduce the likelihood of recurrence, Socure is pursuing the following: 1. **NAT Capacity Controls:** We are improving monitoring and alerting for NAT utilization, connection drops, and port exhaustion risk so that capacity issues can be identified and remediated before they impact customer traffic. 2. **Infrastructure as Code:** We are continuing to move RiskOS network configuration to a version-controlled model so that every change is reviewed, repeatable, and can be recovered quickly. 3. **Environment Isolation:** We are separating network resources and reducing shared capacity dependencies so that the impact of any capacity issue or network change is contained. 4. **Safer Network Operations:** We are strengthening pre-change validation, rollback planning, and maintenance window selection for network changes that may affect customer traffic. 5. **Operational Runbooks:** We are updating operational runbooks for NAT capacity exhaustion and Cloud NAT changes so that response and recovery steps are repeatable and clearly documented.

Is Socure down right now?

No — Socure is up. All systems operational as of Jul 13, 11:43 PM UTC.

What is Socure's current status?

Socure: All Systems Operational. Alert24 checks Socure's status page continuously and can notify you the moment it changes.

How do I get alerted when Socure goes down?

Alert24 monitors Socure and 3,700+ other cloud and SaaS providers. When an outage is detected it updates your status page automatically and pages your on-call team — no manual checks. Start free at alert24.net.