Swapcard Status Page

Mar 19, 2026 · resolved Mar 19

### **Summary** Approximately two weeks ago, users began experiencing disruption when accessing analytics data within App Studio. Initial investigation did not immediately identify the root cause; a deeper analysis ultimately traced the issue to a routine infrastructure consolidation that was capping database capacity below the threshold required to handle specific event analytics queries spanning long date ranges. Full service has since been restored following a targeted increase in database resources. ### **Timeline** * **2026-03-11 19:32 UTC** — Incident created. Initial investigation launched following a rise in user-reported loading issues on analytics, despite no prior performance degradation. The initial investigation did not surface a clear root cause at this stage. * **2026-03-18 10:45 UTC** — Automated monitoring triggered alerts for elevated database error rates, corroborated by incoming user reports of analytics failures. * **2026-03-19 09:30 UTC** — Deeper analysis by the Engineering team identified the root cause: high latency and query failures isolated to specific event analytics queries over long date ranges, linked to the earlier infrastructure consolidation. * **2026-03-19 09:40 UTC** — Remediation plan executed: database resources scaled up to meet query load demands. * **2026-03-19 10:05 UTC** — Database performance stabilized; analytics functionality verified as fully restored. ### **Resolution** To restore service, database resources were immediately scaled up to accommodate the throughput required by analytics queries. Once additional capacity was provisioned, query timeouts ceased and application stability was reestablished. The system remained under active monitoring until no further degradation was observed, at which point the incident was marked resolved. ### **Root Cause Analysis** The incident was caused by a capacity mismatch introduced during a routine infrastructure consolidation. Such consolidations are a standard part of our infrastructure lifecycle and are conducted with active monitoring in place to ensure continued performance across all services. In this instance, however, the capacity allocated to the Analytics Database was capped at a level that, while adequate for standard workloads, could not sustain the computational demands of specific event analytics queries executed over long date ranges. The initial investigation did not immediately surface this connection; it was only through deeper analysis that the consolidation was identified as the underlying cause. When these queries were executed, they exceeded available processing capacity, resulting in database timeouts and repeated failures of the analytics service; conditions that persisted until capacity was restored. The investigation also surfaced a broader pattern: a gradual uptick in user-reported loading issues over the preceding two weeks, despite analytics having previously performed reliably. This indicated the capacity cap had been quietly eroding performance well ahead of the full outage. ### **Conclusion** We are straightening our processes to prevent recurrence. Monitoring coverage applied during database consolidations will be strengthened to detect query latency degradation earlier, and capacity planning will be refined to better account for the demands of specific event analytics queries over long date ranges. Looking further ahead, the team is actively working on delivering a fully revamped analytics pipeline to production later this year. This is not an incremental improvement; the new pipeline is being built to deliver deeper, richer metrics that give event organizers a comprehensive view of their event performance.

Mar 13, 2026 · resolved Mar 11

## Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. ## Timeline | Time \(UTC\) | Event | | --- | --- | | Mar 11 – 11:04 | Infrastructure update deployed to production. Changes confirmed as applied. | | Mar 11 – 13:21 | Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. | | Mar 11 – 13:42 | Release step complete. Data refresh running, expected duration ~3–4 hours. | | Mar 11 – 14:49 | Configuration mismatch errors first logged internally. | | Mar 11 – 15:33 | Data refresh reported as complete. Spot checks passed. Final release step initiated. | | Mar 11 – 16:52 | Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. | | Mar 11 – 17:29 | Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. | | Mar 11 – 18:08 | Analysis completed confirming the refresh did not produce an indexation with the expected new fields. | | Mar 11 – 18:39 | Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. | | Mar 11 – 18:43 | New indexation confirmed to contain the correct fields. Full refresh in progress. | ## Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. ## Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: * **Ambiguous deployment signal:** A pipeline warning created uncertainty around whether the deployment had completed cleanly. * **Insufficient spot checks:** Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. **Short-term improvements:** * Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. * Update the release process to include explicit schema validation checkpoints for sub field, as a required step. ## Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.

Mar 11, 2026 · resolved Sep 13

This incident has been resolved.

Mar 11, 2026 · resolved Jun 1

This incident has been resolved.

Mar 11, 2026 · resolved May 5

Please see our post-mortem below regarding the 404 error page from May 5h, 2022 at 13:11 UTC through to 13:20 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, May 5th at 13:12 UTC, we experienced a 404 error page our studio due to a miss configuration on the web application \([studio.swapcard.com](http://studio.swapcard.com)\). On Tuesday, May 5th at 13:12 UTC, our infrastructure team has been automatically alerted of an un usual 404 error page on the web application \([studio.swapcard.com](http://studio.swapcard.com)\). Swapcard monitoring detected the start of disruption and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the incident by checking any recent changes on the web application \([studio.swapcard.com](http://studio.swapcard.com)\) configuration and revert to an older version. In parallel, the cause of the issue was investigated and short & mid term plans were put in place. ## **Mitigation deployment** At 13:13 UTC our infrastructure team has immediately revert the configuration changes for studio.swapcard.com. The switch process took around ~6min. The error reporting has stopped as the update propagated through our infrastructure. Swapcard engineering then monitored [studio.swapcard.com](http://studio.swapcard.com) to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the 404 error messages. At 13:20 UTC, Swapcard confirmed that the update was completed and no further error detected or reported. Swapcard’s Engineering team identified the root cause, and by ~14:31 UTC has redeploy the configuration change corrected. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 1 minutes Time to identify the source of disruption: ~2 minutes Time to initiate recovery: ~6 minutes Time to monitor and restore service pre-crash: ~1 minutes ### **Events of 2022 May 5th \(UTC\)** \(13:11 UTC\) | Initial onset of the error rate increase \(13:12 UTC\) | Disruption identified by Swapcard monitoring \(13:13 UTC\) | Configuration revert has been initiated \(13:12 UTC\) | Swapcard status post activated \(13:19 UTC\) | 404 error page decrease and recovered \(13:20 UTC\) | Incident mitigated \(13:20 UTC\) | Status post resolved \(14:31 UTC\) | Swapcard Engineering redeploy the configuration change corrected Affected customers may have been impacted by varying degrees and with a shorter duration than described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability. **We consider the likelihood of a recurrence of this issue to be extremely low.**