HyperTrack Status Page

Mar 18, 2026 · resolved Mar 19

A platform deployment caused an elevated rate of API errors between 21:08 UTC to 21:47 UTC. New systems were provisioned to mitigate the issues. API degradation is fully resolved. All APIs are operating normally, and we are monitoring closely.

Feb 3, 2026 · resolved Feb 3

A system update caused an elevated rate of API errors between 15:21 and 15:48 PST ( 23:21 to 23:48 UTC ). The update has been mitigated and the issue is fully resolved. All APIs are operating normally, and we are monitoring closely.

Sep 10, 2025 · resolved Sep 10

# **Summary** During the [incident](https://status.hypertrack.com/incidents/rslmyh4y3yy3) on Friday, September 5th, attackers gained access to our AWS account using a leaked key that had been inadvertently exposed in a JavaScript build. Within minutes, the attackers created concealed backdoor access resources. AWS and HyperTrack cleanup operations removed them, though a few resources initially went undetected. On September 10, attackers used the remaining backdoor resources to access the AWS account used by the HyperTrack platform and launched large-scale crypto-mining operations across multiple non-default AWS regions. The attack used compromised third-party AWS accounts unrelated to HyperTrack. This triggered AWS automated abuse detection, which temporarily restricted account access at 11:56 UTC. After investigation and remediation, AWS restored full account functionality by 18:10 UTC. Importantly, **no customer data or platform systems were accessed**. The incident was contained to crypto-mining activity. # **Timeline** **September 10, 2025 \(Wednesday\)** * **10:51 UTC** - Attacker accessed account resources from third-party compromised AWS accounts. * **10:53 UTC** - Crypto-mining workloads launched in new regions. * **10:57–11:00 UTC** - Multiple regions disabled again to hide activity. * **11:43–11:44 UTC** - Attacker switched between external AWS accounts from a new proxy IP address. * **11:56 UTC** - AWS issued compute resources abuse alert and started shutdown process. * **12:08 UTC** - Services were temporarily restricted. * **12:10 UTC** - HyperTrack engineers started responding to the incident and engaged with the AWS security team. * **15:07 UTC** - HyperTrack engineers removed all crypto mining resources, including all hidden non-default regions. * **18:10 UTC** - AWS security team confirmed restrictions lifted and account access restored. HyperTrack platform started receiving events and processing API requests. # **Next Steps** We recently detected and contained unauthorized access attempts related to leaked CI/CD credentials. Our investigation confirmed that no customer data was accessed and build pipelines for platform code repositories have been hardened. In response to the two incidents, we have taken immediate actions to strengthen our security posture in the coming weeks. Since the first incident, we implemented a code deployment freeze to address necessary steps to harden platform environments. These steps include the following: * **Improve Incident Containment & Remediation** * Multiple meetings with AWS security team members for account status and configuration review. * Continue to monitor CloudTrail for anomalies with alerting enabled. * **Continue Strengthening Identity & Access Measures** * Apply tighter Service Control Policies to limit usage to approved regions. * **Expanded Security Monitoring** * Review and update GuardDuty, Inspector, Security Hub, and IAM Access Analyzer configurations across all accounts and regions. * Configure additional alerting for high-severity findings and account activity to ensure immediate response. * **Implement Governance & Ongoing Improvements** * Aggregate all security findings centrally for faster triage and remediation. * Review Cognito and network configurations to ensure no hidden backdoors exist. * Evaluate AWS Control Tower and additional protections \(WAF, Firewall Manager\) for consistent org-wide governance. Our focus remains on delivering reliable, outstanding value to our customers. We are hardening our platform, strengthening comprehensive monitoring, and implementing industry best practices to prevent the recurrence of incidents. Thank you for your continued trust and patience.

Sep 6, 2025 · resolved Sep 6

### Summary On Friday, September 5 at 23:33 UTC \(16:33 PDT\), one of our AWS accounts was accessed using a set of unauthorized keys. The intrusion was quickly detected, and the compromised keys were revoked within 30 minutes. During this window, the attackers provisioned a large number of expensive compute resources for cryptocurrency mining, which in turn triggered AWS to lock the account. Importantly, there was no unauthorized access to platform resources or customer data. Our investigation determined the root cause: a misconfiguration in our JavaScript build process inadvertently exposed CI/CD environment variables. This led to the leak of a pipeline key, which the attackers exploited. We immediately cleaned up all malicious resources and engaged with AWS in real time to fully restore account functionality. Full traffic was restored at September 6, 7:07 UTC \(12:07 am PDT\) ### Timeline **Sep 06 2025 00:13 UTC:** The outage started at as AWS locked down account‌ **Sep 06 2025 00:18 UTC:** AWS alerts HyperTrack engineering **Sep 06 2025 04:00 UTC:** * HyperTrack engineering completed actions to remove over 3000 EC2 instances and associated resources * HyperTrack engineering reiterates to AWS urgency of reopening full access to the account and turning traffic back on **Sep 06 2025 05:33 UTC:** * AWS confirms case was escalated to Service team to reinstate account. * HyperTrack explains urgency: HyperTrack SDK used in millions of apps, supporting nurses and essential workers **Sep 06 2025 07:07 UTC:** * AWS turned account back on **Sep 06 2025 10:30 UTC:** * HyperTrack engineers continued working with the production resources and handling scale as mobile devices were coming back online after the outage. HyperTrack SDK caches tracking data and thus core pipeline inbound traffic scaled up by a couple orders of magnitude **Sep 06 2025 17:15 UTC:** * HyperTrack engineers continued investigation to understand the source of the leak. The source of the leak was identified to be a misconfiguration in our JavaScript build process.‌ ### Next steps Our team is conducting an in-depth analysis of the incident and is enhancing our security policies and procedures. These updates are designed to reinforce safeguards and ensure the continued safety of customer data.

May 23, 2025 · resolved May 23

# **Postmortem: System-Wide Outage Due to Database Degradation** **Incident Date:** May 23, 2025 **Time to Resolution:** 85 minutes **Status:** Resolved **Severity:** Critical \(P0\) ### **Summary** On May 23, 2025, our platform experienced a widespread outage due to degraded performance in our database infrastructure. Specifically, a set of read replicas were affected during the period. This degradation resulted in elevated error rates and unavailability across multiple APIs, including Orders, Workers, Places, and SDK-related services. The issue was fully resolved within 85 minutes. We understand how critical our services are to your operations and sincerely apologize for the disruption. ### **What Happened** A query pattern in our system targeting a key Orders API table failed to use a necessary index. This led to full table scans that overloaded some of our reader instances. As a result, several core APIs failed or experienced extreme latency. ### **Impact** * Customers experienced timeouts or errors when accessing Orders, Workers, and Places APIs * Monitoring and dashboard functionality was temporarily unavailable ### **What We Did** * Identified the problematic query * Deployed a hotfix to ensure proper index usage * Applied a secondary patch to reduce load when workers were not actively tracking * Restarted degraded infrastructure and monitored stabilization * Performed a full incident review across impacted components ### **Remediation and Next Steps** We are taking the following actions to ensure this does not happen again: * **Automated slow-query detection**: We’re enhancing our review pipeline with weekly audits and real-time alerting. * **Improved infrastructure alarms**: CPU and query performance alarms will provide earlier visibility into degradation. ### **Final Thoughts** We are committed to providing a stable and resilient platform. This incident has highlighted areas we must improve, and we’re taking swift action to reinforce our architecture. Thank you for your trust and patience.