Teem Status Page

May 14, 2025 · resolved May 16

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. Thank you for all the patience as we worked through this issue.

Apr 14, 2025 · resolved Apr 21

This issue is currently resolved with a release that happened last night. After confirmation we are going to be closing out this Status Page. We appreciate all the patience with our customer base as we have worked diligently to get this update out.

Mar 4, 2025 · resolved Mar 5

**Teem by Eptura detailed Root Cause Analysis | 3/4/2025**  **S1 – Teem Inaccessible**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On March 4th, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 7:34 AM. We received notice of some people not able to access [app.teem.com](http://app.teem.com). At 10:19 AM our Engineering team clearing tables to help alleviate performance. At 11:29 AM we moved the issue to an S2 due to website being up and accessible. At 10:17 PM we are finishing clearing tables meaning performance should now be at full. 3:49 AM website is working as intended. At 9:17 AM the issue was resolved and we entered the monitoring phase for two hours. At 11 AM we cleared the status page.    **Total Duration of Event:**  27 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

Feb 28, 2025 · resolved Mar 5

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

Feb 25, 2025 · resolved Feb 25

**Teem by Eptura detailed Root Cause Analysis | 2/25/2025**  **S1 – Duplicate Events causing downtime**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On February 22, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 8:58 AM. We received notice of some instances of slowed experience and inability to access [app.teem.com](http://app.teem.com). At 8:59 AM our Engineering team is reaching out to AWS team to address this. At 10:30 AM we moved the issue to an S1. At 10:45 AM the site became accessible again, but with slowness. At 12:52 PM we resolved this issue and went into a monitoring phase. At 2:02 PM we cleared the status page.    **Total Duration of Event:**  4 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance