Freshpaint - Processing for Some Events is Delayed – Incident details

All systems operational

Processing for Some Events is Delayed

Resolved
Degraded performance
Started 11 months agoLasted 3 hours 58 minutes

Affected

Destinations

Degraded performance from 4:14 PM to 6:20 PM, Operational from 6:20 PM to 8:12 PM

Management Console

Degraded performance from 4:14 PM to 8:12 PM

Updates
  • Resolved
    UTC
    Resolved

    We have verified that delivery of events to destinations is working normally without delays. The number of events which show traces in Event Verification in the Freshpaint UI may remain reduced for some customers as a mitigation for handling the current load.

  • Update
    UTC
    Update

    We have scaled our ingest capabilities and destination delivery is now working without delays. The Event Verification feature remains degraded due to the load on tracing services, and some delivered events may fail to appear in event verification in the web UI.

  • Identified
    UTC
    Identified

    Delivery of some events is delayed by up to 40 minutes due to degraded performance in event processing caused by increased load. We have increased the scale of our event processing architecture to handle increased load and are actively monitoring performance. The Event Verification section of the management UI is also experiencing some issues as a result of the same processing delays.