Duration: 2 hours and 38 minutes

Affected Components: Destinations

Mar 9, 17:56:43 GMT+0
Investigating - We have identified an incident preventing events from being successfully processed and delivered to all destinations. We are currently investigating the cause of the incident and will provide an update shortly..

Mar 9, 18:14:42 GMT+0
Monitoring - A configuration error with a Google Analytics 4 endpoint resulted in a code failure inside of our ingest processing code that prevented some batches of events from being processed and delivered successfully. We have repaired the error and are working on restoring any events that may have been lost during the incident. Delivery to all events should now be operational again. We will continue to monitor while we backfill events and deploy a permanent fix to our ingest engine..

Mar 9, 20:34:37 GMT+0
Resolved - This incident has been resolved. We are continuing to work on backfilling impacted events and will send a more detailed email message to customers shortly..

Duration: 2 hours and 16 minutes

Feb 1, 00:45:31 GMT+0
Investigating - Freshpaint service may experience intermittent disruption as a result of ongoing security maintenance. This will not impact data integrity but may result in temporary delays in event delivery and the Freshpaint app may be slow or temporarily unavailable. We will update this with more details shortly. If you have concerns or are experiencing issues please contact [support@freshpaint.io](mailto:support@freshpaint.io)..

Feb 1, 03:01:03 GMT+0
Resolved - Maintenance has concluded and all Freshpaint services should now be working normally. Delivery to some destinations may have been delayed during maintenance. Please contact [support@freshpaint.io](mailto:support@freshpaint.io) if you experience any issues or have questions or concerns..

Duration: 3 days and 59 minutes

Affected Components: Warehouses

Jan 23, 21:00:00 GMT+0
Identified - We're experiencing delays with data warehouse syncs. There is no risk of data loss, only temporary delays. We're working on implementing a fix..

Jan 26, 19:15:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 26, 21:58:39 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 31 minutes

Affected Components: Data Collection

Dec 1, 22:46:16 GMT+0
Identified - We have taken steps to attempt to restore service, but it is not yet restored. We are still working to determine the magnitude of the impact..

Dec 1, 23:14:56 GMT+0
Resolved - We have taken steps to mitigate the incident and expect service to be restored..

Dec 1, 21:44:19 GMT+0
Identified - We are currently investigating degraded google ads call conversion tracking. This affects customers who are using freshpaint's proxy to request replacement phone numbers from google. We are working to quickly restore service and determine the magnitude of the impact..

Duration: 3 hours and 59 minutes

Affected Components: Destinations, Freshpaint UI

Nov 7, 20:12:47 GMT+0
Resolved - We have verified that delivery of events to destinations is working normally without delays. The number of events which show traces in Event Verification in the Freshpaint UI may remain reduced for some customers as a mitigation for handling the current load..

Nov 7, 16:14:00 GMT+0
Identified - Delivery of some events is delayed by up to 40 minutes due to degraded performance in event processing caused by increased load. We have increased the scale of our event processing architecture to handle increased load and are actively monitoring performance. The Event Verification section of the management UI is also experiencing some issues as a result of the same processing delays..

Nov 7, 18:20:00 GMT+0
Identified - We have scaled our ingest capabilities and destination delivery is now working without delays. The Event Verification feature remains degraded due to the load on tracing services, and some delivered events may fail to appear in event verification in the web UI..

Duration: 2 days, 11 hours and 11 minutes

Affected Components: Destinations

Jul 8, 03:24:40 GMT+0
Investigating - We are currently investigating this incident..

Jul 9, 16:00:40 GMT+0
Monitoring - Events are sending smoothly to Reddit as of yesterday. We will continue to monitor the situation closely..

Jul 10, 14:35:42 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 3 minutes

Affected Components: Freshpaint UI

Jun 12, 18:26:22 GMT+0
Investigating - We are aware of an issue affecting the ability to log into the Freshpaint web management UI and are actively investigating. Delivery of events to integrations and all other features continue to function without issue..

Jun 12, 21:29:14 GMT+0
Resolved - Firebase has reported that all affected services are now operating normally. If you continue to experience any issues with login, please contact Freshpaint Support..

Jun 12, 20:55:45 GMT+0
Monitoring - Login is impacted by an upstream issue with Firebase, our authentication provider. We have reports that login is working for many customers again. We are monitoring and will provide further updates as we receive them from Firebase..

Duration: 48 minutes

Affected Components: Destinations

Apr 8, 21:16:45 GMT+0
Investigating - Some customers using the Google Ads destination have reported issues with their conversions being reported on Google Ads. This only impacts customers using the Google Ads proxy destination and not customers directly using the Google Ads Conversions API. We are currently investigating this issue and will provide an update when we know more..

Apr 8, 22:04:43 GMT+0
Resolved - We have identified a fix and rolled it out. Google Ads destinations should now be working normally for all customers. Please notify us if you have any other issues..

Duration: 2 hours and 1 minute

Affected Components: Maps

Mar 14, 18:55:00 GMT+0
Investigating - We are currently investigating this incident..

Mar 14, 20:11:00 GMT+0
Identified - The issue has been narrowed down to a third-party CDN, investigating mitigations..

Mar 14, 20:28:47 GMT+0
Monitoring - We migrated CDN providers and have noticed the error rate return to normal levels. Monitoring for stability..

Mar 14, 20:56:09 GMT+0
Resolved - Error rates have returned to the expected baseline. Maps are stable and fully operational..

Duration: 1 hour

Affected Components: Destinations

Mar 5, 17:48:00 GMT+0
Resolved - A bug in our code caused attempts to send events to destinations to immediately fail and be written directly into our retry queue, causing a delay in event sending. This incident has now been resolved..

Duration: 1 hour and 2 minutes

Affected Components: Destinations

Dec 6, 17:27:41 GMT+0
Investigating - We are currently investigating this incident..

Dec 6, 18:01:42 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Dec 6, 18:29:44 GMT+0
Resolved - This incident has been resolved..

Dec 6, 00:14:00 GMT+0
Resolved - We've identified an elevated failure rate in loading Wistia videos via the Freshpaint Video Proxy on some mobile players. We are actively investigating this issue and will provide status updates through this page. As a reminder, the Freshpaint Video Proxy is a sunsetting product, and we recommend long-term that customers migrate to Freshpaint Video Hosting for their video needs..

Duration: 21 minutes

Affected Components: Destinations

Nov 21, 16:31:16 GMT+0
Identified - We have identified an issue that is delaying ingestion and are deploying a fix..

Nov 21, 16:52:40 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 20 hours and 25 minutes

Affected Components: Destinations

Nov 20, 02:14:00 GMT+0
Identified - The deprecated version of SQL transformations are failing in a single environment. Most customers are not impacted..

Nov 21, 22:39:00 GMT+0
Resolved - This incident has been resolved..

Duration: 11 hours

Affected Components: Video Hosting

Oct 15, 16:15:00 GMT+0
Resolved - This incident has been resolved..

Oct 15, 05:00:00 GMT+0
Identified - On Monday evening, we deployed a change to our video infrastructure to enable better availability of metadata on videos. This change contained an error which did not immediately surface due to caching. As caches timed out, some videos experienced issues loading on Tuesday morning. The issue was resolved shortly after we detected it..

Duration: 1 hour and 20 minutes

Affected Components: Maps

Oct 4, 22:00:14 GMT+0
Monitoring - We have identified the issue and have pushed a fix. We are currently monitoring..

Oct 4, 23:20:17 GMT+0
Resolved - This incident has been resolved..

Duration: 15 days, 9 hours and 45 minutes

Aug 15, 06:04:20 GMT+0
Investigating - We are experiencing an uptick in error responses from YouTube. Videos are loading, but are slow to load..

Sep 10, 19:36:30 GMT+0
Investigating - Some YouTube videos are failing to load via the Freshpaint Video Proxy. We are investigating the incident..

Sep 25, 14:37:41 GMT+0
Monitoring - The YouTube proxy seems to be working and we are monitoring the service..

Sep 30, 15:49:26 GMT+0
Resolved - This incident has been resolved..

Duration: 28 minutes

Aug 14, 19:30:00 GMT+0
Resolved - This incident has been resolved..

Aug 14, 19:02:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 days, 3 hours and 42 minutes

Aug 6, 16:41:05 GMT+0
Investigating - We are aware of elevated error rates in the YouTube video proxy and are actively investigating..

Aug 6, 21:05:50 GMT+0
Identified - We have identified the issue and implemented a temporary mitigation. The team is working on a longer-term fix..

Aug 7, 01:21:51 GMT+0
Monitoring - After a setback, we are now able to confirm mitigation of the incident. We are monitoring the fix and will add updates as necessary..

Aug 9, 20:23:03 GMT+0
Resolved - This incident is now resolved..

Duration: 5 days, 23 hours and 27 minutes

Jul 24, 17:39:02 GMT+0
Investigating - We are aware of elevated error rates in the Freshpaint Video Proxy Shield for YouTube videos and are currently investigating..

Jul 24, 18:33:50 GMT+0
Identified - An elevated rate of failures from the YouTube service has caused a cascading chain impacting all videos served by the Freshpaint Video Proxy, including those served by Vimeo and Wistia. A fix is in progress for Vimeo and Wistia, but no fix has been identified for the YouTube issues..

Jul 24, 18:54:01 GMT+0
Identified - We have deployed a fix for the errors serving videos hosted on Vimeo and Wistia. YouTube videos continue to experience an elevated failure rate..

Jul 24, 19:37:12 GMT+0
Investigating - YouTube videos served through Freshpaint continues to be down. We're investigating our options and we expect to have an update in the next several hours..

Jul 24, 22:58:55 GMT+0
Monitoring - YouTube videos are starting to load again for customers using Freshpaint's video proxy service. We are continuing to monitor the situation..

Jul 25, 20:59:36 GMT+0
Monitoring - We are continuing to monitor things on our end. The issue is temporarily mitigated, and the team is working on a longer-term fix. We have also communicated with affected customers directly..

Jul 30, 17:05:47 GMT+0
Resolved - We now consider this incident resolved, but will reopen should the need arise. The team is continuing to work on long term options..

Duration: 1 hour and 38 minutes

Jul 23, 19:35:42 GMT+0
Investigating - We are currently investigating this incident..

Jul 23, 20:30:02 GMT+0
Investigating - We have scaled our services, which has partially reduced the error rate. We continue to see more errors than typical for the YouTube video proxy. We are actively investigating the root cause of the incident..

Jul 23, 21:13:50 GMT+0
Resolved - The elevated rate of failures for the YouTube proxy should now be resolved for all customers..

Duration: 1 hour and 34 minutes

Affected Components: Destinations, Freshpaint UI, Warehouses

Jul 11, 22:33:55 GMT+0
Resolved - This incident has been resolved..

Jul 11, 22:27:58 GMT+0
Identified - We've located the source of the issue and implemented a fix. We are beginning to see remediation..

Jul 11, 21:00:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 4 hours and 44 minutes

Jul 9, 22:40:42 GMT+0
Resolved - This incident has been resolved..

Jul 9, 17:56:17 GMT+0
Investigating - We are currently investigating this incident..

Jul 9, 18:48:55 GMT+0
Investigating - We have several leads that we're looking at, but no certainly identified cause yet. The team is continuing to investigate..

Jul 9, 19:41:40 GMT+0
Investigating - While we've made some progress, we are continuing to investigate the issue..

Jul 9, 22:30:08 GMT+0
Identified - We have identified a fix and are rolling that out. Some users may see videos working immediately, while others may see it shortly..

Duration: 1 day, 16 hours and 50 minutes

Affected Components: Destinations

Jun 27, 02:12:00 GMT+0
Identified - Error rates for several destinations have climbed. The team identified an issue with our ingestion and mitigated the issue..

Jun 28, 19:02:00 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 31 minutes

Jun 7, 14:30:00 GMT+0
Investigating - We are currently investigating this incident..

Jun 7, 21:01:00 GMT+0
Resolved - This incident has been resolved..