Our event backlog has been processed. No events were lost. We've made some code changes to make our service more resilient in the face of higher load. Thank you.
Posted Aug 09, 2016 - 16:57 PDT
Update
We are continuing to investigate the cause of our event write delays. We are certain no data is being lost -- only slowed down. Queries for some customers are noticeably slower than usual as well.
Posted Aug 09, 2016 - 14:20 PDT
Investigating
Writes are again occasionally being delayed. We are looking into this.
Posted Aug 09, 2016 - 13:20 PDT
Update
We have caught up with all events. Again, no events were lost. We will monitor throughout the day but anticipate no further performance degradation.
Posted Aug 09, 2016 - 11:19 PDT
Monitoring
We have flushed out much of the backlog of events waiting to be written. The backlog has not been fully cleared yet but we believe it will be soon. Read traffic is also returning to normal levels for the impacted customers. We will continue to monitor the situation.
Posted Aug 09, 2016 - 10:40 PDT
Update
We are still investigating the best way to reduce the write event delay. We are continuing to take measures to improve the customer experience. Thank you for your patience.
Posted Aug 09, 2016 - 09:50 PDT
Investigating
We are experiencing a delay in writing events. No events have been lost. Additionally queries from some customers are slower than usual. We are investigating now.
Posted Aug 09, 2016 - 08:49 PDT
This incident affected: Stream API and Compute API.