We experienced significant instability in our database cluster today which caused delays in both our Data Collection and Data Analysis APIs. We've taken steps to mitigate the instability and have stabilized both APIs. While we are stable right now, we're still working to fix the root cause and will continue working until we've remedied the issue completely. All of that being said, we're happy to report that no data was lost as a result of today's instability :)
We appreciate your patience and apologize for any inconvenience these delays have caused. Please feel free to reach out with any questions or concerns you might have and we will be happy to help. Have a great evening!
Posted May 18, 2016 - 17:02 PDT
Update
Unfortunately response times for our Data Collection API are increasing again, and are definitely in a degraded state. We're aware of what's causing the issues and are working as quickly as we can to resolve them, but response times will continue to be elevated for a bit. We are keeping an eye on things, and will continue to update you if things get worse or as we work towards resolving the issue.
Posted May 18, 2016 - 12:56 PDT
Monitoring
Good news! Response times for the Data Collection API have returned to normal. Response times for the Data Analysis API are still elevated, but have decreased significantly from where they were a few hours ago. We will continue to monitor and will update you as we have additional information to share. Thank you again for your patience!
Posted May 18, 2016 - 12:11 PDT
Update
We continue to experience elevated response times for both our Data Collection and Data Analysis APIs. We have made configuration changes to mitigate the issue and are seeing improvement, but we're not out of the woods yet. We'll continue to pass along information as it becomes available. Thank you for your patience!
Posted May 18, 2016 - 10:05 PDT
Investigating
We are seeing elevated response times from our Data Collection APIs and Data Analysis APIs, as well as increased write latencies for successful writes. Customers may experience timeouts on requests to both APIs, but no data is being lost for successful API calls. We are investigating now, and will provide updates as we have more information.
Posted May 18, 2016 - 08:44 PDT
This incident affected: Stream API and Compute API.