Partial outage writing events
Incident Report for Keen
Resolved
All incoming events are being processed normally. We will continue to keep a close eye out and work on providing more detailed information.
Posted Feb 18, 2015 - 20:34 PST
Update
We're seeing improvements with query performance while continuing to work on the backlog of events from earlier.
Posted Feb 18, 2015 - 19:25 PST
Update
We're making good progress on getting the write path back up and stable, but some queries from the West Coast are now failing. We're in the process of shifting query traffic to fix those failures.
Posted Feb 18, 2015 - 18:39 PST
Update
We're continuing to process the written event backlog in our Dallas datacenter. We are receiving and safely storing events in San Jose, but the insert time is slow enough that you may still see failures (which is our system letting you know we didn't safely store it). We are rehabilitating the infrastructure in San Jose to process those queued events and make them available for querying. Thank you so much for your patience.
Posted Feb 18, 2015 - 18:22 PST
Update
We've rolled out our Write Path infrastructure to the functional datacenter and the backlog of received events is now being processed again. We're continuing to tackle this!
Posted Feb 18, 2015 - 17:09 PST
Update
All incoming event write requests to our San Jose datacenter are currently being rejected. Writes to our Dallas datacenter are still being queued. This means some customers will be experiencing failed writes.
Posted Feb 18, 2015 - 15:55 PST
Update
We're still experiencing an outage with the part of our infrastructure responsible for making new incoming events available for queries. We're approximately one hour delayed in persisting your data. All hands are on deck to fix this for you!
Posted Feb 18, 2015 - 15:12 PST
Update
We’re continuing our work to solve this problem and will continue posting updates as the situation progresses.
Posted Feb 18, 2015 - 14:18 PST
Identified
We’ve identified the problem in our write path and are working hard to solve it.
Posted Feb 18, 2015 - 13:42 PST
Update
We are seeing elevated write times to our San Jose datacenter. The team is continuing to triage and work on the issue.
Posted Feb 18, 2015 - 13:12 PST
Investigating
We are currently experiencing issues writing events to keen.io. The team is investigating.
Posted Feb 18, 2015 - 13:06 PST
This incident affected: Stream API and Compute API.