Query Service Errors

Incident Report for Keen

Resolved

We're back up and running. Our internal load balancer (HAProxy) got stuck in an ornery state, and it took us a while to realize it was the load balancer instead of the actual services causing the errors. Sorry for the intermittent failures!
Posted Mar 31, 2015 - 16:07 PDT

Monitoring

We identified the root cause in our infrastructure management setup that caused the 503 errors in one of our data centers. We are monitoring, but believe the problem is resolved.
Posted Mar 31, 2015 - 15:57 PDT

Update

The team is at work continuing to resolve this issue!
Posted Mar 31, 2015 - 15:47 PDT

Investigating

We're still seeing 503 errors for some customers and are digging in.
Posted Mar 31, 2015 - 15:01 PDT

Monitoring

We've deployed the internal query service and things look back to normal. Sorry for the inconvenience!
Posted Mar 31, 2015 - 14:51 PDT

Investigating

Some users are seeing query error messages saying the query service is overloaded
Posted Mar 31, 2015 - 14:42 PDT
This incident affected: Compute API.