Intermittent site outage
Incident Report for Chit Chats
Postmortem

The database server started refusing many/most new connections with an error related to free lock memory.

This appears to be a recurrence of a problem first encountered when we massively increased the background worker queue.

I quadrupled the setting that controls lock-related memory and restarted the database software to resolve this problem for now.

I have ticketed an issue to try to find a way to monitor free lock memory to try to get advance warning of this problem in the future.

Posted Apr 26, 2018 - 10:17 EDT

Resolved
At around 8:30 EST we became aware of an intermittent error affecting the site. We have identified and corrected a resource issue on one server that was causing this problem.
Posted Apr 26, 2018 - 09:03 EDT