The longer term solution has been implemented and background job processing is now stable. So sorry about the disruption today, everyone.
Posted Sep 18, 2020 - 00:29 EDT
Update
The delay in processing background jobs was caused by select background jobs that were blocking majority of background workers, and causing the backlog. We have a temporary fix while working on a more permanent solution.
Posted Sep 17, 2020 - 23:23 EDT
Monitoring
At around 18:53 PT background job processing has considerably improved. By 19:03 PT most of the backlog has been processed. We will continue to monitor the situation.
Posted Sep 17, 2020 - 22:08 EDT
Investigating
As of 18:10 Pacific Time we are experiencing issues processing background jobs. This impacts operations such as buying postage, shipment imports, and other tasks that happen in the background. The issue is currently being investigated.
Posted Sep 17, 2020 - 22:02 EDT
This incident affected: Web applications (Web Application and API, Shipments and Postage).