Performance Matters (Unify) loading with a 502 Bad Gateway error message
Incident Report for Performance Matters
Resolved
Last night we released an update to our online testing application that helped reduce the load on our databases. The update was low risk and targeted a specific set of queries that were causing extensive load on our system. These queries were originally intended to add reporting for start time, end time, and duration in our OLA Student Admin page for each student, but the feature was never released. Fortunately, this meant commenting out the execution of these statements reduced the database load while having no negative impact on the end user's experience.

We monitored OLA and Unify throughout the day and will continue to do so, but we are going to resolve this incident for now.

Thank you all for your continued partnership and patience while we work to enhance the system to provide a quality testing experience for all involved.
Posted May 22, 2019 - 15:46 EDT
Update
This testing period represents (and historically has) the highest usage of the Performance Matters system. Based on our system monitoring, the majority of students testing are able to do so without issue. A portion of students have experienced latency due to the volume of testing done from 9:30am-12:00pm ET. For the remainder of this testing window, some reports of slowness may be experienced during these times.

Our senior engineers and architects are actively evaluating any improvements or optimizations that can increase capacity for the rest of this testing period. Long-term solutions to improve performance are in development and currently planned for the summer.

The highest contributing factor to the load on the system has been the use of Baseball Card and the act of releasing tests. If possible, users should avoid running large reports in Baseball Card and releasing tests during the hours of 9:30am-12:00pm ET. This will help mitigate reports of difficulties using the platform.
Posted May 21, 2019 - 13:04 EDT
Monitoring
Performance Matters is functional, but users can experience some pages spinning or slowness at times. We are continuing to monitor and will continue to provide updates as we have them.
OLA (student testing platform) is working properly.
Posted May 21, 2019 - 11:16 EDT
Update
We are still investigating reports of slowness on Performance Matters, but the users should be able to login to the site without error now.
Posted May 21, 2019 - 10:23 EDT
Investigating
We are currently investigating reports of Performance Matters throwing a 502 Bad Gateway error when users attempt to access the site. We will update this page as we have more information.
Posted May 21, 2019 - 10:11 EDT
This incident affected: Student Platform.