AWS Migration
Incident Report for Performance Matters
We keep this incident open on the status page to confirm that the global issues with scanning and responses moving to reporting were resolved. We have not experienced any widespread incidents since November 8, so we are resolving this incident.

Thank you for your patience while we worked through the issues after the AWS Migration.
Posted Nov 27, 2019 - 12:08 EST
We have completed our validation testing on the scans that were reprocessed and the results from submissions through OLA. All results should now be showing in reporting.
Posted Nov 10, 2019 - 23:52 EST
We have made significant progress and have completed our scan re-processing efforts. If you happen to find any successfully archived scans that aren't showing in Student Item Analysis, please submit a support case so it can be researched.

Following last night's builds, we are also validating the OLA results in the system. We will post an update when our QA of the OLA results is complete.
Posted Nov 10, 2019 - 13:04 EST
We have made good progress today, and our teams continue reprocessing the scans and working through our QA efforts to verify the reprocessed records.
Posted Nov 09, 2019 - 19:50 EST
The builds ran as expected last night and all auto-imported files processed. We are still working on re-processing scans and loading them into reporting. Districts will see more and more results populate reporting as the day goes on. Our QA effort is underway, and we will post an update when we have confirmed everything has completed processing.
Posted Nov 09, 2019 - 11:04 EST
Today’s system performance was very encouraging. We saw scanning and OLA performing at the expected speeds. The engineering team is continuing to clean up smaller details regarding the migration.

We will be continuing the re-processing efforts to get scan results loaded into the reporting over the weekend. We have a large QA effort to ensure accuracy that all individual level results are processed in reporting for the start of the week.
Posted Nov 08, 2019 - 16:01 EST
Due to the hotfix last night, the automated files did not run during the nightly builds. The builds will run correctly this evening and moving forward.
Posted Nov 08, 2019 - 11:03 EST
The hotfix to correct the processing delay from scanning to reporting did go out last night and was successful. We have begun the re-processing efforts to get all scans from the last two weeks into reporting. This will continue throughout the weekend. We will post an update when this has been completed.

The system is looking good this morning with no delays on OLA or scanning.
Posted Nov 08, 2019 - 09:56 EST
Scanning has continued to go well today. We are still planning on releasing the fix to correct the delay in scan results showing in reporting. We will confirm tomorrow morning that this was released and had the desired impact.

For districts experiencing performance issues with OLA and login errors, engineering confirmed this is related to the fix that is scheduled for tonight.
Posted Nov 07, 2019 - 18:16 EST
Scanning capacity continues to handle the current volume. The engineering team has figured out the issue with results not displaying in reporting. They have a fix for this that is in its final stages of testing. As long as the testing passes, we will be pushing out the fix this evening. That will ensure that the scan results process to the reporting. That being said, we do have results from prior days that we will need to send to reporting. Our plan is do this over the weekend to get all successful scans from the last two weeks processed in reporting.

The slowness districts experienced in OLA and OLA Student Admin are connected to the issue delaying scan results to reporting. The fix for the scan results will help with the slowness on OLA as well. We have done some interim work to help alleviate the slowness on OLA today, but districts could still experience slower performance today as well.

We are also getting reports of students not being able to login to OLA or not being able to start tests. Engineering is currently investigating this issue.
Posted Nov 07, 2019 - 12:13 EST
We have received reports from a few districts experiencing slowness in OLA Student Admin as well as students getting a message that the test is no longer available to them in OLA. We are investigating this currently. It is not impacting everyone at this time.
Posted Nov 06, 2019 - 14:33 EST
An OLA performance issue was identified at 11:00 ET today and was resolved by 11:27 ET.
Posted Nov 06, 2019 - 12:37 EST
A timing issue with the Nightly Build process has been discovered, which is preventing display of archived scan results. This is currently being addressed in conjunction with the capacity issue, that is causing the scanning backlog. Once the scanning issue has been resolved, all scans since Oct 27, 2019, will be reprocessed en masse, to ensure all received scan sheet results are available.
Posted Nov 05, 2019 - 09:45 EST
The Support Environment is operational again, allowing internal teams to work on Data Load cases. We have pushed out a hotfix to address errors that the users had experienced while using Student Filters in Student Item Analysis. The errors that the school-based users were receiving upon login, as reported by the Districts, were also corrected.
Posted Nov 04, 2019 - 13:30 EST
Our internal support teams are unable to access the tool needed to execute data loads. We are currently working towards restoring the ability for the teams to work these open tasks. We apologize for the delay in completing data loading tickets and appreciate your understanding as you will experience longer than normal resolution times.
Posted Nov 01, 2019 - 12:56 EDT
We have resolved the issue that was causing some scans to show as 'Failed Processing' and have reset all scans for reprocessing. We are continuing to add additional server capacity to address demand.
Posted Nov 01, 2019 - 08:38 EDT
New scans from Nov 1, 2019 are showing as failed processing following the addition of new servers for added capacity. We will have the connection issue resolved soon and will reset all transactions so that no one has to rescan.
Posted Nov 01, 2019 - 08:11 EDT
We are continuing to monitor for any further issues.
Posted Nov 01, 2019 - 08:07 EDT
The identified issues related to Scanning and Search have been resolved. Your pending scans are queued and being processed, so no re-scanning is necessary. The issue with the nightly builds and core files not processing has also been resolved. We will confirm in the morning and provide an update at that time.
Posted Oct 31, 2019 - 20:03 EDT
We identified some issues after PowerSchool migrated the Performance Matters Systems to an upgraded infrastructure on the weekend of Oct. 25, 2019. We thank you for your patience as we work to resolve the issues that occurred during this significant transition. We will continue to provide regular updates until this has been completely resolved.
Posted Oct 31, 2019 - 20:03 EDT
This incident affected: Student Platform.