Central Host 5.155 and Management Console 5.148

July 17th, 2024, a few versions of Central Host were deployed to production. There were issues with the release. JIT account access for Azure ID was broken. Here is an excerpt from the internal document on what happened.

There was an internal data structure that changed the type of a variable as part of some internal code clean up. A guid that was sent as a string in json was changed to being sent as a guid in json. We did a lot of testing around the json serialization and deserialization of this type change to ensure it was compatible, both forward and backward. We tested it in our testing environment and it worked without issue. We missed one of the backward compatible cases that caused the json deserialization to fail. The central host changes were deployed at 4:30 pm EST. At 6:00 pm was the first report of an issue. At 8:30 pm we had found and fixed and deployed a version to fix the reported issues. At 8:45 we got confirmation from several clients that it was working. We continued to investigate the root of the problem and by 9:15 pm believed we had found a few more places where the issue might occur, but we didn’t find evidence of it occurring in production, so we decided to not do another deployment and continue testing and review of the code so as to not cause any more problems. Around 1:00 AM a call came into the support phone line that someone was still experiencing issues with JIT and Azure. We were able to quickly determine that they were seeing one of those other places causing an issue. At 1:30 am we deployed the new changes and shortly after got confirmation from them that it was all working.

What are we doing to prevent this from happening in the future, is the obvious next question.

We will continue to evaluate our response of the course of the next week. At this point we have identified two problems and will be making changes to address them in the coming weeks.

The first issue is that the error logs on the production servers were not making it the right place to notify us of an ongoing error in production. We are modifying the error logging and reporting system to be louder.

The second issue is that the automated tests for released code has blind spots. The automated tests we run before a release are being expanded to include more older versions of the agents and TechClients. 

We value any feedback you have on our response during and after this incident. We look at all issues as opportunity to grow and become better and this is for sure an opportunity to become better.


The result of yesterday is that version 5.155 of the Central Host and version 5.148 of the Management Console were deployed to AWS to better serve all existing and future TechIDManager clients. No action on clients parts is required to use the newest version. 

This release is a release with mostly customer facing new features and a few internal changes. If you have any issues, feature requests, or ideas, let support@ruffiansoftware.com know and we will work on them.

  1. Internal code clean up.

As always user feedback is welcome and if there are any issues, contact us at support@ruffiansoftware.com to resolve them ASAP.