Top Tips for a Successful Software Upgrade

By Chase Mory  |  Director of Operations

We released Galaxy 7.8 last year, the latest version of our Galaxy Ticketing and Guest Experience solution. To get the version, obviously, you need to upgrade. Through thirty plus years of selling ticketing software and countless new versions and releases, we’ve mastered the best practices for managing a software upgrade. Our Delivery and Managed Services team members are the feet on the ground helping our customers through the process, so who better to weigh in on recommendations for a smooth and successful upgrade? And while these tips specifically apply to our software, most are relevant for major system software upgrades in general. Here’s what they had to say:

A TEST ENVIRONMENT IS ESSENTIAL

First and foremost, you should have a test environment with a copy of your production data that is as recent as reasonably possible and in an environment that as closely simulates your production environment as possible. The reason for this: you’re now able to apply the new software version into a close simulation of your production data, so if you run into an error, you haven’t impacted your live production environment. You also have the most accurate prediction of what will happen when you actually upgrade.

With any upgrade, regression can happen. So in the test environment, you want to test as many of your real world business scenarios as reasonably possible to see where it does. And what does ‘reasonable’ mean? Depending on the complexity of your configuration, you could spend two years testing every single transaction type on every type of point of sale and hardware in your test lab. You could also spend tens of thousands on a duplicate super server for your test environment. But is that reasonable? Obviously, the closer you can simulate your production environment, the more realistic your results will be. But you also need to strike the right balance between time, budget and your anticipated results when you move this new version to production.

DOCUMENT YOUR TEST SCRIPTS

When simulating transactions in your test environment, you should always document your test scripts so you can refer to them when you go to execute the same scenarios during the live upgrade. As an added benefit, when standard procedures change at your organization or new products or services are added to your offerings, you now have a test case to refer to and apply against that software versions to make sure things don’t break when they go into production.

We don’t recommend a standard way to document a test script – whatever works best for you and your organization. But as a rule of thumb, document the steps of the scenario, who tested it, when they tested it, and whether the test passed or failed.

YOU NEED THE PROPER RESOURCES TO DO THE PROPER TESTING

We run into this more often than you would imagine. An organization has an appropriate test environment that reasonably simulates production, and they have the right test scripts. But they don’t actually have the right amount of time and labor resources to execute a thorough testing in order to uncover where the issues really live. Planning and preparation are just half the battle. If you don’t execute appropriately due to limited capacity, you’ll never have a comprehensive understanding of the upgrade and you’re more likely to go live with regression. Don’t skip testing the transactions and scenarios that are most important to your business because you don’t have the time or manpower.

ESTABLISH AN APPROVAL COMMITTEE

It may seem like common sense, but it’s important to know who has authority to make upgrade decisions and who makes the ultimate ‘Go, or No-Go.’ call. We’ve seen projects derailed and exceed time and budget due to inefficient decision-making that could have been solved with more clear communication of roles.

These key decision-makers need to review the testing results and identify the key areas of the business that are impacted by regression. Are issues effecting revenue, guest satisfaction, or the ability of your employees to do their job? Are issues effecting the two-day combo ticket that comprises 40% of your revenue, or just mislabeling a retail item that is purchased 5% of the time? This committee should decide if the upgrade is a ‘Go’ or if issues are critical enough that the upgrade should be postponed due to engineering intervention.

ESTABLISH A MAINTENANCE WINDOW AND BACKUP YOUR DATA

So your team decides the upgrade is a ‘Go’. If possible, you now want to establish a maintenance window to perform the upgrade, and the system downtime during this maintenance window needs to be clearly communicated to internal and external business partners.

The best practice is to stop transactions from coming into your system at the beginning of your maintenance window, then backup your database, then begin running the upgrade script and installer. You don’t want any other transactions coming in once you’ve backed up your database. So if something goes wrong with the upgrade and you have to rollback to the original version, you haven’t lost any transactions that may have occurred after the start of the maintenance window.

When planning the maintenance window, make sure you build in rollback time. Plan for the upgrade to go smoothly, but anticipate contingency time in case it doesn’t. Also make sure you capture certain details when you simulate the upgrade process in the test environment: like, how long does it take to run the test script and how long does the upgrade physically take on one node, including the time it takes to access that node. Then multiply this by the number of nodes you have. Few data points or small details are too irrelevant when planning the length of your maintenance window.

Here’s a small trick of the trade: zip up the files on each node that you upgrade prior to the upgrade and set them off to the side (both the program files and the local journaling configuration files). Now the node can work in semi-offline state with its own set of localized configuration and a sales journal to operate. Then if you do have to rollback, you can just unzip the old files and replace the directory of what was just upgraded.

And another one: prior to the conclusion of your maintenance window, make sure you do a cursory check of the system. Are pivotal services back up and functioning properly? You don’t want to complete the upgrade then shut down for the night only to find a horrible error in the morning and no maintenance window for rollback.

We have plenty of customers who operate 24/7 and don’t want to interrupt the online guest experience with downtime during a maintenance window. If this is the case, we recommend an in-depth analysis to understand the day and time that an upgrade or potential rollback will least impact your business. For example, if you don’t want to take your webstore down, determine when your site traffic is the slowest. This could be the best time to minimize the impact to your business.

We’ve also seen 24/7 customers double their testing efforts to be doubly sure all regression is identified and either known or fixed before upgrading. Another option we’ve seen: upgrade the database on just one node and go live with it. If there is regression or an error, you’ve caught it without taking the entire operation down. Then move forward with the rest of the upgrade when you deem fit.

UNDERSTAND WHAT HAPPENS ON THE SQL SERVER OVERNIGHT

When preparing for the production upgrade and planning your maintenance window, ensure that you consider or work around anything happening on the SQL server overnight. Once we were attempting to run an upgrade script when a backup process started running on the server, taking away computer resources and delaying our script. The upgrade was taking longer than the anticipated maintenance window because we weren’t aware of other planned processes on the server. We’ve also seen this happen with nightly virus scans that run at the exact time we’re performing the upgrade.

THINK OUTSIDE THE SCOPE OF JUST THIS SOFTWARE UPGRADE

Upgrading one software impacts many systems, so make sure you cover your basis on programs that connect in or out of the software you are upgrading. For example, in our world the accounting export is incredibly common, so we recommend running an accounting export from the staging environment to make sure the file can still be digested by your accounting system. Keep in mind, you’re not just testing functionality. Another example: make sure you test the graphics on your webstore and ensure they’re not negatively impacted after upgrading.

KEEP THE LINES OF COMMUNICATION OPEN

Keep an open line of communication with your provider in terms of why you’re upgrading, what you’re upgrading and when you’re upgrading so they’re on standby during your maintenance window. If a customer upgrades and reports an issue, but we still have them on record as using a much earlier version of the software, then we may be speaking two different languages when trying to identify a fix. The same goes for the approval committee, as identified above. Open lines of communication go a long way towards a successful upgrade: among departments impacted by the upgrade, among front-line staff and decision-making management, and if necessary, among your organization and your external partners and guests.

FINALLY, HAVE A DEBRIEF

Just like it’s important to document your test scripts, it’s also important to document the process of the upgrade as a whole. Hold an after-action review meeting with all the key stakeholders involved and go through lessons learned. What went right and what went wrong? What did you forget and what did you not foresee? Document that information somewhere easily accessed so you can improve upon your process next time. And maybe stash this article in that folder as well. Upgrades can be difficult, but by following these tips, it can be a relatively smooth process with little impact to your business.