Upgrading 1000 sites
Heads Up!
This article is several years old now, and much has happened since then, so please keep that in mind while reading it.
Part of the Umbraco Cloud offering is that you get automatic patch upgrades for the products included in your project. This includes the Umbraco CMS, Umbraco Deploy and Umbraco Forms, but also a variety of addons that are included, like the Blob storage provider (for storing media on Azure Blob storage), and Umbraco Id (Single Sign-On).
This means that every time one of the products or components are upgraded, we want to add the new files to all eligible Umbraco Cloud projects. In this article, I’ll try to touch on a few important aspects
- Finding eligible projects for the upgrade
- Upgrading one project to the latest version - (Azure Web Apps, NuGet upgrades and Unattended Upgrades)
- Executing and scaling the upgrades - (Azure functions and Azure Container Instances)
In short, I'll try to explain what happens when you see a tweet like this:
Finding eligible projects for Upgrade
On Umbraco Cloud we have lots of different projects running, figuring out which projects should get the latest patch, requires us to know exactly what version a project is running.
This might seem straightforward - We install Umbraco, and we manage upgrades, so just keep track of it, right? Unfortunately, Umbraco Cloud is structured in a way, where the code repository is owned by the users, and they are not forced to only upgrade through the automatic upgrade process. This means that some might upgrade manually, and then we would not know about it in our systems. Therefore, we need to ensure and synchronize the versions a project is running before we can find out who gets an upgrade.
The Umbraco Cloud currently holds a wide range of different versioned Umbraco solutions, and we recently changed the underlying platform infrastructure to Azure Web Apps, as well as introduced Umbraco 9. The infrastructure change meant we needed to develop a new way of upgrading sites, and from Umbraco 8.12 and up we utilize the Unattended Upgrades feature in Umbraco CMS, when running the upgrades.
So, to sort everything out, we run a nightly batch run, which will figure out what version any project is running, and what version of which components are installed
- What platform are you running? - We have two platforms - the old platform, and Azure Web Apps
- What Umbraco CMS version are you running? - We run Umbraco projects ranging back to 7.2.8(Please upgrade if this is you ;-) and until the very latest released version (9.1.1 as of this writing)
- What type of project are you? - The Umbraco Cloud platform supports all of our cloud offerings, meaning regular Umbraco Cloud projects, Umbraco Uno and Umbraco Heartcore
- What versions of components do you have installed? - Each Umbraco project has a range of components installed, like Umbraco Forms, Umbraco Deploy, Umbraco Courier, StorageProvider, Umbraco Id, various Contrib packages etc.
With that information we can sort out which projects should receive the new upgrade.
Upgrading one project to the latest version
When you would upgrade an on-premises Umbraco site the process would look something like this
- Open your project in your IDE.
- Update the Umbraco package via NuGet.
- Rebuild your solution.
- Make sure it can boot.
- Run the upgrade wizard to ensure the latest patch and migrations are applied.
- Validate that your site still runs as expected.
- Deploy the change to your development environment and run step 5+6 again.
- Deploy to the next environments you would have and run step 5+6 again.
The process of upgrading a site on Umbraco Cloud is similar, but to scale the process to thousands of sites, everything needs to be automated.
Let me start by taking you through a little de-tour, talking about some of the challenges we had with the old platform and older versions of Umbraco
In previous versions of Umbraco CMS, prior to Umbraco 9 and early versions of Umbraco 8, the upgrade process was more complex. We did it by downloading the .zip package we always provide of Umbraco - Then we would compare each file in it with the files a project had, and then overwrite the existing files with the newer ones.
Once that copying the files was done, we needed to commit the changes to the git repository, and afterwards copy the changes to wwwroot, where the site would boot. Once the site had booted, we needed to run the upgrader.
This caused yet more issues, as Umbraco did not have a way to automate running the upgrade, so we had to implement custom code to manage that process. Without it, the site would end up in a state called AuthorizeUpgrade where you as an Admin would have to log in to finalize the upgrade.
Once the upgrader had finished, we would verify that the site was running as expected. This last step is also the first time we would notice if the site's code was compatible with the upgraded Umbraco files, if not the site would throw a YSOD, and we would have to revert.
Current upgrades
The current upgrade process has been simplified a lot compared to previously. Three major changes have helped with this
- Moving to Azure Web Apps
- Unattended Upgrades
- Umbraco 9 and NuGet upgrades
1. Moving to Azure Web Apps
The move to Azure Web Apps gave us a challenge, but also fixed some of the complexity we had seen previously. In the old setup, we had direct access to the fileservers via a UNC share. This made it easy to just go to the file share, modify some files and commit them. No git cloning or pushing was needed. The drawback was that the fileserver was shared among all customers, meaning that writing to the fileserver took resources away from everyone else having their files on that fileserver. Furthermore, we had to maintain the fileservers ourselves, with all that follows in terms of monitoring and maintenance, and if that fileserver broke, many sites on Umbraco Cloud would stop working.
While moving to Azure Web Apps we had to change that process. We no longer have access to the file share, so rather than doing the git operations directly on a fileserver, we changed the flow. Now we "just" do regular git operations against the git repository that lies behind the Azure Web App. This means the process of updating files has been boiled down to this:
Once the files have been pushed, Azure Web Apps handles copying files from the git repository and to the wwwroot, and we are ready to check if the site runs as expected.
2. Unattended Upgrades
In Umbraco 8.12 we introduced Unattended Upgrades. This change allowed us to no longer have our custom code that would run the Umbraco Upgrader whenever an upgrade was detected. Instead, we make sure that Unattended Upgrades is enabled on the site, and then we just have to boot the site. This will automatically apply whatever migrations is needed for the upgrade.
3. Umbraco 9 and NuGet upgrades
With the introduction of Umbraco 9 and .NET 5, we started changing the way we run projects on Umbraco Cloud. Previously the git repository would contain everything needed to boot a site. This would include all files from Umbraco, and the custom code that was implemented, compiled to dll’s.
In the new format, we do a build on the servers, so the files that need to be in the git repository are just raw source files, like .csproj and .cs files. If the dependencies are reachable via NuGet, nothing else is needed. For upgrades this makes the upgrade step super simple, we just update the NuGet reference in the .csproj file(s) and run a build. This ensures that the customer's code is compatible with the latest version of Umbraco. Then we commit that change.
Running the upgrade
With those three points in place the process can be simplified to
- For each environment
- Clone down the repository
- Update package references to the latest version
- Build the project
Executing and scaling the Upgrades
Now we know what it takes to upgrade one project on Umbraco Cloud, then how does it scale? This is where we utilize the power of Azure and Serverless computing. We did package all the code for upgrading one project into one single docker image, called the UpgradeExecutor - We have one image pr. product type we run on Umbraco Cloud, so an Umbraco 9+, Umbraco 8, Umbraco 7, Heartcore and Uno. Then we send a list of eligible environments onto a queue, which in turn will trigger a Durable Azure Function Orchestrator to start processing the environments. We tell it about how many we want to run concurrently. Then the Azure Function Orchestrator is in charge of keeping that amount of UpgradeExecutors alive until there are no more environments to upgrade.
The UpgradeExecutor is executed via Azure Container Instances – This is an Azure service for running docker images - a Container instance will take a docker image, run it to completion, and then destroy the container instance again.
We can try to visualize what happens in this short gif
We currently run between 15 and 25 UpgradeExecutors simunltaneously - this image is from a recent upgrade, showing the Container instances cracking away at UpgradeExecutors
This architecture is perfect for us running upgrades, it ensures that we do not have UpgradeExecutors' resources lying idle around and that we only spend money on resources that are actually utilized. Usually, upgrades are mainly run on Tuesdays, so these UpgradeExecutors should only be active during that time. Also, a bugfix or new feature added to one of these UpgradeExecutors will automatically be applied the next time we run an upgrade, if only the Docker image is updated, we just take the latest one.
Final words
In the above I do try to explain a lot, in a fairly short article - It’s also only scratching the surface of some of the super exciting services (according to me at least) we have running on Umbraco Cloud. I tried to keep it at a level it is easy to follow if you're interested but haven't been part of the whole journey. I hope I succeeded 🙂 I also hope you can see there has been tremendous development on the inner workings of Umbraco Cloud. We're constantly improving all parts of the experience based on experience and on feedback. This year has seen some of the biggest changes yet with the introduction of a new and improved infrastructure, Umbraco 9 with better project structure and build process and finally more efficient and reliable upgrade services - just to name a few - and we'll continue to add improvements and new features in 2022 and beyond.
If you have questions about the process or would like to learn more - do not hesitate to reach out to me or anyone from my team. We love to talk about the tech we develop and implement daily.
Speaking of the team, I'd like to give a huge thanks to the Umbraco Cloud Platform Team's proof reading and comments for this article, and coming up with this amazing solution, and many more - credit goes to Dan Lister (for amazing visuals), Morten Christensen, Martin Clausen, Mikulas Tomanka and Rasmus Pedersen
