De-risking a Jenkins upgrade
Have you ever encountered the following problems?
Assume you have your own dedicated VM that uses Jenkins to manage its operations.
- After upgrading to a new version of Jenkins, some build jobs stop working.
- After upgrading to a new version of the Jenkins plugin, your script is not compatible with the new version plugin.
- If the Jenkins workspace gets corrupted, you need to restore the workspace.
If you answered yes to one of the above questions, this article is of interest to you.
For the last two years, I have been working in a small team which supports the operational aspects of SAP Precision Marketing (SPM) a personalized marketing platform that runs on HANA Cloud with a bi-weekly release cycle. I was a mobile and a back-end developer, and I learned Operations from A to Z on this project. Automation, deployment, Continuous Integration, JMeter and Jenkins were new concepts to me when I first started. Jenkins is the backbone for all these. It is a great, simple build server which is used for continuous integration. As the test-driven approach is getting popular, continuous integration plays a key role in the short development cycle. Achieving high availability of Jenkins becomes a very important question for both the development team and the operations team.
So, how do you de-risk a Jenkins upgrade? Of course, I started off googling around, and digging in SCN, looking for a good practice guide to upgrading Jenkins servers. That was not successful. So I had to figure out the process. After talking to the Operations manager in my team, I need to have at least two Jenkins instances – a production Jenkins and a backup Jenkins set up. A backup Jenkins is mainly used for testing the Jenkins upgrade and as a temporary build server whenever the production Jenkins is down. I spent a good deal of time trying different ways to install two Jenkins in the same VM. I will explain each one in the following.
Experiment one — Multiple WebApps in a Tomcat instance
You can install two different versions of Jenkins in Tomcat. Just unzip Jenkins to the WebApp folder in Tomcat and give it a unique name. You can access the Jenkins instance with this URL localhost:8080/test or localhost:8080/production. The drawback of this approach is that all of Jenkins’ configuration files, plugins, and working directories go under <USER_HOME>/.jenkins by default. Of course, you can change it to any other location. Beware that since the Jenkins workspace is shared by different versions of Jenkins in Tomcat, you may get a configuration folder collision.
Experiment two – Jenkins Master / Slave machines
Master/Slave was a new concept to me. I was curious if this approach can solve the problem, so I set up master and slave machines on my local machine for the test. When creating a slave machine, there is an option to define a home directory for the master on the slave machine. This can solve the configuration folder collision problem mentioned in Experiment one. However, this solution has a bigger problem. Slaves are computers that are set up to build projects for a master. Jenkins runs a separate program called “slave agent” on slaves. In other words, you can’t install a different version of Jenkins on a slave node.
Experiment three – Multiple Jenkins instances in separate containers
One of Jenkins greatest features is that despite being packaged as a standard .war ready to be dropped into your web container of choices, it also contains an embedded web container that makes the .war everything you need in most situations. Simply run java -jar jenkins.war. It starts Jenkins on port 8080 by default. If you want to run multiple instances on the same machine, this requires a little more effort. It turns out this is a piece of cake, but the docs are hard to find. Jenkins will use a JENKINS_HOME path environment variable for its configuration files if one is set, so a simple change means we can run Jenkins out of any directory we desire: java -DJENKINS_HOME=C:\production_jenkins_workspace -jar jenkins.war. So is the port number environment variable. Winston, the embedded web container used, has a simple property for this too: java -DJENKINS_HOME= C:\ production _jenkins_workspace -jar jenkins.war –httpPort=9090
With these commands, we can have a production and a backup Jenkins running on the same VM in a self-contained environment. This approach not only solves workspace collision but also has different versions of Jenkins installed and started. So Experiment three works.
After having two Jenkins install on the same VM, we need to back up the Jenkins production server home regularly. This allows us to recover Jenkins workspace when accidently messing up the production Jenkins workspace. I automated this task with Jenkins. I created two Jenkins jobs -Backup and Restore. Each job contains two commands.
Backup
set PATH=C:\Program Files\7-Zip
7z a -tzip E:\Jenkins_home_backup\%DATE%.zip D:\Jenkins_home\
Restore
set PATH=C:\Program Files\7-Zip
7z x -y E:\Jenkins_backup\%date%.zip -oD:\Jenkins_temp
I also tried to use WinRAR to compress the Jenkins_home, it turns out 7-zip is more efficient. My workspace had 20GB storage. WinRAR took 2 hours to compress it while 7-zip took 50 minutes.
And there is even a Jenkins plugin handles backup and restore workspace that you can use.
https://wiki.jenkins-ci.org/display/JENKINS/Backup+Plugin