Prior Post: https://blogs.sap.com/2019/08/07/02-naive-sherpa-project-conciletime/
Next Post: https://blogs.sap.com/2019/09/09/04-naive-sherpa-gathering-data/
In my last post I described the conciletime.com project and laid out the main modules and their purpose. I skipped quite a few files sitting at the top level of the project because they pertain to the building of the project itself. We'll get into a this shortly but first let's talk about why we want to do things automatically.
When embarking on a new project there is a huge temptation to dive in and start coding. You're spending money with each passing hour and you want to show some progress. Usually in the beginning it's just one or two developers sitting in a room or just a chat post away. The quick turn around of writing some code, compiling/deploying it, and seeing the results is intoxicating and I understand the lure. The problem is, as things grow and get more complex and as your team grows and you start dividing up the tasks to other developers, the process of building and testing starts becoming problematic. Now suddenly somebody commits a change somewhere and the whole thing breaks. Worse than that, you make a change and it affects some other part of the application than the one you're working on currently and you don't notice it (because you're only testing what you're working on). This is where continuous integration/continuous deployment(CI/CD) becomes critical to your success.
Even while you're still working on the initial stages of a project as a single developer, I encourage you to work through the issues of setting up a functional CI/CD system and start using it. At the very least it will save you from that accidentally deleted semi-colon. By delegating the building of your project to a Ci/CD system you free yourself up to focus on the problems at hand while building trust in your project code at the same time.
There are many types of CI/CD systems so I'll just mention the ones that SAP uses as a basis for it's offerings.
Overview of SAP Offerings for CI and CD
If you're using an existing CI/CD system or need to use a set of tools not covered by SAP, I recommend reading the CI Best Practices Guide. This is an extensive doc and will also get you thinking on what's needed once you need to start serious testing and production planning.
Since I'm trying to follow SAP's best practice, I opted for using my own CI/CD system based on Jenkins and adopting the Piper library. The Piper library give you a good balance of control and uniformity to your build process. There is an evolving option of using a SAP Cloud Based service for CI/CD but currently it is focused on two SAP focused development use cases so I won't cover it here.
Jenkinsfile : This is where the build process starts by defining the stages the need to happen in order to get your project deployed. While setting up the prepare, build, and deploy stages in that order, most configuration details are left to the configuration files.
JenkinsfileXSA : I want to be able to create a different Jenkins job to build the on-prem version of my application from the same set of sources, so I created this alternate version of the Jenkinsfile. The only difference is that is points to a different set of configuration files.
.pipeline : Is a folder with configuration files that the Piper library picks up for it's configuration.
.pipeline/extensions : Is a folder for overriding step behavior and isn't working yet.
.pipeline/config.yml : Is the main file for configuring the Cloud Foundry deployment.
.pipeline/configXSA.yml : Is the XSA version file for configuring the on-premise deployment and is included from the JenkinsfileXSA file.
pipeline_config.yml : Is where the configuration for the SAP Cloud based CI/CD service mentioned above. It's not used by the current Jenkins process so it's effectively ignored.
webhook.txt : This file has some hints about using the SAP Cloud based CI/CD service mentioned above.
Jenkins does all the tedious tasks for building my project and doesn't forget anything(or fails). The glue between me editing my project and Jenkins is the code versioning tool(you are using a code versioning tool right?). I'm using Git and specifically github.com. Every time I commit to the master branch in the git repository, a new build is triggered. When the process is complete, it notifies me via a Slack message but you use email notification or other mechanisms. Let's take a look at Jenkins.
Follow the directions in the links above for setting up Jenkins to work with the Piper library.
We see here that I've set up 2 jobs within Jenkins. The first one which has recently been building to success(sunny) is the one that is deploying to Cloud Foundry. The second one is for building to on-premise XSA and since I have my server turned off, it's failing(stormy) and I've even temporarily disabled the job so that it doesn't continue trying to build until I can get back to it.
In my github account, I need to configure a webhook trigger. Log into Github and select Settings.
Next select Webhooks.
Select Add webhook.
Fill in the details paying attention to the path of the URL and the type of application/json.
You should now see a new webhook entry.
Jump back over to Jenkins and create a new Pipeline job.
Don't forget to hit OK. Now configure it like this.
Select that you'll be using SCM and Git and use your repo url and the master branch.
Specify which file in your project contains the Jenkinsfile. and Save.
Test your setup by make a change in your project and committing it to git. A build should trigger in Jenkins and you can watch the progress by selecting the jab and "Console Output". You can also check on the progress by using the cf cli tool as the deploy progresses.
You'll notice that all of this takes some considerable time. Currently my project is taking nearly 10 minutes from commit to testable build. Think of the CI/CD as a background process that keeps you honest and is watching over your project. It takes a long time to put all the modules in place and hook everything up to give you a full multi-target application. Most of the time you'll be working on just one module at a time. When you're doing this, there are some other techniques that you can use to speed up your change/build/test cycles. Take a look at this blog post where I get into the details for a NodeJS based module.
Don't. I know it seems like a lot of hassle to get all of this set up and working properly, but do it now and save yourself a lot of headache later. I haven't gotten into adding a bunch of code tests, but you can do that in addition to open-source scans, and end-to-end tests and automate the whole she-bang.
What's even more important is that you start thinking about your software development as a continuous process. You want your code to move from your development environment into testing and then to user acceptance and finally production without any surprises. What don't want to do is get caught up in merging changes from various versions when bugs are caught and fixed. You want development to continue working on a new version while an older version is being tested and an even older version is getting previewed by your customers as well as the most stable version running at scale in production. You can even manage zero-down-time upgrades in-place in production without fear if you trust your process. So take the time to at least think about what your full dev-ops pipeline is going to look like before you get too far into your coding. There is a book(The Phoenix Project) that I highly recommend that takes a fictionalized look at what happens when things go wrong and how good life can be when they go right.
In the next post we'll talk about thinking about your value proposition and gathering some data so that we can start poking at it.
Until then...
-Andrew (Your Naïve Sherpa)
GitHub Project:
https://github.com/alundesap/conciletime
https://github.com/alundesap/conciletime/tree/post03
Prior Posts:
https://blogs.sap.com/2019/07/22/00-naive-sherpa-meet-the-naive-sherpa/
https://blogs.sap.com/2019/08/02/01-naive-sherpa-the-journey/
https://blogs.sap.com/2019/08/07/02-naive-sherpa-project-conciletime/
Next Post: https://blogs.sap.com/2019/09/09/04-naive-sherpa-gathering-data/
Next Post: https://blogs.sap.com/2019/09/09/04-naive-sherpa-gathering-data/
Doing things automatically.
In my last post I described the conciletime.com project and laid out the main modules and their purpose. I skipped quite a few files sitting at the top level of the project because they pertain to the building of the project itself. We'll get into a this shortly but first let's talk about why we want to do things automatically.
When embarking on a new project there is a huge temptation to dive in and start coding. You're spending money with each passing hour and you want to show some progress. Usually in the beginning it's just one or two developers sitting in a room or just a chat post away. The quick turn around of writing some code, compiling/deploying it, and seeing the results is intoxicating and I understand the lure. The problem is, as things grow and get more complex and as your team grows and you start dividing up the tasks to other developers, the process of building and testing starts becoming problematic. Now suddenly somebody commits a change somewhere and the whole thing breaks. Worse than that, you make a change and it affects some other part of the application than the one you're working on currently and you don't notice it (because you're only testing what you're working on). This is where continuous integration/continuous deployment(CI/CD) becomes critical to your success.
Even while you're still working on the initial stages of a project as a single developer, I encourage you to work through the issues of setting up a functional CI/CD system and start using it. At the very least it will save you from that accidentally deleted semi-colon. By delegating the building of your project to a Ci/CD system you free yourself up to focus on the problems at hand while building trust in your project code at the same time.
There are many types of CI/CD systems so I'll just mention the ones that SAP uses as a basis for it's offerings.
Overview of SAP Offerings for CI and CD
If you're using an existing CI/CD system or need to use a set of tools not covered by SAP, I recommend reading the CI Best Practices Guide. This is an extensive doc and will also get you thinking on what's needed once you need to start serious testing and production planning.
Since I'm trying to follow SAP's best practice, I opted for using my own CI/CD system based on Jenkins and adopting the Piper library. The Piper library give you a good balance of control and uniformity to your build process. There is an evolving option of using a SAP Cloud Based service for CI/CD but currently it is focused on two SAP focused development use cases so I won't cover it here.
What files do what?
Jenkinsfile : This is where the build process starts by defining the stages the need to happen in order to get your project deployed. While setting up the prepare, build, and deploy stages in that order, most configuration details are left to the configuration files.
JenkinsfileXSA : I want to be able to create a different Jenkins job to build the on-prem version of my application from the same set of sources, so I created this alternate version of the Jenkinsfile. The only difference is that is points to a different set of configuration files.
.pipeline : Is a folder with configuration files that the Piper library picks up for it's configuration.
.pipeline/extensions : Is a folder for overriding step behavior and isn't working yet.
.pipeline/config.yml : Is the main file for configuring the Cloud Foundry deployment.
.pipeline/configXSA.yml : Is the XSA version file for configuring the on-premise deployment and is included from the JenkinsfileXSA file.
pipeline_config.yml : Is where the configuration for the SAP Cloud based CI/CD service mentioned above. It's not used by the current Jenkins process so it's effectively ignored.
webhook.txt : This file has some hints about using the SAP Cloud based CI/CD service mentioned above.
Jenkins is my Minion.
Jenkins does all the tedious tasks for building my project and doesn't forget anything(or fails). The glue between me editing my project and Jenkins is the code versioning tool(you are using a code versioning tool right?). I'm using Git and specifically github.com. Every time I commit to the master branch in the git repository, a new build is triggered. When the process is complete, it notifies me via a Slack message but you use email notification or other mechanisms. Let's take a look at Jenkins.
Follow the directions in the links above for setting up Jenkins to work with the Piper library.
We see here that I've set up 2 jobs within Jenkins. The first one which has recently been building to success(sunny) is the one that is deploying to Cloud Foundry. The second one is for building to on-premise XSA and since I have my server turned off, it's failing(stormy) and I've even temporarily disabled the job so that it doesn't continue trying to build until I can get back to it.
Setting up the trigger.
In my github account, I need to configure a webhook trigger. Log into Github and select Settings.
Next select Webhooks.
Select Add webhook.
Fill in the details paying attention to the path of the URL and the type of application/json.
You should now see a new webhook entry.
Jump back over to Jenkins and create a new Pipeline job.
Don't forget to hit OK. Now configure it like this.
Select that you'll be using SCM and Git and use your repo url and the master branch.
Specify which file in your project contains the Jenkinsfile. and Save.
Test your setup by make a change in your project and committing it to git. A build should trigger in Jenkins and you can watch the progress by selecting the jab and "Console Output". You can also check on the progress by using the cf cli tool as the deploy progresses.
You'll notice that all of this takes some considerable time. Currently my project is taking nearly 10 minutes from commit to testable build. Think of the CI/CD as a background process that keeps you honest and is watching over your project. It takes a long time to put all the modules in place and hook everything up to give you a full multi-target application. Most of the time you'll be working on just one module at a time. When you're doing this, there are some other techniques that you can use to speed up your change/build/test cycles. Take a look at this blog post where I get into the details for a NodeJS based module.
Boost your Cloud Foundry NodeJS Edit-Build-Test Speed!
OK, I'll get to that CI/CD stuff later...
Don't. I know it seems like a lot of hassle to get all of this set up and working properly, but do it now and save yourself a lot of headache later. I haven't gotten into adding a bunch of code tests, but you can do that in addition to open-source scans, and end-to-end tests and automate the whole she-bang.
What's even more important is that you start thinking about your software development as a continuous process. You want your code to move from your development environment into testing and then to user acceptance and finally production without any surprises. What don't want to do is get caught up in merging changes from various versions when bugs are caught and fixed. You want development to continue working on a new version while an older version is being tested and an even older version is getting previewed by your customers as well as the most stable version running at scale in production. You can even manage zero-down-time upgrades in-place in production without fear if you trust your process. So take the time to at least think about what your full dev-ops pipeline is going to look like before you get too far into your coding. There is a book(The Phoenix Project) that I highly recommend that takes a fictionalized look at what happens when things go wrong and how good life can be when they go right.
In the next post we'll talk about thinking about your value proposition and gathering some data so that we can start poking at it.
Until then...
-Andrew (Your Naïve Sherpa)
GitHub Project:
https://github.com/alundesap/conciletime
https://github.com/alundesap/conciletime/tree/post03
Prior Posts:
https://blogs.sap.com/2019/07/22/00-naive-sherpa-meet-the-naive-sherpa/
https://blogs.sap.com/2019/08/02/01-naive-sherpa-the-journey/
https://blogs.sap.com/2019/08/07/02-naive-sherpa-project-conciletime/
Next Post: https://blogs.sap.com/2019/09/09/04-naive-sherpa-gathering-data/
- SAP Managed Tags:
1 Comment
