Going Off-Script with SAP Tools

[andrew_lunde]

...and then getting back on the same page.

 

The post was going to be called "Mitigating SAP Black-Box Anxiety." but I didn't want the reader to be left with a sense that where was no solution to what created the anxiety in the first place.

The following is a recent experience I has as an internal SAP employee but I'm sure partners and customers can relate as well from their own experiences.  But first some thoughts.

I think that developers often get trapped in their own assumptions about what is possible given a certain vendor tool set.   So often they pick something up and study how it's recommended that it be applied and make their best effort at getting to the solution they need.  Along the way they often loose sight of how the tool can be modified or extended when the fit isn't quite right.

I know this may come as a surprise to many, but the developers at SAP are people.  They are people that are organized and directed to create products and tools for some specified purpose.  That purpose may align exactly with your needs, or it might be close enough to invest your time and efforts in.

This brings us to the age old "Make vs. Buy" proposition.  The assumption is that if you decide to make it yourself from scratch, you'll have to learn a lot (coding time) but you'll be in complete control as your requirements change(or are discovered).  Also by making it yourself, you'll be re-creating the wheel for many things that are mundane but needed for a full solution.  The buy option promises to remove the need to learn something, but this is a fallacy.  What you have to learn is the vendor's tool methodology in addition to taking on the risk that the tool doesn't do something that you need it to do.

Which brings up the fear of "vendor lock-in".   This is a real fear since the risk you thought you were avoiding by purchasing a tried-and-true vendor solution might actually be adding risk in addition to the expense of buying a tool and learning it(and then finding out you have to extend it).

I know that historically SAP has been seen as a vendor of licensed products that were essentially a black-box that while highly customizable, the user didn't have insight to the inner workings(let alone the ability to change them).  We've significantly changed this stance over the last decade for sure and now openly participate in open-source projects which then become the basis of our tools and offerings.  Take a look at this link for an idea of what we've been up to.

https://developers.sap.com/open-source.html

Now back to my story.

 

As part of my Naïve Sherpa blog series, I have been working on a companion application that goes along with the discussion.  It's getting pretty large with 12 modules currently and currently requires nearly 30 minutes to deploy completely.  That's not necessarily a problem, but I was running into 2 issues that were causing me grief.

I'm an advocate for get your continuous integration system set up early so that you don't find out issues late in the development cycle.  I'm using my own Jenkins server but wanted to adopt SAP's best practice for this,  While SAP is still working on a cloud-based solution for CI/CD (coming), I know it will follow closely what is recommended as best practice.  SAP has created the Piper project for this purpose.

Project Piper

So I've been noticing that my long running deploy would time-out with a message saying the token used for the cf command line tool had expired.  If I retried the operation manually, it would finish fine(new token, new expiration) but there were some things my project needed post deployment that were then never performed.  I dug around in the CloudFoundry docs(remember that SAP's CloudFoundry is based on the open-source CloudFoundry so most docs apply) and found this document.

https://docs.pivotal.io/pivotalcf/2-6/devguide/deploy-apps/large-app-deploy.html

It suggests several things to improve large project deployments, but mentions an environment variable that the CF CLI tool will pick up if it's available.  CF_STAGING_TIMEOUT seems to fit the bill since my deploy's were timing out and failing at about 16 minutes and the default for this setting is 15 minutes.  Worth a try but how to override its default setting?

I went back to the Piper project to see about how to make this timeout change.

To use the Piper library, you need to configure Jenkins to pull the library source and define a library configuration.

In your Jenkins system configuration, you define an external library with a name piper-library-os and point to the github repo that sources the library code.



Since this project is open-source, we can visit that repo and see how it's organized.

If you look at the documentation you'll see that your project requires a special file called Jenkinsfile.

When you set up your Jenkins build job for your project, you need to tell it that this is the file to use to kick off the build process by specifying the file name as the Script Path in the job configuration.

The first thing your Jenkinsfile need to specify is which library to pull in.  This is where you specify the library you set up in the Jenkins general config.  Also note that you can specify the configFile so you can use a file name different from the default of .pipeline/config.yml.



If at all possible, use the documented features of the Piper project to get the behavior your project needs.  However, as we've discussed so far, there is something that's not working so we need to use our own copy of the library code and make a modification.

A call for collaborators.

 

Now you could just make a copy of the jenkins-library github project and make whatever changes you deem needed and be on your way.  However, I'd like to make a case for forking the repo.

Make sure you're logged into GitHub with your account and click on the fork button as shown below.



Once this process is finished, you have a linked copy of the repo in your own account.  You can still make changes, but it also opens up the possibility to contribute your changes back to SAP.  See the contribution guidelines link for more info.

Since we'll want to use our own copy of the library, we need to define it in the Jenkins system config as well.  Add a new library definition and fill it in with your repo and give it a unique name.  I'll add -acl(my initials) to the library name.



Now we can adjust our Jenkinsfile to refer to our version of the library.



Without getting into all the details, we can dig into the library code and find that the cloudFoundryDeploy step is defined in vars/cloudFoundryDeploy.groovy file.

We can see near line 417 the actual command that will be executed is being composed and there are a couple of environment variable being set (HOME and CF_TRACE).  The actual deploy command is in the cfDeployStatement variable.



Let's specify the CF_STAGING_TIMEOUT variable and give it a value 60(twice the default).

Now our file looks like this.



Save the file. Commit the change and push it to your github account.  Now trigger a new build of the project.  If all goes well, a new(your) version of the library will be used(with increased STAGING_TIMOUT) and the large deploy will complete properly.

Statically setting a value isn't the best practice, so you'll want to create a new configuration setting for this and pass it into the library from your config.yaml file, but you get the picture.

 

The biggest point to understand here is that you weren't stuck waiting on SAP to fix something that you could affect yourself and continue on with your development.

Getting back on the page.

 

Once you're satisfied that your changes are working and tested, you'll want to contribute them back to the original repo.  After all, you won't likely be the last person to develop a large enterprise application.  This is easy by creating a pull-request.  What you're doing here is to asking the owners of the source(what your forked) repository to pull your changes into their code (so that others can benefit from your work).  This sets up a dialog between you and the original owners to discuss your suggested changes and hopefully merge them into the original repo.

The take away.

 

One of the biggest concerns of adopting a vendor's tool/product/best-practice is worrying about somehow getting trapped waiting for the vendor to fix/enhance something that is broken(or not well documented or understood).  By incorporating open-source projects and actively participating in them as well as offering many of it's own tools as open-source, you can take advantage of SAP experience in enterprise-grade solutions with minimal risk to the success of your project.