8 Steps to Safe Daily SAP Releases
How can organizations running SAP achieve the seemingly impossible – Safe, daily releases into production?
I’ve talked in the past about how businesses need to better respond to the demands of the digital economy and how embracing DevOps for SAP can be instrumental in achieving this.
Here I’m going to explore in more detail how organizations running SAP can actually implement the cultural, organizational and process changes required to implement DevOps for application delivery.
Those who’ve worked with SAP as long as I have will no doubt remember the chaos that was generated many years ago when changes went into production every day. It was carnage. So Application Lifecycle Management (ALM) processes and tools were put into place to control, govern and audit changes as well as to protect production systems.
We can’t have stability and agility
The bad experiences of the past have made people fearful of change to the point that few can deliver anything at pace. Overly stringent approval processes kill agility and this is commonplace in many of the organizations I talk to today.
Well we’ve come full circle now and the effect of consumer demand on businesses is driving us to do it fast all over again.
This is going to be a nightmare right?
Actually no because this time we’re prepared. Processes and tools are here and available to support what we need to do. The solution is DevOps. The outcome is fast AND safe.
The business case is far too strong to stay with traditional ALM and long release cycles. Of course, I’m not saying that everyone should be looking to move to daily releases but I’ve witnessed many great examples of customers moving from 6-12 month cycles to monthly and weekly releases. They could release daily if they wanted to and the fact that they don’t is largely down to culture.
Where do we start?
I’ve spoken to a dozen or so organizations over the last few weeks who want to start working at pace but don’t really know how to start. The words agile and DevOps come up a lot but they’re unsure where to begin.
So if you run SAP today what can be done to move to a faster, better, safer way?
Here are 8 steps that can be taken to implement DevOps and massively improve IT delivery.
1. Increase deployment frequency
Switch to smaller and more frequent release cycles first.
Delaying releases to 6-12 month cycles causes huge delays in the delivery of new functionality.
Focus on moving towards weekly drops of fixes and small changes and monthly drops of larger changes rather than bundling things up into big projects.
2. Organize teams around projects and products focussing on business outcomes
Make business process owners into product owners and start small.
Giving them ownership of products and projects promotes a deeper involvement in the full development lifecycle and allows input into design, QA and delivery.
But don’t try to do this for everything. Get exec and management buy in and start small with a specific project. Learn from mistakes and continuously evolve the process.
3. Evolve to Agile
Focus developments on business outcomes and working solutions.
A switch to agile involves the creation of cross functional teams where collaboration and communication is fundamental.
Create a planned and prioritized backlog where requirements are described as stories that produce business outcomes for specific personas.
When requirements are broken down into smaller manageable chunks they can be delivered in short development cycles. These aim to deliver working software quickly but also support scope changes.
4. Shift Left QA and operations
Engage QA and Basis from the beginning. No silos = No surprises.
Traditionally QA and Basis teams operate as distinct functions with responsibility across many projects and products. Including QA and Basis into project and product teams ensures that every function in involved in the development lifecycle.
Shifting left QA and operational involvement to start as early as possible in the process ensures that there are few surprises later.
5. Streamline approvals
Make approvals slick and informed so they don’t kill the process.
Most people would agree that they want a robust approval process to control and audit the changes being made to SAP systems. The issue is that the process often adds unnecessary delays and the approver doesn’t understand the risk and impact.
Having a slick process where approvers and testers are informed as soon as they need to take action means less waiting and a slicker development process.
Shifting left and automating quality, risk and impact checks into the process provides the approver with contextual information about the item being approved and its dependency on other changes. The approver can then make a conscious and informed decision as to what to do.
6. Automate everything you can
Manual processes always introduce delays and errors so replacing these with automation is a no brainer when speed and quality are a key factor. So automate as much as possible.
Continuous integration involves the automation of unit testing and the build pipeline.
Continuous deployment involves the automation of change deployments into downstream systems.
Both processes require SAP specific tools that support the required components but also require a shift in mindset and development methodology. What it does is create a streamlined development process where problems are identified early and where human error is massively reduced.
7. Create the right QA environment
Continuously refresh QA with production data and config.
To allow the DevOps processes of continuous integration and deployment and the automation of testing activity it’s vital to have the correct SAP environments available at all times.
Everyone would agree that they want to put live exactly what they built and tested. That only works if the environments made available for build and test reflect what production looks like.
8. Integrate change management
Understand the implications of changes across applications and at an organizational level.
The pace of SAP change is often disconnected from the pace of change of other applications. Changes to Systems of Engagement need to happen quickly and frequently to respond to consumer demand but related changes to SAP are disconnected and slow.
Release cycles across pace layers need to be synchronized so SAP changes can be connected and delivered in-line with other non-SAP applications.
It’s not a threat, it’s an opportunity
Any transformation is a journey and these steps will take time and an open mind to come to terms with. Whether you choose some or all and how fast you go will depend on individual circumstances, but the results and benefits that can be realized are clear.
It’s about making continuous marginal gains that add up to substantial improvements over time.
Yes, DevOps for SAP is different. Common tools used in the non-SAP world don’t work with SAP so you need tools that both understand SAP but also to allow you to be agile.
But along with speed comes protection of the crown jewels. As I said at the top you need to be both fast and safe to make your systems into an active enabler for your business goals.
And this is where I see DevOps processes transforming SAP change delivery.
I’ve also published a white paper on this subject so to find out more about the background and the 8 steps to success please download it here:
At the SAP Australia User Group conference this year we had a keynote speech from two guys from Amazon Web Services. The main speaker was Irishman Connor McNamara aided by Nam Je Cho.
As might be imagined the Amazon Web Site is a massive application, with as many web pages as there are stars in the sky.
How often do they make a change to their productive system you might ask?
It turns out, on average, once every 11.6 Seconds.
I thought once a week was bad enough.....
They must have some sort of all singing, all dancing release system, or they are doing something really simple which escapes the rest of us.
Really the obvious point is - has the change been tested? Has it really? Really? Really Really Really? Regression testing as well? Usually the answer is no.
If you have automated HPQC test scripts for example, which take 24 hours to run, that is good for regression tests for weekly changes ... but maybe not so good if you want to make a change to your live system every 12 seconds.....
Thanks for the comment.
You are indeed correct in that Amazon deploys changes every 11 seconds or so and it sounds pretty impressive but also scary. The thing with Amazon is that they changed their IT and software architecture many years ago to allow them to use continuous integration and automated testing. By bringing unit test automation to the core of what they do and by ensuring that they follow the correct software and infrastructure design principles it allows them to make changes to application components that can be automatically tested and deployed rapidly. The principle is that if a component that's used in 100 places is changed and auto tested they can be sure that anything that uses that will also work providing that there's good code coverage in the unit test. Also, Amazon can roll out changes to specific servers to "test" them so only a small user base is affected if something does go wrong and if so they can quickly roll it back.
Now in the SAP world it's different in that changes, once rolled out, affect the entire SAP system so testing is very important. But the application design principles used by webscale IT companies can also be applied to SAP.
We build our SAP software using SOLID OO principles with test driven development and unit test automation embedded from the start. This allows us to use continuous integration and deployment of SAP solutions.
Of course, there is still regression testing to be done where required and automating that will bring further benefits.
The other key part of what I'm advocating is that SAP changes don't need to be bundled up into long 3-6 month release cycles. They can be built in more of an agile way, and once tested they can be deployed to production for business use as long as they don't have dependencies on other changes that need to move together.
Being able to be agile, prioritize requirements, understand dependencies and automate as much of the process as possible can massively accelerate SAP application delivery. I'm not saying that all changes will be built, tested and deployed in a day. What I am saying that what has been built and tested can be safely deployed to production faster and more frequently as soon as it's ready not when a long project is fully signed off.
Organizational culture and team structures need to change and the right tools need to be used but the combination of these can bring huge benefits to make IT much more responsive. It also helps SAP changes that are linked to rapidly changing customer facing applications to be managed in a much more holistic way.
I hope this helps explain my thinking around SAP and why Amazon, although very different, is not so far away from what can be achieved in the SAP world.
I am in 100% agreement with you. I see no reason at all why you could not have daily releases into production without bringing the world to an end.
Unit Testing is the most under-rated thing in the world, it really helps spot obvious errors. I am still waiting for a really good SAP based automatic regression test thing like what ECATT and HPQC were supposed to achieve but failed to.
What tends to happen in real life is that you have a quarterly release with no proper testing, or a weekly release with no proper testing. Given that - what would be the difference between the status quo and a daily release with no proper testing? There would still be no proper testing, you would just annoy the users and find the bugs faster.
Or maybe do some proper testing? That sounds crazy, but it just might work.....
Yes you are correct Paul. Any good process does require proper testing otherwise you'll end up breaking things more frequently!
Being able to automate unit testing to spot obvious low level issues is important but also important is the ability to understand the impact of changes so the correct testing can be targeted. Part of the issue is that people don't know what to test and then regression testing become a real bottleneck. If you can understand which parts of SAP are affected (e.g. which transactions and programs, etc.) then they can be targeted for testing rather than having to run a full regression suite.
Automated regression testing does really help and there are a number of tools available to do this but it's also important to ensure that the tests are kept up to date as the applications evolve.
At the end of the day, proper testing, as you say, needs to happen and the more it can be automated and targeted the simpler this will become.