An open letter to SAP regarding DevOps and administrator enablement #sapadmin
After attending SAP Teched 2013 in Las Vegas, I noticed a few things which are frustrating and worrying in equal measure, I have spoken at TechEd several times now on several subjects – although they have a common theme, DevOps and several concepts within DevOps.
DevOps as a movement is a few years old now, and is approaching maturity within the wider IT community, although it still surprises me how few people within the SAP community have heard of it (I include SAP as a company and an ecosystem). Some of the largest companies in the world use these practices to improve their development and operations practices to massive effect – a recent report reflected DevOps practices had saved Development and Operations costs by 17% and increased quality of deployments by 21%. This is impressive, although it does take time and investment to realise these gains and requires some mental gymnastics for some.
DevOps counts companies like – Facebook, Etsy, Google, Amazon, Flickr, Twitter, IBM, SAP (Internal IT), HP as followers of the methodology which is centred around culture, communication, change and tools in equal measure. Below is a list of areas that companies typically address when using DevOps principles.
1. Infrastructure – should be deployable through tools and pre-defined configurations
2. Code – should be version controlled in a repository which is updated on a commit by commit basis
3. Testing – should automated as much as possible with nightly/weekly build tests
4. Change – should be automated, pushing released changes from Dev to QAS for testing and QAS to Prod to go live
5. Tools – common tools should be used and deployed across all the teams
This leads to a number of short hand statements
1. Infrastructure as Code – Systems administrators, should be able to define their infrastructure artefacts as code – this makes server builds and configurations repeatable, auditable, sharable and consistent
2. Using common tools creates common language which creates common culture – when teams use the same tools they communicate better, for example it is better for the SAP Basis team to use Tivoli for system monitoring because the Operations teams, Database teams, Operating systems teams all use it – in doing this you create a common language and culture. Similarly, using ChARM within Solution Manager also produces the same result between the Developers, the Technical team and the Support Teams, because they are looking at the same data, in the same format, using the same terms which creates common language and understanding
3. Don’t break the build – doing automated testing every night/week (using tools like Selenium, eCATT, HPQC, Rational) ensures that code standards are kept high, and anyone who breaks the build should buy a failcake or wear something embarrassing
4. Continuous Integration pipeline – a logical construct which maps development changes through a process into testing and then into Production.
There are areas within DevOps where SAP is way ahead of the game – for example
1. Change management, within the main ABAP, JAVA, BObj layers SAP has a massively mature change management system – they can be and should be proud of that fact.
2. Solution Manager, many people will mock me for this, but I can defend the concept if not the execution – Solution Manager is a DevOps’ wet dream
- It can monitor and record pretty much anything with in the SAP application and infrastructure landscape
- It manages change throughout the SAP application landscape
- It is a central configuration management database, recording the versions and patch levels for every connected application
- It has a reporting tool and reporting content build into it
I feel a well set up and configured Solution Manager is an amazing tool to have within an organisation practising DevOps.
Where I feel SAP is letting itself and it’s customers down is in 3 main areas
1. Little integration with existing infrastructure automation or configuration management tools – Yes SAP has products like LVM, but these are not Enterprise products – that is to say they are not used across silos. DevOps demands common tools to create common language, this means tools like Cisco ITPA, IBM Tivoli, OpsCode Chef, PuppetLabs’ Puppet. It also means SAP providing an API to extend products like LVM, SWPM and SUM the same way it does with it’s business applications – failure to do so would be discriminatory given the enablement spoken of in favour of developers so often within SAP.
2. Little integration or documentation on working with existing web world automated testing frameworks – Typically if you said automated testing to SAP customers, they would reply HPQC or IBM Rational or SAP TAO. Now if you said automated testing to an non-SAP person, they would use words like Selenium, Cucumber, HPQC or IBM Rational, which is interesting. As SAP encourages customers to use platforms like HANA, NetWeaver Cloud, SAPUI5 and adoption rates increase, the requirement to use bulky testing applications like HPQC or Rational decreases as the Web world already has built flexible and agile testing frameworks for themselves which are easily shared, reused and adapted.
3. Not talking openly about their own efforts in these areas through their “SAP Runs SAP” marketing brand – it may surprise you to know that a lot of these concepts are already being used within SAP already, internally within their IT organisation, for example
(presentation link http://devopscon.com/wordpress/presentation/continuous-delivery-at-sap-it/)
- SAP Uses OpsCode Chef for Infrastructure as Code definition and automated deployment
- SAP Uses JUnit and Selenium for functional testing
- SAP uses Atlassian Bamboo for Continuous integration
These are great things, it shows leadership in a grass roots culture which matches and builds upon the current Developer enablement focus within SAP, but I could not find anyone willing/capable to speak about them this week in Las Vegas, this included both senior management and people on the show floor.
This lack of capability to talk to these items concerns me, I know SAP is a large organisation and there are many pieces to it, but it also has a massive culture of “It wasn’t invented here” in many places – so I am writing this open letter to a number of people within SAP, people like Bjoern Goerke , Michael Golz , Jeff Anders ,Janko Budzisch, Markus WINTER and Martin Heisig – please continue to work with these products and develop capabilities in them so we can use them across our enterprises to create a common language and culture amongst our peers in the wider world.
You (SAP) have strived to make it easier for non-SAP developers to come into our ecosystem using common methodologies, like Agile, common tools like Eclipse, encouraged the use of version control systems like GitHub for platforms like HANA and the HANA Cloud platform to improve sharing. Surely it is not a stretch for the Technical teams to want to have the same experience because box hugging/manual processing does not scale. In order to meet and exceed the expectations of organisations in their quest to “Run better” we must create repeatable and shareable end to end automated processes the same way developers did – but they did it with your help and support which does not appear to be as focussed on technical enablement in the same way it has been for developers.
Most of the companies and tools that I have mentioned above are either already your partners or do not compete with you in any way, but complement you – there is no need to redesign the wheel, to create an SAP version of something which already exists – you (SAP) did not do it for your development tools, rather you used both existing tools (Eg Eclipse) or extended your own tools to provide web world capability (HANA XS engine).
I feel that this is a brilliant inflection point for SAP, the developer enablement process is gathering momentum and there have been valuable lessons learnt from it in terms of engaging outside of SAP as well as inside it. Bjoern Goerke brings with him a wealth of experience in developer and Agile practises which can be applied and encouraged within SAP’s internal organisation to help define SAP Administration technology strategy as part of it’s “SAP Runs SAP” tag line – after all SAP IT faces the same technology challenges as it’s customers.
Thank you for your time and I look forward to your comments.
Great article, it really made me step back and think about the overall DevOps landscape and how SAP (as a company and in terms of their products) sit within it. A lot of the clients I've seen could really see large benefits IMO from looking at integrating SAP more into their existing DevOps infrastructure (or conversely look at leveraging SAP to improve their DevOps in the first place), where up to now they've tended to treat their SAP systems as an independent silo/black box that's completely separate from the rest of their IT landscape, even when they have a pretty good level of DevOps infrastructure in place for everything else.
I'm also looking forward to some responses from people within SAP on this, now that you've made me think about it.
this is a fantastic all encompassing piece of work.
Having been supporting SAP Java Development from the Basis perspective for the last 6 years, setting up NWDI creating tracks, CTS+ integration, I can agree with everything you say regarding the tools which SAP has out of the box which are not trumpeted enough.
What you have written is not only an open letter to SAP but a very valuable all encompassing resource for Basis Architects tasked with implementing a strategy for supporting both ABAP and Java custom development and laying down the infrastructure to do so, from the code versioning and control through transporting and change control, and (automated) testing. Your piece about the nightly builds and who breaks the code buys a cake is very nice, because the underlying message is, everybody should take care in what they are doing and own what they are doing and lack of care costs time and money.
Slightly off your topic, one gap i see when comparing the SAP tools for ABAP development with SAP tools for Java development, with ABAP nobody can develop without a Developer key, in Java if a Developer has the right roles in NWDI, SLD, and the Java RunTime eg Dev Portal or Dev CE, then that person can Develop, Java is missing a Developer Key. In companies where I have been we have implemented a procedure for giving out the Developer Authorisations to SLD/NWDI.Dev Java RunTime as strict as the procedure for requesting a Developer Key.
Supporting Development is often a grey area for the Basis Team and hopefully your all encompassing work will draw more attention to it and doing it properly and the tools and possibilities.
Just a note on NWDI. My experience is that NWDI is brought in for initial implementations but, like automated testing (see comment below), is dropped by the implementation support team. Some of the reasons: buggy, difficult to use for a few. However none of these companies have signed up with CTS+ and generally have the separate ABAP and Java change management processes managed externally to SAP tools.
Kevin, your comments on NWDI, automated testing, having up-to-date production data for testing and even the lack of a truly cohesive change management approach that would involve the likes of CTS+ all marry up so well with my own experience that I had to laugh. Good to know it's not just the clients I've worked on then 🙂
Hi Kevin and John,
companies I have been at for the last 6 years have used NWDI for all SAP Java custom development and later with CTS+ (obviously that wasn't available when NWDI first came out) and it works very well, yes it is work, and yes it needs to be looked after, but it ensure code control and versioning and transport and adherance to standards which are taken for granted in the ABAP area.
Regarding automated testing, I have seen two scenarios:
Companies committed to automated testing with a full time testing team and infrastructure and more importantly test scenarios properly maintainted, test Users properly maintained and even more important, test master data for the Users and scenarios properly maintained.
Other companies, where suddenly budget came available for testing eg load testing of a scenario and then a big push to install eg Load Runner probes across the db, SAP ABAP and NetWeaver applications etc, create Users and create Master Data, and everytime I have seen this, the time line for the project ran out before the Master data had been created for the test scenarios and believe it or not, all that got tested was the logon !
The lesson is, if a company is going to go for implementing and supporting autmoated testing tools there is no half measure, they must go into it eyes wide open and with the budget and timeline to support setting up and keeping the infrastructure for the testing maintained, setting up and keeping the test scenarios maintained (recording the scenarios), and setting up and keeping the test users and test master data maintained. With any one of these missing the testing will not happen.
The problem there is that the companies all invested in license heavy tools, when they should have been working with either a smaller vendor or open source tooling. Granted SAP Gui is nearly impossible to simulate, but with the HTML Gui, NetWeaver Business Client etc... you should be able to simulate something close to reality.
That sounds like you think "licence heavy" tools are always wrong. Why do you say that? I'm not sure I believe that HP-QC, for example, is always the wrong choice. Perhaps a discussion about which tools are right in which circumstances would help people make a more educated choice?
You misunderstand me Steve, I say license heavy in relation to the comment from Andy that customer projects often run out of money during the project. Often the license costs are a large drain on these projects, so that was the only issue I was raising.
Tools like HPQC can be most beneficial, in fact the ability to collect, track and report on defects is better than most things on the market - there is a reason why it has such a massive market share.
Sorry. My mistake. I didn't read far enough back through the thread. 🙁
Thanks for this overview. This is an important topic in view of the new focus on cloud-enabling SAP applications. To take advantage of the flexibility and scalability of cloud environments SAP will need to bring LVM, SUM and similar tools up-to-speed rather quickly AND do a bit more to share the best practices with the customers. I suspect that there are some very interesting tools and techniques used internally for deployment that could be of use to those of us still working mostly with on-premise installations.
A few comments about testing: in the medium and small enterprise spaces I have worked in the past 8 years, I have found almost no investment in automated testing tools. Any automation was usually brought in during the initial implementation and soon was cast aside as budgets strains made the upkeep of the test scripts too time-consuming ( ie, expensive).
That said, testing is now the bottleneck, or shortcoming, in most upgrade projects I have worked in recently. Lacking comprehensive test scenarios, getting end user buy-in and participation --and my all-time favorite -- needing current production data for testing are just a few factors that extend the upgrade project timelines and can result in under-tested projects.
Thanks Chris for getting the conversation started. I will be interested to see how SAP will respond.
PS As a developer I worked with used to say:
thanks for this great blog.
As you know, I have been moving from the R&D side of the [SAP] house to the operations side, now being responsible for SAP's global Cloud delivery, HANA Enterprise Cloud being part of it, and SAP's internal IT infrastructure as SAP CIO.
Now, I have been with SAP for over 20 years and have seen the roots of the "empire" and how it was built. Clearly, back then in the 80s and 90s, the approach taken was exactly right -- and the success of SAP till today proves that many decisions have been without any doubt to the point. Nevertheless, many of the things that were differentiating back then have become commodity today and have been standardized to an extent that makes any attempt to come up with something different plain counterproductive. Perhaps, many of the attempts to still reinvent the wheel till today are a tribute to our "engineering" roots as a company where people rather want to invent than integrate, rather build than buy. So it's a cultural aspect that needs to change as well.
As part of my former R&D responsibility, we have moved to Open Source usage and contribution significantly and with great success. We're using Eclipse, Git, Maven, Hudson, Chef and many other Open Source projects intensively to run our internal development processes and have turned them into the foundation of many of our products. I have transformed the NetWeaver R&D unit using standard agile Software development methodology with Scrum where the overall sentiment initially was that "we are different" -- which was of course complete nonsense. We are not different at SAP.
If it comes to our product or technology portfolio, there are for sure a number of things one might discuss still from the perspective of "standard vs. SAP specific" -- not to say that the answer is always straightforward or easy. And the same applies to operations infrastructure.
In my new role responsible for our global Cloud delivery across all Cloud products, whether pure SaaS or managed Cloud environment, I am now up to making the decisions about the Cloud management infrastructure we are using. And let me tell you: we have many of them! So it is about settling on fewer infrastructure components, on harmonization, on "embedding" into different DC strategies etc. The clear goal here is to get something that we can share with big customers and partners. To make our best practices and operations optimizations available to them as well. And there is only one way I see to succeed here: follow open standards and embed our tools and infrastructure in what the rest of the world is using. Especially in Cloud operations, the lower level Cloud Infrastructure is currently getting commoditized -- whether it is players like VMWare, Cisco, HP, IBM or others delivering products to cover the management of compute, network and storage at large scale in an efficient manner or whether you look at the massive momentum of solutions provided out of the Open Source space with Open Space. If you move up the "logical" software solution stack, the solutions become less standardized and this is where SAP can actually add differentiating value to the management of large IT solutions. This is where we will concentrate our efforts and then have to make sure we simply tap into the "lower stack level" infrastructures that are broadly used. It simply does not make sense to compete with 1000 Open Stack contributors to see who's faster. Lets rather join efforts and contribute what we can contribute best...
I have been a big promoter of Open Source and Open Standards within SAP. You can rest assured that this is the only way I see forward if it comes to infrastructure management. Especially if it comes to our Cloud offerings.
Also, as SAP turns into the Enterprise Cloud Company, we will see that practices like devops will become more and more dominant. Like we have changed R&D processes with agile and Lean software development methodologies, there is no other way to be successful if you want to go Cloud full steam. And we will.
CIO & Global Cloud Delivery, SAP
I like the Open Source parts of your reply, not a fan of the Cloud ones.
I am not sure I understand, why don't you like/approve of Bjorn's Cloud comments - I thought they were spot on and identify clearly with both SAP's strategy and the overall market progression.
I'm not saying I don't like/approve his comments, personally I'm just concerned that Cloud will have an impact on non-Cloud offerings.
thanks for clarification. I can understand that there is some uncertainty what Cloud will bring to the non-Cloud world. And while I agree that not everything can be measured by the same standards, there are two important aspects that make me believe that there will be net positive effect on the non-Cloud world as well:
So I expect that with Cloud and On-Premise worlds living in parallel for the coming years, there's a significant benefit out of our Cloud investments making it into the classical on-premise world as well. And last but not least -- the on-premise world shares many many aspects of the massive scale Cloud infrastructure world as well. At SAP, we manage 60.000 virtualized servers world-wide...
I am sure you have good reason to not be a fan of the Cloud ones. Will you share those reasons with me?
A theoretical example would be that the development tool of the future would be XYZ because it is widely used/accepted in the Cloud, on-premise people would just have to deal with it. That said, I don't want to derail the discussion any further since the topic was DevOps.
I have to admit, I nearly swooned when I read your reply - it was like being a kid in a candy store, I was getting all my wishes and they were going to come true :-). So I slept on your reply and thought about it, then the critical thinking kicked in - the questions like when and how entered my head, although I am not going to ask questions like that for the simple reason that you probably do not have the answers right now because it's a new job, we're in the middle of conference season and Christmas is coming up, but I will come knocking on your door in the New Year :-).
I look at the nearly organic elegance of the design of the ABAP stack, where the entire SAP system can be simply resurrected from a database backup and a quick installation or from your own history, the ITS server - I can see where SAP's engineering decisions and inventiveness have been amazing and have stood the test of time. Making the balancing act between 'make or buy' is difficult, especially when companies like Google and Facebook and Twitter invent their own stuff - but they do it because the scale of their operational infrastructure demands it due to it's centralised nature. Your our customers do not operate on that scale and are demanding to use tools they already use in other parts of operational IT, so continue to extent your reach with APIs into all your tools to allow tighter integration at the Ops layer the same as we have seen in the Dev layer and Business layers.
As a SAP Mentor, I have seen the changes Agile and Scrum have made to SAP - to the outside world the best examples are the explosive progression of HANA as a platform and it's growing maturity and feature list as a database platform, also the NetWeaver (Now HANA) Cloud platform. I look forward to seeing how you bring that idea and experience into SAP Operational IT, I suspect you will find a hungry audience for a more DevOps style of working - or at least I hope you do.
In terms of tools, I have always been a big fan of defined tool kits and not mandated specific tools - because like programming languages, fashions and functionality change, so providing flexibility is critical to maintaining velocity. I absolutely agree that SAP provides good value and differentiation as you move up the stack, the capabilities of Solution Manager and other products are amazing, and if integration can be made into lower level tools (internal & external) then you will close the circle for your customers. If I may offer some advice, your partners (IBM, HP, Cisco etc..) are already massive contributors to the operational/development tools we are talking about, perhaps by leveraging both their experience and advice for your own tools would be a good way to start.
So my next questions are -
As a SAP Mentor - what can we as a group do to help you on your journey, enabling SAP to share and educate the community on integrating SAP into these OpenSource lower level tools from their Operational IT infrastructures. Perhaps a Mentor session to pick up those areas to help with.
As a community member - how do we keep up to date with the SAP runs SAP story on this type of activity, because many customers believe SAP runs SAP to be authentic and like to hear how your challenges match their own.
thanks for your reply (which even is more detailed than my own reply to your blog ;-)...
I'd appreciate any input on expectations for a future state, major pain points from today's perspective and perhaps shedding some light on the vision you'd have for the kind of infrastructure we talk about. Changes in approach will not happen over night and it will require a certain transition. I am not naive to assume that we will change the world over night. But going for open source, going for APIs, going for integration rather than trying to do everything on our own, is clearly the vision I follow. Details need to be discussed and a roadmap needs to be laid out. So it's some work ahead of us. I'd like to continue my interaction with the SAP Mentors community as in the past and am more than happy to particularly spend time on the topic of SAP runs SAP plus our Cloud Infrastructure efforts.
Standing ovations from the community! 🙂
What about a "formal blog" that would wrap all this for normal mortals that missed this article and the comments? That would be very very cool!
Thanks for all this, highly appreciated!
this is an awesome discussion and subject
it's a demonstration of the power of SAP and the SCN and the openness of the two
All the best,
After working 7years in SAP Basis and Security it really feels
SAP hasn't innovated in itself, from simply post configuration steps to be
followed by a Basis admin once the SAP system is built from scratch.
Every time a new System is built the Basis admin needs to
configure from technical settings like:
- Applying Support pack after installation.
I DO NOT UNDERSTAND why an SAP basis admin would need to bang
his head on desk to configure MOPZ and configure satellite system to on which the
basis admin needs to carry out support pack upgrade,.
Why can't it be simple as - Admin can simple hook to internet
and apply the support pack for first time.
- Solution manager set up - Needs so many technical setting
configuration,. why can't it be so simple where I give the Satellite SAP system
message server details to SOLMAN, it detects its self as to what kind a Satellite
SAP system and its usage type of SAP it is.
Anyways I just mentioned 2 of them.
SAP has to really improve a lot to come to a point from where it
used to be to where it is now.
rD - rIsHI DeVaNoOR
I appreciate Your statement about using SAP Solution Manager. Doing Solution Manager since 2004 I want to emphasize to use this mature and integrated methodology.
hi Chris - First, great blog. I read through some of comments, obviously you generated lot of good discussion.
Question - Have you seen any POV from SAP on DevOps?
Does SAP believe that their products can be implemented in more Agile, DevOps approach.