Context: This article is follow-up material to the course ‘Writing testable code’ but also of general value for all developers and development managers.
Why this article?
Maybe you or some of your developers have participated in the course ‘Writing testable code for ABAP’ or otherwise got interested in how to make ABAP development more agile. Teaching just the technical aspects of unit testing, isolation etc. is not enough to really implement this work practice well and get the full benefits in a development organization. This article is about what else is needed to make that happen.
The target group for this article is development managers. We are addressing topics like:
- Why should we care about agile development / – software engineering?
- The importance of clear business goals
- What are the key topics in technical trainings?
- In what way is an agile team process such as Scrum / Kanban important?
- What else is needed besides technical trainings / knowledge?
- Tips for piloting and sustaining the changes
Important disclaimers up-front: We are describing our own experience at SAP and what we learned from our journey. Improvements didn’t come easy and our journey is not over. While the technical topics are the same for all companies, the aspects of culture, starting point, what works best etc. is different and needs to be designed / discovered for each case anew. Still, we hope that this article gives you some interesting points to help you on your journey toward improving the way you develop and test your software. We also give some references to other companies’ stories at the end.
Motivation: Why should we care?
One of the big challenges for managers and product owners is how to balance the demand for features with investments into development efficiency and quality. This ‘quality’ can be external quality such as functional correctness, performance, usability etc. Internal code quality affects development speed and significantly impacts maintenance effort. Unfortunately, the adverse effects of poor internal quality and inefficient development processes is not immediately apparent. The temptation is to just ‘push a little harder to take this next hurdle’ but making the long-term situation worse and worse.
The picture below shows how development productivity can evolve over time:
At an early stage of product development productivity of development is pretty high. It seems that you can add features very quickly – assuming you have the technology learning curve behind you. But over time things slow down.
One part of this effect is simply the time that is needed to analyze bugs, fix them, and bring the fix back into production. It is not uncommon to hear that development teams spend up to 50% of their time with this kind of work, and the effort can be measured as ‘bug fix effort’ or ‘maintenance’ (not counting functional enhancements in legacy code).
Another part of this effect is hidden: Because of the complexity of the code, it becomes harder and harder to make enhancements without ensuring that you didn’t break something. In this way ‘new feature development’ slows down and the slow-down effect can be just as bad, even though it is hidden in ‘development time’. It can become quite dangerous if your competition is much faster that you in this dimension.
In the worst case, a code base can become so unmanageable that it is basically not maintainable anymore and must be rewritten. This is a total loss and has led to very high costs and even software companies going bankrupt.
All this may sound familiar to experienced software managers. Is there a way to avoid this trap and reverse the effects? In our experience: Yes! In this article we want to share what we learned on th way.
Agile Software Engineering at SAP
In 2009 SAP has introduced LEAN as an overall management method and philosophy in product development, and specifically Scrum for development teams. This was a massive change that required adjustments of location strategies, roles and responsibilities, work methods, internal release planning etc. Overall, the change to ‘agile development’ was a success that brought many improvements.
However, like many others, we soon discovered that Scrum alone is not enough. Scrum deliberately omits technical practices in its definition. These are however crucial to be able to deliver high quality software in short cycles. As Martin Fowler stated in his famous ‘Flaccid Scrum’ article you also have to pay serious attention to the internal code quality and this means that you need agile engineering practices that enable Scrum as a project management method to actually work in software development.
Thus, in 2010 we defined ‘Agile Software Engineering’ as “the set of values, principles and technical practices that a developer needs to master to work successfully in an agile context“. The picture below shows some of these key technical and scrum practices.
This was the beginning of the ‘Agile Software Engineering Program’ that had the goal to teach developers these skills in a hands-on way.
Note: We internally called it ‘ASE’ even though SAP had just acquired a database called ASE. In the rest of this article we always mean the ‘Agile Software Engineering’ practices or the education program when we use the term ASE.
Our internal advertising and positioning of ASE read as follows:
‘Agile Software Engineering’ builds on Scrum / Kanban and focuses on the technical development practices that make Scrum work. The Agile Software Engineering program provides training and coaching to development teams that teach modern agile software engineering practices such as unit testing, test-driven development (TDD), test isolation, refactoring, pair programming, exploratory testing, etc. and the related tools. These development practices have many proven benefits such as increased quality on all levels, better design, more stable code-lines, less integration efforts, significantly less maintenance efforts and even increased development speed overall. This change in development methods is profound and has significant impact. …
For some information about the content of the trainings, see
- Information Brochure on Agile Software Engineering (ASE)
- Intro video on the key elements of ASE(ASE youtube channel)
Guidance for development managers
What this article is about
As stated above, ASE technical development practices are a deep change in development methods and culture for most teams. Just taking a course usually does not change anything, especially when change management is involved. This article focuses on the question:
What else is needed besides technical training to really implement these practices in a sustainable way that realizes the business benefits that we really want.
In that way, this article is primarily addressed to development managers that are responsible for development teams / organizations.
In this blog posts we want to give specific advice about what we think are key factors in successfully introducing agile software engineering practices in development teams. The focus here is on introducing the technical practices, not on Scrum / Agile which we consider a necessary foundation.
It starts with a business problem or goal
Why should you care about Agile Software Engineering practices in your teams? Usually one or more of these problems trigger the interest:
- Too much maintenance / bug fix effort reduces our capacity for new features
- Too high manual test effort (often for regression testing) à very high cost
- Innovation cycle times are far too slow, we want weekly instead of quarterly updates –without bugs
- Automated tests are running too long and are unstable
- It takes too long to resolve issues in the system
- Our IT is too expensive
- The competition seems to innovate much faster! How can we speed up?
While a good implementation of ASE addresses many of these, you should have one or a few clear business goals when you start an ASE implementation.
It is also very important to have some metrics on these goals so that you can see if and how much you are improving. If possible, it is also worthwhile to convert these metrics into money, e.g. “too much test effort” into “X amount of Euros/… in test effort” and “Y days of testing that delays delivery”.
Then, as a manager you may decide “we have to invest into our development methods”. At this stage, management has to very publicly state they want this ‘agile / agile software engineering’ transformation and are making it a priority. Use your agile coach to design a ‘big story’ that works in your context, emphasizes the goal and benefits and acknowledges the learning / change efforts.
The key challenges
Introducing agile development practices has to address these main challenges:
- Though not strictly necessary, a good implementation of Scrum / Agile is highly recommended to have a solid foundation for ASE. Not having this foundation massively increases the risk.
- Developers need to learn the technical skills involved, but that part is relatively easy. The harder part is to change the individual and team work practices and make the change stick. This is a hard ‘change project’.
- Managers need to truly support the change because (a) any change is hard and needs energy, and (b) it takes effort that would otherwise be spent of feature development. For managers to be able to do this, they need to understand enough to believe in the benefits and know how to support it. This requires good information and supporting guidance for managers.
- Some level of external coaching for the Agile and ASE training and transformation topics is usually necessary. But you can also use some good internal people to drive this.
1. Scrum / Agile as foundation
How to do an agile transformation of your development teams is out of scope for this article. For that you would need either internal coaches you can use or get an external one. It is not strictly necessary to have a good agile team process, i.e. a good implementation of Scrum / Kanban / … for ASE methods to work. But there are several aspects of a good development team process that are important in our context:
- Stable teams (less than 20% turnover per year)
- Co-location, i.e. also the product owner is co-located with the team
- Team code ownership as opposed to individuals
- Empowerment to decide how to work
- Team is responsible for the quality of what they deliver (not some separate testing group)
- Team drives internal improvement process by retrospectives and the like
- Development and internal delivery in short cycles, i.e. sprint lengths of 1-2 weeks
- Fast feedback from production
- A culture of engineering excellence over ‘just get it out the door’
Part of such an agile transformation is looking at the real incentives in your company / development teams. What do people really get commended and rewarded for? Getting out the door on time (since the date was communicated) and fix it later? Be a good fire-fighter that fixes the issues in weekends and night-shifts (hero syndrome)? Or for very high quality / ‘virtually no bugs’? You cannot change something where the culture and real incentives push against.
There is a lot more that needs to be said about ‘agile development’ on the team and product level but that is out of scope for this article. Again, we recommend to hire an external coach.
2. Teaching the technical skills
Then comes the task of teaching the technical skills to the developers. The main topics to be covered are:
- ABAP Objects, i.e. a practical knowledge of object oriented programming & design, is often not sufficiently strong in ABAP teams.
- Agile Software Engineering methods as described above. Internally, we used a 5-day ASE course (on-site, team training) with additional 3 weeks of coaching in the teams. For customers and partners we recommend the openSAP course ‘Writing testable code for ABAP’ (3 days) that covers the core topics.
- Dealing with legacy code is an additional challenge. Here we recommend Michael Feathers’ book “Dealing effectively with legacy code” that not only covers refactoring for test isolation but also improving legacy code systems in general over time.
Test automation also has the challenge of covering ‘all those possible combinations’, especially for legacy code that has automation only on the system / UI level if any. For that we recommend out ‘combinatorial test design’ videos on youtube that shows how to handle large parameter spaces very efficiently.
There are many ways to teach / learn something: classroom, videos, books, … We are not saying that you cannot learn these things by book – some people certainly have. But when they did, they wanted to learn and wanted to change their work methods or try out new things. To effect a change in a team or development organization is a totally different matter and orders of magnitude more complex.
In our experience you need:
- A small ‘core team’ of enthusiastic developers who already know / learned the topics, or external experts you can hire as coaches to teach and coach your developers. These are the people that must be ahead of the rest in experience and help the others to get started quickly without much frustration – or they will give up.
- We think that a training course is necessary that shows the techniques and the new way of thinking in a coherent way. This gives the team a common language to discuss the topics, share problems and solution approaches etc. We also think that the course should be a class-room training because there is a lot more happening than showing how a tool works. There are discussions – some topics will be controversial – within the team, there must be room for alignment and exchange of views to create a common view and understanding. A classroom training also avoids the frustration of being ‘stuck’ at an exercise, bug / tool or other challenge. We also emphasize pair programming during our courses which is another powerful instrument to support learning, for change and to strengthen team spirit. Even when a course is only accessible as a set of videos and exercises, the team should at least get together, watch the videos together, do the exercises together, have some help from a coach that has some more knowledge / experience – and make it a fun experience. The latter is also important! Spend the budget for cookies!
- We already stated a few times above: It is essential to do trainings as a team training. Just imagine what happens when one person of a team comes back and says “I know it now – we all have to work differently”. You can imagine the effect of that. Of course, there will always be someone sick or on vacation but it is important that most of the team (80%) is together because the team as a whole will have to decide what to focus on later, what the ‘done criteria’ are to be, and exert the peer pressure needed to bring everyone along.
- A course alone is also not enough because people need help to apply the patterns in their own code / product. If you don’t provide this in some way then the training time is usually a complete waste (we have had this experience). For our internal ASE courses, we had 3 weeks of coaching after the course where the trainer of the course would go into the team and practice all that was covered in the course.
For the smaller ‘Writing testable code’ course, there surely can be other ways. The main point here is that there must be support for the team for technical questions but also to keep up the focus. In the section ‘Scaling and Sustaining’ we will discuss some options for this.
3. Piloting – Getting started
Assuming you have an expert available to teach and coach, this is how we suggest to get started:
- Since the learning effort will take effort / time, there will be a lot of skepticism if this is worth it. You need to set up a small project team that is tasked to drive this topic in all aspects. They will have to promote it, explain to managers what this is about, be coaches to the developers etc. In this team you need a few very technical people but also some soft skills.
- It is always a good idea to have a pilot project that proves the point that it really works within your context. For that you will need to select at least 1, better 2 teams that are open to try new things, open to these particular topics, are not in escalation or high-pressure mode at the time, and have features to develop in their backlog right after the training time.
- If at all possible, get an external expert or at least 1-2 very good internal people wo know the techniques and tools already that can coach the teams and make the pilot a success. This is also where you find out what needs to be tuned to your specific context of projects, product types and development culture.
- After the pilot you, i.e. the project team, should very openly ask about feedback, issues, problems but especially also the positive experiences, benefits they saw etc.
- In the pilot you will most certainly hear that applying the ASE techniques for new code is relatively easy but ‘in our legacy code it is very hard’. This is true! And this is the reason for the needed coaching discussed above and the ‘further reading’ references at the end of the courses. Apart from that however, our experience and recommendation is that you should learn the techniques when writing new code / new features. Once you have the techniques and patterns down, you can then learn how to apply them to legacy code. For this reason, the courses focus on the new code / new feature case and leave references for the legacy case.
- As a corollary, this means that your development teams should take this course just before / when they are developing new features, not when they just have to handle small bug fixes or the like. They need the immediate practice after the course so you have to plan this carefully.
Apart from all the planning there is also the people aspect to consider and this is where it gets really ‘complex’ and you have to show sensitivity and prudence.
- Motivating developers: People react differently to new topics, especially when you have ‘done it for 20 years’ and now someone comes along and says you should learn something new. Some are always eager to learn, others may feel threatened by something they don’t know well (can do intuitively) and it may create resistance. You will surely have to deal with resistance in some ways.
- There are many great books about this. My personal thoughts about this:
- Don’t force change but make it a desirable goal by focusing on the benefits. Many years ago, the HP CEO called out the goal this way “Wouldn’t it be nice if we had only 10% of the bugs and maintenance effort we have now. If anyone has ideas on how to improve, you can get budget for it.” That was a goal for several years and they did I and they increased feature output by 8x.
- Developers have to be convinced technically by other developers / technical leads. So, it is a good thing to drive the initiative as a technical bottom-up topic – with management support. Let developers do the advertising.
- Make it easy to do the right thing. When people struggle to apply the learnings to a complex situation in their product and there is no coaching / support from experts they will give up and this will create resistance! Therefore: Don’t skimp on the coaching by internal experts that help their peers and/or get some external help.
- Look for early adopter teams that are open and start with them. Let them then tell their story.
- If someone totally resists you can finally say that this is more and more considered the ‘professional way to work’. You can use the references at the end for that.
4. Scaling and sustaining
When the pilot went reasonably well, you learned what you can from the pilot and the developers of the pilot team are positive, you should let them do the advertising to their peers.
When you want to scale to many teams, e.g. a whole product, you need more. We suggest:
- From the pilot, select a few developers that have shown some passion for it and have some level of training / coaching skills to become your internal experts / change agents. They should take some time to get more experience, get deeper into the topic (do some reading, try some patterns in real products) to get a solid understanding. After 1-2 months they can then be your ‘internal trainers’ and teach their peers. This will be your internal team that drives the topic.
- You should view these internal trainers not just as a means to ‘get all developers to take the course and be done with it’. While this is certainly a necessary first step, they should take a long-term view of your product architecture and code quality and see how e.g. decoupling for local testability can make your tests much faster and more stable. In the end you want them to go after the business goals you had when you started this process.
- We suggest to invest 2% of your development capacity (i.e. 2 people for 100 devs) for at least 1 year and then re-evaluate.
Important: Celebrate and be public about every improvement and small victory towards your business goals!
5. Transparency and Metrics
It is important to have transparency about the ‘reality’ and about the progress you are making. This needs a mindset of ‘never hide the facts’ – which is usually not the strongest trait in a company culture. This is of course requires that the messenger of bad news doesn’t get punished, even if they are responsible themselves! But there is also a lot of wishful thinking. It can happen that you find out that the maintenance effort for bug fixes is e.g. 3X as high as you assumed. These kinds of things must come to the surface when there is to be any chance of sustainable change for the better.
Another matter is one of the key elements of Scrum, the retrospective. When real problems are not named there, or the named problems are not addressed – and therefore people give up and don’t take retrospectives serious anymore – then you have no chance to change anything.
In addition to that there is the topic of metrics, i.e. KPIs that measure things. Here you have to be smart: Very briefly:
- Transparency is important for motivation and keeping up the focus on what you want to improve. Sometimes it is good to set a target as a project goal, in other cases (like overall code coverage), just show the chart and let everyone enjoy the line going up.
- Be clear about the business goals, how measure progress for individual goals? Put some metrics in place that show if you are really making progress.
- Asking people for time (=effort) recording is a bad idea – people hate it. That should only be done as sample every few months. Preferably use metrics that come out of automated tools.
- Focus on outcome metrics (e.g. bug fix efforts) and do not use internal metrics like ‘code coverage’ as hard goals.
- Do not ever evaluate KPIs for individuals (that will kill all honesty in the company) and never compare KPIs between teams to say one is better than the other. Every team should compare itself against its own history.
Beyond Unit Testing – where do you want to go?
There is always more that can be said but for now this should be enough to get you going. Unit testing is not the end of the story, nor should it be the focus. In the end everyone wants more new features delivered faster with less bugs. And developers should have less bug fix effort and more fun doing the creative feature development. What is covered in the course and described above should put you in the right direction.
If you wonder where the trip could ultimately lead, consider Continuous Delivery also for your internal SAP applications. For some inspiration you can watch the video The Rationale of Continuous Delivery, David Farley, one of the authors of ‘Continuous Delivery’.
Finally, here a short list of references that we found valuable:
- Kent Beck: Test Driven Development by Example
- Robert C. Martin: Clean Code
- Michael Feathers: Working effectively with legacy code.
- Gerard Meszaros: xUnit test patterns
Stories from other companies for further inspiration:
- Google has extensively documented their journey in this area with ‘Development productivity’, ‘testing on the toilet’ (and all the related articles and blogs), ‘From QA to test engineering’ and ‘How to write clean and testable code’.
- Microsofts ‘Scaling Agile’
- Spotify Engineering Culture
I wish you great success improving your development practices.