Skip to Content
Personal Insights

Agility in Software Engineering is not an Option

First word of caution: this blog is in the category ‘personal insights’: it is thus PERSONAL, it should be the start of a conversation, not the end… What is more useless than a blog with no comments?

Imagine you are working for a large enterprise software company: you are a manager of a team of say 100 persons, and one Monday morning, you receive a call from your manager saying: “Something important came up: you have one week to provide me with 20 names to work on this new topic!”. We will refer to this as the ‘Monday topic’.

Rings a bell? Not everything has to be that fast, big, or brutal, but as true as the sky is blue, these situations will happen: it could be because there is a customer escalation, because there was a bad analyst report, because a competitor announced a new stunning feature, because there is a new head or product, because a development manager has left and his team is in disarray…

How did we get here?

The first reflex is to ask: ‘but how did we get here?’. In the true spirit of agile methodologies, and the use of retrospectives, the first reaction is to try to understand why something so important was not considered high in the backlog on Friday and appeared on top on the next Monday. The truth is that backlog ranking is always subjective. You may try to organize committees between product managers to get a consensus, or rely on a single individual with charisma, or do customer surveys and users councils: None of these rankings is truer than the others because there is no absolute scientific truth about these rankings. Do not get me wrong here: you need a backlog ranking because you need to communicate something for the development teams to work in sequence if the workload is bigger than the available resource (it will always be the case). But a ranking is like any other plan: it may change, and some may argue that it should change (and it may even be because of incremental changes over months that the important ‘Monday topic’ was pushed to the bottom of the stack).

So, the practical advice is do not spend too much energy on understanding out you got there, there will be always unexpected triggering events, and black swans cannot be predicted, and they may not even be rare! Change is constant. The source of the change is not that much important.

How easy it is to handle that situation?

Now, you must engage with the development managers and the development teams. You must face a situation of forces pulling in different directions.

  1. Most software organizations consider agility as a goal, so the reaction time to the ‘Monday topic’ is in your KPIs (Key Performance Indicators).
  2. It is not as if your teams were not already working on some features expected by some customers, so the ‘Monday topic’ will choose some customers over some others. Of course, the Net Promoter Score of these customers will be lowered, and the NPS is also in your KPIs.
  3. Switching to the ‘Monday topic’ will mean that you will have to stop developments that were going ahead, and it is likely that, when stopped, these developments will be half-baked, not reaching the viability that you were shooting for, and this leads to lower overall quality level, another of your KPIs.
  4. Context switching will reduce productivity on a short term because of the knowledge transfer costs, and longer term because it may introduce dependencies (since you need to find the knowledge somewhere) and productivity is one of your KPIs.
  5. Teams will feel less engaged because they have the feeling of losing their autonomy (which is a fact in our case), and not recognized for their domain knowledge on the topic they were working on before. The employee engagement will thus be reduced, and it is also one of your KPIs.

The way that your organization handles the fact that you (and your team leads) will have to lower 4 KPIs to be more agile is a clear sign of the overall maturity of your organization with respect to ‘agility’, and therefore you need a special leadership culture for agile. Agile Organization means much more than having scrum ceremonies or a Kanban board: it is a known fact and has generated the thinking around ‘Fake Agile’. Out of these 4 lowered KPIs, the most important is the employee engagement. How to manage that employee engagement? It boils down to the one agile currency: trust. You know that the ‘Monday topic’ is a disturbance is that this disturbance will have costs. It is important that you show that you know about these costs, and, eventually, that you tried your best to avoid it. Concerning the feeling of autonomy loss, it is worth remembering that autonomy of development teams does not mean that they live in the vacuum, there ARE pressures from the external world, and that developments teams should be as proud of having some domain knowledge as their ability to become knowledgeable on some other domains, knowing that the spectrum of these domains cannot extend to infinite. Finally, and this will be the subject of the next blog, you will have to watch the consequences of these disturbances to, in a way, check the costs.

Agility and Road-maps

The point on the satisfied customers and less satisfied customer is linked to the enterprise culture with respect to ‘commitments’ and thus road-maps. Road-maps describing features to be delivered for the next 3 or 4 quarters are not compatible with an agile world when things change fast. As noted in the agile methodologies, road-maps should focus on outcomes rather than outputs and talk in themes and epics rather than features: They must not stand for tactical plans. The large cloud providers understand this as noted in the road-map style of Microsoft Azure or the style of Amazon (a public example can found in Amazon blog) where the development themes are described under three labels: “shipped”, “in preview”, or “in development” (sometimes ‘in research” is added). When we talk about enterprise culture it even goes to the sales team which should not be trained to sell road-maps to win a deal. If the upper management is advocating both for agility and road-map adherence where road-maps are described as tactical plans, you know this organization is not setup for success.

Conclusion

In our world of large and complex enterprise software, there is nothing, absolutely nothing, that is carved into stone, and there is no absolute truth on what is needed by the customer enterprises. Therefore, agility is not an option: it is absolutely vital and mandatory. But changes will require a proper management of the customer (or internal stakeholders) dissatisfaction for the ones who were expecting something that will not happen (and thus leading to coming escalations), a proper monitoring of overall quality (since the developments that were stopped early will lead to quality issues and will eat some upcoming resources in the same development teams), a proper monitoring of the productivity loss (and do not mix activity with productivity), a proper monitoring and specific actions to elevate employee engagement. This will be exposed in a next blog.

10 Comments
You must be Logged on to comment or reply to a post.
  • Hi Erik,

    I wouldn’t agree when you say “Teams will feel less engaged because they have the feeling of losing their autonomy (which is a fact in our case)".

    I don’t think that teams lose autonomy when asked to move to a “Monday topic”. I think of autonomy in the sense of Daniel Pink in Drive (https://www.amazon.co.uk/Drive-Surprising-Truth-About-Motivates/dp/1786891700/) i.e. autonomy relates to how to solve a problem, not what problem to solve. As you say, we don’t work in a vacuum when it comes to business priorities.

    And moving to a “Monday topic” doesn’t diminish the team’s freedom to solve problems any way we see fit; teams still decide how to tackle the problem, and are completely autonomous still in how to get the job done.

    I agree though there can be a loss of mastery – e.g. subject matter experts moving on to a new domain, or worst-case someone with a career of experience for example as a backend Java services dev being asked to move to a frontend stack, or vice versa! That’s a tough one 😉

    Cheers

    Paul

    /
    😉
  • Hi Erik,

    without considering the reason why this happened and the sense of loss for working out of our comfort zone, my personal opinion is that the as developer I always (most of the time) hope to work to something really helpful/required for the customer. So my engagement is related to that.

    Agile does not mean everyday anything can happen without any issue for the team, but, instead, working in an Agile environment allow the team to react in a rapid way to the requirements changes.

    It works if we respect few simple rules, one of all: "If we plan the sprint correctly (all the task finished at the end of it), when the new one begins the team can start with a fresh new goal (if we have the backlog correctly filled)"

    This means  that instead of "Monday topic" we could say "new Sprint topic".

    Thanks

    Raffaele

     

    • Thanks Raffaele Sangiovanni , I think you bring an interesting point that I am touching upon in the following blog, and this is linked to the fact that I am focusing on large and complex software where it is very unlikely that all functions will be delivered in one sprint/wave, by one team, so, even if you manage to have user stories delivered every sprint or wave, they will not always reach the minimum practical usability scope. This will lead to low quality perception by the customers expecting the full scope to be delivered.

  • I would add that team members should be aware of those possible changes. It is part of our package when we enroll in a dev team : priorities can change.

    Maybe this point is not said clearly and often enough => it could be included in a kind of N Practices Developer Manifesto (adaptation of 12 factors agile manifesto)

    But in a cloud development I think that priorities should not change so fast between feature dev : as we should be in a ci/cd process, we should be able to deliver a task (small part of a feature) fast enough so that it is not necessary to let it in the middle of nowhere. Once it is finished, another task on the new priority can be started (maybe not day+1 but day+3 or day+4) => I cannot imagine a feature is so urgent that 3 days lost would change the game (not talking about issues that are top priorities)

  • Hi Erik,

    Very interesting read and thanks for sharing these insights.  I particulary welcome the very clear transparency to the balancing act and difficult decisions that you and the management team have to make regularly. I’m glad you are making the decisions on the “Monday topics”!

    Sorry I still don’t understand the main proposition that “agility is not an option”.

    In our world of large and complex enterprise software, there is nothing, absolutely nothing, that is carved into stone, and there is no absolute truth on what is needed by the customer enterprises. Therefore, agility is not an option.

    Surely most of the blog and even the 1st sentence in that quote is advocating that agility is necessary?  I am missing the “therefore”.  From my reading, the blog title and most of the content contradict the “agility is not an option” line.

    You are clear that agility can be a negative contributor to KPIs and the balancing act involved definitely rings true.  That comes down as you say to the maturity of the organisation in how it copes with change.

    The learning I get from that is rapid and constant (disruptive) change is bad but gradual change (adjustments in course) at a lower cadence is good.

     

    The reason I think “agility is a necessity” is the more we understand the customer and the priorities of our customers then (hopefully) the less “Monday topics” there will be.

    There will always be organisation changes and external influences that require flexibility.  I think that it is more environment flexibility vs. what we call agility in terms of software development.

    In terms of building software, the “business as usual”, we build solutions to meet our customer’s requirements.  We try to understand what will make them happy and get their adoption.  As per the revelation of agile, that is an ever-evolving thing as we understand the customer requirements better and adjust our priorities accordingly. Agile puts customer value front and centre.

     

    On roadmaps –

    The azure link looks more like release notes to me (like SAC what’s new).  The AWS link looks pretty similar to the SAP roadmap explorer.

    The “subject to change” disclaimer in SAP roadmaps is the key bit. That’s where the agility aspect comes in even though it is normally in tiny text at the corner.

    The roadmap should evolve as we understand the customer better and pivot with improvements/innovations/ideas that might better meet their needs than the existing solution.

    There is also the “build it and they will come” approach too so that covers things like the technical foundational aspects. A good example of that would be the BTP transformation (e.g. the Data Plane topic) and other cool innovations.

    I think internal roadmaps are still extremely useful for development teams.  This helps dev leads and architects work out the technical and non-technical blockers and carry out due diligence (e.g. PoC’s) before hitting a blocker.  The blockers could be purely technical (good in theory but not practise) or just a lack of resources from a dependent team (unfortunately more common).  If the blocker doesn’t go away we can pivot the plan based on customer value.

    thanks,

    Alan

    • Thank you very much Alan McShane , I think you bring an interesting point around smooth changes versus abrupt changes. I think that my starting point is that there will be abrupt changes: And all this comes from the fact that when we say ‘put the customer at the center’ it is as if there was one prototypical customer, which is not the case for complex software, we have many different personas and use cases, and the switch of focus between personas and use cases is like a lot of natural processes: it is not linear! At one time, you will reach a threshold, and bam! you will have to deal with an abrupt change.

      • I agree with Alan on Roadmap too. We have also the notion of Agile Project Management that will support the organization above the ToT. Its role is to put in place or run a process framework that will provide high level governance and should hopefully prevent some Monday topics.