Skip to Content

This is a post I have written and rewritten many times over the last few months, I still find it imperfect and will probably catch many flames for it – but what the hell, “If you’re not creating trouble, you’re not creating much” as Hugh MacLeod says.  This blog will attempt to talk about certain developer practices, how it affects Basis/Technical people. I am writing this to start debate so we can explore a more common language and understanding that build a culture supporting philosophies like Lean and #DevOps.

Also I fully expect, in fact I demand, that someone write a rebuttal to this post :-).

I have spent the last 9 months immersed in #DevOps stuff, trying to find ways to engage the SAP ecosystem with it and see how SAP, customers and consultants can learn the lessons of #DevOps the same way it learnt and adopted Agile. So as we have seen in previous blogs, #DevOps is built around a concept of

C – Culture

A- Automation

M- Measurement

S – Sharing

At first glance these are quite technical and operations oriented things, I was a lot more interested in the Developers and how I as a technical person can help/work with them to create something amazing. My first step was to start to learn to talk their language, something which is like a difference dialect of SAP. Then to make matters even more complex, I also had to learn the language of the Web world as well – something which was a lot harder and by no means complete, I reckon I am talking and reading WebDev stuff at a 5 year old level :-). I have been ably assisted in this journey by a lot of friends, DJ Adams, Ethan Jewett, Steve Rumsby, Matthias Steiner and others (you know who you are)

Many administrators I have come across have no idea about developer practices, they know the rudiments of how developers work in the ABAP stack, but very little about Java, ADS or WebDynpro. I am nearly sure that some administrators, especially those who are not co-located with their developers, think that there are sacrifices made at the full moon to create the code or the code is produced through Ouija boards. So as I have said above this a miss-mash of thoughts to try and kick start closer collaboration and also start to alter some practices to make life easier.

1. I need you to help me speak your language and I need you to learn mine.

Especially non-ABAP developers – you know who you are, you have ignored us for long enough, we have let you get away with it and it stops now! We all spend time working on projects, and our managers make us sit in different areas so we can be with our teams. Is it any wonder we have lost the language of the birds, now I am not talking about being fluent or fully understanding how things are done but as a minimum I think we should have a common understanding.

2. I need to be a part of your change process.

Between us we must be able to agree on how we get code into the Production system. There are people who set down rigid rules, there are people who use 3rd party tools to enforce a process. There are those who just fly by the seat of their pants. Bottom line – there HAS to be a process and we all have to abide by it, live with it. Stop trying to go around it in order to get your ‘little’ fix into production because someone did not test it right! Before the integration of many of the SAP products into CTS+, I wonder how many Basis people actually understood or could support moving changes through the landscape of NWDI, CE and Business Objects. I propose that between developers and Basis people, we begin to understand the following – and yes this is very simple stuff, but if we do not understand the basics, then we cannot build good lean processes.

  • What does a change look like in each type of system
  • How does that change get migrated through the system
  • How do those changes get applied
  • How do those changes get tested
  • How do we ensure the quality of the changes that enter the process

Someone recently asked me – why do Basis even do transports surely our time could be better spent on other things. Something else to ponder as you discuss the points above together.


3. An extract of the transport history IS NOT a transport build list.

Let me say it again, just in case I have not made myself clear – the next developer who tells me that I should use the Transport history of a system as the release build list, will be punished in the most severe way possible. You and your team are the developers, it is your development – do your job and track your changes, I will help you – I will even provide some tools or tricks which will make life easier. Do not treat me like your mother picking up after you – you are not a rock star and I do not work for you, I want to work with you.


4. Object Management and conflict resolution

If you are working in a BAU environment, please, please, please – release your tasks quickly and often, not your transports (although this would be nice). This ensures that the object locks get released and other people can use the objects. Of course this creates dependencies  between transports with overtakes and undertakes. You know what, if we have worked together and have a solid process, we can mitigate that through both tools and processes. I have refereed enough bun fights between developers who need the same object but one has it locked – bored of doing that and I want to do sexy fun work.

5. Integrate your code quickly, the job is not done until the system works.

Often as a Basis guy, I am supporting the development work of multiple vendors – and everyone is doing their own development in their own environments. The nightmare starts when Integration testing starts, I will admit that we in SAP have a major deficiency in testing. We often do not use automated testing, and that is a travesty in it’s own right. So testing is a pain, when dealing with multiple code streams with multiple vendors, it has always appeared to me – that all developers keep their code streams as separate as possible and then at the last second merge them. This has 3 effects

  • Development have to scramble like mad to develop fixes, increasing the number of changes
  • Transport sequencing becomes ultra important, and someone  decides to use the import history as the sequencing method
  • The flurry of change can derail the change process as priorities get escalated.

I have been thinking for a long time of two potential ways to improve this situation.


  • Everyone can have a development system, but there is a single QAS, Pre-Prod

Multiple_Dev.jpg


  • A dual track landscape, one BAU and one Project. All code co-exists within the landscape

 

/wp-content/uploads/2013/02/2_track_190972.jpg

The bottom line is this – your development job is not done until the code works properly in my Production system. The sooner you get your code integrated with everything else, the sooner it can be tested and reduce the last minute panics. Of course that does assume that all the developers from all the parties play nicely with each other.

6. You are not always getting your own environment

This is an extension of the item above, I do not have limitless hardware or resources. I have to make a judgement call on what resources you get to run your project. This means that you may not get your own environment to work in. That does not mean I hate you, or do not value your project enough – often it means I actually can’t do it, or it means I am not going to stretch my team further in supporting  another set of environments so you do not have to be nice to other developers (or so it looks to us) Work with me, educate me on your requirements – do not try to pull the wool over my eyes and try to get more than you need. Let me help to support you with a brilliant service, rather than stretched to within an inch of my life and providing a crap service to everyone

7. Stop complaining to me that there is no test data in Development

This is an old one, but it is one of the most frustrating for both of us – I know you guys need good data to test with but it is not acceptable to do your unit testing in QAS. I want to help you get decent test data, I know that generating it is a nightmare and often belongs to the functional consultants as a task. Lets stop fighting with each other and gang up on the functional people to get them to actually do their job in supporting us for a change!

8. You manage to turn around code fixes in no time flat

I am always amazed by how developers know their systems and can turn around fixes to problems in no time. In fact if the truth be told, it makes me a little jealous sometimes.

9. You guys have bigger teams than we do

Often on projects or in BAU, developers outnumber the Basis guys – there are times, I am the only Basis guy assigned to a project. You have development colleagues you can bounce ideas off if you get stuck. I like the way you do stick together and support each other – a sweeping generalization I know but it is more often true than false.

From reading above, it looks like I find developers do more things wrong than right – nothing could be further from the truth, but the things above are mostly things that drive me nuts because we could be doing things much better. I look forward to reading your comments whether they be positive or negative.

To report this post you need to login first.

42 Comments

You must be Logged on to comment or reply to a post.

  1. DJ Adams

    Nice post Chris 🙂

    Looking forward to the discussion that will ensue.

    One initial comment from me: “You manage to turn around code fixes in no time flat”. In some cases (by no means the majority), I’m wondering if it’s because the developer knows that the area of code was a bit dodgy and was extra-wary of it, so a quick zoom into the problem was possible, especially if the problem scenario triggered a new thought in their head on a recent change.

    dj

    (0) 
    1. Chris Kernaghan Post author

      DJ,

      Thanks for your help and advice on it, just took a long time to distil all the thoughts and feelings on it into something useful 🙂

      Agreed that I have seen a lot of developers whipping out a fix as a result of a negative test – and to be honest I am not going to punish that behaviour as long as transports and documentation are all in order and no-one takes the Michael.

      Chris

      (0) 
  2. Former Member

    Can I also add, i’ve just e-mailed you my transport request, can you have a look at it ASAP please.

    Within 5 minutes of sending said request.

    Anyway, Nice Blog !

    (0) 
    1. Chris Kernaghan Post author

      Jamesy

      I would put that one under Number 2 – trying to sneak your code in quickly 🙂

      Although everyone deserves a few strikes, as we make mistakes occasionally.

      Chris

      (0) 
  3. Steve Rumsby

    I’m going to have to digest this for a while, but one thing immediately jumped out at me. You said “my (basis) production system” and “your (developers) change process”. With respect it isn’t your production system, it is the business’ production system, and therefore it isn’t the developers’ change process, it is the business’ change process. Somebody in the business should be signing off change requests for production – that’s not the job of anyone in Basis. And if a developer can get the relevant person to sign off on a quick change, it isn’t the job of Basis to complain.

    Get the process agreed by everyone and owned by the relevant people, and then Basis and developers will have fewer fights over quick fixes.

    (0) 
    1. Chris Kernaghan Post author

      Steve,

      That is a valid point and to be honest one that is probably looks worse due to poor explanation than anything else.

      Of course I do not actually think the Production system belongs to me – although I would say that the stability of the system responsibility rests first and foremost with Basis. We get kicked 1st if the system is unstable or there is a problem.

      You are correct that it is the business’s change process and they should sign off the change, which they do as part of the CAB, but 3 of the 5 points mentioned above do not require business intervention and I would argue that it would actually slow down the discussions if they were involved. On top of that Steve, it is my experience that Basis and development actually do more of the process engineering of the change management process.

      Also I am now of the opinion that Basis should not even be involved in the implementation of the change, a previous project used a 3rd Party tool sitting on top of CTS+. One of the change team, someone who had never used SAP, was responsible for putting the changes through the landscape. It worked very well and enables Basis to have proper separation of duties as it were. So that fits well with your last point.

      I look forward to the rest of your rebuttal 🙂

      (0) 
  4. Former Member

    Great to get the debate started!

    The Developers of which you speak, and I am one of them, are the people who bring alive the ‘dreams and promises’ of the Functional Consultants. Once the configuration is in place and assuming that some non-standard coding is required, the ‘grunt’ work is performed by the created/changed code.

    The Developer also has to be able to talk to and understand the Functional Consultant as well be able to converse with the more Technical minded colleagues in a language that everyone feels comfortable with.

    Some ‘blue sky’  thinking needs a little translating, before it becomes anything like code!

    Regarding Transports, I have been working with a client that separates Basis from Transports, so that the activity of performing the Transports becomes purely mechanical. Transport AND Task release is performed by people who have no Basis knowledge – a Configration Management (CM) team – they purely perform a script based on a supplied and verified Transport list with sequence and blocks pre-defined and if anything goes wrong (RC=8+) they stop and seek advice.

    This takes the purely physical Task away from Basis, and moves the responsibility for creation of the Transport list, with dependancies and blocks, back to the Development team – ensuring that the building blocks are presented in the correct order.

    This approach relieves the busy Basis guys of an un-enviable ‘Admin’ type task, but more importantly focusses the Developers to understand Object dependancies, Transport sequence, et al.

    Up to now is has worked well with rigour being put into the Control aspects BEFORE the Transports are done and nobody now ever sends a ‘history’ list, but sometimes the inablity of the Developer to Release a Task can cause a dependancy block!

    There is more to digest and discuss but I will leave it there for now.

    Regards, Iain

    (0) 
    1. Chris Kernaghan Post author

      Iain,

      That is a great thing to hear, through the change in 1 area, your customer has changed the behaviours that I talk about in another 2 points.

      Thanks for the feedback and looking forward to more

      (0) 
    2. Steve Rumsby

      I really like the idea of moving the physical transport task away from the Basis team. My only hesitation is that there are sometimes timing issues – please don’t put that enormous change request in at the same time as job X is running, or similar. Otherwise, as I said above, once the business approves it, there’s no reason not to transport it. And Basis do generate occasional change requests, so having them unable to transport them would good segregation of duties practice.

      And I do like the idea that whoever does the transports doesn’t understand the process technically, forcing developers to pay more attention and take more care 🙂 .

      Steve.

      (0) 
  5. Steve Rumsby

    We are currently looking at our development landscape, and at moving away from a simple 3-system landscape to something more like your examples above. Virtualisation technology, for both server and storage, has meant that creating new systems is relatively cheap and if project based sandpit/dev/qa systems make sense it is no longer too hard to create them. I’m a big fan of merging everything together as soon as possible, though. I’d rather push everything into a single QA system than have project developments continue on longer in blissful ignorance of conflicts.

    As for “I don’t have the resources to give you your own dev system”, I find that’s becoming less and less of a problem. Virtual servers are cheap, and cloning storage means you don’t need 100s of GB for a new system. Yes, you do still need some resource, but nowhere near as much as you used to. If you aren’t virtualised yet, then I agree it is a big problem.

    This last point can also address the test data issue. If you can reasonably cheaply clone production, you can refresh dev from prod frequently enough to keep developers happy. The only other solution I see to this is automated data slicing tools like TDMS. Asking functional people to produce enough test data isn’t realistic, in my opinion.

    (0) 
    1. Chris Kernaghan Post author

      Steve,

      Sometimes it is not a case of physical/virtual resources in terms of infrastructure, but also a person to even do the copy. Also as you say the tools which can easily create additional clones are available, I have not seen much if any penetration into the market. Mostly because a lot of companies do not calculate the indirect costs of what a system refresh is as they do them infrequently for many reasons- they do however look at the direct costs of the tools and then have a conniption.

      I think there is a wider piece of work to define what data the developers need, versus what the business will give them access to, versus what the technical team can provide quickly and effectively. This differs wildly between companies and so I hate people talking about best practices in this scenario trying to enforce a way of working (consultants do this all the time)- but good practice or “what works for you at this point in your system life cycle”

      (0) 
    2. Martin English

      If you can reasonably cheaply clone production, you can refresh dev from prod frequently enough to keep developers happy. The only other solution I see to this is automated data slicing tools like TDMS. Asking functional people to produce enough test data isn’t realistic, in my opinion.

      I have no problem with the developers having their own sandboxes, but if you want to get integration as early as possible, consider refreshing them regularly with the latest version of all production code, config, security, etc, as well as a subset of appropriately scrambled and subsetted (via TDMS) data, with any changes not backed up (via the local CTS) considered unwanted changes (and of course, since anything you do more than once is automated, then you reduce the impact on the support teams) 🙂 .

      As for physical resources, this touches on another issue I’ll address in my full reply, but…. my first thought is why can’t you have these developer systems on a local (infrastructure) cloud with access via VPN ? A serious question here; From a developers viewpoint, how close does the development platform have to be to the production environment (say, Linux / MaxDB for Dev and AIX / DB2 for prod) ?

      I would suggest that using virtualisation / cloud hosting to enable ‘gold copies’ of both development and testing environments would be a very useful tool for Developers …

      HTH

      PS a reply / support / rebuttal (depending on what mood I’m in) to the whole post is on it’s way as well !!

      (0) 
    3. Former Member

      Hi Steve, Interestingly enough thats what we do now. Helps Payroll, devs & testers. no end.

      What we do is clone production.

      In dev, create a clean client Lets call it 299. Then do a Client copy from the clone of production to development,

      Becasue you have a stable copy of production, with no users, you can run a parallel client copy from development, and you can leave it running quite happily for a week or so without time pressures.

      You don’t have to lock down the production client for inconsistancies etc.

      Everyone is generally happy with these sort of timescales due to the quality of data then available in development, no prossures on business users to create testing data.

      Its a massive win really.

      We also use the same principal in QA.

      So in effect we have five copies of production data.

      1 Production system.

      2 Clone of production for testing production on and upgrades.

      3 A “Secret” copy of production for Client copies.

      4 A Full copy of the data in a Development client

      5 Full copy of production in QA.

      Its a real win.

      We have the full copy of prod sepeate from dev and QA so we can test activation of SAP Business functions and the like, seperate from dev.

      As once you activate business fucntions, you can’t undo them.

      To be honest we used the exact landscape chris was describing in our ECC5 – ECC6 EHP4 Upgrade.

      Dual Dev and QA systems.

      Then we rebuilt the STMS configuration and discarded the old landscape.

      (0) 
      1. Steve Rumsby

        If you are clever, you can streamline it a bit more than that, at least for ABAP instances. With the right storage technology, and if you don’t do the system copy the “official” way, the process can be very quick – hours, not days!

        I’ve never had to do it for Java instances, but I don’t see why it would be significantly more complicated.

        (0) 
        1. Former Member

          Hi Steve,

          Our Java instances are what stop us using the full system copy process, and makes the landscape complicated……

          One SAP Portal can only point to one R3 Client. We have two QA Portals. One  Normal QA Portal and one Training Portal.

          These point to respective clients on the QA backend. 300 and 360 respectively.

          Our 360 Client is refreshed from 350 which is a “Training Master” client.

          Our QA Portal points to client 300, which has certain test data in it.  Don’t forget we run full HCM which is a pain, as HR like to use fully annomous data for Training and Demo purposes. Which you can respect.

          Thus doing client copies is easier rather than system copies, for certain areas.

          IMHO were too nice to our Dev and Business teams……………

          (0) 
  6. Tom Van Doorslaer

    You asked for it 😆

    I believe I’m at the other end of the spectrum. In the first place, I’m a developer, trying to understand and collaborate with BC.

    I’m relieved to tell you that I’ve always kept a very clear excel list of all transports, the order in which they have to go, which transports must go at the same time and whether there are any manual actions after a transport. So I suppose I won’t get punished in the most sever way. The only reason I do this, is because once, I was responsible to get these transports safely into the next system. So I understand the finer subtleties and problems that pop up when you do not strictly adhere to the right transport sequence. And No, the transport History is NOT the best source.

    Contrary, I’m amazed at how sometimes, the BC guys completely ignore my list and simply release all transports at once. *JawDrop* When I consequently get a panic call from a customer that nothing works any more in production, and that I was responsible over the transport sequence list, so it’s my fault… hmpf! Not a happy day.

    But mostly the collaboration works quite well. I even tend to take initiative and talk to the admins before doing something funky and new (like creating indexes, activating services in the SICF, importing notes, using the reverse proxies, or importing my own transports from hobby projects…)

    When it comes to locking objects, I agree that you have to release tasks sooner, even if it were to simply structure your task set better, but if someone else wants to work on your object, they can actually. It will just create a new task in the same transport request for their user. They can’t put it in a separate transport as long as you don’t release your own transport. That’s where object inheritance comes into play and saves the day. 🙂

    On the topic of integrating code: that’s where architecture comes into play. I prefer to have developers that want to work alone. Those are the guys that will structure their code in such a way, that they can work on their own part without impacting the others. This usually delivers nice, clean objects with little or no dependencies. As an architect, the job becomes a lot easier to then lay down rules to the developers in terms of interfaces, so that you can link everything together at the end.

    Why do I not prefer developers that actually like working together? Well, as long as there are only two developers, they will agree on a good way to integrate their work, but if you have more, they all have their own idea on how the integration should happen, and you end up with spaghetti. Or worse, they all work together on the same code, ending up with a procedural mayhem. Fortunately, most developers are loners when it comes to their work.

    I already mentioned architecture, code dependencies and object inheritance. These are actually vital factors when multiple projects are running on the same system. You have to clearly structure code and responsibilities. Modern development techniques offer plenty of possibilities to limit the impact of your code changes, to only your specific project. Even if the objects are shared over multiple projects. Again, Architecture plays a vital role here, but I don’t often see architects really diving down to that level of detail. I think architecture also plays a vital role in connecting the developers with the basis guys. At least that’s what I try to do.

    The test data bit brought a smile to my face. Yes, we should pummel the functional guys 🙂

    Oh PS: to understand your language better: “What’s a BAU?” (Basic and Ugly, Brilliantly Automated Upgrades,…)

    (0) 
  7. Robert Shiels

    Nice article. I was interested by your comments at the start about Java/Webdynpro which I think would be good to expand on.

    Moving ABAP code with STMS, or 3rd party tools like Transport Express, is well understood by the old timer basis guys, like me. The issues with merging code streams never go away, but we can dig around and work it out fairly easily, even if sometimes it takes a while.

    However, Portal transports, Portal themes, Java – these objects don’t have the same well known tools, and it’s much more difficult to manage them, especially when CTS+ isn’t setup.

    How do the Basis Administrator and the developers keep track of what has gone where? Spreadsheets seems the only way.

    (0) 
  8. Tobias Hofmann

    I’d already be very happy if basis guys stop interpreting basis as in having a very basic understanding and everything that is more than just going through a checklist is ignored.

    And I like it when basis tasks like creating a WS or JCo connection are not done by basis, claiming this is developer task and NOT giving then the needed permissions to the developer to create the needed configuration; especially when later basis thinks they are smart and change the parameters (JCo without a default language? Oh no, let’s change this …).

    A goal needs to be to eliminate pure basis tasks completely so everybody can focus on what really matters instead of blaming each other.

    (0) 
    1. Former Member

      Tobias,

      They key is ‘One Team with One Vision’.

      I have been stopped in my Tracks disrupted by Authorisations |Colleagues whereby I have to justify having SE80, SE37 and SE11 as a Developer, as well as not being allowed SM59 (view access) because it is seen as a purely Basis Function – its not easy sometimes!

      (0) 
  9. Former Member

    Chris,

    Some more thoughts/questions…..

    1, Dual Track landscape.

    I am on a client site using exactly this set up however several challenges present themselves.

    The Project Track Transports stop at QAS until the Project is happy/allowed to move into the BAU/LIVE track. This also means that any on-going BAU/LIVE Track changes have to be managed and ‘re-keyed’ into the Project Track to ensure consistency when the code reaches Pre-Prod.

    The process of ‘re-keying’ strikes me as an anathema from several perspectives given that we normally use Transports as a one off to capture any changes.

    • It is prone to potential ‘human error’
    • It increases Testing effort
    • It is an additional, sometimes hidden, cost to the Development.

    Are you aware of any more robust and elegant way to incorporate the ongoing BAU changes into the Project Track?

    The BAU/Live changes must be there for any Testing to be valid and for the Integrity of the BAU/Live track code

    2, Tools and Control are key on all the Transports moves in Projects that I have been involved with – a requirement for ANY change HAS to be recorded and a Change Control Form (CCF) raised, which states:

    • Who?
    • Why?
    • What?
    • Where?
    • When?
    • Risk (of doing the change and NOT doing the change)?
    • ‘Backout’ approach, should there be any issues
    • Cost/Resource assignment can be captured.

    Once the Change is discussed and Approved by the relevant system owners, the CCF number is used to drive the Change and link all appropriate Documentation/Transport names, etc.

    There are many way to do this and to avoid ‘Admin’ overload:

    • Project – there is usually a very High level CCF which will cover ALL Transports and Manual Config Changes
    • BAU/Live – each individual issue encountered could generate individual CCF numbers.

    3, Development of Utilities

    Each Project is different but sometimes pre-dependant processes are not yet available.

    E.g. a Web Service is used to fill in a Form which is then routed through to an Adobe Form in SAP. This Form is evaluated to see if the data is suitable – this may be for many purposes such as field validation, stopping data duplication, Fraud audit etc. Once the evaluation has happened then the data is sent to the SAP Database for Business Partner creation with concomitant acknowledgements

    With luck and reasonable planning, the Web Service data Interface will be developed in parallel to the Form creation and BP creation process but if these do not happen then Utilities have to be written to enable the Unit Testing of code.

    Whilst this may not affect the landscape, the Utilities may only have to progress as far as the Integration ready Client to enable Unit testing of the individual components – it is another hidden cost in time and Development and potentail confusion point as these Utilities have a life span of the Project only, so will not be specced for the Live Track.

    Is there a better way of handling this?

    4, Performance and Volume testing (PVT)

    In my experience the BC guys become heavily involved in this in the Pre-Prod environment – lots of system copies to reset data and management overhead to ensure that the System will work for the client at the Volumes+ that they expect.

    Do you have any comment ot this area?

    Regards,

    Iain

    (0) 
    1. Former Member

      Hi Iain

      With Regards to dual landscapes.

      One possible solution to move transports across is relatively easy…….

      Make sure the transports are released.

      Make sure the cofiles and datafiles are present under /usr/sap/trans.

      Copy the cofiles and data files for the transport manually from one dev system to the other.

      Add the transports manually into the new dev queue. Bit Fiddly, but it can be done. And you need to ignore all warnings about component versions.

      You can also manually rename the transport files so no confusion can occur over transport numbers, depending upon the new systems configuration.

      This stuff is relatively easy, and will stop all keying in. However you need basis people who know what they are doing……

      and helpful.

      James

      (0) 
      1. Former Member

        James,

        Thanks for the ideas.

        Unfortunately, the Project Track will have lots of extra code in it that could be overwritten by the BAU/Live Transports.

        We take a copy of BAU/Live as late as possible and at a point in time to serve as a Baseline onto which we add the Project code; however the BAU/Live track does not contain any Project code until they meet in Pre-Prod.

        Unless I am mistaken, simply Transporting the BAU/Live changes on an ad hoc basis has the potential to overwrite new Project code – hence the re-key effort.

        I will go and talk to my friendly Basis guys to see if is a possibility and to see what we could do to make the above process run using a manual script/Work Instruction, so that the CM team can do it easily.

        Regards,

        Iain

        (0) 
        1. Former Member

          Hi Iain

          Thats correct the potential does exis, but if the code in either track is clean, then it could be a good method.

          The real question is whats in the transports, what does it affect and is the time saved by moving the transport files quicker than re-keying the code / config.

          Your developers need to know what they are doing as do the project teams.

          If the code from tranports in either system has the potential to create problems, then the same issue could arrise at Pre-Prod stage,

          It could be a option for lots of configuration for example. In everything in SAP, if it could save people a vast amount of time, give it a go and see what happens.

          It can be done. Its just a question of rick / reward etc.

          James

          (0) 
  10. Susan Keohan

    Chris,

    You definitely channeled my #workhusband when you wrote #7. 

    He’s a Basis guy, I am a developer.  I whine to him about the data in Dev.  Our Dev system has been around for 10 years, and to say that the data is stale is an understatement. 

    And yet, between the two of us, even if we were given a month, we could not produce ‘good data’ in our Dev system. 

    I have no answers, I am just glad someone else whined about this.

    Thanks,

    Sue

    (0) 
  11. Former Member

    As a security person, what I really get “nuts” about is the sometimes lack of implementing a “Authorization Check” into the codes written for the new program or variant etc etc etc……….

    Or worse….you need to trawl through the code yourself and find any funny embedded calls……

    Ah the joys of having a principle to maintain SU24 and authorisations in roles to the fullest possible! 😐

    (0) 
    1. Steve Rumsby

      Good point! Most people only think about what they want to be able to do, not what they shouldn’t be able to do. Security is usually very much an afterthought.

      Those of us who think of security first are a rare breed 🙂 . Some might say a strange breed 😉 .

      Steve.

      (0) 
      1. Former Member

        “A Few Good Men” Steve…”A Few Good Men” (and Women)……

        I think i’m generally seen as the crazy strange and rare breed when I go asking the developers if they have considered Auth checks…..but have to admit that I have worked with some good developers in the past who have actually actively sought for my advise for developing auth checks into the program….always felt chuffed by such moments 😛

        (0) 
  12. Brian O'Neill

    How do you know whether or not I am a rockstar? You don’t know what I do off hours! The Nerve! We’ll see what happens when I develop the new facebook!

    I have noticed a lot of people saying that “these transports must move in this order”. We don’t move our own transports to production, but in non-production environments I move all my transports at once. I select all my transports and let SAP sort out what to do and that has been working fine for me.

    We have a dual landscape like you described and it’s been working pretty well. We put the large projects in the project system and require them to check out code from the BAU system and make their changes.

    We had issues though where a few projects did half the work in one system and half in the other. The project transports were loaded in the BAU landscape after moving to production, which overwrote the BAU transports. Good processes and making sure all the developers and consultants understand it is a must.

    (0) 
    1. Tom Van Doorslaer

      Whatch out with moving them all at once. I have seen situations where there were multiple transport that contained the same object(s) with some interdependencies making the whole thing go tits up.

      It’s not unimaginable that you end up with version 1 in production instead of version 3 like it was supposed to be.

      (0) 
  13. Martin English

    Hi Chris,

    I remember reading somewhere that large business systems last an average of 7 years. Even if you allow that a large scale implementation may last 12 – 18 months, then the major implementation phase is still only 20% of the total life of the system. It probably sounds like a leading question, but what’s more important – the development phase or the ongoing operations ? My experience suggests that the main (perhaps only ?) motivation for the developers is the set of functional requirements  – i.e. are reports accurate and complete, do transactions run to completion, with appropriate error reporting etc… whereas the successful implementation and ongoing operation of a development or system also needs to concern itself with the non functional or operational requirements (FYA, see http://scaledagileframework.com/wp-content/uploads/2011/09/Types-of-NFRs.png from http://scaledagileframework.com/nonfunctional-requirements/ for a good working list). The real answer is (of course), it depends… Remember that both ongoing SAP and ongoing Custom maintenance are part of the post-implementation phase – changes are required and need to be integrated speedily and accurately – BTW, did you see what I did there 🙂

    I came to SAP from an MVS Systems Programming background, so sometimes there’s a part of me that sees SAP as the application, and the whole SAP ecosystem as one of developers (despite being a very limited programmer, myself). While your view of SAP #DevOps is Developers and BASIS, don’t forget to include the teams that are closer to the infrastructure (in whatever form it is these days…); the DBA, Systems and Network people etc. This is more than good manners !! In many cases, your beloved SAP system is a pimple on the whatever of the total infrastructure, and the people who are responsible for that infrastructure don’t necessarily know anything about SAP. For example, in my early days, I saw an experienced Oracle DBA (with no SAP experience) go through a production instance changing various parameters, table parameters etc, because they weren’t best practice (for Oracle). One of the side effects was the database grew by over 40% (because all the unused tables now had minimum extent sizes). Another, very common these days, example is where SAP instances are running virtualised with the memory and CPU resources being way over allocated. For what it is worth, I am a fan of virtualisation and using the vendor tools to speedily snapshot databases and to create / copy / delete multiple development and testing environments, but the physical resources need to be in place to support the virtual systems.

    The key point here is that I’m sure I have done stuff that, when viewed through the eyes of a Developer or Infrastructure person, has looked equally stupid.

    This lack of knowledge is almost always driven by a lack of communication. Coming from an SI background, I remember projects where you never met anyone outside your grouping (Developer / BASIS / Infrastructure); These were extreme examples, but the TL;DR version is You don’t know what you don’t know, and the only way to fix that is to go and ask. And you get other people to do this by modeling the behavior you want them to follow.

    However, it’s not just finding out what everyone else is doing; To read SCN, you would believe that all ABAP developers know about OO programming, Unit Testing etc, and that all specialist (i.e. SAP CRM, PI/PO, Workflow) developers know what they are doing. Unfortunately, this is not so. And despite our angst (woe is me, BASIS has to fix up someone else’s mess again…), it is very rarely due to stupidity or carelessness. It is almost always a lack of knowledge of the concepts, or a lack of understanding of the importance of these concepts; i.e. it’s a Training and Hiring problem. If an organisation is going into a new (for them) area of Development, they need to get external help filling the roles; By definition, there is not enough in house knowledge of the new area to recruit the right people.

    This leads me to the elephant in the room. Excuse the language, but I do get pissed off with the managers who have made promises or want to ‘big-note’ themselves with the board…. These are the people who have an innate belief in magic (i.e. if I insist loud enough or long enough, I can change reality), insisting that you speed up the restore or backup, or use their corporate authority to override test protocols (i.e. move it from development to production now). For some reason, the worst seem to be the ones who got some sort of SAP certification in 199x and have since been promoted as far away from the keyboard as possible (I could name names, but I want to work in this country again !!!).

    Management has become a synonym for control, and this stifles the development life cycle. It is no accident that the same sort of thinking that bought us Agile Development is being brought to bear on the entire development life cycle. Simple things like reducing communication barriers, leading to streamlined processes. In turn, these become faster yet more predictable while retaining control and traceability. A simple example is that the Business doesn’t need two or more senior managers to sign off on a test, but they do need to see that the system has passed the tests that they were involved in developing. And automating the tests to ensure that the correct tests are done, in the correct order, every time.

     

    In short, investing time and effort up front, leading to less time being spent overall, for a better result.

    HTH

    (0) 
    1. Tobias Hofmann

      “investing time and effort up front”

      Congrats, solution and problem identified in the same sentence. Problem here is, that the manager that is doing the upfront investment (test cases, automation) won’t be the same as the one that will benefit from it: promotions, transfers, etc and enough egoism make sure that this effort won’t be done until it is too late.

      I like QA teams that are really large, as their test scripts have to be run manually and need some sort of interpretation (what ever that means)

      (0) 
  14. Graham Robinson

    Hi Chris,

    I love this post, and have been considering for some time how best to respond. I don’t disagree with anything you are saying and in fact welcome a more open and confrontational approach to clear the air between developers and technical teams.

    So in that spirit let me suggest what Basis/Technical people do well and what drives me nuts!

    Basis bods are great at keeping things working. This is a major achievement. In an enterprise environment where the business quite literally relies upon the enterprise systems to function it is vital that they just work and keep working. 24×7 and 365 days per year if necessary. This means clear, well thought out, and complete change management procedures need to be put in place, documented, enforced, audited and adhered to. Reality is that most system breakages can be traced back to “someone changed something” so it makes perfect sense to minimise the risk that this will happen through systemised testing, quality control, etc.

    Basis bods are terrible at change. The attributes that make you guys so good at keeping things working make you resistant to changing anything – especially in a hurry. I understand why we don’t rush changes into production. I don’t understand why we can’t make quick changes to support the development system – and especially sandbox environments. Why does it take 3 months to get a SAPRouter configuration change so I can connect to a system? Why does it take 4 weeks to get a simple DNS change implemented? Why does it take 6 weeks to setup inbound email to a system? All these are real examples I have experienced in the past few months. Why would the same rigorous procedures that protect the production system have any place in a sandbox environment? It becomes really frustrating when the red tape that governs the production systems is wielded as justification for glacial change in non-core systems.

    Cheers

    Graham Robbo

    (0) 
    1. Chris Kernaghan Post author

      Graham,

      This is exactly the comment I was looking for amongst all the others – and you are right – Basis (and Operations people) are interested in keeping things running, and often they do not like change because that makes it difficult to maintain system stability.

      As a Basis project person, I have to embrace change, but I also recognise something which a lot of people don’t within projects

      Often hosting SIs use mutualised teams across multiple clients, which effectively means that the support/operations people are working across multiple and heterogeneous environments. The theory goes that the best way to ensure stability (and least amount of service credits given away) is to use process, documentation and e-mail to make sure everyone does what they are supposed to do. It often fails because

      • These people enforce the changes across the whole landscape in the semi-mistaken belief that the systems must be representative of Production at all times. reducing effectiveness on non-production systems
      • There is so much documentation on the minutiae of the task, but nothing on the context of the task – which means outside factors are not considered by anyone (diffusion of responsibility strikes again)
      • We (ERP) threw people at problems and not technology, and we have people centric heavy processes – when the web world has people light, technology implemented solutions which are repeatable, auditable, consistent.

      The purpose of philosophies like DevOps, Lean and Agile is to encourage failure and experimentation in the goal of embracing change. I have a customer who is in the middle of a renewal cycle – they are asking what innovation we have brought to their contract. The interesting thing is that in the event of any issues, these people jump up and down so hard there are holes in the floor – this actively discourages change as it infects the whole process with fear – especially with non-technical people who manage the account.

      This situation can improve by coming closer together and working in cross-disciplined teams with greater transparency, with less reliance upon documentation and greater respect – then we can move faster and on the edge of turbulent flow of change :-).

      (0) 
  15. Former Member

    Chris,

    Other questions I’d like to raise are:

    ‘What are the core activities that BASIS need to do and be Responsible for?’

    ‘What ‘non-core’ activities can be delegated from Basis (given good governance)?’

    ‘How does the internal Customer/Supplier relationship work best between BASIS and Developers?’

    We all have internal Customer/Supplier relationships in our everyday work.

    These functions and roles are not always very well understood or defined, but give us an expectation of what we all need to do for others and what we expect other to do for ourselves.

    E.g.

    1, A Developer cannot work on any code until a Developer Key is made available to them.

    This simple Task has a Customer – the Developer – and a Supplier – the guardian of the system, who is usually with the BASIS Team.

    2, The Functional Consultants ‘dream’ of how a system will work may need code that is written by a Developer – The Functional Consultant is the Customer in this case and the Developer is the Supplier

    All very simple stuff, but without setting out Role expectations and some Service Level Agreements (SLAs,) the processes can become frustrated.

    Another example:

    In the past I have been on a client site where access to:

    • OSS,
    • Developer Keys
    • Object Keys

    all have to go through BASIS.

    This can be a blocker to progress – not through unwillingness to do the Task but because it is more ‘Admin’ than ‘Action’ – and there may be something more pressing happening.

    This is where the principle of delegation of Responsibility for the ‘Admin’ can come in.

    As you know, a client site has a site specific key that is used for Installation Licence purposes and enable BASIS to obtain software downloads etc.

    There is also a need to be able perform other activities based around the Installation Key and these can be delegated to another less skilled team, who can perform the process of obtaining Developer keys, etc. to a pre-written script. This other Team should only act on a suitably authorized request and should stop and seek advice if they hit any issues.

    Note that this does not take the final Responsibility way from Basis for the integrity of the systems but allows them to get on with more core activities, and goes some way to ensure that clear lines for the expectation of who performs an ‘Action’ and ‘When by’ can be drawn.

    I realize that I may be assuming that there may be more than one person available to do the activities described, however it does NOT mean that the principle of ‘who’ is responsible for ‘what’ cannot be defined clearly and so avoid any frustrations and confusions – the ideas and principles should be adapted to the size of the relevant team(s).

    Regards,

    Iain

    (0) 
  16. Kevin Grove

    Chris:

    It seems that you accomplished your goal of creating a robust discussion, though it seems that only a few developers and functional people have added their input — I’d like to see more!

    The issue of transport – really change – management always seems to be one of the more contentious aspects of an SAP project or production support environment. If I had a dollar [insert your favourite currency here] for every time I was told, “Of course there will be no impact to the production users”; I would have retired years ago. 😀

    I would like to add just a few other observations:

    1. After reading twice and then a search, I find no direct mention of the business user. Of course, it may be “assumed” that the goal of a smooth change management process is to minimize the impact to the business. My advice is to start the design of change management processes from the end user viewpoint and keep focused on that viewpoint and less on the IT-centric viewpoint.
    2. Testing is way too undervalued in most SAP teams, particular in those that are in the “BAU”, or run and maintain mode. Usually very few, if any, people are allocated specifically to testing in today’s era of tight budgets. Test plans and test cases require nearly continuous care and maintenance. If left to gather dust for 6 months, tests (especially automated tests) that “used to work” become nearly useless.
    3. It seems to me, that combining these comments, one could produce some very good recommendations for change management best practices for which SAP does not really provide much guidance, In particular the client copy or TDMS methods for bringing more test data to the DEV environment comes to mind.
    4. One of the best tools I have used was a home-grown program that checked a given list of transports. Among the things it checked was for predecessor transports not in PRD, authorization objects not assigned to new programs or to specific tables. The type of things that can cause problems once moved into production. It was not fool proof and the advice could be ignored, but it saved disruptions on many occasions.

    Finally, I have encountered more than a few characters in my working with SAP teams over the years. One developer/ functional person produced one of my favorite quotes:

    “I don’t often test my code, but when I do, I test it in production.” Sadly for most of the business users of the SAP systems around the world, that statement may seem all too true. Perhaps more discussions like this will move us closer to providing our business customers a more stable working environment.

    Regards,

    Kevin

    (0) 
    1. Chris Kernaghan Post author

      Kevin,

      That blog post is on it’s way – it will be a selection of Good Practices, not best practices (you know my feelings on that subject), as taken from this comment thread.

      You are correct we do not deal with business users, something both you and Steve Rumsby have picked up on, my only explanation to that is that I have to start somewhere.

      Also I agree, we need more developers on this thread, or on a separate thread roasting my a$$

      Chris

      (0) 
      1. Graham Robinson

        It’s hard to roast this post when I agree with most of it. 😉

        Dev guys will always want more. They need to consider alternatives first, then better justify their requests later.

        Basis guys always push back. They need to become better enablers and less rule enforcers.

        (0) 
  17. Former Member

    Chris,

    Good debates, the key theme for me (a developer) is that there needs to be clear communication between all stakeholders – developer, business, basis, change board, testers. SAP puts solution manager (and sometimes CRM) as the key tool to request,manage and approve changes. (caveat I haven’t actually used SolMan 7.1 yet) – IMO SolMan falls well short in user experience for all stakeholders.

    I say put some lipstick on the pig (see earlier caveat). 😛 Have a look at how other agile tools are being baked into the development and change process of competitor products (Outsystems, Mendix, JIRA) and you will see devops in action.

    Regards, Warren

    (0) 

Leave a Reply