Skip to Content

Improving Development Productivity by leveraging Cloud Computing

Introduction

Throughout the last years I have been involved in numerous initiatives aiming to measure and improve total cost of development (TCD) in Composite Application projects. What we found out is that one (if not THE) main influencing factor on TCD is development productivity, which – in a nutshell – refers to the time it takes a trained professional to execute a given development task. Leaving the proper skill set aside (which is why I explicitly said trained professional) it all comes down to how efficiently the development platform – in our case the SAP NetWeaver Composition Environment (NWCE) – supports the developer in said tasks.

Influencing Factors on Development Productivity

Yet, before we dig deeper let us have a quick look at all of the influencing factors there are:

  • Application Design (Componentization, patterns,…)
  • Development Method (Agile methodologies, continuous refactoring, …)
  • Reuse (Technical components, frameworks, …)
  • Tooling support
  • Infrastructure


Especially in an SAP-dominated environment people tend to point out how efficiently a developer can code within the ABAP workbench and as such are quick to compare any other development platform to it. I certainly do admit that the ABAP workbench has set a very high standard in regards to development productivity and one of the main reasons for this is probably the fact that there is one central application server (for both design and run-time) shared by all developers. In the Java space we have separated the design-time environment (NetWeaver Developer Studio – NWDS) and the run-time server (NWCE)  and usually every developer has his/her own IDE and server. In the many discussions I had with my fellow colleagues this central development environment is one of the first differentiators being called out and put down as an advantage for ABAP. Having experienced both I tend to agree with them… (on that particular point!)

Yet, it’s not my intention to turn this blog post into a heated discussion about platforms. I believe that the individual application servers that comprise SAP NetWeaver have been designed to complement each other and usually the characteristics of the applications being developed with them are essentially different. The decision on which platform is best suited for a particular project is typically made at the very beginning. The reasoning to apply when facing this decision isn’t the scope of this blog either as it has been covered elsewhere already.

So, now that we know what this blog is not about – why don’t we get to the point now? Well said! Here we go… as the title hinted, in today’s blog post I’d like to talk about how-to improve development productivity in distributed development projects by leveraging new infrastructure options – namely the cloud.

Development Productivity in Distributed Development Projects (DDPs)

In 2010 I’ve been on a distributed development team, which developed an XL Composite Application on top of SAP NetWeaver Composition Environment. At peak times we had around 20 developers in 4 different locations (2 EMEA and 2 APJ) and a close window from a timeline perspective as always. One of the first things that was brought to my attention was that in the past cycle the team has been struggling in regards to productivity – mainly caused by long deployment cycles (due to the size of the application.) When looking at the magic project management triangle we realized that the only two constraints we could somewhat bend were scope and budget as the timeline was fixed. So, while our product management team focused on applying reasonable cuts to the scope (aiming to reduce it to the famous 80% mark), the development team faced the challenge of finding ways to improve development efficiency within the given budget. As we had done our very best to improve in all other influencing factors (see list  above) we realized that infrastructure was the only domain left for improvements.

Classic Development Infrastructure Setup – On Premise (Phase 1)

In the past it has been quite common to have each member of a development team setup their own workspace consisting of one machine running the IDE (NWDS) and another to run their local server on (as the hardware requirements of these two components do not really allow to run both on a standard developer PC.) Typically, these developer machines rank on the lower spectrum of the required hardware requirements in regards to CPU power and memory. In addition to these local servers maintained by the developers themselves, there usually are several central servers for test, QM and demo purposes, which are running on real servers.

In such a scenario the following inefficiencies exist:

  • n-times the setup and configuration efforts to maintain local servers
  • not all developers may have a dedicated local server (restricted budgets)
  • remote connections to the central development servers and backend system may be slow
  • slow deployment times due to limited hardware specifications of local servers
  • n-times the synchronization efforts due to individual code basis

Some of these bullet points can be compensated by using advanced configurations scripts or by leveraging images, yet inefficient hardware and slow remote communication still remain a burden. As mentioned above, the one thing that troubled us the most were the long deployment cycles we faced due to the insufficient performance of the local servers. Consequently we were looking for ways to improve this aspect and the most promising solution was to try out cloud computing.

Cloud Servers (Phase 2)

Fortunately we were given the opportunity to participate as pilot in an internal cloud initiative that allowed us to host NCWE instances on the Amazon Elastic Compute Cloud (Amazon EC2). Given the hardware power (2 CPUs and 8GB RAM) the deployment times in all locations were reduced significantly plus using a unified image basically reduced our configuration efforts to zero. There was still room for improvements however… in this 2nd phase we only moved the “local” development servers to the cloud while the IDEs still remained on premise. As such, when remote(!) debugging the application we experienced some lag – even with debugging proxies turned on.

In this early development phase we focused on the development of our domain model, the service interfaces and the persistence layer. However, we knew that sooner or later we would need to connect to the involved backend systems residing within the corporate intranet (on premise). For this purpose we would need to establish a bi-directional communication between the corporate landscape and the cloud – a concept known as Virtual Private Cloud (VPC.)

Vishal Sikka at TechEd 2010: “… I thought a Virtual Private Cloud is a cloud that only rains on me! ;)”

Virtual Private Cloud (Phase 3)

By moving to the virtual private cloud we were able to also put the IDE to the cloud instances as the bi-directional communication allowed us to connect to all our backend systems and link the NWDS in the cloud with our central NetWeaver Development Infrastructure (NWDI) within the corporate landscape. As a consequence we no longer had to do remote debugging anymore, as both the IDE and the server were running on the same machine. The performance of the cloud instance we used was more than capable of doing so and as such allowed us to do “desktop-less development” – all our developers needed was a Remote Desktop connection to their assigned cloud instance. Even better, this scenario enables really convenient hand-overs between developers residing in different locations as they would only need to communicate the IP number and logon credentials in order to have a colleague continue the development started by someone else, which brings a whole new meaning to the term “working around the clock/globe.”

The next logical step – Custom VPC

While the infrastructure setup of using a VPC has eliminated or at least greatly reduced the inefficiencies of the classic on premise setup used in the old days, we are still not there yet. Especially in scenarios where the development of the Composite is done in-house and then shipped to the customer (which is still the standard process in my organisation to minimize travelling costs) we still face the issue of lacking direct connection to the backend systems residing in the customer landscape during the development phase. So typically these systems need to be replicated or mimicked during the development phase and there’s the need of an explicit implementation project once the development is complete. Furthermore, such a scenario does not really cater to agile development nor is it optimal in projects where the Composite may be developed in parallel to a Service Enablement/Provisioning project happening at the customer site. Again, ABAPers can only frown upon such problems as they have gotten used to simple remote connections to customers landscapes via transaction STFK.

But there’s hope… as the adoption of cloud computing increases, customers and especially customer IT departments will understand the value-add of virtual private clouds and may setup their own. Once this has started  to happen on a bigger scale we will have reached a whole new level of convenience and ease-of-use in regards to some of the end-to-end challenges we have been facing in composite application development.

Development Productivity Improvements through Cloud Computing


Special thanks to Emil, Konstantin, Dimitar and Boris and their teams for all the support!

/
Development Productivity Improvements through Cloud Computing
11 Comments
You must be Logged on to comment or reply to a post.
  • as the overhead in development of the Netweaver stack in Java at least is significant.

    I would however like to throw into the mix that though your blog talks about a really nice approach to deal with the challenges you list, the really fundamental question is: Why do these challenges for development exist in the first place within the SAP stack? It doesn’t need to be as heavy and complex as it is.

    Perhaps a combination of a VPC and dealing with the underlying performance and set-up issues in the stack would give you a great dev environment instead of a good one with just the VPC.

    But really like the idea you present here…will be interesting to take this approach under consideration in future projects as an alternative to the traditional development models, but as you stated, it does require a buy in from a much wider group than just the development team as we need to work together with the in-house infrastructure and security teams to name a few and they might not be quite as open minded towards the new approaches.

    So the challenges here might be one of culture and education as much as it is purely technical, but as you stated at the end, this might just be a matter of time before it starts to work out in your favor.

    Cheers

    Kalle

    • Hi Kalle,

      thanks for your thoughts.

      First off, I’d like to emphasize that the project I talked about here has been about extremes. As I have been developing composite applications since 2004 I find this particular one to be the biggest and most complex – the opposite of simple light-weight scenario-driven consumption composites. When building such a large application in a distributed manner you have to squeeze out every “little bit” you can get. 😉

      I’m not sure what you refer to as the overhead in particular though. I would say that any so called “overhead” is typically added for a reason, right? 🙂

      Having worked on all releases of SAP NetWeaver Composition Environment I have seen continuous improvements in the tooling. Developing composites based on timeless software principles that are flexible to adjust to changes in your IT landscape is a complex topic. Distributed transactions, compensation and roll-backs, forward error-handling, backend abstraction, extensibility… WS* standards, J2EE standards.  I find the NWCE application server to be a great environment to develop enterprise-scaled applications and so far I found little what I would call “overhead.”

      So, as already said – in this project we had a fix end date and we needed every advantage we could possibly get. As such, the blog was more to point out the development infrastructure improvements to be gained by cloud scenarios in contrast to on-premise workstations with mixed hardware specs.

      And yes, it’s just a matter of time me thinks.

      Until that day… I will keep “evangelizing” about SOA and Composites as long as my record backs up my statements 😉

      • Hi,
        I understand that your main point was to do with the concept of using the cloud to your benefit on this type of a project and I agree as stated that it is an interesting way to resolve the issues (and seems to have worked for you).

        Even more I would like to agree on your point that there are scenarios where overhead created is of benefit (to certain types of projects).

        However, at the same time, I could not resist commenting on the fact that for generic Java dev. on the SAP stack (which might have very different requirements from composite applications as am no expert in that area) the performance and overhead of the dev environment, etc, when compared to say a more generic Java dev stack based on a mix of open source and commercial products, is far from optimal as the set-up and development overheads are quite significant. To be fair, most big players in the market suffer from very similar issues due to catering for the large projects instead of the smaller projects which only have a half a dozen of devs at most and very limited budgets.

        But as you stated, the size and complexity of your project warranted you to look for a solution that would allow you to deliver on time on the technology that was available to you.

        By no means was I commenting on your choice to use cloud computing to resolve this issue and I assume that a few teams have as a result of this blog at least looked into that as an option.

        Last point to your post: Could you quantify how much you managed to speed up the development through the use of the cloud (in %)?
        Was there a cost involved with the cloud or did the time saved give the project a reduced spending as well?

        Important factors when trying to get buy-in for this 😉

        Cheers

        Kalle

        • I really appreciate your comments and you bring up lots of valid considerations again. Thanks for sharing with us!

          Your last two question do not come as a surprise either. Good ones!

          Unfortunately this project did not really allow us to do a scientific benchmark to track the exact imrpovements in percentages. Furthermore, there was a magnitude of influencing factors such as generic network issues and various other influences on benchmarking. Yet, the most reliable measure in this regards (to me) is the “developer gut feeling” and all locations reported back a significant improvement 🙂

          Concerning the budget. Very true. Please keep in mind that this project participated in a pilot program for cloud-based NWCE development workstations. The costs for server instances are publicly available by the cloud providers. It’s easy math to compare that against traditional hardware infrastructures. I’d also assume that prices will drop over time as the adoption of cloud computing increases.

          Ultimately, I think it all boils down to the convinience such a cloud-based infrastructure brings to the table and it keeps your development team happy – #priceless 😉

  • Hi Matthias,

    Thanks for this insight into real-life problem solving 🙂
    As much as I am with Kalle, and would rather solve the cause than the symptoms of development inefficiency, I understand that in complex SAP projects, the complexity if probably there for a reason. So no way to work with limited yet lightning-fast test containers instead of full NWCE, no way to switch to scripting environments where “deployment” means replacement of a source code file rather than creating a full archive, copying the archive and then extracting it on the server side. Remembering the time of Blue Ruby (built in ABAP and fairly complex), I am not sure that server-side development is always the answer. I clearly recall Daniel’s email saying “I am refactoring the entire VM, be prepared to not do any work for the next two or three days”…

    But actually, I have a question: You mention access to (customer) backend systems only in the last paragraph. How did you provide the required enterprise services to developers of your composite app in the various deployment scenarios? Did you deploy an SAP backend along? Or did you just provide mock-implementations? Or were there so little touch points with the backend systems that if was possible to develop and test significant portions of the composite application without ever having to call a web service?

    thanks, Juergen

    • Hi Jürgen,

      good points! I truly believe that picking the best tool for any given operation is vital in this regard. Given the constraints we faced and the complexity of the application we opted for NWCE. Having used the tool for years we felt it was the best option as it provided a well-known toolset where the entire team had plenty of experience to work with. I stated the tight deadline and given the nature of the app (= it’s composite character) and as such it was a simple choice. Given the fact that we successfully rolled this out we seemed to have made the right decision. 😉

      Being a long-time Java developer I got to know and work with a lot of other application servers, OpenSource frameworks and runtime environments. Lots of the innovation we saw in the Java enterprise space from these influences has been incorporated into EJB 3.0, which was a huge improvement to the old standard. Those who have been following my blog know that I also promote bringing in other techniques such as “Aspects (as in AOP)” to the table to separate some of the cross-cutting concerns(SoC.)

      One of the main misunderstandings I see is that people just see the one-time development costs and may not look at the big picture. Sure, you may be able to use a light-weight container or the like to build simple apps super-fast. Yet, when building enterprise-ready apps targeting complex landscapes you are well-advised to provide some abstraction to gain the flexibility needed to adapt the application over time (timeless software principles). This is when tightly build apps suddenly reveal long-tail issues. Been there, done that – swore “never again!” Build it right – right away.

      I think it certainly depends on the scenario and personally I have been mostly involved in XL applications and complex scenarios and while I call it Composite – it’s indeed far off the simple light-weight apps for which the tools have been designed for.

      So, the right tool for the right job. I’ll be following closely on what River may bring to the table when it comes to light-weight consumption scenarios for sure though 🙂

      About your questions: Well, in our case it was a phased approach. At first we focused on the domain model, interfaces, persistence layer and some fundamental framework pieces. No backend integration was required at this time. Once we needed to work on the integration functionality we were already in Phase 3 and the VPC allowed us to communicate with our backend systems.

      In this particular scenario we didn’t have to worry about custom specific backend systems and as such we could simply connect to existing systems (ERP, CRM, KM) we have available anyway. Yet, you mention the topic of mocks – makes my day. That’s one of my fav. topics – backend abstraction and extensibility. We typically provide one abstraction layer that encapsulates the tech communication and domain model transformations when talking to the backend systems. This allows us to exchange the active  implementation on the fly (at runtime) very similar to BAdIs. As such, it’s a proven technique to develop mocks that can be used for unit-testing and at the same time provides the flexibility needed when looking at ever-changing backend systems.

      Especially in regards to EJBs and WS, I believe that NWCE really rocks. I can create a skeleton implementation for any given WSDL with a simple wizard. I can create a WS consumer proxy in 2 minutes. As such mock objects are a great asset for such kind of development projects and until we have reached phase 4 they serve an important role. You can develop the app using mocks or standard implementations and during the implementation project at the customer site you can replace those with custom implementations unique to the customer landscape. I have done this many times and it works 🙂

      If you have any further questions, please do not hesitate to contact me. I covered a whole lot of these and related topics in my TechEd Session CD301. You may want to check it out…

      Best regards and many thanks for sharing your thoughts,
      Matthias

  • I liked this blog, because it is based on experience from an actual project – too often blogs about cloud-based development scenarios are purely hypothetical.

    D.

  • Sounds great.  I like the idea of leveraging the lower cost infrastructure and only paying for the cycles in Amazon’s cloud.  I’m working on how to move our clients’ SAP environments to private cloud.

    In this case, how are you backing up the daily work?  The value of the day’s work can be in $10Ks and $100Ks. Are there any concerns about privacy or security?  Does SAP have any licensing issues or are we free to run NWDI anywhere we’d like and in any quantity?

    As you move to private cloud which will solve some of these problems, you will begin to pay by the reservation, rather than just the consumption.  Does this hurt your economics?

    • Hi Chuck,

      first of all – sorry for the delay in response, but I’ve been out of office for a few days over the Easter holidays.

      You raise some good questions, yet unfortunately I cannot provide concrete answers at this time. (maybe at SAPPHIRE NOW we will have the official information.)

      From our experiences we had no issue with loosing work as we are syncing our sources against our central NWDI.

      Being on the development side I rely on our IT folks to make sure that the privacy and security aspects are taken care of according to our standards.

      I’m not sure I got your last statement correctly. Could you may rephrase it or elaborate a little more please?

      BR, Matthias