Improving Development Productivity by leveraging Cloud Computing

Introduction

Throughout the last years I have been involved in numerous initiatives aiming to measure and improve total cost of development (TCD) in Composite Application projects. What we found out is that one (if not THE) main influencing factor on TCD is development productivity, which – in a nutshell – refers to the time it takes a trained professional to execute a given development task. Leaving the proper skill set aside (which is why I explicitly said trained professional) it all comes down to how efficiently the development platform – in our case the SAP NetWeaver Composition Environment (NWCE) – supports the developer in said tasks.

Influencing Factors on Development Productivity

Yet, before we dig deeper let us have a quick look at all of the influencing factors there are:

Especially in an SAP-dominated environment people tend to point out how efficiently a developer can code within the ABAP workbench and as such are quick to compare any other development platform to it. I certainly do admit that the ABAP workbench has set a very high standard in regards to development productivity and one of the main reasons for this is probably the fact that there is one central application server (for both design and run-time) shared by all developers. In the Java space we have separated the design-time environment (NetWeaver Developer Studio – NWDS) and the run-time server (NWCE)  and usually every developer has his/her own IDE and server. In the many discussions I had with my fellow colleagues this central development environment is one of the first differentiators being called out and put down as an advantage for ABAP. Having experienced both I tend to agree with them… (on that particular point!)

Yet, it’s not my intention to turn this blog post into a heated discussion about platforms. I believe that the individual application servers that comprise SAP NetWeaver have been designed to complement each other and usually the characteristics of the applications being developed with them are essentially different. The decision on which platform is best suited for a particular project is typically made at the very beginning. The reasoning to apply when facing this decision isn’t the scope of this blog either as it has been covered elsewhere already.

So, now that we know what this blog is not about – why don’t we get to the point now? Well said! Here we go… as the title hinted, in today’s blog post I’d like to talk about how-to improve development productivity in distributed development projects by leveraging new infrastructure options – namely the cloud.

Development Productivity in Distributed Development Projects (DDPs)

In 2010 I’ve been on a distributed development team, which developed an XL Composite Application on top of SAP NetWeaver Composition Environment. At peak times we had around 20 developers in 4 different locations (2 EMEA and 2 APJ) and a close window from a timeline perspective as always. One of the first things that was brought to my attention was that in the past cycle the team has been struggling in regards to productivity – mainly caused by long deployment cycles (due to the size of the application.) When looking at the magic project management triangle we realized that the only two constraints we could somewhat bend were scope and budget as the timeline was fixed. So, while our product management team focused on applying reasonable cuts to the scope (aiming to reduce it to the famous 80% mark), the development team faced the challenge of finding ways to improve development efficiency within the given budget. As we had done our very best to improve in all other influencing factors (see list  above) we realized that infrastructure was the only domain left for improvements.

Classic Development Infrastructure Setup – On Premise (Phase 1)

In the past it has been quite common to have each member of a development team setup their own workspace consisting of one machine running the IDE (NWDS) and another to run their local server on (as the hardware requirements of these two components do not really allow to run both on a standard developer PC.) Typically, these developer machines rank on the lower spectrum of the required hardware requirements in regards to CPU power and memory. In addition to these local servers maintained by the developers themselves, there usually are several central servers for test, QM and demo purposes, which are running on real servers.

In such a scenario the following inefficiencies exist:

Some of these bullet points can be compensated by using advanced configurations scripts or by leveraging images, yet inefficient hardware and slow remote communication still remain a burden. As mentioned above, the one thing that troubled us the most were the long deployment cycles we faced due to the insufficient performance of the local servers. Consequently we were looking for ways to improve this aspect and the most promising solution was to try out cloud computing.

Cloud Servers (Phase 2)

Fortunately we were given the opportunity to participate as pilot in an internal cloud initiative that allowed us to host NCWE instances on the Amazon Elastic Compute Cloud (Amazon EC2). Given the hardware power (2 CPUs and 8GB RAM) the deployment times in all locations were reduced significantly plus using a unified image basically reduced our configuration efforts to zero. There was still room for improvements however… in this 2nd phase we only moved the “local” development servers to the cloud while the IDEs still remained on premise. As such, when remote(!) debugging the application we experienced some lag – even with debugging proxies turned on.

In this early development phase we focused on the development of our domain model, the service interfaces and the persistence layer. However, we knew that sooner or later we would need to connect to the involved backend systems residing within the corporate intranet (on premise). For this purpose we would need to establish a bi-directional communication between the corporate landscape and the cloud – a concept known as Virtual Private Cloud (VPC.)

Virtual Private Cloud (Phase 3)

By moving to the virtual private cloud we were able to also put the IDE to the cloud instances as the bi-directional communication allowed us to connect to all our backend systems and link the NWDS in the cloud with our central NetWeaver Development Infrastructure (NWDI) within the corporate landscape. As a consequence we no longer had to do remote debugging anymore, as both the IDE and the server were running on the same machine. The performance of the cloud instance we used was more than capable of doing so and as such allowed us to do “desktop-less development” – all our developers needed was a Remote Desktop connection to their assigned cloud instance. Even better, this scenario enables really convenient hand-overs between developers residing in different locations as they would only need to communicate the IP number and logon credentials in order to have a colleague continue the development started by someone else, which brings a whole new meaning to the term “working around the clock/globe.”

The next logical step – Custom VPC

While the infrastructure setup of using a VPC has eliminated or at least greatly reduced the inefficiencies of the classic on premise setup used in the old days, we are still not there yet. Especially in scenarios where the development of the Composite is done in-house and then shipped to the customer (which is still the standard process in my organisation to minimize travelling costs) we still face the issue of lacking direct connection to the backend systems residing in the customer landscape during the development phase. So typically these systems need to be replicated or mimicked during the development phase and there’s the need of an explicit implementation project once the development is complete. Furthermore, such a scenario does not really cater to agile development nor is it optimal in projects where the Composite may be developed in parallel to a Service Enablement/Provisioning project happening at the customer site. Again, ABAPers can only frown upon such problems as they have gotten used to simple remote connections to customers landscapes via transaction STFK.

But there’s hope… as the adoption of cloud computing increases, customers and especially customer IT departments will understand the value-add of virtual private clouds and may setup their own. Once this has started  to happen on a bigger scale we will have reached a whole new level of convenience and ease-of-use in regards to some of the end-to-end challenges we have been facing in composite application development.

Special thanks to Emil, Konstantin, Dimitar and Boris and their teams for all the support!