Why DTR ? Why not Perforce or ClearCase...?

[former_member183805]

The other day, I struck up an interesting conversation with a colleague who had some strong views about SCM tools, especially concerning SAP's new Design Time Repository ( DTR ).

"I've taken a look at DTR" she said, "and I still cannot figure out why we had to build our own Version Control tool when there are so many mature tools in the market. I mean, take Perforce or ClearCase - they've been around for years and have proven their strength in supporting large scale distributed projects. So why reinvent this technology ?"

"There are a number of sound reasons behind that" I replied.

"I'd really like to hear them out" she said. "To me, DTR looks similar to Perforce - you have the same file/folder checkout-checkin model, 'changelists' in Perforce are called 'activities' in DTR, and Perforce Branches map to DTR workspaces. Old wine in a new bottle, isn't it ?"

"Not really. I'll explain the differences in a moment, but regarding the abstractions like 'activities' and 'workspaces' - these are from the DeltaV standard which DTR implements and not- "

"What's DeltaV ?" she interrupted.

"DeltaV is the versioning extension to the WebDAV standard. WebDAV defines a standard way to access and manage files over the web. DeltaV adds versioning support to this and defines the protocol to perform tasks related to configuration management and version control. Implementing an SCM tool based on this standard - and not on some proprietary technology the way Perforce does - is in line with SAP's strategy of creating 'open-ended' systems."

"That sounds okay, but it doesn't justify the development of a full blown SCM tool to meet this end !"

"True. The reasons for that are different. You work with Perforce from Bangalore, don't you ? How fast is it ?" I asked.

"Irritatingly slow. But that's understandable considering the fact that the Perforce server is in Walldorf and we access it over the WAN." She seemed to have accepted her fate calmly.

"Understandable yes, but acceptable ? I think not. Perforce has only one model for distributed development - a central server ( what they call Depot ) to which clients connect from different places, even across continents. But with DTR, you can also replicate sources between repositories in different locations. So if you were to move your sources to DTR, there will be a DTR Server configured in Bangalore to which you will connect, and the sources you modify will be replicated to the DTR Server in, for example, Walldorf, and also in the reverse direction. "

"How does that work ?" she asked. "I mean, what is it in DTR which allows it to replicate sources in a way Perforce cannot ?"

"Perforce has no concept of propagation of sources in a distributed landscape. You can integrate your changes from one Perforce Branch to another one in the same depot, not across depots. In DTR, the analogous abstraction 'workspace' is not tied to a physical repository - so integration of changes is possible between workspaces in different repositories. If the target workspace is in another repository - in another location - then you only need to export your changes from the source repository, import it into the target repository, and then integrate the changes into the target workspace."

"Much better, I agree. But what about ClearCase - they've been supporting replication for years!" There was a twinkle in her eyes; she probably thought she'd got me here.

"True. ClearCase has what they call a VOB - Versioned Object Base - which is a source tree in the repository. So in a way it is something like a Branch in Perforce or a Workspace in DTR. And a VOB can be distributed over a WAN - this is done by having multiple replicas of a VOB, and allowing users in different locations to work with their replica of the VOB. Then, at regular intervals, you can synchronize sources between these different replicas and thus get the changes made in other locations."

"Just what we need, isn't it ?!" she asked, still excited.

"For distributed development, perhaps yes." I said. "But SAP has other requirements which ClearCase does not fully meet. The most important one is support for delivery of sources to customers. As you know, SAP delivers its application sources to customers, and these sources can be modified by them. So we need an SCM tool which supports the functionality of propagation of sources to customers in such a way that it satisfies some important attributes."

"What attributes ?"

"Firstly the tool must support upgrading software delivered earlier such that customer modifications are not overwritten. This is possible only if the tool is able to automatically detect conflict situations caused by propagation in any direction. In DTR, this is possible since it maintains a global version history - a version history of a resource that spans multiple repositories. So if you deliver a specific version of a file to a customer, additional meta-data related to its version history is also transported so that at the customer site the full version history can be calculated. This is used to decide whether or not a conflict must be reported when the next version from SAP is received."

"Perforce has a linear revision history containing revisions in a single branch, right ?" she asked.

"Correct. Perforce uses what it calls 'Inter file branching', which means that when you integrate a file to a new branch, a new file is created. In DTR, this only results in another version of the same file."

"But Perforce does maintain the branching information, doesn't it ?"

"It does, but that is just a pointer from the source to the target branch. It does not keep track of subsequent integrates that were performed in both directions, and hence cannot indicate conflict situations correctly. For instance, back integration from a 'correction' branch to the 'development' branch always brings up a conflict, even though the development branch version was never modified."

"Oh ! So that's why we always had to do double-maintenance with our projects in Perforce ! If we could integrate in the reverse direction it could save us a lot of manual overhead!!"

"Yes." I smiled. "With DTR you can simply integrate your bug fixes of older releases into the new releases. A conflict is reported only if the file was modified in both workspaces."

"What are the other attributes related to this propagation functionality ?" She seemed more curious now.

"Another important attribute is sequence independent propagation. In a distributed landscape where there could be more than one provider of software - like SAP, and some partners of SAP - and more than one receiver of software - like SAP Partners and customers of SAP - the SCM tool cannot depend on the sequence of propagation. Current SCM tools which support propagation work on the assumption that if change A was done first, followed by change B, then these are transported and received in the same order."

"Hmmm..."

I was not too sure if she understood the importance of this functionality. I continued anyway.

"Also, the granularity of propagation should be flexible. With DTR, changes are grouped in 'activities', and these activities - which contain versions from a set of files and folders - are units of propagation. Additionally, we can select arbitrary versions into another unit of propation called a 'Propagation List', which can contain any number of versions of distinct resources." I looked at her to see if she had understood.

"So normally all changes recorded in activities are propagated, but we havethe additional flexibility of choosing precisely what versions we want to propagate - is that it ?" she asked.

"Thats right."

"All this is fine, but if I remember correctly, ClearCase also has branches and supports merging across branches - right ?" She really was persistent.

"In ClearCase, a branch is something you create for every individual resource - there is no "global" branch which applies to all resources in a given code-line. This causes problems with administration and control."

"How ? I did not understand. "

"If you remember, a VOB in ClearCase is like a source tree. But each resource - a file or folder - in this source tree can have complicated version graphs, based on the branches created for each resource. (So in this sense a VOB is like a virtual repository). Now let us say that a bugfix for release X needs a modification of file F1, F2 and F3. This must be performed in a separate branch for each file. To keep things under control, ClearCase recommends to use the same branch name while creating branches for changes that belong together. If this is followed, then a ClearCase View can be configured such that elements on this branch are visible in the context of this view. Since this decision - of naming the branches for individual files - is left to the user ( or the person who configures the View ), it is not comfortable in organizations where centralized and automated control is important."

"What is a ClearCase View ?" she asked. I had used this term without explanation, and she probably liked to have a clear idea of everything.

"A View in ClearCase is like a 'Virtual Workspace', which provides isolation through access to a specific set of versions of distinct resources in a VOB. Think of it as a private storage - it allows different users to work independently on the same VOB."

"So when I create a View, can I select any version of a resource in a VOB ?"

"Yes you can. So if you want to work on a bugfix of a release X, then you can create a View by selecting the latest version from the Bugfix branch of resources which must be modified for this fix, plus the latest version from the Main branch of all other resources."

"That conveys a lot of flexibility, doesn't it ?"

"It does, but that turns out to be a disadvantage in our case. We want to have control over what versions our customers can see. If we do not have this control, then customers could define their own views to the repository and ignore versions shipped by SAP, which could frequently lead to inconsistencies in the state of the software."

"So how is this situation avoided when one uses DTR ?"

"DTR is well integrated into the Java Development Infrastructure of the Netweaver platform, and it is the Change Management Service - CMS in short - that creates and configures workspaces in a landscape. Once the workspaces - the codelines like Development, Correction, Consolidation - are created, things proceed in a controlled manner. So users cannot create workspaces on their own - they work on workspaces created and administered centrally. This applies to customer landscapes as well. So when an upgrade from SAP arrives, the workspaces - codelines, or branches - into which the changes must be integrated are known, and so we do not have any surprises after the upgrade."

"So if the upgrade contains a new version of a file which the customer has modified, a conflict is reported, right ?" she asked.

"That is right. Since DTR maintains a global version history, it can calculate - based on predecessor successor relationships in the version graph - whether or not a conflict must be reported. And of course, DTR provides tools to view the differences between the two conflicting versions, and merge them - either automatically or manually."

"That is fine. But assuming the customer resolves his conflict in his 'development' system - or workspace - then when the upgrade is imported into his 'consolidation' workspace, the same conflict will be reported there also, right ? So he has to resolve it there again ?"

"No. When you resolve a conflict in DTR, the merge arrow is persisted, and propagated along with the merged version that was created. So if you resolve a conflict in the 'development' workspace, and then integrate this change into the 'consolidation' workspace, the conflict in that target workspace will be automatically resolved due to the merge arrow that was also part of the integrated change."

"Sounds cool ! I get the impression that apart from covering the functionality offered by major SCM vendors, DTR addresses rather well the specific issues that arise in the development areas within the landscape of SAP and it's customers." She seemed convinced, finally.

"Quite correct. You've probably noticed that we follow the 'main-line' model for development, which means changes in all branches are periodically integrated back to the main code-line ( or branch ). This was adopted based on our experience - it was found that most of the development within SAP follows this pattern. Now if you look at the strategy adopted by Perforce - Inter File Branching - it becomes clear that they do not expect integrates back to the main line of development ( and hence their model does not support such integrates well enough ). That strategy is good when you create a branch to work on a related but independent area, like, for example, a branch for platform dependent development. Since platform dependent code of a product is placed into a different branch, it is not expected to be integrated back to the main branch ( which contains platform independent code ). Since most of the development in SAP does not follow this pattern, this strategy of Perforce is ill-suited for us."

"And the features needed to support delivery of software to customers - they seem to have been mostly ignored by leading SCM vendors !" She seemed a bit surprised that other vendors had not ventured in this direction.

"Again, this is specific to our requirement. Typical customers of SCM tools only want to use the tool to support their in-house development, and this need is met to varying degrees by different tools. As we saw, some tools like ClearCase are quite generic and flexible, but when it comes to our special requirements - like the support for regular delivery of sources to customers - they fall short of the functionality we need."

"It is clear now." she said, with a smile.

"Well, DTR is the result of years of experience with complex, distributed development landscapes. We learnt a lot from the predecessor of DTR - a versioning system called Mobile Application Repository ( MAR ) which is used by the Mobile Application Studio to store and version the CRM Mobile applications shipped by SAP. The MAR already contains a few of the features I've talked about, and having gone through the experience of building and productively using a large scale versioning system across a few releases has given us valuable insight into what works and what does not. Its a pity that SAP does not sell technology - otherwise DTR could have given nightmares to all major SCM vendors."

"Considering all that you have said, I'm beginning to think so too. But I still have one complaint - why such a name like 'Design Time Repository' ? It sounds so ... so technical, doesn't it ?!"

"Er..um..well..."