What will Gateway really deliver?
Over on Bluefin Solutions, DJ Adams wrote a great blog on the history of SAP Integration, finishing up with a discussion on Project Gateway. You can find his blog here.
I have to admit that I’m a little skeptical of what Gateway will bring. Admittedly, I’ve never touched it, and have only seen it in action at Tech Ed, but the buzzwords that I keep hearing are Atom, OData and RESTful. This sounds great, but surely the questions to ask are why Atom, why OData and RESTful – are you sure?
Firstly, Atom as a representation has proved itself to be valuable outside its original domain of news articles and blogs. Almost everyone has an Atom library, and its metadata allows clients to add searching and sorting etc without needing to understanding the source of the data. That said, should every list be a feed and each item an entry? Doesn’t this generic representation of the data confuse the original meaning. We work hard in our code bases to use intention revealing names derived from our Domains. Should we just accept that we will lose this in the Atom representation of our data. Surely parsing XML or JSON is not a complicated task, so why should our only representation be Atom (In saying this I admit that I’m unaware of any other representations – other that Atom as JSON – being supported by Gateway at this time – can I custom code a simple XML representation of my resource – no idea – happy and hopeful to be proved wrong). I guess the crux of the point is this – Why use Atom when its not an appropriate representation for your resource, adding overhead when its not required. I’d like to have this type of choice with Gateway. If I just want a list, I want to be able to get just a list.
My other concern is the adoption of OData. OData, like GData, is an extension of Atom. While GData is used specifically for Google APIs, OData is being pushed out to the community by Microsoft as the Open Data Protocol. What do I get from using OData? I get a lot of stuff in a Microsoft namespace – and metadata that is based on the Entity Data Model (does anyone really use that???). Apparently if you are rolling your own OData messages, the EDM Metadata is optional, but if you plan on generating Proxies, some form of Metadata is needed. I’d assume that it’s the same Metadata that is driving the code generation tools shown at Tech Ed. Will this be a mandatory step in Gateway, or if I never plan on generating a proxy/iPhone app, can I just build a Atom representation without this additional metadata?
Assuming you want the additional metadata, in OData the easy way to build this Metadata is the EDM – has SAP built a similar tool in Gateway to build this? Are we looking at another repository to add to the growing set in our landscape? Are we modelling Resources in the REST sense, or Entities in the Domain sense – we may choose to represent our domains in different ways and expose them to the world, but the list of what is considered a Resource is much larger that what is considered an Entity. Can we model both in Gateway?
Atom schema extension through OData (and maybe SData if this is what SAP choose to call their protocol) need to be considered carefully. Extensions need to be done in such a way that they help specialised clients that understand the extension, but don’t break generic clients. My concern is that we’ll go down the specialised client path if the impacts of our extensions aren’t understood by those using the tools.
Finally, the OData protocol breaks REST in a number of ways – will Gateway be the same? Is it really RESTful, or is it just Atom (with extensions) over HTTP. The OData protocol doesn’t use the standard HTTP interface (MERGE is added via the HTTP header), its Batching capability breaks Caching, and the OData query language does some nasty things with URIs rather than just using Forms and Links – which may mean ongoing maintenance as resource locations change – it also means Clients may be building URIs which is not really a good thing…
And I know this is for backward compatibility, but I have to admit that when I saw the screen scraper being used to activate a transaction for use via Gateway I was a little uncomfortable. I’m hoping that while all this is going on, SAP is also addressing the application architecture such that there is no business logic in the GUI, so tools like Gateway and WebServices can operate at the Application/Services layer rather than at the GUI layer. This will help both those SOA and ROA people to access the underlying SAP data and functionality.
I guess we’ll need to wait and see – right now Gateway is just a word with much promise – hopefully it will deliver some of this – I’m just hoping that its done right, and not already compromised.
wow, you know how to make a good first impression! Great blog and lots of good questions. Let me tyr to point this to someone from the Gateway team to get some (official) answers...
Cheers,
Matthias
"I saw the screen scraper being used to activate a transaction for use via Gateway I was a little uncomfortable." +1
The idea of having a separate ABAP instance for the service consumption layer takes me back to the early BSP days.
Cheers
JSP
I normally try not to tread in these waters but your comment in regards to Screen Scraping set the hook.
You mention that you would hope that SAP would solve the issue of removing business logic from within the GUIs. You do realize that over the past 10+ years thousands of BAPIs have been created with just this in mind. On top of this in the past few years thousands of Enterprise Services have been created making SOA architecture possible with SAP.
That said, how many thousands of older SAP installations are productive out there that don't have those BAPIs/Enteprise Services? More likely the number is in the tens of thousands. Gateway and more specifically Gateway's Screen Scraping services will make it possible to easily expose the business data on those older systems. I have to think there is a huge market for this personally knowning the complexity/difficulty of what it takes to do this (what ever happened to those RFC libraries anyway?).
But that's just my opinion. 🙂
Great point, and I don't doubt the motivation for such an approach.
Given the large steps forward that SAP have made API enabling the Business Suite to provide Service Enablement, could some of these APIs be back ported to reduce the need for a screen scraping approach?
Depending on your feelings about REST, you may or may not agree that Transactions have a place in a RESTful architecture. By providing the Screen Scraping Service, I believe Gateway is making this decision for you. I guess, depending on your release, you could just choose not to use it 🙂
There are possibly other compromises that have been made in getting Gateway to market, such as whether Verb tunnelling has been used, and how stateless it really is.
Right now I'm just going on the brochure - can't wait to get Gateway installed and answer some of these questions. Personally I'm very interested to see how well aligned Gateway is with REST and the use of the Web as an Application Platform.
AT
great to see you venting this in public! In your investigations did you find out if JSON is going to be allowed as SAP-ized OData/Atom transport?
From the tiny bit of info on the SAP site - http://en.sap.info/gateway-apps-mobile-rest-duet-enterprise/43463/3 I could only assume that outbound from Gateway is going to be exclusively XML and not JSON.
Also interesting to see the SAP site defining REST using the 4 accepted verbs and then referencing OData... Perhaps SAP won't be polluting the waters with MERGE.
Great blog, I look forward to seeing any responses from SAP.
One of the Tech Ed Sessions (CD108) has some good info, and does refer to both XML and JSON representations. That said, the proof will be once we get our hands on Gateway and can see for ourselves 🙂
I have to admit I hadn't read the article you linked to. I wonder if its time to let them know that SOAP hasn't been called Simple Object Access Protocol since V1.2 of the SOAP standard. The W3C recommended this change in June 2003!!!
http://en.wikipedia.org/wiki/SOAP
🙂
AT
Dear Alister,<br/><br/>In your blog you raised numerous questions about what Gateway will really deliver. <br/> <br/> <br/>As the presenter of SAP TechEd Session CD 108 and a member of the Project Gateway solution management team, I would like to take the chance to clarify some misconceptions that seem to stem from our presentation back then. Please note that Project Gateway is still under development and features and functions are thus subject to change.<br/> <br/>Let me start with your concerns regarding the usage of the Atom Publishing Protocol. When using OData, there is no reason to fear that names derived from your domains will not be revealed. If intention-revealing names are used for OData properties they become part of the Atom content. As a result they are not lost.<br/> <br/>There are numerous representations that can be chosen for a resource. One of them is Atom, and it is required as it is the one being used by OData. The additional overhead for choosing an Atom representation is not that big. The simplest XML representation is available within the OData m:properties element within the Atom content. <br/> <br/>If we look at the following example:<br/> <br/><atom:entry><br/> <atom:content type="application/xml"><br/> <m:properties><br/> <d:carrid>AA</d:carrid> <br/> <d:connid>0017</d:connid> <br/> <d:countryFrom>US</d:countryFrom> <br/> <d:cityFrom>NEW YORK</d:cityFrom> <br/> <d:airportFrom>NEW</d:airportFrom> <br/> <d:countryTo>US</d:countryTo> <br/> <d:cityTo>SAN FRANCISCO</d:cityTo> <br/> <d:airportTo>SAN</d:airportTo> <br/> <d:flightTime>361</d:flightTime> <br/> <d:departureTime>11:00:00</d:departureTime> <br/> <d:arrivalTime>14:01:00</d:arrivalTime> <br/> </m:properties><br/> </atom:content><br/> <atom:id>http://example.com:50022/sap/iw/rest/sflight/FlightService.svc/FlightSchedule(AA,0017)</atom:id> <br/> <atom:link href="FlightSchedules(carrid=AA,connid=0017)" rel="self" type="application/atomxml" /> <br/> <atom:link href="FlightSchedules(carrid=AA,connid=0017)/Flights" rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Flights" type="application/atomxml;type:Feed" /> <br/> <atom:title>Flight Schedule AA0017</atom:title> <br/> <atom:updated>2011-02-04T16:59:12Z</atom:updated><br/> <atom:author>SAP AG</atom:updated><br/></atom:entry><br/> <br/>The mandatory atom wrapper is relatively small; a feed is just a list of entries and adds almost no overhead.<br/> <br/>One of our main aims is that services provided by Project Gateway can be easily consumed by non-SAP developers. Atom is a well known standard that is easy to parse. Moreover, there are several OData SDKs available that allow for easy consumption of OData based services. Using a proprietary data format would make this more difficult and would leave the heavy lifting to parse the service results to the developer of the consumer application.<br/> <br/>What do you get from using OData? You get a language for describing an entity relationship model including a type system that is well suited for publishing enterprise data. <br/> <br/>Although you are not forced to use the metadata on the client side, you have to provide a data model for the data you want to publish using Gateway. So you would lose valuable information by not using this metadata. <br/> <br/>We are also planning to provide modeling tool support in Gateway, but a programmatic approach is also planned to be available for ABAP hardcore developers who do not like modeling; they can express the data model via an API :-).<br/> <br/>The generators whose prototypes were shown at SAP TechEd will create such a model plus connectivity to the backend based on existing content, without the need or possibility 🙂 to code anything.<br/> <br/>With regard to your question on whether you have to model in a REST sense or in the domain sense, I am not sure whether I got your point. But you will primarily model structured data as entity types, and unstructured data as media resources; both are addressable as resources in the REST sense. <br/> <br/>SAP plans to extend OData in strict conformance to the OData specification, so generic OData clients will ignore, or should I say gracefully omit, the SAP annotations when generating proxy code. But they will not break. <br/> <br/>OData extends the HTTP interface with the MERGE verb for partial updates and will replace MERGE with the HTTP verb PATCH as soon as this is standardized; see http://www.odata.org/blog/2010/10/18/support-for-http-patch for details. So it is not really breaking the standard to use MERGE but rather anticipating what is necessary. <br/><br/>The same is true for usage of the batching capability if done in a correct way. We intend to use BATCH only for transactional updates (LUWs), so this will not break caching. While performing a bunch of GET requests in a row would not break caching, it would be bad programming style to wrap these in a batch request, since no caching could be leveraged. We cannot, however, prevent anybody doing so. For update or creates, use of the batch request makes perfect sense. If someone tries to book a hotel, flight and car for a journey, either all or no booking would have to take place.<br/> <br/>The OData query language does not do nasty things to URI's nor are clients required to build them. <br/>Finding collection resources via a service document (and finding a service document via a catalog service) is completely hypermedia driven. If a client wants to express a query, it can construct the query part of the request URI by following well-defined rules (independent of resource locations). In fact we really like the quite powerful query language of OData :-).<br/> <br/>The "screen scraping" tool that was shown in our TechEd session seems to have caused some misconceptions. While the code generators for custom transactions (aka "screen scraping") and the Business Object Repository we presented at TechEd represent the bottom-up content generation approach, there will also be a top-down content creation approach that will be a programmatic one. As my colleague Jeff Gebo pointed out, both generators are meant to serve the needs of those customers who have build a considerable amount of business logic in either custom Dynpro screens or custom BAPIs. The generators will serve the need of a request such as the following: "I want to see the following transaction on an iPhone or Web Page". If the use case cannot be implemented leveraging BAPIs or Dynpros, a model-based approach (either tool-based or programmatic) will cover more complex business scenarios.<br/><br/>
I really appreciate you responding to my questions. I think as a community it's important to know that SAP is listening and willing to participate in our discussions.
I'd like to respond to your answers - unfortunately this isn't the best forum for this - I really wish I'd gotten more time at Tech Ed to have had this discussion with you!
Firstly, the choice of Atom/OData. Sure, if were using OData, then Atom it is. My point on whether this is the appropriate format for Representing all of our resources still stands. Although the attributes in the properties node retain their names, they are still just a collection of properties. From your example I can guess that the entry is probably a flight, but because the parent node is called "entry" its meaning can be ambiguous. If content extension rather than metadata extension was used, the type could then become application/vnd.flight+xml and the content could be a well defined XML message describing a Flight. This also gains an advantage over application/xml in that application/xml is not hypermedia aware, while our content extension can be. Of course, a proliferation of new content types is probably as bad as excessive metadata extension. Personnally I like the approach taken by Webber, Parastatidis and Robinson who lean towards metadata extensions for application agnostic extensions, and choose content extension for domain specific information.
I know that this may sound pedantic, but will Gateway still require all the microsoft namespaces? Your example uses a linked relationship - http://schemas.microsoft.com/ado/2007/08/dataservices/related/Flights - but the namespace really confuses the semantic meaning. It is my resource afterall, so surely I can use a namespace with some real symantic meaning? The rel attribute is the only way to determine what the href link actually does, so it needs to be good 🙂
You mention that OData provides a type system for defining enterprise data. Will this include the ability to use the GDT/CDT model that was part of the Service Enablement standardisation? Unfortunately I don't have access to your FlightService.srvc$metadata file so I'm curious if all the types for the Flight Service come from the Microsoft EDM? If that's the case, why are these types better than any others?
Part of the defintion of REST is the use of a Uniform Interface. I can't accept that using MERGE doesn't violate this constraint. That said, I fully support the idea for OData to include PATCH once it becomes part of the HTTP Protocol. Of course, then there is the issue of supporting MERGE for backwards compatability, which means it will probably always be there...
I'm still not convinced on the batching approach. From what I understand from the OData documentation I can call $Batch against my service, and I pass the commands/payloads as a multipart MIME. Does this mean that I can't batch accross multiple services? So in the scenario you described below I would need access to the Hotel, Flight and Car resources through the single OData service to allow my batch request for a Hotel, Flight and Car to be processed? This is starting to sound more SOA than ROA. I don't agree that performing 3 requests sequentually is bad programming style. Each 2xx response tells me that it's OK to proceed, and the hypermedia in the response from say my Hotel bookling can offer the URI needed to make the Car booking. Of course, if I decide to cancel, I may need each resource to have a compensating command - maybe as simple as a DELETE to the created resource, or I could use a transaction resource (like a shopping cart) to wrap the LUW. Similarly, I could define a resource represening say an itinerary or trip which captures my Car/Hotel/Flight requirements and sends these in one go - similar to the Batch idea, but more Resource focused. Caching will be effected differently by the approach taken, and so depending on the use case this may drive the approach used.
Finally, the query language. I see these as template URIs (transparent URIs), They will be fairly simple to use, and tell us something about the structure of our resources. But as you said, "if a client wants to express a query, it can construct the query part of the request URI". The trouble with this is that the Client is now coupled to the implementation. It also means that the client may build URIs that are invalid, and on the other side, the Server is forever bound to honour any template it has published. Once again, I like Ian Robinson's approach of only recommending Template URIs as entry points as these are much more stable over time. This is why I really like the decoupling that can be achieved with Forms and Links 🙂
Once again, I really appreciate you responding to my blog, and hopefully we can meet up sometime to discuss - this is such an interesting topic. I feel kind of bad responding like this as I don't want this to become one of those he said/she said discussions. What I think is important is to find out as much about Gateway as we can, as our customers are asking questions that we're struggling to answer.
Thanks Again,
Al
Al - Great first blog - I had to comment so I can see the responses to this blog (I wish weblogs had a notification option...)
Cheers,
Matt
Hope to see much more of such great blogs from you.
Cheers,
Rohit