At TechEd 2017, I attended a presentation on DX113, presented by matt.borges . In particular, it was clear to me that there are quite SAP Mobile offline capabilities and limitations that users are not aware of and I’d like to highlight them here.
This blogs focuses on my experience implementing offline mobile , offline SDK evolution , best practices and improvements needed.
Design consideration plays very important role while designing offline apps , i have included more details to what was presented in TechEd while designing offline app.
Offline SDK Evolution
Below are the new updates in offline SDK i learnt from from TechEd session by matt.borges
Before we get into the details of design considerations lets first lets discuss about the Architecture and features of SMP/CPms and Mobile SDK for offline.
SAP Cloud Platform - Mobile Services / SMP Architecture
Picture shows high level architecture to connect CPms to on-premise system.
SAP Cloud Platform requires setup mentioned in the picture
SAP is now mainly focusing on CPms rather than On-Premise version of SMP.
SMP On-Premise version will go end of maintenance in 2020.
Challenges customer are facing today with on premise setup.
With CPms all above challenges are addressed and customer can focus on app development rather than worrying about infrastructure needs.
Features of SAP Cloud Platform
Below picture shows the features of SAP Cloud Platform.
Now going back to design consideration and best practices when defining offline mobile app.
OData modelling plays a very important role when designing mobile application.
Data model and mapping of backend call (RFC) is very important , lot of customers and partners design backend and then they adjust the mobile design, instead first data model should be created and then RFC should be mapped to the data model. Designing the data model requires understanding of backend as well as understanding of offline store in mobile.
Offline store in mobile is created looking at the oData model and metadata , all the tables (keys and relationship) in offline store are created from metadata.
.ini configuration in CPms/SMP helps to define indexes, delta token lifetime to optimize the performance.
Performance Improvement
Upload & Download
Its very important to understand the sequence for synchronization first the upload need to performed then download , if done vice versa user changes in offline can get lost.
Upload of information is done by Flush method in SDK which sends the information from request queue and Download by Refresh method which downloads the information to entity store.
Upload
It is important to understand what happens when data is created and updated in offline store. During Uploads( flush method in SDK) the operations in the request queue are sent to the server. New records which are created with local IDs are resolved with real IDs.
Records which failed to execute in backend for some reason, offline store keep track of error information and puts affected entities into the error state, user can query the errors from error archive to show the reason for the failed records.
Upload is the frequent activity so users can upload information several times in day.
Download
Download also resolved local entities/relationships with backend data.
Download activity is not frequent , it is required during the start of the day or couple of times depending on scenario. Delta plays very important role during download of information which i have highlighted below and various mechanism to implement delta.
During initial download when user is onboarding all the data need to be downloaded which includes master data , configuration data and transactions. Transaction data is needed be downloaded frequently when on field, so its very important to separate out master data and transaction data when defining offline store or defining request.
Several offline stores can be created depending on scenarios.
Defining request is which tell what should entities should participate in synchronization during download.
Deltas
The data which is changed should only be downloaded and sync to mobile.
Delta can be implemented in backend by Syclo Exchange Framework or by Gateway.
Determining delta on gateway is more expensive call compared to Syclo Exchange object.
There are two places where deltas come into play:
To maximize performance, the OData delta protocol SHOULD be implemented by the OData backend to ensure only the changes are sent to the SMP/CPms server.
By default, if no OData deltas are provided by the OData backend, SMP/CPms will compute the delta to be sent to the client.
There are two main approaches to implement Delta Token functionality in backend
It calculates deltas at modification time, wherein the ABAP system tracks relevant changes, as they occur. At request time, the deltas are already prepared, making them available. Though this approach needs higher development efforts, however, it’s more scalable and ensures an optimized overall performance.
In this type of approach, the system compares old and new states to check which records have been modified/deleted. Here, though the implementation effort is small, it however does not optimize the backend performance. This implies, if you have more records in the full collection, the response time of the request tends to get longer.
However, with both the approaches, the payload of the response is minimized, though the first approach seems to be more prudent, as it ensures optimization of the backend performance.
Batch Calls
Using batch in offline store is extremely important , it reduces the number of roundtrips between the end-user and the SAP NetWeaver Gateway server. As a result, network time is reduced.
It helps to reduce CPU time of server in case of update calls with single atomic unit of work.
Always consider to use batch calls against the single call to improve performance of server and mobile client.
Shared Data
Its very important to understand use of Shared data , if same data needed to be downloaded for group of client should be made as shared.
Shared defining queries are cached in the SMP/CPms server so that all clients can retrieve the shared data from the cache instead of all going to the backend.
Using shared data will reduce the load on backend and help to improve performance
Offline relationships without using $expand
Maintaining cardinality is equally important to define offline store to improve performance. Associations in the $metadata (except many-to-many) can be exposed as referential constraints. Offline OData can build the relationships for the Offline store using referential constraint.
$expand is only supported for ODataV4 ( NOT OData V2) but implementation can get very complex.
Example
Original defining query
Separated after exposing referential constraint
Reading data in pages
Reading the data in pages allows users to display complete data using client and server driven paging. Client-driven paging; add $top and $skip to the local queries .
For server-driven paging; specify the page size (using a Prefer odata.maxpagesize header or store option) and the SDK will return a next link if there is more data
Creating and updating records with relational entities in offline store
When working with parent/child objects in the offline data store, updates which is performed are done against the "__metadata/uri" field of the object.
SAP Offline OData store works as below:
Example using batch:
batchChanges.push(oModel.createBatchOperation(uri, "PATCH", oEntry, {sETag: etag}));
Repeatable Requests
The client can make the request multiple times with the same RequestID and RepeatabilityCreation header values and get back an appropriate response without the server executing the request multiple times
When an upload is called with failed requests in the queue, a merge algorithm is run to merge the failed requests with new requests referencing the same entity to attempt to create requests without the errors.
Repeatable requests is a feature that OData servers can support to prevent the same operation (for example, a POST) from being executed more than once.
Repeatable requests is important for all OData backends, but in particular for those used with Offline OData.
Offline Store and Request queue
With the new version two databases are created Entity store and Request queue, this is helpful when any schema changes are done.
Mobilink protocol is used for communication between SMP/CPms and mobile.
Entity store contains defining requests and if delta is enabled, delta link information from response of entities so that mobile can use delta queries to optimize download. It stores backend data which ultralite DB.
Request queue contains create , update and delete information and response of CUD operations. It also stores error information and temporary local key and real key mapping.
Picture besides is picked up from TechEd session and shows the high level details about offline store.
Schema Upgrades
Schema changes are done frequently even after the roll out, user have to upload the data to get the new updates but it is not possible with several 1000 devices in the field .
This is new update i learned from SAP TechEd where schema changes does not impact offline store as long as OData standard are followed.
Changes are automatically updated to offline store without recreating it.
Operations in the request queue that had not been uploaded to the backend should not be affected by the schema change.
Ability to create new entities while offline
Often there is need to store local data when offline , cool feature was added to SDK to create new entities while offline and entities can be modified and read.
Rather than assign arbitrary key values (or rely on values that could be changed), generate a unique entity ID for newly created entities and use that ID in the edit/read link returned in responses until the real ID/keys become available
Rather than construct URLs from key properties (which may not exist or could be changed), simply use the URLs that are returned
Below is the JSON Example to demonstrate how to use it.
Conflict Resolution & Error Handling
Using etags helps to handle conflict when several users are modifying the same records. Etags uses timestamps to compare the server data with client data to determine the conflict. There are weak and strong etags can be implemented depending on usecase.
Data related error returns response as 200 which is considered successful by mobile client and removes it from request queue , backend need mechanism either manually fix the data and logical error in the backend or IDOC's to reprocess the errors.
There are several uses to do Conflict Resolution on server ( who wins when two user update same record ) and users on server has ability to decide and overide the changes.
Limitations .
Limitations which we see today hopefully Mobile product development team matt.borges addresses it in future release or suggest better options to implement.
This blogs focuses on my experience implementing offline mobile , offline SDK evolution , best practices and improvements needed.
Design consideration plays very important role while designing offline apps , i have included more details to what was presented in TechEd while designing offline app.
- OData Modelling
- Performance improvement
- Structuring Download / Upload calls
- Delta calls
- Using Batch instead of single call.
- Repeatable requests
- Shared Data
- Offline Relationship
- Reading Data in pages
- Repeatable request
- Conflict Resolution on mobile - Using ETags.
- Conflict Resolution on server ( who wins when two user update same record )
- Error handling on mobile.
- Error handling on server ( Reprocessing errors in backend using idocs)
Offline SDK Evolution
Below are the new updates in offline SDK i learnt from from TechEd session by matt.borges
- Entity Store and Request queue DB
- Schema upgrades
- Ability to create new entities when offline
Before we get into the details of design considerations lets first lets discuss about the Architecture and features of SMP/CPms and Mobile SDK for offline.
SAP Cloud Platform - Mobile Services / SMP Architecture
Picture shows high level architecture to connect CPms to on-premise system.
SAP Cloud Platform requires setup mentioned in the picture
- Cloud Connector to connect On-Premise system such as CRM , ECC and non SAP system.
- CPms - It stores minimal information about the app.
SAP is now mainly focusing on CPms rather than On-Premise version of SMP.
SMP On-Premise version will go end of maintenance in 2020.
Challenges customer are facing today with on premise setup.
- Time required to setup on-premise version of SMP
- Load balancing and DMZ nuisances
- Maintenance and patch level, version upgrade.
- Server outage due to incorrect sizing.
- Increase in number of users
With CPms all above challenges are addressed and customer can focus on app development rather than worrying about infrastructure needs.
Features of SAP Cloud Platform
Below picture shows the features of SAP Cloud Platform.
Now going back to design consideration and best practices when defining offline mobile app.
OData modelling plays a very important role when designing mobile application.
Data model and mapping of backend call (RFC) is very important , lot of customers and partners design backend and then they adjust the mobile design, instead first data model should be created and then RFC should be mapped to the data model. Designing the data model requires understanding of backend as well as understanding of offline store in mobile.
Offline store in mobile is created looking at the oData model and metadata , all the tables (keys and relationship) in offline store are created from metadata.
.ini configuration in CPms/SMP helps to define indexes, delta token lifetime to optimize the performance.
Performance Improvement
Upload & Download
Its very important to understand the sequence for synchronization first the upload need to performed then download , if done vice versa user changes in offline can get lost.
Upload of information is done by Flush method in SDK which sends the information from request queue and Download by Refresh method which downloads the information to entity store.
Upload
It is important to understand what happens when data is created and updated in offline store. During Uploads( flush method in SDK) the operations in the request queue are sent to the server. New records which are created with local IDs are resolved with real IDs.
Records which failed to execute in backend for some reason, offline store keep track of error information and puts affected entities into the error state, user can query the errors from error archive to show the reason for the failed records.
Upload is the frequent activity so users can upload information several times in day.
Download
Download also resolved local entities/relationships with backend data.
Download activity is not frequent , it is required during the start of the day or couple of times depending on scenario. Delta plays very important role during download of information which i have highlighted below and various mechanism to implement delta.
During initial download when user is onboarding all the data need to be downloaded which includes master data , configuration data and transactions. Transaction data is needed be downloaded frequently when on field, so its very important to separate out master data and transaction data when defining offline store or defining request.
Several offline stores can be created depending on scenarios.
Defining request is which tell what should entities should participate in synchronization during download.
Deltas
The data which is changed should only be downloaded and sync to mobile.
Delta can be implemented in backend by Syclo Exchange Framework or by Gateway.
Determining delta on gateway is more expensive call compared to Syclo Exchange object.
There are two places where deltas come into play:
- Between the SMP/CPms server and the client device (MobiLink protocol)
- Between the OData backend and the SMP/CPms server (OData delta protocol)
To maximize performance, the OData delta protocol SHOULD be implemented by the OData backend to ensure only the changes are sent to the SMP/CPms server.
By default, if no OData deltas are provided by the OData backend, SMP/CPms will compute the delta to be sent to the client.
There are two main approaches to implement Delta Token functionality in backend
- Syclo Exchange Framework based approach
It calculates deltas at modification time, wherein the ABAP system tracks relevant changes, as they occur. At request time, the deltas are already prepared, making them available. Though this approach needs higher development efforts, however, it’s more scalable and ensures an optimized overall performance.
- Delta determination at request time approach
In this type of approach, the system compares old and new states to check which records have been modified/deleted. Here, though the implementation effort is small, it however does not optimize the backend performance. This implies, if you have more records in the full collection, the response time of the request tends to get longer.
However, with both the approaches, the payload of the response is minimized, though the first approach seems to be more prudent, as it ensures optimization of the backend performance.
Batch Calls
Using batch in offline store is extremely important , it reduces the number of roundtrips between the end-user and the SAP NetWeaver Gateway server. As a result, network time is reduced.
It helps to reduce CPU time of server in case of update calls with single atomic unit of work.
Always consider to use batch calls against the single call to improve performance of server and mobile client.
Shared Data
Its very important to understand use of Shared data , if same data needed to be downloaded for group of client should be made as shared.
Shared defining queries are cached in the SMP/CPms server so that all clients can retrieve the shared data from the cache instead of all going to the backend.
Using shared data will reduce the load on backend and help to improve performance
Offline relationships without using $expand
Maintaining cardinality is equally important to define offline store to improve performance. Associations in the $metadata (except many-to-many) can be exposed as referential constraints. Offline OData can build the relationships for the Offline store using referential constraint.
$expand is only supported for ODataV4 ( NOT OData V2) but implementation can get very complex.
Example
Original defining query
- Customers?$expand=Orders/OrderItems/Product
Separated after exposing referential constraint
- Customers
- Orders
- OrderItems
- Products
<Association Name="Product">
<End Multiplicity="*" Role="ItemEnd"
Type="SAP.OrderItem"/>
<End Multiplicity="1" Role="ProductEnd"
Type="SAP.Product"/>
<ReferentialConstraint>
<Principal Role="ProductEnd">
<PropertyRef Name="ID"/>
</Principal>
<Dependent Role="ItemEnd">
<PropertyRef Name="ProductID"/>
</Dependent>
</ReferentialConstraint>
</Association>
Reading data in pages
Reading the data in pages allows users to display complete data using client and server driven paging. Client-driven paging; add $top and $skip to the local queries .
For server-driven paging; specify the page size (using a Prefer odata.maxpagesize header or store option) and the SDK will return a next link if there is more data
Creating and updating records with relational entities in offline store
When working with parent/child objects in the offline data store, updates which is performed are done against the "__metadata/uri" field of the object.
SAP Offline OData store works as below:
- When new data is posted to the offline store, it creates in internal ID/URI which represents the location of this data in the offline store ("__metadata/uri" of the object/response).
- Any updates must be made to this object or else the context may get lost when flush happens. PATCH/MERGE/PUT to the metadata uri.
- Include the etag as a parameter (this is also in the __metadata).
Example using batch:
batchChanges.push(oModel.createBatchOperation(uri, "PATCH", oEntry, {sETag: etag}));
- Keeping the keys blank is the essential part (if the key is going to be generated on the backend at sync time). To perform any updates to the record use the record locator url and etag to reference the item to update instead. (example "/Entity(lodata_sys_eid = X'047AD786A01011E4800095AE655F2D3900000000')" . If there is need to display a temporary key field in the offline app until sync happens, I would recommend creating a secondary field to store the temporary display name so you can the leave the key field blank.
Repeatable Requests
The client can make the request multiple times with the same RequestID and RepeatabilityCreation header values and get back an appropriate response without the server executing the request multiple times
When an upload is called with failed requests in the queue, a merge algorithm is run to merge the failed requests with new requests referencing the same entity to attempt to create requests without the errors.
Repeatable requests is a feature that OData servers can support to prevent the same operation (for example, a POST) from being executed more than once.
Repeatable requests is important for all OData backends, but in particular for those used with Offline OData.
New features which were demonstrated in TechEd Session
Offline Store and Request queue
With the new version two databases are created Entity store and Request queue, this is helpful when any schema changes are done.
Mobilink protocol is used for communication between SMP/CPms and mobile.
Entity store contains defining requests and if delta is enabled, delta link information from response of entities so that mobile can use delta queries to optimize download. It stores backend data which ultralite DB.
Request queue contains create , update and delete information and response of CUD operations. It also stores error information and temporary local key and real key mapping.
Picture besides is picked up from TechEd session and shows the high level details about offline store.
Schema Upgrades
Schema changes are done frequently even after the roll out, user have to upload the data to get the new updates but it is not possible with several 1000 devices in the field .
This is new update i learned from SAP TechEd where schema changes does not impact offline store as long as OData standard are followed.
Changes are automatically updated to offline store without recreating it.
Operations in the request queue that had not been uploaded to the backend should not be affected by the schema change.
Ability to create new entities while offline
Often there is need to store local data when offline , cool feature was added to SDK to create new entities while offline and entities can be modified and read.
Rather than assign arbitrary key values (or rely on values that could be changed), generate a unique entity ID for newly created entities and use that ID in the edit/read link returned in responses until the real ID/keys become available
Rather than construct URLs from key properties (which may not exist or could be changed), simply use the URLs that are returned
Below is the JSON Example to demonstrate how to use it.
{
"__metadata": {
"uri": "Customers(113)",
"type": "SAP.Customers”
},
"ID": 113,
"Surname": "XXX",
"GivenName": "XXXX",
},
{ "__metadata": {
"uri": "Customers(lodata_sys_edt=X'5CD949C1F0F11343443AAA7C200000000')",
"type": "SAP.Customers",
},
"@com.sap.vocabularies.Offline.v1.islocal": true,
"ID": null,
"Surname": "QWER",
"GivenName": "XXX",
} Conflict Resolution & Error Handling
Using etags helps to handle conflict when several users are modifying the same records. Etags uses timestamps to compare the server data with client data to determine the conflict. There are weak and strong etags can be implemented depending on usecase.
Data related error returns response as 200 which is considered successful by mobile client and removes it from request queue , backend need mechanism either manually fix the data and logical error in the backend or IDOC's to reprocess the errors.
There are several uses to do Conflict Resolution on server ( who wins when two user update same record ) and users on server has ability to decide and overide the changes.
Limitations .
- Offline SDK does allow condition based synchronization for upload meaning synchronize when certain condition are met- Example upload only when order is completed in offline.
- Does not support 1: N relationship for upload for offline , SDK support deep insert with 1:1 relationship. Limitation results in creation of Staging table in backend which unnecessarily increases overhead on development.
- SDK does not allow to batch multiple entities in single batch.
- OData V2 does not support deltas when $expand are used in defining queries.
- Mobile app connects to backend system to sychronize data , it has to go through severals systems like SAP Gateway, CPms, Cloud Connector and Private mobile network and so forth its very hard to find out the network lag during the peak hour. A tool to trace the network hop will help customer to trace the network issues.
Limitations which we see today hopefully Mobile product development team matt.borges addresses it in future release or suggest better options to implement.
- SAP Managed Tags:
2 Comments
