Skip to Content

Supported versus tested

Mobile app developers do not have to buy a mobile device to actually run their app on it and test it. You get simulators for free. You can test your development on iOS, Android, WM or Blackberry without spending money on a physical device. As soon as your project gets more concrete and you identify your target group, you also start testing your development with real devices. Why on a real device? After all, you don’t want stumble over a bug that only manifests in a real device. There still may be a hardware / software / mobile network combination that break the code. That’s not different when you develop with SUP.

Here is the good news: SUP supports the main mobile operating systems: iOS, Android, BlackBerry, Windows Mobile. Here is the bad news: Sybase did not test SUP on physical mobile devices. For the Object API and iOS, the documentation details what was tested:

  • iPhone 4 (iOS 5.1), iPad and iPad 2 (iOS 4.3)

This means that the iPhone 4S and new iPad are not tested. If you now suspect that Sybase has no money to buy new devices from Apple, take a look at the page for the hybrid web container. The list of tested devices lists more combinations:

  • iPad (iOS 4.3.x), iPad 2 (iOS 4.3, iOS 5.1), iPad 3 (iOS 5.1), iPhone 4S (iOS 5.1), iPhone 4 (iOS 4.3.5), iPod (iOS 5.0).

Pad 3 is now tested, as is the iPod and iPhone 4S.

What does that mean?

If you decide to develop your SUP app for iOS 5.1, using the Object API and an iPhone 4S, you will run your app on a device that wasn’t tested. It’s supported, but not tested. You are transforming yourself into a beta tester.

Other mobile operating systems

The next problem you run into is how mobile devices are sold: they are customized. The operator puts some custom apps and other stuff on it, and for non-iOS devices you get a different UI/UX from venders like Samsung or HTC. As Sybase supports for instance Android, does it support the customized Android from Samsung? It is too much to ask a company to test against every OS modification, but the best sellers should be tested.

And what about the simulators mentioned in the beginning? What simulators Sybase used to test SUP? On the BlackBerry page you can download almost every device / OS RIM released. Now: what did Sybase tested? Yes, Object API for BB OS 5.x, 6.x 7.x is supported, but did Sybase test the  device using the simulator? That leads to another question:

How to interpret the phrase tested in the online documentation? A physical device was tested or just the simulator? Or both?

What about Windows Mobile devices? A use case for them is field services. And that means that you won’t use a WM device you can buy in a shopping mall store. It’s these heavy, brick-like mobile devices like the one here made for outdoor usage, with a keyboard, scanner and RFID reader. Use case: offline scenario. For SUP this means: MBOs.

Sybase does not list one of these in the tested device section. As these devices are slightly more expensive than consumer WM phone, it would be nice to have a test suite before buying them in large quantities.

Test suite

It’s impossible for Sybase to test all devices as it is for others: here is an example for phonegap and jquery mobile. There is a test guide available on SCN that lists some test tools (most only available to Sybase and consulting) for you app, not for SUP. How to know that the device chosen for the solution works with SUP, before start writing code? What if only after a few months into the project an error gets identified and traced to the actual device / OS / SUP combination? How to get a test suite to test the correct execution of SUP on the mobile device?

What can currently be done is to use the sample applications and run them manually to find out if they work on the device. But that’s far from having SUP passing (automatic) tests to ensure a functional SUP on the device.

Here is an idea SAP: release a test suite for SUP and together with a developer license people can start testing SUP with their simulator and physical device. They will test their device using a real internet connection, different OS, configurations, locations, providers, and so on. This way SAP can use the crowd to test SUP in an unbeatable way: real life.

Update

Idea at IdeaPlace created: https://cw.sdn.sap.com/cw/ideas/11555

4 Comments
You must be Logged on to comment or reply to a post.
  • Hi Tobias,

    excellent points! Buy I fear you forget that you do actually need specific hardware to implement for one of the platforms. It was for this reason that I now have a Mac Book Pro.

    With the announcement of SAPPHIRE that free developer licences will come (I've not heard anything since, but honestly I've been too tied up with Successfactors stuff to check) we are almost half way there.

    Does this point merit an idea place idea? Or do you not see the value?

    Thanks for the excellent ideas and points.

    Chris

    • You mean iOS? You can also buy Mac OS X and use a virtual machine 🙂 With a public test suite you won't have to buy 4 or more devices from Apple as you can look up if someone from the (no doubt) hundreds of developers on Apple run ever into a SUP on iOS problem.

      Idea place? The place where ideas go to die? Rarely get a feedback what is happening to ideas that were posted there. If you really think its worth the effort ... feel free to post the idea there.

      Yes, it was announced, and since then: silence. Let's hope for the best. Worst case: we have to wait for the next big SAP event (TechEd in October) 🙁

  • Presumably, if you're testing to a specific device, it's for a reason; that reason should be enough of a business case to get a physical example of the specific device. Alternatively, you code within the device limitations (at least for that particular device - do you have access to feature detection facilities within SUP ?)

    BTW, this supported but not tested scenario is not that uncommon;  I was installing SAP production systems running on VMware and ESX before SAP was supporting them. Not to be bleeding-edge or anything, it was because the customer had run out of physical space in their datacentre.  You do need to know what the costs and risks are - sometimes it's limitations in functionality that you can live with (in the VMware case, the ST06 CPU Utilisation was always being reported as being 100%), sometimes there are, in fact, none (i.e. SAP on CentOS instead of Redhat).

    hth

    • The business case for Sybase to test SUP MBO for iPhone 4S is simple: that's the device Apple is selling. I expect that a company that is selling a solution for mobility at least has an impressive number of physical devices to actually test their product on. And that they are proud of their test suite and communicate the effort they are doing to ensure that you, as the customer, can be assured that the product was tested.

      In your example you get at least some test tools from VMWare, linux, SAP, and so on to find out what the actual usage is. For mobile devices the situation isn't as good: you depend on what the device, OS and application vendor give you.