The thoughts and opinions expressed in this blog post are my own, and do not necessarily represent those of Sybase or SAP.
This past week an article written by my colleague, Eric Lai (Twitter), entitled Multitenancy & Cloud Computing Platforms: Four Big Problems, caused quite a bit of controversy. An example of this is the formal rebuttal by independent consultant Frank Scavo (Twitter) in his blog post, Mischaracterization of Multitenancy in an SAP-sponsored Blog Post.
As Eric noted in his article, this post was motivated by a discussion that he and I had about multi-tenancy around Sybase’s forthcoming SQL Anywhere OnDemand cloud database. In this post I will respond to Frank’s comments, attempting to frame the discussion in the context that the article was intended. I think we will find it is all a matter of hats. To that end, I invite you to put on my hat for a moment.
I am a product manager on the SQL Anywhere team. For those who are new to SQL Anywhere, it is an embedded relational database that has been in development at Sybase for over 20 years. Although you may never have heard of it, there is a good chance you have used it. This is because it is typically so deeply embedded in an ISV’s product that you do not know it is there. An example of an application that embeds SQL Anywhere is Intuit’s QuickBooks.
Over the last two years, we have started to see a shift in the ISV community. ISVs who previously used to deploy an application to their end users (and have it run on-premise) are feeling pressure to have a hosted version. This turns an ISVs core competency on its head. After all, ISVs have expertise in deploying software, and managing deployed software. While hosting removes those challenges, it puts the challenges of hosting in their place. This is an entirely new world for many ISVs.
To capture this shift (and other shifts within IT), many companies have launched full-stack, public platforms-as-a-service products. These are multi-tenant platforms where the resources are shared between hundreds of different applications running on the platform. The promise of the PaaS is that it will take care of the hosting, and it will scale your application automatically.
There is no such thing as a free lunch. The tradeoffs come in four places: flexibility, security, power (or functionality), and cost. Now that we are all wearing my hat (also known as the ISV-looking-for-a-platform-to-host-their-application hat), let’s take a look at the objections raised in Frank’s response.
When choosing a platform, any platform, you have entered a garden. Some of these gardens have quaint white picket fences with lots of gates and a nice breeze. Others have tall, thick, brick walls.
On a PaaS, the ISV will be limited to the physical locations, certifications (HIPAA, PCI-DSS), hardware, technologies, and terms-of-service of the platform provider. This rigidity causes problems if the ISV has current (or future) requirements that can not be met by the platform provider.
So what does multi-tenancy have to do with this inflexibility? Whether speaking of a platform or an application, it is often the multi-tenancy aspect that limits the amount of flexibility that an application or platform can support. I am not saying that multi-tenancy is bad, just that it is usually inversely correlated with flexibility.
It’s Less Secure
The question of multi-tenancy for an ISV has two facets. There is the question of the multi-tenancy of the platform they are running on, and there is the question of their ability to create a multi-tenant application.
I am not aware of any published security breaches between separate applications running on a multi-tenant platform. Each application running on the platform will likely have its own database, and so the risk of data leakage is mitigated. I believe that the platform providers will have done a lot of work and testing to ensure that separate applications on their platforms are isolated.
But the platforms do not provide any help in isolating the data within an ISV’s own application. When an ISV deploys an application on-premise, each customer gets their own instance of the application (with their own instance of the database). When the ISV pulls all of those customers together to host the application, does it make sense to combine all of the customers’ databases together into a single database?
Most of the PaaS stack’s databases have been designed to scale with the absolute size of a single database. This metric of scaling suggests that it would be best to combine all of the customers’ data into a single database. This creates a potential risk because it is possible that the ISV could introduce a bug that accidently exposes one customer’s data to another. The most likely cause of this is a coding error. (eg. forgetting to filter the data to just that customer, a bug that causes confusion of the customer identifier, etc)
Cases like this have happened in the wild (emphasis mine):
- Microsoft BPOS cloud service hit with data breach
“We recently became aware that, due to a configuration issue, Offline Address Book information for Business Productivity Online Suite (BPOS) Standard customers could be inadvertently downloaded by other customers of the service, in a very specific circumstance,” said Clint Patterson, director of BPOS Communications at Microsoft.
- Hulu.com users were logged into someone else’s account
“Hulu said shortly after launching its Facebook Connect feature Friday that it noticed a small number of users were seeing someone else’s account information upon logging in to the site.”
“We’re still drilling down on the precise nature of the issue, but we know that it was a coding and configuration error on Hulu’s side, and not the result of hacking, or other third party actions”
Both of these are large companies who I expect have the resources to design and test their multi-tenant solutions, and yet both had (thankfully, limited) data breaches.
The concern of many smaller ISVs who are brand new to hosting is that they will not do it correctly. I would expect that data breaches of this nature are more common, but it is just that many smaller ISVs do not have the profile to have their breach featured in ComputerWorld or VentureBeat.
One solution for the ISV is to keep total isolation of the data between all of their tenants. One tenant, one database. The application layer may be multi-tenant, but the database is single-tenant. While some of the platforms will allow you to maintain multiple databases, it is not cost-effective, and they do not have any tools to help manage thousands of separate databases.
This is exactly the use-case for which SQL Anywhere OnDemand was designed: a multi-tenant application layer, backed by a single-tenant database layer.
I want to make it clear I am not suggesting a multi-tenant application is inherently insecure. (After all, we are enabling our ISVs to create multi-tenant applications!) Instead I am suggesting any developer should only include multi-tenancy up to the level they are confident that they can make secure. For many ISVs who do not have experience in multi-tenancy and hosting (and whose apps are already written as single-tenant applications), it may be prudent to keep the databases single-tenant.
It’s Less Powerful
As Frank points out, the platforms allow for huge improvement in productivity. I have no argument here. However here as well, there is no free lunch. The productivity gain is inversely correlated with power and functionality.
When I have to write a quick script, my language of choice is Python. I love Python. Our SQL Anywhere database is written in C/C++, with some performance critical routines written in assembly. Would it be a wise choice to rewrite SQL Anywhere in Python? No, it is not the right tool for the job.
Many of the ISVs using SQL Anywhere have very database-intensive applications. They move large amounts of their code into stored procedures in order to reach their performance goals. Moving the logic out of the database, and going through an abstraction layer (eg. Object-Relational Mappers) may not be an option for them.
This really comes down to the same conclusion as the flexibility argument. If the restrictions in functionality (which allow the boost in productivity) are acceptable to you, great. If they are not acceptable, that platform is not an option.
It May Be More Costly
As Frank asserts, this is speaking of the cost to the ISV, not the end customer. The reason for this is that many ISVs are smaller shops who do not have the bandwidth to fully re-architect their application to fit the constraints of a platform. They need their application hosted, and they needed it done yesterday.
Many of the ISVs that I have talked to plan to accomplish by doing it in stages. The first stage is to move the existing application and database up a hosting provider, and use remote desktoping technologies to remote the application’s GUI to the end-user
I can almost hear an audible groan of disdain from cloud purists:
“You can’t do that! The application must be totally re-architected in order to take advantage of the cloud.”
That is true, but pragmatism is holding the trump card. Don’t let the perfect be the enemy of the good!
For many of these ISVs, the end goal will be to re-write as a “cloudy” application (and thus reap all of the cost savings to both them, and their customers), but the direct path may not be the most cost effective.
Now let’s take off my hat, and put on the Enterprise-end-user hat. To understand the reaction wearing this hat, I invite you to read Frank’s blog post.
As Frank points out, when the original post is read as an enterprise end-user (or even an enterprise developer), a lot of the arguments do not make any sense. This is because enterprises and ISVs are different beasts.
An enterprise knows its requirements. It knows what local data centers it will need, and it controls all of its end-users.
An enterprise does not have to consider what would happens if a new customer appeared in a country that had strict data laws, and there was no data center for your platform located there.
An enterprise does not have to consider that they might suddenly find out their enterprise application needs to be HIPAA compliant because they were able to score a new customer in the health care space. (I am not saying they would not ever have to be HIPAA compliant, but they would be better able to plan for it).
It’s Less Secure
The question of multi-tenancy within the application is meaningless here. All the data in that application is for that enterprise. There no risk of having your enterprises’ data accidently exposed to another enterprise due to your programming or configuration error.
It’s Less Powerful
An enterprise is in control of all of its users, and is able to limit functionality by mandate. An example of this is IT departments that often mandate “This is our list of supported browsers”, or “This is our list of supported devices”.
Most ISVs are not in a position to make mandates to their users. If they cannot support a certain feature, they lose customers. That customer will not care if the excuse is, “My underlying platform does not support that.”
It May Be More Costly
From time to time, enterprises need to do overhauls of their applications. While these are disruptive, there is nothing you can do except grit your teeth and wait for the disruption to pass.
It is much harder for an ISV to tell their customers:
“We have to do a major internal rewrite. This means our next release will contain almost no new features, and will probably be late.”
(In reality, ISVs still actually have to do this, but they try to mitigate it by doing it in smaller chunks.)
Taking the Hats Off
Several months back, there was a blog post at the SAP Community Network from Richard Hirsch entitled What is the relationship between Sybase’s ‘SQL Anywhere OnDemand Edition’ and SAP’s other OnDemand offerings?. After delving into the question, he concluded:
- There were other use cases in the market beyond those that were being met by SAP OnDemand offerings on which I usually concentrate (OnDemand Core, OnDemand Edge, SAP NetWeaver OnDemand, etc)
- The SaaS market is varied / more complicated than many assume.
I think this hits the nail on the head. The original post was targeted at the group outlined in his first point.
The second point is a good reminder for me. I spend so much of my day wearing my own hat (after all, it is comfortable), and I failed to anticipate how these ideas would be interpreted if read wearing a different hat. I apologize for the confusion it has caused.