Comparing Different Application Runtimes on SAP Cloud Platform Cloud Foundry
In this blog post, I conducted an experiment, which shows one of the many advantages the Cloud Foundry Environment can offer to you. It will demonstrate how to use the existing resources more efficiently and therefore, how to lower your infrastructure cost.
I recently read a well-written blog post by Ivan Femia “Why SAP Cloud Platform is a real Cloud Foundry Platform” in which he showed how easy it is to deploy a hello world application (written in Golang) on SAP Cloud Platform. This is only one show case of the power of the Cloud Foundry Application Runtimes.
Now you might wonder “Why should I care about the different application runtimes in Cloud Foundry? My apps are written in <insert your favorite programming language> and they work fine”.
Well, each (or at least most) programming language has been designed for a specific use-case and performs extremely well on those tasks. This doesn’t mean that they are suited for all tasks. So yes, your apps might run fine, but they might also waste computational resources which eventually costs you money.
This blog post is about a small experiment I have conducted to test this hypothesis (I also got some inspiration from Tim Nolet)
I Experiment
Apps
I’ve implemented the same program in three very popular languages: Node.js, Java and Go. The app is a simple web server with two HTTP endpoints, one endpoint simply returns a random number (not much computation needed, right?), while the other one returns the first 75 Fibonacci numbers (computationally more intense). The code is implemented in the same manner in all languages (as far as it is possible). I choose those easy tasks, same as Tim, for one specific reason. I didn’t want to compare the performance of web framework X and Y. I only want to compare the application runtimes and discover how they behave in load scenarios.
Code
You can find the code of all three applications (including instructions how to deploy them) on Github.
Test
I deployed the apps easily with ‘cf push’ to my trial landscape of the SAP Cloud Platform and performed a simple loadtest with an appropriate npm module. Each run triggered 30,000 requests, whereby the number of concurrent requests is variable. Those load tests were performed for both endpoints (random number and first 75 Fibonacci numbers) separately. I used ‘cf app <appname>’ to see how the CPU utilization and the memory consumption of the applications altered during the experiment.
II Results
Memory / Disk consumption
To me, the most surprising finding was the memory / disk consumption. I knew that the Java Virtual Machine (JVM) need to reserve a lot of RAM just to start up (even though our actual program might not need it) while Go is very economical with resources. Still, it was astonishing to see how little RAM the Go app actually needs to run (and bear the load of 30k request):
Disk Usage | Mem Usage | |
Go App | 7 MB | 6 MB |
Java App | 123.2 MB | 195 MB |
Node App | 56.5 MB | 82 MB |
Those are the max values that occurred during the test
CPU Utilization
The following chart shows the CPU utilization over the course of the experiment. We can see that all requests to the ‘Fibonacci’ endpoint eventually grow to a CPU utilization of 100%, it’s worth mentioning that the Java app reaches this limit only during higher stress levels whereas the Node implementation maxes out quite early.
The mean latency time shows a similar behavior. The mean latency of the Node app (Fibonacci endpoint) grows almost linearly after its CPU utilization reaches 100% whereas the Go and Java implementation have a mean latency time which is almost constant.
As a consequence of the previous charts we can also see that the throughput (served requests per second) starts to grow almost linearly with the number of concurrent requests. Once the CPU utilization hits 100% the throughput stops growing and might actually drop (as we can see in the case of the Fibonacci endpoint of the Node implementation).
III Conclusion
We have seen that the sample implementations in Java, Node.js and Go show very different behaviors during the load test. While the Go app is very economical with memory and disk, the Java app required the by far the largest memory and disk consumption. The Node app reached the maximum CPU utilization very early in the experiment, whereas the Java app could handle the load with the lowest CPU utilization.
All three apps reacted with slower response times and a larger average latency once the CPU utilization climbed over 90%.
It’s crucial to keep in mind that a CPU utilization of 100% “only” results in slower response times. Overloading of the memory results in a crash of the application!
If we would like to be able to bear the same load on all implementations, we needed a faster CPU / greater memory for certain implementation. This upscaling causes a more expensive cost of ownership! This shows that the question of the optimal programming language is not only a technical decision, but also business decision.
Luckily it doesn’t matter what the answer to this question is: The SAP Cloud Platform is able to run the most efficient implementation of your application or (micro-)service.
Disclaimer: The purpose of this experiment is NOT to bash any programming language / runtime or to tell you which one is the best. The main conclusion is: Different runtimes result in very different resource consumption behaviors. This post should raise the awareness that there is no single / best programming language. The choice of the used runtime should be evaluated carefully before one starts a new project.
I hope you enjoyed reading this blog post and I would appreciate it if you could leave me your thoughts on this in the comments below.
Update: Compared to the previous version of this blog post (in which I used express.js instead of the low-level http module) most results remain unchanged. The only big different is, that the CPU utilization of the ‘node/random’ endpoint did not explode with a growing load. This means the application was able to serve more requests faster.
I also noticed that the memory/disk consumption of the node app decreased, since external dependencies have been removed and don’t consume memory resources anymore.
Hi Marius,
very nice comparison about the performance of the different runtimes! Could you provide the source code to reproduce your results?
Hi Nils,
thanks for your feedback! We're currently discussing internally what's be the best way/channel to publish this source code. I'll get back to you as soon as we come to a decision 🙂
Regards,
Marius
Hi Nils,
unfortunately I cannot publish the entire repo yet (we're currently in the process of releasing the repository on Github).
Regards,
Marius
Thanks for you patience Nils! You can find the code on github now.
Thanks for providing the source on github 🙂 But am i blind, i can't find any github link in the blog post. All github hyperlinks are pointing to this blog itself ?
my bad, fixed it
Hi Marius,
Thanks for this comparison.
I have a some questions with regards to the memory values you showed and what SCP CF reserve by default.
In a recent experiment, our finding was that a minimum of 1024 MB are by default allocated by SCP CF to a Java app, even when the app requires far less memory.
i.e 2 simple java "hello-world" app will consume 2 GB from your SCP CF Quota. Obviously in this case the memory should not even reach 1GB.
Thank you!
Hi Olivier,
that's a great comment! I also think that this requirement (for java apps) is really strict. Per default all CF apps that you push to the SAP Cloud Platform will consume 1024mb memory. I'm sure you know that you could change the default memory allocation in your manifest.yml file or via the "cf push -m" parameter. This works fine for apps written in Node.js or Golang, but not for Java apps, as you found out in your experiment.
The reason for this behavior is the java buildpack which is used per default for java apps. Roughly spoken: The buildpacks define the application runtime, including all parameters. One of those parameters includes the memory your JVM should reserve during the startup of you app. And this is the reason why you app crashes when you provide less than 1024mb in the Cloud Foundry enviroment:
Output:
I guess (I haven't tried it myself) substituting the default buildpack with a "slim version" could solve you problem. You "just" need to write a new buildpack or find one online and specify it in the manifest.yml file. Here are some official Cloud Foundry resources for you:
Official Cloud Foundry Java Buildpack
Java Buildpack Memory Calculator
I hope I could help you with your problem
Regards,
Marius
Looks good old Java still has some kick left on him. What about C#? 🙂
Great comment Joao! It would be intersting to find out how C# performs (my guess would be that it's in "between" of Go and Java). I'm afraid my C# skills are very limited, so I cannot write that code myself, but maybe you can create a gist of the corresponding code (I've posted some Gists in a comment above)?
I'll try and do that, was reading up on Cloud Foundry and C#, I'll try to do it whenver I have some free time. From a coding perspective and runtime, I prefer .Net to Java any day.
Hi,
Finally got up to do it, here it is https://github.com/jgsousa/cfnetcore.
Hi Marius,
nice experiment definitely.
I would not got so far to derive here any conclusions on performance. Probably unexperienced developers will do, especially as you are part of the SAP Developers Relation Team. Why not?
First the example is very basic, I think it is not more than a “Hello World” althought also some processing is done. But it has really nothing to do with an enterprise application use case with more layers (database,…), where performance is important.
Second and that is already relevant for this experiment, you are comparing here apple with pears. I was able the have look at the code snippets in the gists (unfortunately I cannot access them anymore, why?).
For node you are using express package to provide the http endpoint. For Go and Java you are using low level api’s to provide the http endpoint. Express is a complete web application server with default settings, features and which comes with a whole bunch of further npm packages. By using express you cannot compare the results with Go and Java anymore.
It’s like you throw Tomcat in for Java, which when I think about it, is the case for Java anyway?! Is Tomcat the runtime for java buildpack? So probably we compare here: Express vs Tomcat vs Go?
What we also compare is multi-threaded Java/Go with single-threaded Node, which implies a complete different approach and challenge on scaling up and when it comes to performance.
Performance is a very complex topic which depends on many factors and this experiment is coming to short.
Could you provide the code snippets again?
Also of interest would be the options we have to influence garbage collection, memory consumption of the different runtimes.
Have fun and best regards, HP
p.s. Sorry for providing such feedback on your first post. I do not want to discourage you. It is a complex and tough but important topic. So carry on!.
Hi HP,
Thanks for your feedback!
You’re right, this experimental setup is a group of extended “Hello World” applications, which are not comparable to real-world enterprise apps. I would even go that far and say that a fair comparison of enterprise apps is impossible:
That’s why I decided to implement this easy example, since it was suited to show what I wanted to show in this blog post: Different runtimes result in very different resource consumption behaviors. I think I should re-write the conclusion (and maybe some other sections) to avoid confusion. I didn’t mean to say “Node.js has a bad performance, don’t use it” (by no means, I’m a huge fan of Nodejs myself!).
Also, your comment about the express package is absolutely valid. Somehow, I use express so often, that I subconsciously assumed it was a low-level API (which is obviously not true). I will re-implement the node app with the http module and update the results!
Currently we are comparing Express vs Java (Main Container) vs Go. The Tomcat container would only be used implicitly when there was no main() function and a directory named “WEB-INF” in the project directory (btw you can also use spring boot container in a similar manner). We’re getting really deep into Cloud Foundry now, this is super exciting, right J?
It’s true, I’m actually comparing apples with pears here. It’s by no means “fair” to compare the single-threaded Nodejs with multi-threaded Java / Go. All three runtimes have very good arguments why there are around and used in the industry. The purpose of this blog post show that there are significant differences in those three runtimes and those differences will reflect in your resource consumption / monthly bill. In my option this fact is often neglected in practice and people start to implement projects in a language, which is not suited for their problem (just because they prefer this language, or because their boss told them to use it). This blog post should raise awareness that there are different runtime behaviors for different tasks. I will try to emphasis this in the updated version of the blog.
To summarize, I will rework the blog post today and emphasis that I want to raise awareness that the choice of the runtime / programming language is important. The results of this experiment should support that claim and not be interpreted that some runtimes are bad per se.
Thanks for this interesting and very technical comment, I appreciate that you had such a deep look into my post and shared your concerns in such an honest way! Please let me know if you agree / disagree with me.
Regards,
Marius
It's funny that I assume people know this is just a PoC, not some definitive proof about which language is best or has the best performance ... and yet HP is right. People will probably conclude things they shouldn't 🙂
I tried to analyse the difference and changing from Express to basic HTTP module won't really make a difference, Node.js is just not the language you should use for calculations like Fibonacci. Node.js excels at small tasks and io calls, while long calculations will just block the request loop and break the application.
It does go to show that "legacy" languages like Java and C# still have a space to fill in the todays microservices/container architecture.
I have to admit I never gave a second look at Go. Maybe I should.
Agreed! I think the "node.js revolution" was incredibly huge and therefore many "legacy" languages were pushed into the background.
About Go: At first, go seems kind of weird when you're coming from the Java / JS world and have little experience with low-level languages like C (as I do personally). But you will find many very nice features of the language once you do your first steps with Go.
The "revolution" was too huge in my opinion. People start using node.js for everything because it's "hip" and "modern" instead of using languages better suited for the job.
It's the same with blockchain, where for some use cases I just think "Would it just be simpler to use a traditional solution?"
Blockchain is particularly strange as it shouldn't appeal to the envrionmentally consciencious crowd. It shows that "fight the establishment" is what's really important.
Hi Joao,
I personally think in the context of Cloudfoundry and HANA XSA (SAP on-premise version of cloud foundry) Node.JS is a perfect match doing the heavy lifting on big data in HANA Database and have light-weight very scalable service layer with NodeJS.
The "burning cpu only" sceanrio used here with recursive functions, is most far from the mayority of all business use cases.
The revolution with node in SAP eco system is hopefully just started as SAP for now only migrated mainly one to one the XS Classic approach with xsjs/xsodata/cds with the xsjs compatibility layer for node. The full async power (and complexity!), which NodeJS provides, is still to be harvested by SAP providing new frameworks for building business applications.
Have fun, HP
Node and Hana is a scenario of IO where the cpu burning is happening at the database level. I use a lot, I don’t use it everywhere .
What do you mean by XSA is cloud foundry on premise?
I think he's referring to the fact that XSA mimics many of Cloud Foundry's mechanisms to provide a similar dev experience (both have a similar CLI interface, you define services and bind them to an application, both make use of a UAA etc)
And about the source code: We will publish the repository on Github soon
Thanks for you patience HP! You can find the code on github now.
Hi Marius,
as the link seems to be incorrect. I guess the correct one is:
https://github.com/SAP/cf-web-server-benchmark-tests
I really wondering again, after all the discussion, that this "burning cpu only" Scenario is now called "Benchmark"!
BR, HP
There are use cases that are CPU intensive. May not be the most common use case with SAP Hana but it is a common benchmark.
perhaps if you want to measure performance of CPU's or GPU'S. But runtimes, platform, frameworks?
Does not make sense to me, sorry.
As it called web server benchmark. It would make sense to have something like user session in there. And see how much memory is used per session and how many user session could be served in parallel. This could be also measured without any another layer and isolated.
It the current form is does not provide any value as benchmark. Instead many people will derive wrong decision from it by looking on the nice charts!
Let's agree to disagree. This is relevant if you are using Cloud Foundry to run CPU intensive tasks. It shows that Node is probably the wrong framework to use for those kinds of tasks.
Are you saying there are absolutely no use cases where your microservice relies on CPU instead of IO ?
Nice comparison. I was comparing node vs go, with similar results on memory consumption.
Do you have any experiences in connecting to the hana db from go?
Hi Rafael,
Unfortunately I haven't tried this experiment yet, but it would definitely be interesting to know.
I'll put it in my back log, but I cannot guarantee if I'll have time to so this any time soon 🙁