Technical Articles
SAP HANA Express Edition on AWS for XSA development space issues
Hi Folks,
I am writing this BLOG about my experience trying to do XSA development on the SAP HANA Express edition on AWS and the space issues I ran into and how I addressed it. If you are using the HANA Express Edition on AWS, and if you are running into strange behavior and space issues, this BLOG may be helpful.
I launched the SAP HANA, Express Edition on AWS through CAL.
Once it ran, I ran across many issues.
A few things to help out so that I could get start developing:
- started nodejstools explicitly.
- create a space and explicitly redeployed the builder using the di-space-enablement tool
I was encountering a lot of strange issues and then found out the they could be happening due to disk space issues. The biggest issue I had was the the logical volume /dev/mapper/vga-sysfiles was getting filled up. I started deleting log files and any files that were not necessary. But this did not help.
I was having the issue of /dev/mapper/vga-sysfiles getting filled up to 100%. I cleared the log files, and would start the system at 95% as shown here – but that would fill up to 100% relatively quickly.
sid-hxe:~ # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 10189112 3035020 6613472 32% /
devtmpfs 15721704 8 15721696 1% /dev
tmpfs 40897116 0 40897116 0% /dev/shm
tmpfs 15730604 9900 15720704 1% /run
tmpfs 15730604 0 15730604 0% /sys/fs/cgroup
/dev/mapper/vgcal-sysfiles 42985920 40757896 2228024 95% /hana/shared/HXE
/dev/mapper/vgcal-dbdata 60819392 8848024 51971368 15% /hana/data/HXE
/dev/mapper/vgcal-dblog 33553344 2002172 31551172 6% /hana/log/HXE
So here are the steps.
Through the AWS EC2 console, click on Volumes under the ELASTIC BLOCK STORE option on the left sidebar window and then click on Create Volume:
I chose a 50 GB Magnetic volume type:
The volume is created. Note the Volume ID.
Now attach this Volume to the instance (SAP HANA Express in my case)
Log into the SAP HANA system.
Extend the volume group vgcal (in my case) to include this new volume. If you need the volume group, run vgdisplay and get the VG name
sid-hxe:~ # vgextend vgcal /dev/xvdt
Physical volume "/dev/xvdt" successfully created
Volume group "vgcal" successfully extended
Here is the file system:
Once the volume is attached, we need to extend the logical volume /dev/vgcal/sysfiles to use this volume – note the change from 41 GB to 91 GB:
sid-hxe:~ # lvextend -i 1 -r /dev/vgcal/sysfiles /dev/xvdt
Size of logical volume vgcal/sysfiles changed from 41.02 GiB (10500 extents) to 91.01 GiB (23299 extents).
Logical volume sysfiles successfully resized
meta-data=/dev/mapper/vgcal-sysfiles isize=256 agcount=16, agsize=671984 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=10751744, imaxpct=25
= sunit=16 swidth=192 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=5264, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 10751744 to 23858176So the additional 50 GB is available now for /dev/mapper/vgcal-sysfiles
sid-hxe:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 9.8G 6.5G 2.8G 70% /
devtmpfs 15G 8.0K 15G 1% /dev
tmpfs 40G 0 40G 0% /dev/shm
tmpfs 16G 18M 15G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/vgcal-sysfiles 91G 41G 51G 45% /hana/shared/HXE
/dev/mapper/vgcal-dbdata 59G 8.5G 50G 15% /hana/data/HXE
/dev/mapper/vgcal-dblog 32G 2.0G 31G 6% /hana/log/HXE
sid-hxe:~ # lvdisplay
--- Logical volume ---
LV Path /dev/vgcal/dbdata
LV Name dbdata
VG Name vgcal
LV UUID trUgvA-oNiA-QHUM-HYMY-2URJ-bp4z-5YKYOQ
LV Write Access read/write
LV Creation host, time ip-10-0-22-11, 2017-09-01 11:26:20 +0000
LV Status available
# open 1
LV Size 58.03 GiB
Current LE 14856
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 3072
Block device 254:0
--- Logical volume ---
LV Path /dev/vgcal/dblog
LV Name dblog
VG Name vgcal
LV UUID 7TzNYT-bRRX-wbpA-L5xx-xAEO-RO5R-GpB2aN
LV Write Access read/write
LV Creation host, time ip-10-0-22-11, 2017-09-01 11:26:26 +0000
LV Status available
# open 1
LV Size 32.02 GiB
Current LE 8196
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 3072
Block device 254:1
--- Logical volume ---
LV Path /dev/vgcal/sysfiles
LV Name sysfiles
VG Name vgcal
LV UUID zCjmzn-yjgf-VENQ-HCW1-Af5J-TPAx-fOkEm0
LV Write Access read/write
LV Creation host, time ip-10-0-22-11, 2017-09-01 11:26:27 +0000
LV Status available
# open 1
LV Size 91.01 GiB
Current LE 23299
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 3072
Block device 254:2
After I did this, the system was behaving much better and I could work on the XSA development. More BLOGs to follow on the SAP HANA XSA development. I did refer to the following article: Increase disk space on AWS for SAP CAL system –
https://archive.sap.com/discussions/thread/3739935
But i added the additional steps to get this working for the instance.
Enjoy,
Jay
You can connect with me at – Jay Malla on LinkedIn
The blog was extremely subjective and it clears the thought regarding SAP.
Hi Dipanwita,
The system is behaving much better now that the SAP HANA express edition version is:
2.00.023.00.1513691289 (fa/hana2sp02)
Dec 19, 2017 2:55:07 PM
SUSE Linux Enterprise Server 12.1
However, I am still having issues with running the applications. The build works fine. The system overall is behaving fine. But when running the applications, it fails half of the time and we see that the memory of the linux system is completely consumed by HANA. So we are in the process of upgrading the system to see if that addresses the issue. If it does, I will post another BLOG.
Regards,
Jay
Did you face any I/O performance issues while deploying custom applications on XSA in AWS?
Lots of issues with HANA Express on AWS. Make sure you update it with the HANA Express download manager. But the system is very unstable and has lots of issues.
Hi, Jay,
Thanks for sharing how to add space, however, there are some things that are clearly not related to disk space so I would not advise anyone perform this for those cases. It would be interesting to know what led you to think that volume was a problem.
The XSA space part is one of them. Why did you need to create a new one? What was the error with the development one?
I’ve seen the Node.js problem happening with disks that are not SSD, you kept the standard default config for these instances right? Did you recall what the logs said? Did you enable the service from the cockpit or just the feature from Web IDE?
As for running out of memory, I would recommend stopping applications you are not using. I, for example, stop the cockpit and any services for custom apps I don’t need running. Also, to confirm this is the actual problem, you can try to run the garbage collection script hxe_gc.sh in /usr/sap/HXE/home/bin after an issue and see if you can repeat the action without problems.
Some crashes from Web IDE are solved in further releases and are not related to memory. Some crashes ARE related to memory and you see these by using command xs app di-core and xs app di-builder. All of this is already documented in different places (generally Q&A addressing these issues).
What has stopped me from compiling this in a blog is that each scenario is different so advising for example to add space in the applications volume when the problem is related to memory or a bug may cripple an instance even more.
BTW, I’ve followed up on your Q&A regarding the password issues.
Regards,
Lucia.
Hi Lucia,
I really appreciate your response. The space problem happened when we uploaded the new patches that we downloaded using the HANA Express download manager. We quickly found out that we filled up the disk space on as shown here - actually this number was getting to 100%
/dev/mapper/vgcal-sysfiles 42985920 40757896 2228024 95% /hana/shared/HXE
We moved the installation files form /hana/shared/HXE but were still running into issues. So we just added 500 GB and that took care of that.
But even then, we had issues with the system running really slowly. So we changed the instance on Amazon from a 30 GB on a quad processor to 60 GB with two quad processor - basically the next level up.
In terms of the memory issue - we notice that we will get our applications to build fairly quickly - the run still takes time but is much better than it used to be - and then everything seems to work well for a while. And then the run takes longer and longer. And then the logging into the WebIDE also starts taking so long that it's almost not working. We did see some of the processes in the system as using 99% of the CPU. I think there seems to be an issue with the system. We can run the garbage collection and see if this improves the performance.
As for the di-core and di-builder - it looks like we need to restart them when we see performance degradation?
Is there anyway to make the run process faster and smoother? Where can we make those adjustments.
Thanks for your help in advance.
Regards,
Jay
Hi Jay,
I'll reply to each reply.
The HXE free license is up to 32 GB. Even though you could add some more GB for the OS to run more comfy, the database will not use it so this should not change anything.
As I stated before, the issues you are describing are not related to disk space. You would be getting different errors for a full disk.
Hi Lucia,
I do notice in my XSA audit log:
Also, the di-core is only running in the SAP space and not in the development space. I am assuming that the di-builder needs to be in the development space but not the di-core.
Regards,
Jay
I would not worry about that error in the auditlog. Yes, those apps (di-core and di-runner) should run in the SAP space, they are backend processes to Web IDE. You will have one di-builder per space and its creation part of what the space enablement tool does.
I don’t see a reason to restart these processes unless you are changing something.
Errors in Web IDE like “Request failed: Bad Gateway URI: /che/project/workspace …” may be related to those particular processes crashing. Commands xs app di-core and xs app di-runner will tell you if that was the case (you will see the word “CRASHED”). If you are not running out of RAM, you can scale the RAM allocated to them, but you need to make sure you really are not running out of RAM (the garbage collector will clean it up and also tell you how much you were consuming before running it). This is why I was suggesting you run both commands after one of those errors.
Hi Lucia,
Very strange - but even after we added the 50 GB, the hard drive filled up again. There is some issue with the internal apps filling up the disk space it seems.
Can you tell by this, what is causing the problem?
Regards,
Jay
Hi Lucia,
What can we do to clean up the /xs/app_working/vhcalhxedb/executionroot folder? It seems that a new folder with the build is being generated every time and not getting deleted when no longer in use.
Regards,
Jay
Hi, Jay,
Can you please post this in the Q&A (that's where it belongs) so I can route it to the proper teams? Please document the part in which each build creates a new folder for an app.
Thanks,
Hi Lucia,
I will create a question for this topic - /xs/app_working/vhcalhxedb/executionroot filling up. I will also add the details. We are having lots of issues developing on HANA Express even with the latest patch from HANA Express downloads. It is working much better on a HANA installation from scratch. Looks like HANA Express has not been tested very well - at least on the AWS instance.
Thanks,
Jay
Hi Lucia,
I restarted the server. I think the server processes have memory leaks that is causing a lot of the issues. When I check the logs i see:
What is this log for?
This is just the XSA log.
Hi,
I have the same issue about the directory , my file system /hana/shared/HXE/xs/app_working/hxehost/executionroot/
gets full in few minutes.
Maybe did you discver the solution?
Thanks
Fabiola
Hi Fabiola,
Developing on AWS HANA Express has been very painful - even with upgrading the system to the latest available through the HANA Express downloads. We deleted the files in /hana/shared/HXE/xs/app_working/hxehost/executionroot/ and then restarted the system which at one point did not work. We are currently working on a new HANA install (not HANA Express) with 128 GB RAM and most of the issues are resolved.
It seems to me that HANA Express is just not a good development environment (or at least SAP has a nicer and simpler way to address these issues). We upgraded our HANA Express AWS system to 64 GB memory (not the SAP HANA instance which is still at 32 GB) and added 100 GB drive and a dual quad processor but still have performance issues. Half the time the run does not work.
Regards,
Jay
Hi FABIOLA AURELI ,
Please post this in answers.sap.com. That is the proper channel to get support for HXE.
Unfortunately, this blog post mixes performance issues related to lack of RAM memory (apparently solved with a proper sizing) with this particular behavior that needs attention. Anything documenting the problem and environment helps troubleshooting.
Best,
Lucia.
Hi Lucia,
The issue is definitely not with the lack of RAM. I have increased the memory of di-core, di-runner and webide to 1 GB which is a work around.
The issue is that the logging by XSA is just adding gigs of log files which is totally unnecessary. I think there should be a simple way to clean up these files. I am tempted to just delete all folders that are older than a few days old but I am not sure if that will cause an error with existing running applications.that are logging to these files. Maybe I can delete files that are older than a few days?
Have you looked into deleting these files? The Google Cloud Hana Express instance we have been working on was at 36 GB but then we increased to over 70 GB and it is getting full again. Our code and data is a fraction of this so it's definitely all of the logging.
Regards,
Jay
Hi everyone, I also ran into space issues on my Hana Cockpit 2.0 SP09 system (based on HanaExpress).Also fighting high volumes in /hana/shared/HXE/xs/app_working/hxehost/executionroot/ ...
Apparently some more customers complaining as a note got released: https://launchpad.support.sap.com/#/notes/2775467
Take your advantage, I tested and consumption went down from > 10Gb to < 4Gb..
Regards, Patrick Rieken
Hi Patrick Rieken and Lucia Subatin ,
I am running into the same issues again. This time I am using HXE on Google Cloud. It's SP3 and a higher release than the one I was using when I entered this issue for the HANA Express on AWS.
Patrick - the note the you pointed to is very insightful. Here are the note details
It definitely summarizes the issue. But there should be a better way to clean up these files that are totally filling up the hard drive and also causing the system to be slow. These are mostly junk log files. Yes - sometimes it's handy to have this for troubleshooting but most times it's simpler to just turn on the logging as needed. These folders and files are just filling up the harddrive and Amazon/Google/Azure just charges us per hour for this additional space.
SAP should provide a utility to clean up these unnecessary files.
Any thoughts?
Jay
Hi Patrick - thanks for posting your feedback. It is helpful for sure. I am running into the same issue. Did you try manually deleting folders/files from that directory to free up some disk space?
Thanks,
Jay
Hi HP Seitz and Lucia Subatin ,
I did some more research and dug into the folders and files and noticed that every build that I am doing with my Node application is bringing in the @sap node module which itself is about 350 MB. Then the other node modules are brought in which are much smaller but still take space – a few MBs. So since I have been building the application many times – it’s filling the hard drive just as fast.
So I think the issue is that when you run an app, the previous run build and log files are NOT being cleaned up. That is what the issue is I think. So as you keep on building and running the application more and more, the system keeps on getting filled up with junk.
I think there definitely needs to be an option to not retain previous builds of the applications. The problem now is that I want to delete the builds that I don’t need without deleting the builds that I do need so I don’t mess up the working apps.
Regards,
Jay