Are you seeing unusually high virtual memory usage on your BI 4.x Linux install?

[former_member59613]

I may be able to help explain why!

As I tend to talk too much, I have provided a summary and a details section below.  Choose your own adventure :lol:

SUMMARY

• Webi monitors VM usage as part of it's Memory Analysis feature.  This can cause issues when VM limits are exceeded
• glibc (version 2.10 or higher) which is updated on most newer RHEL versions can cause excessive VM allocation out of the box.
• There are ways to tune/tweak memory allocations using the steps outlined in this KBA
• Another option is to disable the Enable Memory Analysis option for the WIPS servers

DETAILS

We have seen a few issues come into Product Support now where BI System Administrators were reporting abnormally high VIRT (Virtual Memory) usage for the BI processes on their Linux installations.  I didn't really think much of it at first because virtual memory isn't usually a concern now a days with RAM being so cheap and hard drive space being plentiful.  Whelp, it wasn't long before I realized that this IS an issue for BI 4.x.  Why you ask?

Well, it all started a few years back when we were learning to operate within the constraints of a 32-bit operating system.  Windows has a well know limit of about 1.8 - 2.0 gigabytes of memory per 32-bit process.  Once a process hits this limit, it's a crapshoot as to what will happen.  Either the application will crash, hang or if you were lucky enough, it would through a catchable error message.

To work within this 32-bit limitation, we implemented a memory analysis feature into the Webi Processing Server and allowed administrators to set a low/high/max memory setting to help control this.  Since we do not know how much memory will be used when a query is executed on a report, we just have to do our best to stop new queries from coming in when the "high" memory mark is reached.

What type of memory does the Webi Processing Server take into account with this Memory Analysis?  Virtual Memory of course.  Why?  Because on Windows, if a process exceeded 2GB of Virtual Memory, the process would do horrible things.  So, we had to monitor this as best we could.

I could write a whole document on that subject so if anyone has questions around how that works. feel free to message me.  I'll fast forward a little bit to today's 64-bit world.  Today, with BI 4.x being 64-bit, we are not worried about memory as much.  The limits are very high with 64-bit processes and we will likely never reach those limits.

Even though the limits are much higher, we still have this Memory Analysis feature implemented and enabled by default on the Webi Processing Servers.  It is still a useful feature for administrators that wish to keep an eye on their system's resource usage and limit users from executing unusually large requests.  Out of the box however, our default settings are low and this can cause some hard to explain behavior if you don't know what you are looking at.

Enough background, here is a little known issue that I wanted to bring to everyone's attention.

It seems that some versions of Red Hat EL 5.x and 6.x ship with an updated version of the glibc libraries.  Starting with around version 2.10 of glibc, the default malloc() functionality changed slightly.  For those that aren't aware, malloc stands for Memory Allocator and the new default behavior was added to potentially increase performance on some applications.  The new functionality will allocate an "arena" of memory per thread that a process uses and will do this based on the # of CPUs that the server has.  So on multicore machines, we are seeing that a HUGE amount of virtual memory is being allocated right on the startup of our processes.

Right out of the box, my 16 CPU test system was allocating up to 11GB of virtual memory for some processes.  Webi processes (WIReportServer) were using 1.7-2.0gb of VIRT memory right out of the gate and were climbing up to 4-5gb after a single report refresh.  The java processes were way up there too and often started in a range of 8-10gb.  This becomes a concern for our Webi Memory Analysis feature as Virtual Memory is what it monitors for the Webi processes.

This results in stopped/failed Web Intelligence Processing Server and end users start reporting a variety of errors.  Server is Busy, Out of Memory, etc...

So, what's the solution?  I have started a Knowledge Base article that documents the known work-arounds for this issue.  It is linked below:

https://service.sap.com/sap/support/notes/1968075

In short, there are 2 options.

1. Add an environment variable to revert back to the old malloc() functionality (where virtual memory was not blown way up)
2. Disable the Memory Analysis feature in Webi.

I would recommend option 1 personally.  The KBA above gives the details of which environment variables to test with.

The below - shows you the 'top' output of my BI server right out of the box after starting my BI processes with 'startservers'

And the below image shows the memory usage AFTER setting the environment variables.

Notice how much lower they are once we revert the malloc settings back to the old way of doing things.  It's not just Webi that is lowered, you can see the reduction in VIRT is across the board.

So, if you have a RHEL system with glibc version 2.10 or higher, you may want to check your VIRT usage to see if it is abnormally high.  If it is, it might be worth testing these environment variables to see if it helps you keep that under control.

If you read this far down!  Thanks for tuning in.

Jb