Happy Holidays! Hope you've all had a good 2014. Before closing out this Year, I'd like to put up a reminder:
Revisit the sizing of Web Intelligence Processing Servers on your deployment of SAP BusinessObjects BI 4.1
If you've done so in the past eight months or so, that's great! But pencil that task into your calendar for 2015.
If you've haven't done so in a year, then I strongly recommend scheduling a sizing exercise for 2015Q1.
I support Web Intelligence. I've supported it since before BI 4.0 Ramp-Up, through its General Availability on September 16, 2011, and to now with BI 4.1
It's been over four years, and through all the years, the three most important considerations that affect the smooth operation of BI 4.x WebI on your system is (1) sizing, (2) sizing, and (3) sizing.
If the sizing of your BI architecture is wrong, then you're going to face issues with performance, stability and availability, if not now then in the near future as demand on your system steadily grows. You might even find yourself spending quality time with me on the phone, and one of the first things I'll be asking you is "When did you last size your system?"
If the answer is before February 2014, then I'll be pointing you to the Business Intelligence Sizing and Deployment page, where you'll find the BI 4 Sizing Guide, updated February 2014. There's been changes to the recommended configuration of Web Intelligence in the newest edition of the Guide.
Rule: Always start your sizing exercise using the most recent Sizing Guide
When you get to the Sizing page, I recommend you click the "Follow" page, so that you'll be notified whenever the page is updated. Or go to https://service.sap.com/sizing and find the BI 4 Sizing Guide, where you can "Subscribe" to the Guide and be notified when it's updated.
There were changes to the recommended configuration of Web Intelligence services in the latest update, so I highly recommend reading through the entire Guide, just to make sure your BI 4.1 deployment is following recommended practices.
I've spoken with the Platform Product Owner, Sada, and he anticipates no changes to the Sizing Guide until at least BI 4.2. But don't take my word for it. Before starting any sizing exercise, make sure you go and download the latest version of the Guide.
Rule: Do not use XI 3.1 sizing guides or best practices to size BI 4.1
One time, a customer pulled up their architecture deployment sizing document for their BI 4.0 system and in there was a references to XI 3.1. Turns out they had been keeping the same sizing architecture since their XI 3.1 days. That definitely did not work, and they had lots of issues that mostly disappeared once they had redone their sizing exercise anew.
XI 3.1 and BI 4.1 are entirely different beasts, and trying to fit BI 4.1 components into a XI 3.1 architecture plan is like harnessing thoroughbreds to a dog sled. No matter how strong the beasts, they're just not gonna perform well when they keep tripping over each other and getting into each other's way.
The major differences between the two versions:
- BI 4.1 WebIPS is a 64-bit process. XI 3.1 runs as a 32-bit process, even on 64-bit machines.
- BI 4.1 WebIPS delegates query and chart generation to a different Service. XI 3.1 the generation is in-process.
Because of these differences, the sizing best practices developed for XI 3.1 is inapplicable, even harmful, to BI 4.1
Let's walk through the best practices we had for XI 3.1, and compare to BI 4.1:
XI 3.1 - deploy one WebIPS for each CPU core on the machine.
BI 4.1 - deploy one WebIPS per machine.
There wasn't a real good reason why you'd want to tie the number of XI 3.1 WebIPS to the number of CPU cores - the WebIPS threading model parallelized concurrent report processing requests just fine across multiple cores. Each thread would process a single document request, but the threads can run on different CPU cores. WebIPS does not impose CPU core affinity.
XI 3.1 being 32-bit meant that memory resource limited how many processing jobs a single WebIPS could efficiently handle. Back then, proper management of memory utilization was an important part of sizing WebI. Increased usage would quickly demand more processing power to be added to the mix, and because of the memory limits, the only reliable way was to add additional WebIPS to the deployment. Having one WebIPS per CPU core was just a pretty good rule-of-thumb when it came to deciding whether you needed to add additional machines to the system.
But BI 4.1 WebIPS is a 64-bit process, and has a much expanded memory space. If I had a nickel for every time I've had to recommend customers alter the memory handling setting for their BI 4.x WebIPS, then I could afford, well, not even a stick of gum.
As you can see in the Sizing Guide, the BI 4.1 WebIPS is limited by IO. You want to ensure the WebIPS have access to as much IO bandwidth as possible, and that means, especially on Virtual machine environments, having just one WebIPS per machine.
Very early on in BI 4.0, there was an issue with WebI failover across different machines (a problem with the Document Recovery Service), so I used to recommend two WebIPS per machine (failover of WebI Sessions on two WebIPS on the same machine doesn't require use of the Document Recovery Service). But that time is long past.
XI 3.1 - Enable Memory Analysis
BI 4.1 - Do Not Enable Memory Analysis
(added 2015-01-16)
I'm adding this item since it does come up often enough, and I recently got confirmation from a WebI developer that this is what's recommended now.
Disable the "Enable Memory Analysis" option for BI 4.1 WebIPS. It's enabled by default currently (that will likely change soon), but it's a holdover from XI 3.1 that's not needed and may cause issues.
In XI 3.1, it was a noble effort to try and preserve the precious 32-bit memory space. Enabling memory analysis causes the WebIPS to reject any further incoming processing requests, other than saving the document, when its memory utilization goes above the "Memory Upper Threshold" value. The hope was that this would give Users an opportunity to save their work before the WebIPS starts to run out of memory. Beyond "Memory Maximum Threshold", all requests were rejected in the hope that the WebIPS can recover before it crashed.
BI 4.1 WebIPS works in a 64-bit memory space and has been upgraded to behave much more gracefully when it does reach resource limits. The "Enable MemoryAnalysis" is a cure without a disease. In fact, keeping it enabled may cause instability - here's one notable example that's affected some customers.
During your next scheduled BI Platform downtime, log onto the CMC, go to Servers -> Service Categories -> Web Intelligence Services, select each WebIPS, and in the Properties uncheck "Enable Memory Analysis".
XI 3.1 - set the Maximum Connections at 50 to start, increase slightly if necessary.
BI 4.1 - set the Maximum Connections at 200 to start, increase slightly if necessary.
Maximum Connections is generally determined by the number of active users you have on the system. If you anticipate about 200 active users on the system at any one time, then you can set the Maximum Connections to 200. But that might be problematic, since the WebIPS will keep a Connection open till idle timeout. So even if you have at most 200 active users, there might be slightly more than 200 open connections. Thus the recommendation to increase the number slightly above the number of active users.
The change from the older Sizing Guide is that, with testing, the "sweet spot" for the BI 4.1 was determined to be 200 maximum connections - that the WebIPS for 4.1 was more than capable of handling beyond 50 connections.
If you do have more than 200 active users, then bringing in a new machine into the cluster, to deploy new WebIPS, should be a consideration.
XI 3.1 - configure Output Cache Directory for all WebIPS on a common network share.
BI 4.1 - configure Output Cache Directory for WebIPS on local disk and do not use network share.
When the WebIPS first opens a WebI document, it has to unzip the wid file and parse the contents to build an internal representation of the document. It then caches this representation to disk, that it uses subsequently for further processing. This opening/unzipping/parsing does take a bit of processing. The good thing is that the WebIPS, when asked to open the same document, first checks to see if there's already a cached representation from when it opened the document earlier. If there is, it proceeds to process the cached version, so that it doesn't have to spend the time opening/zipping/parsing the wid.
Now, all WebIPS that share the same Output Cache Directory can share any cached representation stored to disk by any other WebIPS.
This improves performance on opening a WebI document on deployments where there are many WebIPSes running. A given WebI doc needs only be parsed once, then the cached representation can be used on subsequent requests for the same doc, regardless of which WebIPS the request goes to.
In XI 3.1, there were usually a lot of WebIPS running around. One per CPU core, one each for 50 active users. It wasn't surprising for me to work on systems with more than 16 WebIPS, clustered. If there were no cache sharing, then the high likelihood of hitting a WebIPS that hadn't already cached the document was high enough to bring down the performance.
But for BI 4.1, there would be far less number of WebIPS running on the system, since it's recommended to have one per machine, one per 200 active users. Instead of 8 XI 3.1 WebIPS, there would be 2 BI 4.1 WebIPS. Far less chance of a cache miss. Furthermore, since what limits BI 4.1 WebIPS is IO, every request to the cache would have a performance hit if the cache was located on a network share.
Because of this, the recommendation for BI 4.1 WebIPS is to configure the Output Cache Directory to fast local disk.
There are other considerations that may discourage setting the Cache Directory to a network share. There's an issue with older builds where, if the Cache Directory points to a network share, cleanup is not done and can fill up the disk (SAP KBase 2050700). Network access issues can cause the WebIPS to hang (SAP KBase 1757824). Network issues affecting cache access can even cause the WebIPS to shut itself down intermittently (SAP KBase 2057341) - I've resolved intermittent WebIPS shutdown issues by moving the cache to local disk.
All in all, it's better in BI 4.1 to have Output Cache Directory point to local disk.
If you do have a very complex and large WebI document that takes significant time for the WebIPS to parse the wid on open, what you can do is create a Server Group containing a specific WebIPS. Then all processing request for that WebI doc would preferentially go to that WebIPS and no other, so would consistently open from cache.
Rule: Revisit Sizing Regularly. And Remember that the Sizing Guide is a Starting Point - a Guide and not the Rule
Remember that usage of your BI system will change with time - new documents will be created, the amount of data reported on will multiply, and the number of users consuming reports will increase. That's a good thing, since that means your users think the BI system so good that they want to use more of it. But that also demands the necessity of continuously revisiting sizing: determining the number and size of documents being processed by reporting off of auditing and monitoring the load on your WebIPS.
Resize your system, at least once a year.
And remember that the BI 4 Sizing Guide is a starting point. The people who wrote the Guide are pretty sharp, and have done a lot of testing to make sure what they recommend is reasonable. But the tests they ran, the level of activity for an "active user" that they considered, the typical size of a "large" WebI document they used, will all differ from your actual deployment. Only you know what's going on in your system, and ultimately, sizing means tuning. Although you might size the system once a year, monitoring the heath of your system must be done continuously.
Rule: Split and Size the Adaptive Processing Server
This is a very important rule, and a subject unto itself. I'll leave this subject for my next blog entry.
I will say this: the out-of-the-box default deployment of BI 4.1 has a single Adaptive Processing Server that is inadequate for any purpose other than a POC on a single machine. If the APS isn't split and sized correctly (SAP KBase 1694041), your deployment will encounter issues.
Summary
The most important consideration that will ensure a stable and performant processing of Web Intelligence on SAP BusinessObjects BI Platform 4.1 is sizing. In this blog, I briefly covered sizing of the Web Intelligence Processing Server. Next blog, I'll briefly cover the sizing of the Adaptive Processing Server.
