Revisit the Sizing for your deployment of BI 4.x Web Intelligence Processing Servers!
Happy Holidays! Hope you’ve all had a good 2014. Before closing out this Year, I’d like to put up a reminder:
Revisit the sizing of Web Intelligence Processing Servers on your deployment of SAP BusinessObjects BI 4.1
If you’ve done so in the past eight months or so, that’s great! But pencil that task into your calendar for 2015.
If you’ve haven’t done so in a year, then I strongly recommend scheduling a sizing exercise for 2015Q1.
I support Web Intelligence. I’ve supported it since before BI 4.0 Ramp-Up, through its General Availability on September 16, 2011, and to now with BI 4.1
It’s been over four years, and through all the years, the three most important considerations that affect the smooth operation of BI 4.x WebI on your system is (1) sizing, (2) sizing, and (3) sizing.
If the sizing of your BI architecture is wrong, then you’re going to face issues with performance, stability and availability, if not now then in the near future as demand on your system steadily grows. You might even find yourself spending quality time with me on the phone, and one of the first things I’ll be asking you is “When did you last size your system?”
If the answer is before February 2014, then I’ll be pointing you to the Business Intelligence Sizing and Deployment page, where you’ll find the BI 4 Sizing Guide, updated February 2014. There’s been changes to the recommended configuration of Web Intelligence in the newest edition of the Guide.
Rule: Always start your sizing exercise using the most recent Sizing Guide
When you get to the Sizing page, I recommend you click the “Follow” page, so that you’ll be notified whenever the page is updated. Or go to https://service.sap.com/sizing and find the BI 4 Sizing Guide, where you can “Subscribe” to the Guide and be notified when it’s updated.
There were changes to the recommended configuration of Web Intelligence services in the latest update, so I highly recommend reading through the entire Guide, just to make sure your BI 4.1 deployment is following recommended practices.
I’ve spoken with the Platform Product Owner, Sada, and he anticipates no changes to the Sizing Guide until at least BI 4.2. But don’t take my word for it. Before starting any sizing exercise, make sure you go and download the latest version of the Guide.
Rule: Do not use XI 3.1 sizing guides or best practices to size BI 4.1
One time, a customer pulled up their architecture deployment sizing document for their BI 4.0 system and in there was a references to XI 3.1. Turns out they had been keeping the same sizing architecture since their XI 3.1 days. That definitely did not work, and they had lots of issues that mostly disappeared once they had redone their sizing exercise anew.
XI 3.1 and BI 4.1 are entirely different beasts, and trying to fit BI 4.1 components into a XI 3.1 architecture plan is like harnessing thoroughbreds to a dog sled. No matter how strong the beasts, they’re just not gonna perform well when they keep tripping over each other and getting into each other’s way.
The major differences between the two versions:
- BI 4.1 WebIPS is a 64-bit process. XI 3.1 runs as a 32-bit process, even on 64-bit machines.
- BI 4.1 WebIPS delegates query and chart generation to a different Service. XI 3.1 the generation is in-process.
Because of these differences, the sizing best practices developed for XI 3.1 is inapplicable, even harmful, to BI 4.1
Let’s walk through the best practices we had for XI 3.1, and compare to BI 4.1:
XI 3.1 – deploy one WebIPS for each CPU core on the machine.
BI 4.1 – deploy one WebIPS per machine.
There wasn’t a real good reason why you’d want to tie the number of XI 3.1 WebIPS to the number of CPU cores – the WebIPS threading model parallelized concurrent report processing requests just fine across multiple cores. Each thread would process a single document request, but the threads can run on different CPU cores. WebIPS does not impose CPU core affinity.
XI 3.1 being 32-bit meant that memory resource limited how many processing jobs a single WebIPS could efficiently handle. Back then, proper management of memory utilization was an important part of sizing WebI. Increased usage would quickly demand more processing power to be added to the mix, and because of the memory limits, the only reliable way was to add additional WebIPS to the deployment. Having one WebIPS per CPU core was just a pretty good rule-of-thumb when it came to deciding whether you needed to add additional machines to the system.
But BI 4.1 WebIPS is a 64-bit process, and has a much expanded memory space. If I had a nickel for every time I’ve had to recommend customers alter the memory handling setting for their BI 4.x WebIPS, then I could afford, well, not even a stick of gum.
As you can see in the Sizing Guide, the BI 4.1 WebIPS is limited by IO. You want to ensure the WebIPS have access to as much IO bandwidth as possible, and that means, especially on Virtual machine environments, having just one WebIPS per machine.
Very early on in BI 4.0, there was an issue with WebI failover across different machines (a problem with the Document Recovery Service), so I used to recommend two WebIPS per machine (failover of WebI Sessions on two WebIPS on the same machine doesn’t require use of the Document Recovery Service). But that time is long past.
XI 3.1 – Enable Memory Analysis
BI 4.1 – Do Not Enable Memory Analysis
(added 2015-01-16)
I’m adding this item since it does come up often enough, and I recently got confirmation from a WebI developer that this is what’s recommended now.
Disable the “Enable Memory Analysis” option for BI 4.1 WebIPS. It’s enabled by default currently (that will likely change soon), but it’s a holdover from XI 3.1 that’s not needed and may cause issues.
In XI 3.1, it was a noble effort to try and preserve the precious 32-bit memory space. Enabling memory analysis causes the WebIPS to reject any further incoming processing requests, other than saving the document, when its memory utilization goes above the “Memory Upper Threshold” value. The hope was that this would give Users an opportunity to save their work before the WebIPS starts to run out of memory. Beyond “Memory Maximum Threshold”, all requests were rejected in the hope that the WebIPS can recover before it crashed.
BI 4.1 WebIPS works in a 64-bit memory space and has been upgraded to behave much more gracefully when it does reach resource limits. The “Enable MemoryAnalysis” is a cure without a disease. In fact, keeping it enabled may cause instability – here’s one notable example that’s affected some customers.
During your next scheduled BI Platform downtime, log onto the CMC, go to Servers -> Service Categories -> Web Intelligence Services, select each WebIPS, and in the Properties uncheck “Enable Memory Analysis”.
XI 3.1 – set the Maximum Connections at 50 to start, increase slightly if necessary.
BI 4.1 – set the Maximum Connections at 200 to start, increase slightly if necessary.
Maximum Connections is generally determined by the number of active users you have on the system. If you anticipate about 200 active users on the system at any one time, then you can set the Maximum Connections to 200. But that might be problematic, since the WebIPS will keep a Connection open till idle timeout. So even if you have at most 200 active users, there might be slightly more than 200 open connections. Thus the recommendation to increase the number slightly above the number of active users.
The change from the older Sizing Guide is that, with testing, the “sweet spot” for the BI 4.1 was determined to be 200 maximum connections – that the WebIPS for 4.1 was more than capable of handling beyond 50 connections.
If you do have more than 200 active users, then bringing in a new machine into the cluster, to deploy new WebIPS, should be a consideration.
XI 3.1 – configure Output Cache Directory for all WebIPS on a common network share.
BI 4.1 – configure Output Cache Directory for WebIPS on local disk and do not use network share.
When the WebIPS first opens a WebI document, it has to unzip the wid file and parse the contents to build an internal representation of the document. It then caches this representation to disk, that it uses subsequently for further processing. This opening/unzipping/parsing does take a bit of processing. The good thing is that the WebIPS, when asked to open the same document, first checks to see if there’s already a cached representation from when it opened the document earlier. If there is, it proceeds to process the cached version, so that it doesn’t have to spend the time opening/zipping/parsing the wid.
Now, all WebIPS that share the same Output Cache Directory can share any cached representation stored to disk by any other WebIPS.
This improves performance on opening a WebI document on deployments where there are many WebIPSes running. A given WebI doc needs only be parsed once, then the cached representation can be used on subsequent requests for the same doc, regardless of which WebIPS the request goes to.
In XI 3.1, there were usually a lot of WebIPS running around. One per CPU core, one each for 50 active users. It wasn’t surprising for me to work on systems with more than 16 WebIPS, clustered. If there were no cache sharing, then the high likelihood of hitting a WebIPS that hadn’t already cached the document was high enough to bring down the performance.
But for BI 4.1, there would be far less number of WebIPS running on the system, since it’s recommended to have one per machine, one per 200 active users. Instead of 8 XI 3.1 WebIPS, there would be 2 BI 4.1 WebIPS. Far less chance of a cache miss. Furthermore, since what limits BI 4.1 WebIPS is IO, every request to the cache would have a performance hit if the cache was located on a network share.
Because of this, the recommendation for BI 4.1 WebIPS is to configure the Output Cache Directory to fast local disk.
There are other considerations that may discourage setting the Cache Directory to a network share. There’s an issue with older builds where, if the Cache Directory points to a network share, cleanup is not done and can fill up the disk (SAP KBase 2050700). Network access issues can cause the WebIPS to hang (SAP KBase 1757824). Network issues affecting cache access can even cause the WebIPS to shut itself down intermittently (SAP KBase 2057341) – I’ve resolved intermittent WebIPS shutdown issues by moving the cache to local disk.
All in all, it’s better in BI 4.1 to have Output Cache Directory point to local disk.
If you do have a very complex and large WebI document that takes significant time for the WebIPS to parse the wid on open, what you can do is create a Server Group containing a specific WebIPS. Then all processing request for that WebI doc would preferentially go to that WebIPS and no other, so would consistently open from cache.
Rule: Revisit Sizing Regularly. And Remember that the Sizing Guide is a Starting Point – a Guide and not the Rule
Remember that usage of your BI system will change with time – new documents will be created, the amount of data reported on will multiply, and the number of users consuming reports will increase. That’s a good thing, since that means your users think the BI system so good that they want to use more of it. But that also demands the necessity of continuously revisiting sizing: determining the number and size of documents being processed by reporting off of auditing and monitoring the load on your WebIPS.
Resize your system, at least once a year.
And remember that the BI 4 Sizing Guide is a starting point. The people who wrote the Guide are pretty sharp, and have done a lot of testing to make sure what they recommend is reasonable. But the tests they ran, the level of activity for an “active user” that they considered, the typical size of a “large” WebI document they used, will all differ from your actual deployment. Only you know what’s going on in your system, and ultimately, sizing means tuning. Although you might size the system once a year, monitoring the heath of your system must be done continuously.
Rule: Split and Size the Adaptive Processing Server
This is a very important rule, and a subject unto itself. I’ll leave this subject for my next blog entry.
I will say this: the out-of-the-box default deployment of BI 4.1 has a single Adaptive Processing Server that is inadequate for any purpose other than a POC on a single machine. If the APS isn’t split and sized correctly (SAP KBase 1694041), your deployment will encounter issues.
Summary
The most important consideration that will ensure a stable and performant processing of Web Intelligence on SAP BusinessObjects BI Platform 4.1 is sizing. In this blog, I briefly covered sizing of the Web Intelligence Processing Server. Next blog, I’ll briefly cover the sizing of the Adaptive Processing Server.
This is excellent. I've been out of sizing for so long myself, I hadn't realized the currently recommendation is one WIPS per machine in BI 4.1.
Thank you for posting this incredibly informative blog.
I've been out of it even longer. LOL Ted did a great job on this. As I was stuck in the old way of thinking how WIPS are deployed as well. Have a great day Carly and tell BT I said hello.
BW
this needs to be promoted everywhere !!
Excellent !! Needed !! Must read !!
Correct me if I'm wrong, but doesn't this contradict the blog post by Matthew Shaw: Performance Tips: Getting the most out of your Web Intelligence Processing Servers, more specifically regarding a single Web Intelligence server being able to take advantage of multiple cores?
Matthew writes:
Matt's blog - Apr 6, 2011 , that is 3 years of analysis difference.
I think Kristof's comment is still a valuable post, which points out an issue which can be found at every edge of the BI Platform Ecosystem: Inconsistent documentation.
I've asked Matthew several questions in the comments of the mentioned blog post in 6/2014 (after the sizing guide was updated) and one of them was the same what Kristof asked:
How does his golden rule match with the sizing guide?
Even if he didn't directly answer the question, he didn't revert his statement, even if it's completely contrary to the sizing guide's statement regarding the WIPS sizing.
And if you ask the same question on the SCN you get eventually a third opinion, provided by another SAP Employee. This happened to me in 2014.
(2 WIPS per Node, but the statement was not based on the document recovery issue Ted pointed out in this post).
In my opinion, if there is outdated information available in the BIP Ecosystem, it should be adapted to the new behavior of the product or should be deleted or at least marked as outdated/only relevant for release X.Y SPxx Patchyy, in any form.
Otherwise it's always hard for the customers to figure out which information is correct.
Krgds
Moritz
Hello Moritz,
Thanks for your comments!
The official documentation and the Sizing Guide has been very consistent for the BI 4 Product - some starting parameters have been upticked, but no truly significant differences as to architectural changes.
Recommendations one may receive on SCN or elsewhere would, of course, vary. But that's just because of the nature of the beast. Sizing is very deployment-specific and certainly not One-Size-Fits-All. What works for one will not work for all.
My point in this blog is (1) start with the official Sizing Guide - in my experience it's a very good starting point, and (2) revisit sizing periodically - it's a continual process.
Sincerely,
Ted Ueda
Hello Ted,
Thanks for your reply and this blog entry!
I'm afraid I can't agree with your opinion, because for me a statement contained in a guide-like blog post published on the SCN by a SAP employee, becomes a kind of official documentation and if such statements are contrary to the official product documentation/guides, they should be revisited, deleted or at least marked as relevant for certain releases only, as stated before.
Otherwise it's very hard for customers to know what to follow.
And Performance Tips: Getting the most out of your Web Intelligence Processing Servers is such a case for me.
Maybe this is not the best example, but there are some which make this issue even more evident.
Krgds
Moritz
My opinion is that I can't speak for other blogs, and that I've taken the opportunity in this blog to re-emphasize the importance of relying on official documentation.
Best wishes!
Ted Ueda
Hello Kristof,
BI 4.1 WebIPS scales. That's from both design and performance testing. The recommendation is to have one WebIPS instance and increase capacity outwards - by ratcheting up its Maximum Connections and Maximum Memory till it reaches the limits of the machine. Then, if more capacity is needed, add another machine with a WebIPS. It's good to have one there anyways, for failover.
Essential reading here:
http://scn.sap.com/docs/DOC-52621
The author Yumiko Hata's with SAP BI PnR, the group that does performance testing for BI WebI.
Sincerely,
Ted Ueda
I have to partly agree with Kristof and Moritz here, I have seen inconsistencies in what I have found in sizing guides, SCN articles by SAP employees, and what I have encountered in support calls. However, I fully support Ted's assertion that one must adhere to the official documentation. I define official documentation as the Sizing Guide, the Admin Guide, and the APS Best Practices doc.
Here is where I have seen inconsistency and I think there is good reason. I'll give my take on both.
The main inconsistencies I have encountered are in the area of how many WIPS to have on the landscape.
I have heard:
My take on this is that they all make sense depending on your situation and keep in mind every environment is different. This is why I am not a fan of the "out of the box" standardized installation. If you are not tuning your environment according to how it is used you are setting yourself up for a big disappointment and you are not taking full advantage of the power that the BI Platform has to offer.
What I would like to see from SAP is an official document or at least a vetted blog that discusses the integration of WIPS, AJS, and APS (DSL in particular). It should also cover when and if it is appropriate to add more than WIPS per node.on 4.x. In the event it does make sense, what are the proper settings that should be in place when running more than one. Again, remember the rule that 3.1 settings don't apply and you wouldn't believe the number of systems I have seen where admins have set things up based on their 3.1 knowledge. Prior to the release of the Sizing Guide and the APS Best Practices guide that's all early adopters of 4.x had to work with. So I understand those who based things on 3.1 knowledge back then, but this is definitely no longer the case!
I would agree in most cases a single WIPS per node (or two if you want to be extra safe in a high availability situation) will do. However, what about when you have an environment where your BW users, for example, are in love with really large record sets and there is no changing their mind to go with another tool? A document that focuses on Webi specifically and goes beyond just the WIPS settings would be awesome. Something that saves us the pain of going through multiple documents and putting it all together. Yes, perhaps I would like to be a little more lazy about it, or I would like to be able to easily demonstrate to support a solid design was done. Last thing I want to do is make a support engineer sift through the environment to arrive at the conclusion you need to add a WIPS and change the settings to X, Y, Z. Especially if I could have picked up a doc and quickly identified that one WIPS doesn't cut it my case.
I have personally found it to be a balancing act and running performance tests to benchmark your environment is critical to getting it right. However, its difficult with the info readily available to back up your plan/recommendations with documentation. I always like to be able to back up my recommendations with official sources and never rely solely on my understanding. I also like to be able to confidently encourage someone else to validate what I did and be able to say wow, you have a solid architecture design.
To illustrate my point a little more,
In Yumiko Hata's blog at http://scn.sap.com/docs/DOC-52621
he makes the statement "as far as Webi scales fine without degrading in reponse time with the above setting the tuning is done. Otherwise adjust the parameters to see if the response time improves. If your load is far over the capacity of one instance then reduce the concurrent users per instance by dispatching the load onto two, then three.. instances. Whichever the boxes, either on a single node or on separate ones, Webi scales in the same manner, i.e. when 1 Webi = 100 users 4 Webis = 400 users. While increasing the concurrency of Webi you may need to go back to tune others such as WebApp Server, DSLBridge, CMS/CMSDB or OS level."
Specifically, "If your load is far over the capacity of one instance then reduce the concurrent users per instance by dispatching the load onto two, then three.. instances. Whichever the boxes, either on a single node or on separate ones" supports the notion that there are instances where you should consider scaling out to multiple WIPS instances and the throttling back the settings on those instances. Some added guidance on determining when the "load is far over the capacity of one instance" would be helpful. There is definitely much to gain from the blog, but if this were discussed a little more in an official doc about when to "dispatching the load onto two, then three.. instances" and when not to do then as well as how to properly re-align the APS sizing and scaling it would be HIGHLY beneficial.
In short my opinion is not that the info is really inconsistent, but rather it is spread out, its not "one size fits all" and when you cross the bridge into a highly active system where users consume lots of data its way more challenging to get the scaling just right.
This is a really good and useful summary, thanks for it..
But why the hell should it be needed at all, at least for small installations? The default design is made for demos and test! The Configuration Wizard for the APS is a blessing, and it should be extensively extended on all this stuff. Any rule of thumb implemented in the wizard would be as good as all these hints tracked on SCN or BOB. The Apache split is supposed to be a huge boost, why is it not possible at install?
I'm not a Java guy, and my added value is in the datawarehouse & universe design or Webi development, But my customers have usually a small user base (<<100 users), and they don't want to pay yet another consultant for days just for installing the product and baby-sitting it for weeks. I'm the "BusinessObjects guy" so all these configuration tasks fall on me, and going through thousands of pages of administration is not funny when it is not your primary job.
3 years after BI 4.0 came out, I'm beginning to understand it, and documentation has improved, but administration and optimization tasks take a huge amount of time compared to XI 3.1. And the customers complain that BI 4.1 is huge, hungry and slow, or that it was created to generate billable consulting time. SAP is losing many small customers this way.
Hi,
These are good points, well made, and we (SAP) understand your pain. Unfortunately, it is difficult to be all things to all people, but we try to offer flexibility in the configuration options to allow for that.
However, one thing is for sure, with the proliferation of add-ons for our innovation products (and their respective installers), these are making the Patch Upgrade process complex .
My understanding is that we have an opportunity to improve here, with the advent of 4.2 coming in future. It is planned to revisit our installers - they may well be updated to bring gains in this area (i.e. merged installers, flexible installers) because 2 key themes for 4.2 are penned to include : optimised patch upgrade process , and minimised downtime .
That's all i can allude to at this stage!
Regards,
H
Hi Chris,
Perfect summary, thanks!
Especially the part about the BW users is very familiar to me...unfortunately.
Just to say that I am listening...
I did forward Chris' comment to Yumiko.
But to be fair to BI 4.1, with XI 3.1 you couldn't even come close to running very large data set reports. The answer then was simple (try adding more WIPS and redesign reports to fit in the 32-bit process space) but not entirely satisfactory. Once sent up and sized, people are much happier with BI 4.1 than the old version. The challenge is placed more on the administrator and installer, however.
Sincerely,
Ted Ueda
I dunno what you mean by "very large data sets", but 4 years ago I could manipulate 1 million lines with Webi in XI 3.1 with a fourth of the ressources (UNV, Webi, unoptimized datawarehouse full of delegated measures). It does *not* seem quicker now. The interface is rather sluggish, the RAM consumption is huge, and it is still full of bugs. And 4 years ago I could live with a default install for my small customers (No, I don't want to discourage, keep up cleaning and optimize all this, keep up the good work!)
the number of rows needs to be expressed for meaningful measurement
My perception - definitely not official development word - is that early BI 4.x focussed on throughput, and there's more focus on latency recently.
Hands down throughput and scaling of BI 4.1 goes far beyond that of XI 3.1
But I do understand your point - end users perceive latency more than they perceive throughput.
And bugs. XI 3.1 was far from perfect, but in 4.0 days we really wondered of we would be able to finish the project. Slowness was the least of our concerns. I still encounter problems with Webi that should have gone through QA. A slow product can be appreciated if he's almost perfect.
Anyway, I was not really aware of the latency/throughput difference, thanks for pointing it.
I do have to say for the added complexity, 4.x is way more scaleable than previous versions. Th only thing I would like to see is some more targeted direction towards Webi tuning rather than assembling the pieces from several places. In particular, the APS is covered well, WIPS coverage is fair (although it could be cleaned up a little to clean up some of the seemingly conflicting info out there--things have to be put in perspective though), AJS however is a bit lacking in terms of how to size and scale it accordingly IMHO.
Hello Chris,
Thanks for your comments - what kind of issues do you run into with AJS wrt sizing and scaling?
AJS itself is just there to ensure there's sufficient number of job server child processes up and running to take on the required number of scheduled jobs.
It actually doesn't do any of the processing work itself, it's sort of like a manager who manages other managers.
Not trying to deflect your interest, just that as a Support person, much of my work is on the WIPS and APS, not on the AJS.
Regards,
Ted Ueda
Understood on the fact that it doesn't really do any processing work. So I would referring to guidance on setting the proper number of concurrent jobs and children. I suspect most customers will try the trial and error approach here and something more definitive would be nice here. I've also seen discussions and even a couple of notes that will suggest splitting the AJS (e.g. one for scheduled report, one for Platform Search one for LCM, Security, etc.). However, most of the info I see points to no don't do this. So is it ok to split the AJS as long as your careful on what is included in each one, or is it an absolute taboo? What relationship is there between what you set for number of concurrent jobs for the AJS relative to its counterparts the APS and WIPS for example (I realize the analogy may be a bit off). Can I arrive at the right anwser by combining a little common sense and technical knowledge? Sure, but it would sure be nice to see it better defined and explicitly laid out as well as it is for the Sizing Guide and the APS Best Practices Guide.
While on this thought, there are a few other subjects where expansion would be nice. For example, it would also be nice to see the resource estimator updated to allow for some inputs on Explorer, the Analysis add-on, Mobile, etc. Also, Analysis for Office does place some demand on the Webapp Server and CMS, so some sizing info relative to that would be nice too. Again, with some common sense and technical knowledge I can arrive at the answers but people newer to the product and dealing with vast expansion of new applications and functionality may feel overwhelmed and not sure where to draw the line between what they should be able to handle on their own, what they need to bring in a consultant for, and what should go to support. Even as a consultant, I don't have quite the inside knowledge on the inner workings of things that someone within SAP does. Just the nature of the beast to some extent.
The level of engagement and quality of support has improved over the years for sure. I have way more ability to get my hands on the needed info that I ever had in the past. So I'll give credit where credit is due as well!
For the AJS, there's absolutely no benefit to splitting. The only thing splitting does is add resource pressure. For fault-tolerance or scaling up for increased concurrent jobs, add a AJS to another machine (and scale up the Processing Servers accordingly).
Explorer has its own Sizing Guide. Analysis OLAP the sizing knowledge is much improved, and Analysis Office there's no guidance, since it doesn't use the Platform reporting servers.
Mobile uses the same backend services and non-Mobile, so the Sizing Guide applies. But there's not much guidance for Mobile web app sizing.
You're correct that the main thrust of the Sizing Guide is with the report processing servers - that's been the significant challenge when it comes to the XI 3.1 -> BI 4.1 upgrade and that's where the focus has been.
Your point that the other aspects of the product need coverage is well taken.
Regards,
Ted Ueda
Hi,
While I understand that there is no reason to split the AJS from a performance / sizing point of view, there is from a administration point of view: splitting the AJS allows me e.g. to stop scheduled Web Intelligence reports from being sent out, while still allowing scheduled Program Objects to go through (just an example, there are other scenarios).
Another possibility is to set specific port ranges for the child processes to use, depending on the type of job server.
I'm sure that these reasons are not relevant in every environment, but when they are, it does make sense to split the AJS.
Kind regards,
Kristof Speeckaert
Hi Chris,
There is a kba available which tells that you shouldn't split the AJS:
SAP Note 1868751 - Sporadic and Random scheduling failures
But again: hard to find.
At the end I had to issue a support message to figure that out.
Krgds
Moritz
Also - Sizing Guide has a section on the Adaptive Job Server that recommends to not split.
There may be special circumstances in an organization that requires one to split - the analogy I usually make is that AJS are managers and APS are workers.
If the workload ramps up, one hires more workers and specialize them. Hiring more managers may slow things down.
Regards,
Ted Ueda
An unverified anecdote, but increasing the -Xmx on the AJS apparently helped a slow LCM job
Hi,
wonderful document covering many topics,
Thanks for sharing,
Regards,
Bansi
First, one of the excellent blogs I see.
I like to add my experience on WebiPS. We have a BO 4 SP5 environnement and try to reduce our number of WebiPS to maybe fix some issue and add stability.
So far , we didn't be able !!! We have 2 BO cluster with 5 webiPS each for a total of 10.
keep in mind if WebiPS is automatically restart for any mysterious reason (yes, it happen in BO, you have to agree!) , it will make all Job attach to it fails. So, the WebiPS need to be very stable.
Hi Ted,
Thanks for such a useful piece of information on WebIPS. Could you please provide us some such information on AJS. I mean how much we can increase the number of concurrent jobs, how can we zero in on a number etc. Anything on AJS will help us a lot.
Thanks in advance.
Aqueel
Hi Ted,
- We have a big deployement with big documents. We have now stability problems ( a lot of Teradata Internal error). We have 2 servers with 2 services Tomcat by server, and 1 server with one service TOmcat.
You don't speak about rules about services Tomcat ?
- The services that take less refreshs are more stable or server 3 is more stable ( with one service Tomcat). I don't know exactly the reason of instability. What do you think about that ?
- We have 5 WebiProcessing server by server and one APS.Webi by server. What do you think about that. The cache document is stored locally.
Thanks for you help
Laurence - if you have teradata errors, they are in no way related to Tomcat servers.
Sizing Web tier is covered in sizing documents you can find here:
Sizing and Deploying SAP BI 4 and SAP Lumira
Hi Ted (or all others), I came to read this threat last Friday.
I'm specially interested of the recommendation "BI 4.1 - deploy one webiPS per machine"
Why my question :
we are running 4.1 SP6 since 19-09-2015 (before, from 07-03-2015, we were on BI 4.1 SP5.2)
We have 4 app-servers, a separate DB-Server (CMS and Audit), and a separate FileStore-servers. All on the same VLAN, all on VM's. All with 6 CPU's and about 28 GB of memory.
Since several months, we have regular WPS-Crashes. A ticket is open @ SAP-Support for months as well, but without real progress.
We have tried many things (we have ourselves about 20 years of experience with supporting quit large BO-environments (200 --> 1000 users), but we cannot find the root-cause.
So I came to this threat and saw this recommendation of having only one WPS on each node. Now we are having on 2 mode 2 x WPS'es, and on the other ones, only one. I wonder whether this really can be the reason. Note that we have multiple WPS'es on each node already from the moment we have upgraded from XI 3.1 to BI 4.0 back in September 2012. From Sep-2012 to about 5 months ago, we did not had these WPS-crashes.
Any remarks and/or suggestions are more then welcome.
Hi Eddi,
...Long time no see 😉
I would still recommend 2 WPS per machine. I had the case that this service will be used by only using semantic layer RESTFul and we had a better experience with the existence of multiple WPS, by having a higher load. But only my 2 cents
cheers
Sascha
Hi Sascha. Indeed, nice to hear form you again 🙂
Meanwhile we have been informed by SAP that our issue is a known bug. Which will be solved in BI 4.1 SP7. But they will also provide a kind of hot-fix for it (a "Pilot Note"). We are waiting for it. Als for the SAP-Note/KBA number which describes this issue.
Anyway, thanks for your answer. We will consider it anyway.
please keep us updated with the Note/KB number, when available.
Brand new info from a few hours ago : See SAP-Note 2213780
Since your WIPS are now not recycling to often anymore,
you could run into memory leak issues, if you are using Excel exports.
We had this issue in our deployment.
See following KBA:
2222230 - When exporting a Web Intelligence document to Excel after a refresh/schedule, the memory of the Web Intelligence Processing Server keeps growing
I would recommend to monitor the memory usage of your WIPS(s) for some days after deploying this critical fix.
Best Regards
Moritz
Just to be clear, the issue fixed by 2213780 didn't cause WebI PS recycling but a process abort. Failover mitigated, but didn't correct under load. Load testing uncovered the issue.
The issue with Excel export was introduced back in BI 4.1 SP04 Patch 7 and propagated forward to the other SP's. For that as well, normal recycling mitigates but under load, when there's less opportunities to recycle, the memory footprint can grow.
But it is indeed recommended to monitor the size of the WebI PS.
Regards,
Ted Ueda
Hi Ted,
Thanks for the info.
Yes, you're right. It were real crashes in our case, no recycles.
I just wanted to stick to the terminology of KBA 2213780,
even if I was wondering that it's mentioning "recycles" 🙂 .
Best Regards
Moritz
The terminology is indeed unfortunate - it retained the title with the original understanding of the issue, and not the later correct understanding that led to the solution.
Regards,
Ted Ueda
I recall internal discussions for 2213780 - it wasn't sizing related, but increased load did increase the chances for encountering the issue.
Regards,
Ted Ueda
Thanks Ted. We will install from tomorrow first on our Sandbox, then on DEV-Environment, then on QA. We'll keep SAP up-to-date through the ticket we logged. And I will update the discussion as well.
Unforntunately no progress today . It appears SAP-Note 2213780 is in status "Document is not released" since yesterday-evening, although it was available yesterday in the afternoon. We suspect somebody from SAP has it open. We have asked though our open incident to release it again but no reaction so far 🙁
It should be public again, it was a version update upcheck.
For anyone else reading this, please do note that SAP Note 2213780 is not a regular fix release deliverable. Please don't apply to your system unless you have confirmation that you are afflicted by the problem described. Any fix described in that Note will be streamed into the next Support Package (SP).
Regards,
Ted Ueda
After doing all kind of tests on the other environments in the second half of last week, we have implemented SAP-Note 2213780 last weekend on the Production-environment. So far all works very well. So it seems this Pilot Note solved our issue.
Plan is now to install SP7 somewhere in Dec/Jan.
Ted, as informed on 27-10-2015, we implemented the Pilot Note on our Production-environment.
From that moment, we did not had these WPS-Crashes anymore and our environment was/is very stable.
When I saw SP7 was released, I immediately looked into the Fixed-issue list, but saw this SAP-Note 2213780 was not mentioned 🙁
So I logged ticket 901895/2015 with the question whether this fix was yes or no in SP7.
Until now, I have no sufficient answer. They first informed me :
"It was integrated with an internal bug and therefore does not show up in the notes but it is included."
Then I asked about the SAP-Note of this "Internal bug", and they informed me it is 2203828 . But also that one is not in the fixed-issue-list 🙁 .
So I still don't know whether it is "save" for us to install SP7 or not, as I do not want to have this WPS-Crash again.
By the way, the original 2213780 has again the status "not released".
If possible for you, can you please inform me what do you know about this WPS-Crash fix and the Pilot Note we have installed ?
Many thanks !
Eddy
Hi,
Note 2213780 is not for public consumption - it is a 'private fix' which is issued by product support in exceptional circumstances according to process.
As i'm sure your support engineer told you, this fix has been ported to SP07. The Note above is for customer escalations who are not on SP07
Regards,
H
Thanks Henry for your prompt answer.
Yes, the support engineer told me that it is ported to SP7. But there is no prove at all. Even 2203828 is not in the fixed-issues list.
So I have no prove at all to tell my customer to go for SP7. If we do all the effort to upgrade to SP7 on our 4 platforms (SandBox, Dev, test, Production), and afterwards we have the WPS-Crashes again, we have a big issue (and have to roll-back again).
That's why I would like to be absolutely sure that this fix is included in SP7.
Thanks a lot
Eddy
Hello Eddy,
Just to let you know, the request is working its way forward internally, and it's been noted there's an error in the SP7 Fixed Issues document generation that dropped listing of that SAP Note.
I don't have timeline for when SP7 Fixed Issues document would be corrected, but I think if you have an Incident, then you would be notified through it when it is.
Regards,
Ted Ueda
Thanks Ted.
Yes I have an open incident. Through that one, I have asked to update me once the fixed-issue list is corrected
Hi Ted,
We are at BI4.1 SP7 patch level.
Is the recommendations still valid..?
BI 4.1 - deploy one WebIPS per machine.
BI 4.1 - Do Not Enable Memory Analysis
BI 4.1 - set the Maximum Connections at 200 to start, increase slightly if necessary.
BI 4.1 - configure Output Cache Directory for WebIPS on local disk and do not use network share
The recommendations are still valid for your edition of BI Platform.
Regards,
Ted Ueda
Hello Ted,
We are experiencing similar issue in our environment with WPS crashing. We set our Memory Lower Threshold 3500MB and Memory Upper Threshold 4500MB, and Memory Maximum Threshold 6000MB.
Can you provide us with fix 2203828? Or do we have to open a ticket and request it through the support rep?
Our Env: BI4.2 SP3 ; 32GIG RAM; WINDOWS 2012 R2
Thank you
Hello,
Haven't gotten a response, however we increase our hardware configuration: 48GIG RAM; 8vCPU; virtual server - WINDOWS 2012 R2. It remedy our single WPS, CMS, iFRS, oFRS, etc from crashing. However, we talked to another well verse SAP support Rep that recommended a second WPS in our node that is a stand alone. What are your thoughts?
Thank you ahead.
Hello Trieu,
Consideration for additional WIPS on the same machine should be driven not by load balancing considerations, but failover considerations, in case a WIPS crashes.
Best Regards,
Ted Ueda
I just have to say that after 15 years of running over 1000 different Crystal Enterprise and BOE Enterprise clusters for companies all over the world, I have found that some of these Web Intelligence sizing recommendations are not wise on MS Windows. Running a single WIP PID and Increase max connections beyond 200 never works out. Disabling "Memory Analysis" is a poor decision. One bad report or big report can cause the WIP to lockup and stop responding to all users. Happens all the time. I still run multiple WIP with 30-50 max connection and keep memory analysis enabled. If memory is running out, I increase the ranges to accommodate. Normally, memory never runs out if you deploy multiple small max connection WIP.
Hi Friends,
We are at BI 4.2 SP4 with 2 nodes (6 CPU & 44 RAM). We want to cross check on the recommendations are still valid..?
Did you even read the blog ?
Because your #1 is directly against what it says.
and you realize that you just allocated 20+gb of memory just to webi out of 44gb on each node ?
We have migrated to BO4.2 SP3 patch9 from 3.1 SP5.3.
4 nodes cluster(8cores 32GB RAM) each. triggering the reports from Java SDK in both the env gives different result. 3.1 completes around 8-9k reports in an hour while 4.2 completes 5k. Both are having same set of reports and pointing to same DB. Adding/removing WIPS on each node doesn't help. Even with one WIPS per server gives a similar result. What else can I tweak in my WIPS to enhance the performance in 4.2?
Hi,
On a customer's system (BI 4.2 SP5 Patch 4) there are 8 processors. In task manager it says "8 sockets, 8 virtual processors". A WIReportServer.exe never takes more than 13% (12.5%) on their system.
Should the advice be 1 x WIPS per processor, rather than node (what you say) or core (what Matthew Shaw says)? Because if I follow your advice, Web Intelligence processing will never take more than one eighth of the processing power available. In my case, these would appear to not be 8 cores of the same processor, so can't share.
Thanks
Darren
Hi Darren ,
did you ever had an answer on the above ? I just re-read several blogs and SAP-Nodes about how many WIPS / core or WIPS / node an environment should have.
Reason : a few weeks back we moved to a completed new environment : New VM's and BI 4.2 SP7.5. We have 5 nodes, each 40GB memory. One WIPS on each node
Before, we were on old-VM's (slower ones) and BI 4.2 SP6 (also 5 nodes, also each 40GB memory) . On this environment, we did not had any issues with WIPS-servers. We also had only one WIPS on each node
But on our new BI 4.2 SP7.5 environment (so with new, faster VM's) we have regularly 100% memory-consumption of the WIPS . With as result the full node out of memory and SIA stops.
We suspect a memory-leak (or not recycling) bug in BI 4.2 SP7.5 WIPS (as happened on older releases/patches) as well. Although we do see the WIPS recycling regularly : Idle Document Timeout (seconds): is set to 3600 ; Maximum Documents Before Recycling: is set to 50; just as it was on the old environment)
We logged a SAP-ticket for this and the support-engineer recommends us to add one WIPS on each node (so having 2 instead of 1). But I wonder if this is a good idea. We were working for years with one WIPS / node (on BI 4.2 SP6 and earlier). So I don't see immediately why to increase.
Eddy