Skip to Content

There are some quick and easy ways to get additional power out of your Web Intelligence Processing Servers.  Most customers don’t bother changing their servers’ configuration but they should. This blog will highlight some of the parameters you really should change.  This is my first blog about performance tuning SAP BusinessObjects Enterprise, more to follow.

Number of instances to run

To benefit from maximum scalability, set the number of instances of this service within a machine (or virtual machine) to be atleast the number CPU cores (or Virtual cores) available to it. If the machine has 4 cores then you should haveat least 4 instances of Web Intelligence Processing Servers running. This is a golden rule not to break!

One ‘Web Intelligence Processing Servers per CPU core per machine’ also helps with load balancing. Let’s say you have 2 machines, one machine of 4 cores, and another machine of 2 cores. You’ll want the 4 core machine to receive twice as much Web Intelligence ‘work’ as the 2 core machine.  To do this you just allocate twice as many Web Intelligence Processing Servers to the 4 core machine compared to the 2 core machine.  And if you follow the one ‘Web Intelligence Processing Server per CPU core per machine’ rule, that’s exactly what end up with! Perfect.

Maximum Connections

Sometimes, however things are not so straightforward and you want a bit more fine tuning without breaking the golden rule “One Web Intelligence Processing Server per CPU core per machine”. For example when the ‘power’ of your machines isn’t determined by the number of CPU cores, but by something else like faster CPUs, what do you do then?  This is when the Web Intelligence Processing Server property ‘Maximum Connections’ should be used to ‘balance’ the load.  Let’s say you have 2 machines, both have 4 cores, but one machine runs at 2.0 GHz and the other at 2.5 GHz (25% faster). You want the 2.5 GHz machine to receive 25% more ‘work’ than the 2.0 GHz. Simply by increasing the ‘Maximum Connections’ for each Web Intelligence Processing Server on the 2.5 GHz machine by the required ratio will result in the load balancing to be weighted towards the 2.5 GHz machine. In this example we would increase this parameter from 50 to 63 (50 + 25%= 62½, round up to 63). Don’t be tempted to reduce this figure, as if there aren’t enough connections a user will get “server busy” messages. You don’t want to add unnecessary software limits on your hardware resources. There is a school of thought that you should drop this parameter to ‘protect’ the server, I don’t follow that school, and would prefer to let the server do as much as it can, potentially giving a slower response time when busy, rather than generate an error when it’s not busy at all.

Caching

There are many wonderful ways to configure the cache to suite your environment. Let’s have a look at the main cache properties on the Web Intelligence Processing Server one by one:

The property “Enable Document Cache” is correctly enabled by default, make sure it stays enabled.

Document Cache can have a very significant performance improvement as experienced by end users. This feature, when enabled, allows a Web Intelligence Job to ‘prime the cache’. The cache is primed when a Web Intelligence job is run, as scheduled, and the presentation view (Microsoft Excel®, Adobe® Acrobat® and HTML, but it’s actually XML) is also generated as part of the job, though the down side is the job will take longer to complete. The cache is generated after the document has been refreshed and in the chosen formats selected.  When scheduling a Web Intelligence document it is important that the cache option is selected to benefit from this feature.  Enabling this feature can provide significant performance improvements.

The property “Enable Realtime Cache” is correctly enabled by default, make sure it stays enabled.

Realtime Cache also has significant performance improvements when enabled. Realtime cache allows the cache to be loaded dynamically. When something is not found in the cache, it is calculated and then saved into the cache for later re-use. This applies for scheduled as well as non scheduled documents.

The “Maximum Document Cache Size”, the default settings is 1048576 K Bytes, or 1 G Byte.

This is typically quite small for most organisations. With a cache size of just 1 G Byte it will not take long for a document that is in cache to be pushed out of cache. This will mean users could experience a good performance one minute and slower the next. I recommend increasing this size significantly. A cache size of 20 GB or even 50 GB is not uncommon. Confusingly the ‘Maximum Document Cache Size’ really isn’t the maximum size, the product doesn’t check, every time its writes a file to cache, the size of the cache! This parameter is used just as a value when the clean-up route runs.  Estimate the amount of cache by clearing out the cache and seeing how much disk space is consumed once a number of documents have ‘primed’ the cache (see above about ‘priming’ the cache).  Carefully monitor the disk space after making such alterations! You don’t want to run out of disk space.

The “Document Cache Duration”, the default setting is 4320 minutes, or 3 days.

This means the cache for a given document is valid for 3 days, which is ok if your documents are scheduled at least once every 3 days. But if you’ve got weekly or even monthly documents, then you should consider increasing this duration to a week or a month, otherwise you won’t be reusing perfectly valid cache content. You want to avoid any extra regeneration of cache as possible.  If most of your documents are scheduled weekly, then a setting of about 8 days would be appropriate. The server is never going to provide a user with cache content that is invalid or out-of-date.

The “Document Cache Clean-up Interval (minutes)”, the default setting is 120 minutes, or 2 hours. (The ‘properties’ page incorrectly says “(seconds)”)

This is how often the process will scan its cache and reduce the size of that cache if it’s too large. It will delete the oldest files first, and reduce the total amount of space to ‘Maximum Document Cache Reduction Space % of ‘Maximum Document Cache Size’ (or 70% of 1 GB with default settings).

So the cache is reduced to a % size of the maximum. Thus when you are determining what the Maximum Document Cache Size should be, you need to add on a fair bit as only 70% of it will kept after a clean-up.

There’s no harm with a short setting (2 hours), but you could probably increase this dramatically if you have plenty of disk space and aren’t worried about the cache size growing well over its limit.

Sharing the Web Intelligence cache across multiple machines. Not set by default.

There’s not much point in each Web Intelligence Processing Server having its own cache when you can share. Sharing a cache has huge benefits; it means one Web Intelligence Processing Server can re-use the cache generated by another. And if you’re ‘priming’ the cache then this is even more important to gain from that extra effort your jobs are performing.

To share the cache the Web Intelligence property ‘Output Cache Directory’ should be set to a shared cache folder, typically hosted on network storage device which is accessed via a UNC or NFS path. You’ve probably already got a network file share for the File Repository Server (FRS) file store. Why not just add another folder onto that share for the Web Intelligence document cache. Set the ‘Output Cache Directory’ property to it. (If running on Microsoft Windows you’ll need to make sure the Server Intelligence Agent is running as a user who has network access).  Sharing the cache across machines can provide massive performance improvements, but watch out for network or disk bottlenecks. If you really want to get the most throughput, which might be important if you’re ‘bursting’ lots of documents via a publication, enable a network share and a separate disk system for each of: Input FRS, Output FRS, Web Intelligence Document Cache.

Customers real-life experiences of sharing a cache are mixed. Some customer report benefits, whilst others report issues generally around the stability of opening and viewing documents. Whilst there’s no conclusive evidence, customers with issues use a network share that isn’t as quick to respond as local disk. For example the latency is high (over 3 ms) and/or the disk is just slow to either create a new file, or read an existing one. So if you enable a shared cache, be careful to monitor any instability. Any issues rollback. When the ‘Output Cache Directory’ is left blank, then each machine will use a single cache folder for all the ‘Web Intelligence Processing Server’ running on it. If you like, there’s always a cache that is shared, but by default only with the ‘local’ Web Intelligence Processing Servers. Setting the ‘Output Cache Directory’ to a true common shared folder means its shared more.  When the ‘Web Intelligence Processing Server’ starts up it calculates how much cache is being used. Prior to about BI4.0 Patch 6.7 and BI 4.1 Patch 1.3, this calculation had to be completed before the process accepted work (it would say in ‘initialization’ status and not move to ‘running’). After these releases a separate thread is doing this task allowing the process to enter ‘running’ state much sooner.

The “Universe Cache Size” the default value is 20 universes.

Universes that are pushed out of cache will require extra processing which is unnecessary. Set this to at least the same number of universes you have, or even a few more! It’s only a tiny bit of extra disk space and setting this to a large value won’t affect the amount of RAM consumed.

Making changes + feedback

As with any changes you make, put them through change control so you can rollback should you get any ill effects. Having said that, I strongly believe that the above settings are pretty universal.

Adobe and Acrobat are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Windows is a registered trademark of Microsoft Corporation in the United States and other countries.

To report this post you need to login first.

47 Comments

You must be Logged on to comment or reply to a post.

  1. Greg Myers
    Love the admin-focused post. This is the type of stuff people only ask about when things are broken.

    Love the ‘golden rule’. I never heard that one before and will be making some adjustments to my environments based on it.

    I’ll also have to look at the Maximum Simultaneous Connections setting, too. I was advised to set that lower a few years ago to “protect the server” and haven’t really looked at it again since.

    Would love to see you do a performance-related blog on Universe settings sometime soon!

    (0) 
  2. Jason Cao
    Hi Matthew,
    I’m glad you’re able to share these tips with the community, and even happier to see experts in our community acknowledge the value of such content. Great job!
    Regards,
    Jason
    (0) 
  3. Mirko Langhorst
    Hi,

    great post so far. Very helpful in getting more internal details about how the platform is working. One question: is the setting for the cache size, e.g. 50 GB, per WebI server or in total? We set the cache to a NFS in share the cache directory between 4 WebI servers.
    Are you going to release performance tips for Crystal as well?

    Regards,

    Mirko

    (0) 
    1. Matthew Shaw Post author
      Many thanks for the feedback, from you and many others.

      Q1: Is the setting for the cache size, e.g. 50 GB, per WebI server or in total?
      A1: If the cache is shared, then the size of the cache size is the total cache size. If the cache isn’t shared, then the size is per Web Intelligence Processing Server.

      Q2: Are you going to release performance tips for Crystal as well?
      A2: I was thinking of writing a blog along the lines of servers in general, including Crystal and Job Servers for Webi and Deski. There are just too many things to blog about… so your feedback will help me prioritise! 🙂

      (0) 
  4. Christian Key
    Hi Matt,

    Thanks for the tips. I have a few comments I’d like to share with regard to this topic.

    We host many customers on our BOE XI 3.1 SP3 platform, each customer having access to their own source data database via a Universe using access restrictions to control which connection they use. We also have row-level security enabled using @BOUSER inside the Universe to control the data that individual users are allowed to see in the reports. We also use custom access levels in the CMC to control which reports they can access.

    Each time a customer runs a report the results will be different depending on which company they work for and the data that they are allowed to see. Therefore caching reports is a meaningless exercise is my opinion in this scenario. I have disabled the real-time cache as the report will be different each time it is run depending on who you are.  I have also disabled the document cache for the same reason. As a result I have noticed siginificant performance improvements on our server each time a report is run.

    If you have any suggestions on how we can schedule and cache reports which contain data which is specific to the user running the report maybe by using report bursting then that would be very useful.

    I also have noticed that the ‘Document Cache Cleanup Interval (seconds)’ value in the Webi Processing Server properties window is referred to as ‘Document Cache Cleanup Interval (minutes)’ in the Webi Processing Server metrics window and posts on the BOB forums have suggested that this value is actually in minutes and not seconds. Similarly ‘Cache Timeout (minutes)’ in the properties window is referred to ‘Cache Timeout (seconds)’ in the metrics window. If you could confirm whether the values are stored in minutes or in seconds then that would be very useful too.

    Thanks for your blog.
    Christian Key

    BO Developer
    e2train Ltd
    http://www.e2train.com

    (0) 
    1. Matthew Shaw Post author
      Many thanks Christian,
      You’re right there are some typo-bugs in the User Interface. To help:
      “Document Cache Cleanup Interval (seconds)” on the properties screen should read “Document Cache Cleanup Interval (minutes)”, as it does in the metrics screen. (I’ve updated by blog to make it more obvious, thank you on behalf of other readers
      And “Cache Timeout (seconds)” on the metrics screen should read “Cache Timeout (minutes)” as it does on the properties screen.

      You raise a great point, caching is only good for when the content is the same for all the users that access the document. For you it appears ever user will have a different security profile and so the cache will never be re-used. So you’re right there’s no point generating cache and you’ve done the right thing by disabling it. Disabling the cache from being ‘primed’ or ‘pre-loaded’ during the scheduling process will naturally improve the time it takes for the scheduler to process it.

      For a cache to primed, both the “Enable Document Cache” must be enabled on the Web Intelligence Processing Server and on the document schedule (cache option “Select the Formats to Pre-Load the Cache with when Scheduling (only applicable if scheduled to Web Intelligence format)”). Now you’ve gone for the global option by disabling it on the server, which is probably a good thing as you want to disable this for all your documents. If however you have some documents that you want to prime the cache with, and others not, then leave the “Enable Document Cache” selected, and just select, or not, the pre-load cache option when scheduling documents.

      You’ve also de-selected ‘Enabled Realtime Cache’ because you want to disable cache for all documents, and that’s fine for this use-case where all users see different content. I would hope by de-selecting this you didn’t see a huge performance improvement, it really shouldn’t take any extra effort to save the output to a cache folder (as long as the disk is nice and quick).

      (0) 
  5. Mike Saunders
    Hi Matthew

    Very interesting reading. I have always worked according to the golden rule and it’s good to see it again here.

    Unfortunately there seems to be conflicting information in the current BO_XI_3_1_Companion.pdf sizing document, or am I misinterpreting what is being said?

    Will look forward to your future blogs.

    Regards

    Mike

    3.1.2.14.1 Service Requirements
    The Web Intelligence Processing server will  expand to as many CPUs as needed and can process
    several documents in parallel, so 1 Web Intelligence Processing server is sufficient per machine. To
    avoid reaching the 2G memory per process limit when running large reports, you may consider
    running more than one Web Intelligence Processing server on the same machine.

    (0) 
    1. Matthew Shaw Post author
      Hello Mike,

      Sadly there is some confusion within that document you mentioned.

      “The Web Intelligence Processing server will expand to as many CPUs as needed and can process several documents in parallel, so 1 Web Intelligence Processing server is sufficient per machine.”

      This isn’t correct. It is true however for Desktop Intelligence!

      “To avoid reaching the 2G memory per process limit when running large reports, you may consider running more than one Web Intelligence Processing server on the same machine”

      This is correct. If you find that the memory consumption for your Web Intelligence Processing Server is heading toward 2 GB, then you could add some more processes (but unlikely if you’ve got 1 Web Intelligence Processing Server per CPU core). More Web Intelligence Processing Servers will spead the documents (and thus memory consumption) across more processes.

      Thanks for the feedback
      Matthew

      (0) 
  6. Denis Konovalov

    In XI3.1 with regular Webi reporting (not dashboards) I found it very hard to have more than 25 Max Connections because Webi is just not stable enough at higher numbers.
    I found it is better to have more WPS instances with <=25 Max Connections each versus WPS with >25 Max Connections.

    When majority of reporting is dashboards, as in small amounts of data that refreshes very fast – yes 60+ Max Connections WPS’s make sense.

    (0) 
  7. Chandrakanth Angannagari

    Hi Matthew,

    This blog does have some really good points, but is there a SAP official document on this? i tried looking for SAP notes (there are too many) . If you could point me to some where i could get all the information in this blog

    I am especially curios about the golden role. one Webi processing server per cpu seems like a lot of webi servers. is this rule only to eb applied when there is a LOT of webi usage? or even with a scanty usage?

    (0) 
    1. Matthew Shaw Post author

      Thanks for the feedback. I’m working with my colleagues to get this type of advice in the official documentation.

      A single Webi process can only be processed on a single CPU core at any one time. It cannot be processed on multiple CPU cores at the same time. A webi process will be processed across all the CPU cores within the machine, which will be determined by the Operating System. Thus, if you had a 4 CPU core machine, and 2 webi processes, then the very maximum overall CPU that could be allocated to processing Webi documents would be 50%. You would still see each webi process being processed across all the CPUs, but the very maximum overall CPU for webi could never exceed 50%. If you want more of the ‘CPU’ you need more webi processes.

      You need 1 webi process per CPU core to gain maximum scalability, that doesn’t necessarily mean you have to have 1 webi process per CPU core for ‘every deployment’. For example, let’s say your BI platform is processing all kinds of content, not just webi documents. Your system is then unlikely to be in a situation where it is JUST processing webi documents. It is likely to process many other things too (Crystal Reports, Dashboards, Data Federation, Explorer, Mobile ….). For these situations there would be no gain in one webi process per CPU core.

      If you had a 64 core server, then it would be unlikely you would need 64 webi processes as the system will need to be processing other stuff too.

      So the Golden Rule is a good rule, applying it for all situations and all deployments may not always be necessary. Any shortfall of the number of Webi processes will mean a loss in scalability (support less users)

      The Golden rule is still applicable for BI4 as for XI3, but not quite as ‘strong’ I would say. You see the added benefit of multiple webi processes is that there will be less user sessions allocated to each webi process (the load balancing should roughly evenly distribute users across available webi processes). This in turn would mean, in general, each webi process would consume less memory (less documents open per process on average). For XI3 this was a great benefit, since it was a 32 bit process it would soon reach the 2GB limit for Windows (different limits for other OS’s, but still low). This would then mean the process would try to garbage collect (consuming more CPU, slowing down processing of the documents) and potentially blow up and fail. So, for XI3 you would not only need multiple webi processes from a ‘processing/scalability’ perspective, but also from a memory management perspective too.  With BI4, a 64 bit process, this is no longer the case. Thus the Golden Rule for BI4 is still valid, but now only just from the ‘processing/scalability’ perspective (because a single webi process can only be processed on one CPU core at any one time)

      How many webi processes should you have? Determine/estimate the ‘maximum’ processing power required of the system to process Webi documents. If you estimate 50%, then allocate 1 webi process for every 2 CPU cores. If you estimate 75%, then allocate 3 webi processes for every 4 CPU cores. Etc.

      Got enough webi processes? Monitor the CPU usage of all the webi processes, if ALL of them reach 100% of the CPU they are allocated then you need more. We would always expect a single webi process to consume up to 100% of the CPU it has been allocated; the scalability of the system is restricted when all of webi processes reach 100%.

      Hope this helps explain a few things.

      (webi process= Web Intelligence Processing Server)

      (0) 
  8. Chandrakanth Angannagari

    Hello Matthew

    Thanks a lot. Was very nice of you to take some minutes to answer in detail.

    Would be eagerly waiting for this official SAP doc

    we have one webi and one APS hosting DSL. When two users run the webi reports, we see in our process explorer that the  all 4 cores seem to be equally used. so just wanted to confirm that a webi process uses only one cpu core and that this load is not distributed accross all cores.

    Also the concept of webi cache is not very clear for me. i thought crystal and dashboard have cache servers which is well documented. but webi has a cache? when does it come into play? when a user views a webi report ? cache is bypassed when webi is refreshed?

    Also, the logic for scalability in this case should apply for any BO processing server correct? would APS hosting DSL also need to be made scalable taking into consideration the cpu/cores?

    Thanks

    Chandrakanth

    (0) 
    1. Matthew Shaw Post author

      Hello Chandrakanth,

      Thanks for the feedback.

      we have one webi and one APS hosting DSL. When two users run the webi reports, we see in our process explorer that the  all 4 cores seem to be equally used.” That’s right, you will see the Operating System allocate the 2 processes to all the 4 cores. You’ll get an even load, about 25% across all the cores.

      However (at least for Web Intelligence) only one webi process can be processed on one processor at any one time. A processor will process more than one webi process, but not exactly at the same time.(A processor, can only process 1 thing at any one time)

      To put it another way, one webi process could not consume 100% of more than one processor at any one time. For example, it could not consume 100% of each of the 4 CPU cores. The maximum ‘total’ CPU across all 4 cores, would be 25%. It could never get to 26%, let alone 100% of the ‘total’.  This is the reason why you need more than 1 webi process.

      I believe the same is true for almost all the processes we have (CMS, APS, AJS), but I don’t know that for sure. I would have hoped the APS/AJS does not have this behavior. However I would make the assumption that it does. I do not know of single process (on the BI Platform) that supports M:N (Hybrid threading), my understanding is that most (if not all) support N:1 User level threading. Certainly parallel processing isn’t supported  (Grid processing is supported with Data Services).

      Some of our processors, spawn ‘child’ processes, so each child does the ‘work’ as told by the ‘main process’. You will have as many ‘childs’ as there are jobs. This means the ‘main process’ doesn’t have a problem ‘scaling’, just one ‘main process’ is enough regardless of the number of CPU cores. This ‘main process’ to ‘child process’ relationship is ‘similar to’ M:N (Hybrid threading), but at the process level, not the thread level.

      Webi cache- Yes, Web Intelligence has a cache that can be shared across multiple servers within one cluster. My original blog describes it, but perhaps not well enough!  If the cache is enabled, then webi will look at the cache and use it. It works just as you expect it to. Once you refresh a document, the cache is no good, but that won’t stop the product writing out new cache. The cache will also be written to when you schedule (and have set it to write to the cache).  Users are guaranteed the right results and pages, a user will never see something they shouldn’t.

      Worth enabling, but watch out for slow performing file servers (where the cache is held) as this can cause problems, in which case disable the cache. Enable cache, monitor carefully, get ready to rollback and turn it off! If ok, users viewing documents can enjoy the performance boost! Others (as in this thread) mentioned its not always appropriate, and that is certainly true, especially when all users ‘refresh on open’ documents.

      Does this help?

      Matthew

      (0) 
      1. Ashish Bhandari

        Hi Matthew

           

        Liked your post on this topic and definitely got a good insight on various PT option in Webi. I am going to be working with the max connection parameter and Thumb rule as suggest,.

        We have BOBJ 3.1 with SP3 on Windows 2008/MS SQL 2005 in our landscape.

        1) My question is around the well-known issue of webi restart post reaching the 2 G mark. My question is can we make multiple webi server to work as one consolidate server adding memory in combined effect  and making the solution to work with less issue. This is to reduce the restart of webi server once it reaches the 2 G mark.

        2) Is there a possibility of distributing a report over different webi server for process having less impact on just one server if they cannot be bundled together as asked earlier by me.


        Regards

        Ashish

        (0) 
        1. Denis Konovalov

          report cannot be distributed over several wrs’s, it runs on single server.

          the fact of webi restarting after it reaches 2gb mark is not an issue, it is a limitation of a 32bit process and result of a system that is not sized properly or a symptom of poorly designed reports.

          (0) 
          1. Ashish Bhandari

            Hi Denis,

            Thanks for sharing your thought, any idea on my first question on consolidated Webi server operation. Just to add the OS and CPU we are using the BOBJ on is 64 bit

            Thanks

            Ashish

            (0) 
              1. Ashish Bhandari

                Let me rephrase, is there a way or configuration that allow multiple webi servers memory to consolidated as one, like a pool. Say we have a report that would need 3 GB of memory in regular case the webi would restart so say if we have webi memory work in consolidation to each other so 2 webi server would easy handle this a kind of load distribution or using memory pool just as how now a days we have the resource management in OS.

                Thanks

                Ashish

                (0) 
                1. Denis Konovalov

                  no, this would be impossible. same as #2.

                  P.S.

                  report that needs 3gb WRS memory should not exist … ever.

                  Any webi report that execues in more that 5 minutes should never exist. Crystal is the platform for larger/longer reports.

                  DI/DS products are the tool for large data exports.

                  (0) 
                  1. Ashish Bhandari

                    Thanks Denis, so what is the exact reason then that the webi servers memory reach up to the it limit of 2 G mark and get restarted. Are they just that the reports are incorrectly designed to pull the data of such large size. Could this be the only reason or there is more to it. Just trying to  understand the factors that jack up the webi memory and toss it over so that I can work on the tweaking part based on the inputs i gather.

                    Regards

                    Ashish

                    (0) 
                      1. Ashish Bhandari

                        Thanks once again. The resources on the system are adequate enough never seen the CPU or RAM got fully utilized. CPU is 8 core RAM is 48 G on each box. I will get the reports checked once again. Bug is some thing that I would appreciate if you can elaborate for us all in this blog.

                        Thanks

                        Ashish

                        (0) 
  9. Rick Rabe

    Hi Matthew,

    Thanks for this information. I’m new to BI 4 administration and I’m wondering about the frequent recycling of this server. Is it necessary/normal to recycle frequently (on BI4 SP5) on a 64 bit system? Also, on the properties page do you have a recommendation for the Maximum Documents Before Recycling setting?

    (0) 
    1. Chandrakanth Angannagari

      Hi Matthew, wanted to see if you have some news on the official documentation about Webi settings for best practice and high performance.

      Currently the only two changes to the default properties of webi servers we did are setting a timeout on webi processing server and increasing the cache size  (apart from the regular splitting of APS services to be hosted individuallly)

      Apart from this there is not much documentation that helps BO adminstrators to fine tune webi performance

      A side question : Do we really need the STS on each APS ? (we are using webi with BICS based on SAP BW bex). For this is it not sufficient to have STS hosted on any one APS? or is it required on each APS?

      Thankx

      Chandrakanth

      (0) 
    2. Matthew Shaw Post author

      Hello Rick

      Its quite normal for the process to re-cycle itself. We need to do this to regain the memory that can not or is not released by the memory allocator. Some operating systems are better than others, but we recycle on a ‘regular basis’ for all operating systems.

      The algorithm is quite complex to determine when to recycle, but its based on the memory consumption and the number of documents it has already processed. When the process is re-cycling itself it can go into a state that makes it look like its hung, but actually its waiting to recycle ‘timeout before recycling’ (default 20 mins) to allow any users on it to save and finish what they are doing. During this time the process won’t accept anything new (still process documents it has open) and its important you’ve got enough other Webi processes around that will still be open for business to process new documents/requests.

      In general best to leave all the settings as they are, except for the ones I’ve mentioned above. Sometimes you might want to increase the “Maximum Documents Before Recycling” but only when you know its receiving a large number of small requests (like calls from QaaWS (when not using Xclisius/Dashboard servers), or calls from Crystal Reports 2008/2011 (when performing universe overrides). Both these requests are just to get the SQL, not actually process a document, but they still count towards that number)

      You need to have a good pool of webi processes to you have a good number ‘open for business’ whilst one is in its re-cycle mode. Because the algorithms are quite complex its very unlikely you would have all your processes re-cycle at the same time. To avoid this, in case you’re worried, just make sure you have about 4 webi processes enabled and running if you’ve only got 1 or 2 at the moment.

      Increasing the number of webi processes will, in general, distribute the load and in general reduce the overall memory footprint of each one. So in general this reduces the changes of the process entering a state of ‘recycle’, as the algorithm will force a recycle when the memory usage is high and won’t come down, and when the number of documents its processes has reached a limit.  So, plenty of webi processes helps quite a lot in a number of ways.

      (0) 
    3. Jon Fortner

      Hi Rick,

        We had a constant issue of WebI servers restarting even when the limits were not reached. In fact a few times per day across 12 WebI servers. This issue was accompanied by an API error for the users running WebI reports where they constantly refresh and change prompt values.  This is not normal and was occurring on BI 4.1 SP1 Patch 3. Our update to Patch 8 fixed the issue.

      (0) 
  10. Moritz Hoedel

    Hello Matthew,

    Thanks for providing this guide, but even after two years of BIP administration (4.1 SP2 at the moment), I’m still not very confident which documentation to believe in.

    Regarding your “golden rule”: you can find a completely different statement about this in the official SAP BusinessObjects BI4 Sizing Guide.

    It says on page 39:

    “When adding additional WebI instances, it is recommended that each instance run on a separate machine in order to maximize the I/O capacity available to the servers.”


    How do these two statements match?


    And when you follow your golden rule:

    If you have a lot of parallel scheduled refreshs running in general, what’s the impact for the users using other tools provided by the BIP, as Crystal for example?

    Or whats the impact on the CMS itself, when all cores are busy with WebI processing?

    You might see, I’m not very confident in applying that rule to our landscape.

    There are also unanswered question threads in the SCN containing the same questions, available for months now:

    Adaptive Processing Server DSL max heap and Number of WebI Processing Servers

    Re: BI4.1 Webi processing server per node

    The one listed first, was one created by me and I got a different (3rd) statement there, also from an SAP employee.

    And if your “golden rule” is really true:

    Can’t this be documented in an official guide after these many years of uncertainity or at least not said completely the other way round in the official documents like the sizing guide?

    This would make it a lot easier for all BIP admins (running WebI) out there!

    Or how to size the “document cache duration” correctly in a real-world scenario?

    We have documents with recurring schedules on a daily, weekly, monthly or even three-monthly basis (and also values in between) as usual I guess.

    So what’s really the correct setting here?

    And again: Why choosing such a small default value for an Enterprise Application Platform?

    Additionally I’m wondering why after so many years and updates these manual changes like setting the “Universe cache size” couldn’t be done in an automated way,

    at least as an option to set.

    This would be so easy to develop from my point of view! 

    Just check the number of available universes and set the value accordingly.

    Shouldn’t be a greater development effort than one hour for a skilled developer.

    Or why is this “Maximum Document cache size” still set to such a low level?

    Times have changed, disks are getting cheaper and cheaper, but the default value is still the same.

    To be honest, I simply can’t understand such things…

    Isn’t SAP the biggest european Software-vendor??

    Besides that I’m also wondering about Denis Konovalov‘s statement in the comments that no WebI report should take longer as 5 minutes to refresh.

    Where is this documented at all?

    Looking forward to your answers.

    Krgds

    Moritz

    (0) 
      1. Moritz Hoedel

        That’s already clear to me, but unfortunately not for our internal customers/our platform management, therefore this doesn’t help me any further, since it’s no official WebI product restriciton.

        (0) 
    1. Matthew Shaw Post author

      Many thanks for your reply.

      Inconsistent documentation

      it is recommended that each instance run on a separate machine

      Yes I see this isn’t very consistent. I think that phrase is trying to say, don’t just add Web Intelligence Processing Servers all on ONE machine, but rather spread over other machines too. I agree it doesn’t make that obvious. I’ll see if I can get that changed.

      Impact on other servers

      If you have a lot of parallel scheduled refreshs running in general, what’s the impact for the users using other tools provided by the BIP, as Crystal for example?  Or whats the impact on the CMS itself, when all cores are busy with WebI processing? You might see, I’m not very confident in applying that rule to our landscape

      It depends on what the machine is used for. If the machine is not dedicated to processing Web Intelligence documents then you may not need to configure the machine to gain maximum scalability for Web Intelligence documents.

      As you mention, it could be processing CMS tasks (such as scheduling, authentication/authorisation, auditing, events, name services etc) or other document types like Crystal Reports etc. So if the server will not need to process Web Intelligence content on ALL the CPU cores at any one time then certainly there is less of a need for 1 Web Intelligence Processing Server per CPU core. You could easily say just have 50% of the scalability by allocating 1 Web Intelligence Processing Server for half of your CPU cores. But remember this, that machine will have a maximum scalability, for Web Intelligence, to be 50% and no more. Can you really guarantee there will be no point in time that Web Intelligence documents will require no more than 50% of the CPU resources?

      There are two schools of thought around this. You could say, well I need 20% processing for CMS, 20% for this and 20% for that, perhaps because of the mix of document types you have. You could then allocate (and restrict the server) to just that 20% for each. It looks good on a spreadsheet, but I’m of a different school of thought. I’m of the thought that is ‘enable the software to be configured to take maximum advantage of the CPU and RAM’ and ‘when the system is busy, the OS will sort it out’ and ‘when the system is slow to respond, its only because the system is busy and not because of some artificial limit that is preventing the software from scaling’.

      I agree that on a machine with a very large number of CPU’s you may not want to have 1 Web Intelligence Processing Server per CPU core, but never forget this will limit its scalability.

      For me, the OS is best at managing which process should run and where. I’m not of the school of thought that something should be reserved for a particular task as those reservations are really restrictions for others.

      Document Cache Duration

      Or how to size the “document cache duration” correctly in a real-world scenario? We have documents with recurring schedules on a daily, weekly, monthly or even three-monthly basis (and also values in between) as usual I guess. So what’s really the correct setting here? And again: Why choosing such a small default value for an Enterprise Application Platform?

      Once a document cache has expired you can no longer take advantage of it. So a duration of 1 day is ok for daily scheduled documents, but certainly not for 3-monthly scheduled documents. You need a much longer time indeed. If you set the cache duration to years, there’s nothing really wrong with it, except the cached documents will not be deleted from the cache when the ‘scheduler’ (defined by Document Cache Clean-up Interval (minutes)) comes around. Instead it will need to work out what are the oldest documents and delete those. The impact is pretty minimal; perhaps a larger cache is the only side effect.

      A small default value isn’t the best I do agree. I also agree with your comments around “Universe cache size” and “Maximum Document cache size”. I’ll see what I can do to get those changed.

      In general any shared Web Intelligence cache should held on a device that responds very well and had a very low latency, otherwise we start to see many errors.



      Regards

      Matthew

      (0) 
      1. Moritz Hoedel

        Hi Matthew,

        Thank you very much for your detailed explanations.

        I hope you can work something out to get the standard documentation/default values corrected/adapted.

        I will talk about this with my colleagues and maybe I will come back with some more questions.

        But for now: Thanks for your qualified support!

        Krgds

        Moritz

        (0) 
  11. Kristof Speeckaert

    Hi Matthew,

    Thanks for sharing this very valuable information. This is the type of advice that should be included in the official administrator documentation!

    If most Webi documents (90%) are set to refresh on open, use prompts and the data returned is dependant on the user’s authorization, is there any point in keeping the caching enabled?

    The only scenario (I can think of) would be is 2 users shared the same data authorization, used the same prompt values and refreshed the same document within the cache duration span.

    Do you think disabling the caching result in a noticeable performance gain? The cache is shared on a network share between the different Webi processing servers within the cluster.

    Thanks again.

    Kind regards,

    Kristof Speeckaert

    (0) 
    1. Christian Key

      Hi Kristof,

      I have a similar situation as yourself and noticed a performance improvement when I disabled my real-time cache and document cache as they are useless as each report will return different results for each user dependent upon the data authorisation they have in the database. See my reply above from April 26 2011.

      Cheers,

      Chris

      (0) 
  12. Ramanaidu Kolasani

    Hi Matthew,

    Thanks for your detailed explanation.

    Regarding Maximum connections concept, could you please elaborate your explanation on below information.

     

    “You don’t want to add unnecessary software limits on your hardware resources. There is a school of thought that you should drop this parameter to ‘protect’ the server, I don’t follow that school, and would prefer to let the server do as much as it can, potentially giving a slower response time when busy, rather than generate an error when it’s not busy at all”

     

    Moreover, today while refreshing the webi report, we received a below error.

     

    Unable to connect to service WebiPlugin.WebiSession from server BOSERVER.com:6400 via CMS osca:iiop://GRZSMS185.andritz.com:6400;SI_SESSIONID=1653977JpB38w9BrGXJIf0opXpgAS1 (FWM 01006)

    I heard that increasing maximum connection parameter in cmc for webi processing server solves the issue.

     

    correct me if I’m wrong.

     

    (0) 
    1. Matthew Shaw Post author

      Hello Ramanaidu, The error message you mentioned doesn’t prove either way if the webi server was too busy or not. My guess is that there’s a problem somewhere else and the webi server logs would be the best place to start looking.  If the issue is easily repeatable then you’d be better off doing an end to end trace as this will enable the logs to HIGH across all processes but only for that session. Then use FlexiLog reader to merge all the logs together so you can the see the entire end to end trace for that users’ session. Regards, Matthew

      (0) 
  13. Andrew McFarlane

     

    A very helpful blog post indeed, Matthew – thank you. We have found that in times of peak usage, users are receiving a server busy error, which we suspect is due to faulty configuration of our 2 WIPS for our 4 core machine. Here are some observations:

    1. Analysis shows that only one of the WIPSes is doing any heavy lifting; load is not being shared with WIPS2. WIPS1 easily reaches its threshold without WIPS2 ever really being ‘invoked’. We’re trying to understand what accounts for this behaviour. I suspect WIPS2 is a clone of WIPS1, and perhaps faulty config was copied across. I guess I can test this by creating (not cloning) a new WIPS.
    2. In a scenario where the WIPS property “Memory Analysis” is disabled, is  only one WIPS therefore required (since it is now free to consume as much RAM as it can)? Or should we maintain several instances for load balancing?
    3. Max connections on each WIPS is = 200. I think this is too low. Assuming point 1, that only 1 WIPS is working, that allows our 115 users less than 2 sessions / connections each. I’m wondering how to calculate what our optimal max session value should be, and have settled on giving each user a max of 5 sessions. So 115 x 5 = 575, so max connections should = 300 on each WIPS, I’d assume. Does this sound about right?

    Many thanks once again.

    (0) 
    1. Matthew Shaw Post author

      Hello Andrew,

      Analysis shows that only one of the WIPSes is doing any heavy lifting; load is not being shared with WIPS2. WIPS1 easily reaches its threshold without WIPS2 ever really being ‘invoked’. We’re trying to understand what accounts for this behaviour. I suspect WIPS2 is a clone of WIPS1, and perhaps faulty config was copied across. I guess I can test this by creating (not cloning) a new WIPS

      Indeed I would do exactly that. Also use the webiadmintool is see what’s going on. It might show something

       

      In a scenario where the WIPS property “Memory Analysis” is disabled, is only one WIPS therefore required (since it is now free to consume as much RAM as it can)? Or should we maintain several instances for load balancing?

      One of the reasons for needing more than 1 WIPS was because the memory of one would need to go over the OS limit of 2 GB (that was before BI 4.x). So whilst the 2 GB limit has been removed with BI 4.x that doesn’t remove the other reason for needing more than 1 WIPS per machine. Don’t forget to view my blog on this subject

      Another reasons is simply performance and scalability. Even before XI3, back in XI release 2, we found that when Web Intelligence was very heavily loaded, to get the maximum performance and scalability, you needed 1 WIPS per CPU core. It was the surprising result from a load test that I was privy to. The architecture since then is basically the same, albeit now 64 bit, but the internal architecture of Web Intelligence is identical. So there’s nothing yet to suggest that these old rules are still not appropriate. Indeed, only last week a colleague had a performance and scalability issue with Web Intelligence. These issues disappeared when they increased the number of WIPS per machine from 1 to 5.  This doesn’t stop other schools of thought or ideas from different experts like myself. Find two different experts, you’ll get 2 different recommendations! Others might say you only need 1 WIPS per machine. Whilst this is true, you only need one, that doesn’t mean you may not require more than one to get the very best performance or scalability. The test results back then and the continuous feedback I receive from customers or colleagues who read this blog tell me that the more than 1 WIPS per machine improves performance for them, especially when Web Intelligence is the dominant document type being processed on a machine.

       

      Max connections on each WIPS is = 200. I think this is too low. Assuming point 1, that only 1 WIPS is working, that allows our 115 users less than 2 sessions / connections each. I’m wondering how to calculate what our optimal max session value should be, and have settled on giving each user a max of 5 sessions. So 115 x 5 = 575, so max connections should = 300 on each WIPS, I’d assume. Does this sound about right?

      A ‘session’ here is a document don’t forget, not really an end user BI Platform session. I wish the ‘max connections’ wasn’t a configurable thing! Just set to high, high enough for a user not to find it! Set it to 1000 say. In some cases you might need to set it higher still. Don’t get bogged down trying to work it out. If a user gets a error because the lack of connections, increase it! Just make sure its the same and consistent across all your servers. If one server is taking all the load, it might be because this is higher on that machine, than all the others.

      Regards Matthew (@MattShaw_on_BI for those on Twitter)

       

       

      (0) 

Leave a Reply