Going Live with ESS

[Former Member]

First, let me welcome everyone to my very first web-log. I hope you will find it and want to encourage you to comment and provide feedback so I can be more effective.

A quick introduction. These days, I am the SAP Enterprise Portal Architect for ChevronTexaco (CVX) and I have been working with the SAP Portal since 2001. I began my career with SAP in 1995 by specializing in the areas of Basis, Performance Tuning, Operations, and Oracle Administration. In addition, I am currently the Group Chair for the ASUG Business Information Technology and Infrastructure process group (http://www.asug.com/groups/group.cfm?group_pk=8) and an active speaker and presenter.

With that said ….

We are LIVE today with an upgraded HR system (4.6B R3/Enterprise) and access to the ESS transactions via the portal. The fact that I actually have time this afternoon to sit down and write my first BLOG is a great testament to the team, our partnership with SAP and the hard work we all put in to make everything happen. I have to admit that last night I tossed and turned and worried about what would really happen when we threw the ON switch this morning. I know I risk jinxing things, but so far we are up, live, and running smoothly (well there are a few small problems). Way Cool!

Implementing ESS via the portal is a major step for CVX since it expands our user base from about 2,000 users to over 23,000. By providing a foundation for reaching a large user population and reducing our per user cost we took a vital first step to insure the SAP Portal is a viable technology at CVX. It took a bit of doing, but it wasn’t nearly as difficult as our first implementation, eFoundation, which started with EP 5.0 GA and ended with EP 5.0 SP4 Patch 2 and took a year of concerted effort.

At CVX we have split the responsibilities for the Portal into three main groups:

• Technical Team who is responsible for the installation of the Portal, patches, and the infrastructure needed to support it (i.e. hardware, O/S, monitoring, alerting etc.)

• Applications Support Team which design the roles and integrate applications/content into the portal (i.e. building iViews, JAVA/.NET development, branding etc.

• Security Team which is responsible for assigning the roles to the proper set of users.

And some of the major aspects of ESS we had to consider were:

• Why access ESS via the portal in the first place?

• What will the design look like? Which transactions will we support?

• Will our infrastructure work under the new load and meet our performance and stability requirements?

• How do we integrate these new roles into the portal and manage the impact of the change on our users.

In the rest of this BLOG I am going to concentrate on the infrastructure, performance, and stability aspects of the Portal and highlight some of our experiences in implementing ESS.

Let’s begin with the infrastructure. When we went live on 1/1/2003 we architected the portal to have 3 Lock Clients which were load balanced to distribute the workload and provide a high degree of redundancy. At the time, we knew there was lots of horsepower to spare in this configuration but we figured that servers were relatively cheap and the insurance policy of having 3 of them was well worth the price. In hindsight, this was a great bet and we were really able to leverage the work we did in setting this infrastructure up ahead of time. When we started planning the implementation in February, we believed that we had plenty of horsepower to accommodate the much larger user base and peak demands of our users accessing ESS.

Believing something is one thing, but proving it is something else entirely. We decided that we needed a formal stress test and budgeted 6 weeks to prove that we could handle the demand while meeting our criteria for response time and stability. And boy did we need every minute of that stress test (and then some) before we were ready to say, GO.

We encountered a number of key issues along the way:

• Discovered that we would have to implement SSL for IIS and the J2EE engine in order for the automatic termination (and release of locks) to work for our ESS implementation. The protocol of the portal and the protocol of the ITS server running the ESS transactions have to match and our IP standards forced us to use SSL.

• Discovered stability problems with the J2EE engine under high load (it would mysteriously hang and could only be restarted dependably with a reboot). At first we thought moving to SP4-Patch3-HotFix 22 would solve the problem, but it didn’t resolve it completely (it fixed something else so we decided to stick with it).

Activating SSL for the portal was one thing I had really not wanted to do! Just looking at the documentation involved sent shivers down my spine, but we no longer had an option given our IP standards. So our team took the plunge, read the documentation, downloaded the cryptographic files, and began working through the issues. Activating SSL is a complex procedure and you must be VERY detail oriented to get it working. It also has a significant impact on performance and in our tests we noticed a 30% decrease in response time with software SSL activated.

Rant #1: Nowhere in the documentation (including an otherwise great write up on Portal Security) does it mention that hardware- based SSL accelerators weren’t supported nor why. Fortunately we were able to query SAP directly on this issue and although it took a while we did get word that it wasn’t supported and some portal functionality might not work. If we hadn’t had such a good partnership with SAP we might have made a wrong turn in our architecture. After all, we already had the hardware SSL accelerator installed and available.

Rant #2: The documentation for activating and testing SSL is greatly improved. But it is still a very complicated process! Our upgrade team was sweating bullets as we went through each item in the task list on all 4 of our servers. It is way too easy to miss a small step and have no clue why things aren’t working! I highly recommend detailed documentation (we built our own based on SAP’s) and lots of practice.

Our next major hurdle turned out to be instability in the J2EE Engine (J2EE 6.20 patch 12) when operated under high loads (about 130% of our expected activity). Strangely enough, simply recycling the JAVA engine did not resolve the problem. We found that rebooting the server itself was required. We reported this to SAP and have been partnering with them to resolve the issue. One recommendation was to try Hotfix 22 and while it helped resolve an unrelated problem, it did not fix the stability issues we encountered. We are still working with SAP on this but decided that we could still Go Live. Having the insurance policy of 3 Lock Clients and the infrastructure that can identify a problem and fail over in under a minute, really paid off.

So it's Monday afternoon. Thousands of people have been into the Portal to check out the new ESS transactions and complete their timesheets. There have been a few small issues, but my phone is quiet. This is a sure sign of a great team, hard work, a good partnership with SAP, and a successful implementation.

So hopefully Tuesday is just as quiet and I can finally get through my email.