Starting January 2nd 2012, COIL will be leaning forward into both finishing projects with a late start in 2011 and ramping up some of the newest projects beginning early in Q1.
With attending to end of year matters and staging projects for 2012, it comes as no surprise to be lacking the time needed to fully reflect back upon the past 12 months here in COIL and what’s been accomplished across so many projects and teams.
Before the sand from this year “runs out” entirely though, I would like to comment on a few 2011 COIL projects. A quick opportunity to share some project highlights, and to offer some insight into how projects like this are managed and to describe how innovation teams working in COIL benefit from tapping into our unique platform of services to drive positive outcomes from the projects performed here. I will start with sharing news about the project known by some as “Monsta”‘
Time permitting, if my colleague Kevin offers blog posts covering some of the other projects he has managed, then this is where I will try to comment further on some of the cool projects we did this year.
BI 4 Large scale Implementation
I wanted to write this blog post a lot given that this project has offered up some interesting things to share despite the fact that the team does not yet have the final results from the main tests of the study. The tests are actually running this week of Dec 19th so we may have our first results before the end of the year. It is nonetheless, the single largest project to become executed in COIL PAL which meant that the project earned itself a nickname “Monsta”.
What is the Monsta Project Exactly?
Developments within SAP and BI as an industry itself, suggests strongly that this growth trend in analytics is nearly epic, and with the release of HANA, the push for BI on mobile devices, and an “analytics anywhere, anytime” direction, the need to scale up is and to be capable of supporting larger and larger environments is predictable. A truly large BOE environment could be anything larger than 2-4,000 users concurrent (usually equivalent to a 20-50,000 total user base for BOE), which would start pushing up against ~100 CPU. This COIL project tests architectural, scalability and performance limits of valuable interest serving as a confidence builder internally, but also externally. Future customers for environments of this size would feel much more comfortable to find that SAP tested this in our own labs prior prove that it works. To develop the tests and to produce such proof points as an orchestrated co-innovation activity among multiple partners from the SAP ecosystem adds validation and opens up new opportunity to accelerate innovation on this front.
The BI 4 project team is working to validate 10,000+ concurrent users leveraging a large load balanced system comprised of 60 individual hosts where the load was comprised of BI system login in, query generation, review and report generation then logout. Our COIL project member Soasta Systems is running its CloudTest platform here in COIL for the Monsta Project. The performance results will be used to support product feature/functional claims and the tacit knowledge derived from the project work is useful to creating best practices and configuration recommendations. The output will also be used to inform the latter phase of the project that will assess energy use of the system and optimizations to reduce energy use by a large BI 4 deployment.
With active participation from COIL Sponsors Intel and F5, plus project members RedHat, Super micro, OSISoft, and Soasta, you quickly assemble a large project team. The complete project team rounds out beyond the participants from each project member or sponsor with the inclusion of the COIL PAL team members and the BI project requestors from service delivery. COIL is additionally supported by a crew from SAP IT Hosting helping us to deliver the infrastructure as a service as well as to participate in many different aspects of the project. Developers from our Vancouver-based BI platform (BIP) team as well as Sybase engineers and development for ASE and IQ are all on the core team which also includes equally strong technical expertise coming from the BI field as well as those representing all the partnering and business aspects relative to this BI 4 project.
To add just a bit more weight, we also have a sub team in the project focused upon the sustainability and green IT dimensions of a large scale BI 4 deployment. One team member who was behind creation of the Power Performance benchmark (SAPS/KWatts) is participating in this project to examine the energy consumption and observe whether the study can contribute to the formation of new optimization techniques for diminishing power consumption without poorly influencing performance. The power consumption and performance studies will be based based upon concurrent loads up to 10,000 users.
We expect to begin analyzing performance results by mid to late January leading the team to generate some useful reports to tell us how the system behaves in delivery of core BI 4 services from our target infrastructure. This analysis by end of January will begin to flow into the creation of planned BI marketing collateral. It will also be used for videos, demos and talks created as part of COIL project showcasing efforts.
These tests are to be re-performed in support of the energy consumption and management study which will equally generate marketing collateral in various forms. This project has yielded some very interesting things from the very start extending beyond the stated goals to obtain specific measurements relevant to BI 4 performance on a large scale and how energy efficient it can run. In looking back, I’ve touched on a bit of the project history as a way to offer some context to the sorts of things COIL and the project team has learned during our time together.
A previously successful COIL project featuring BOE 3.5 triggered an interest from some of the key BI team to learn if COIL could support a project yielding results meant to contribute to the launch of BI 4. A project where the output could be readily used to develop some proof points for understanding how to optimize for a large scale BI 4 deployment where there is concurrent use numbers reaching upwards to tens of thousands. While it was an internal team seeking an opportunity to do project work on this front, this request somewhat neatly coincided with the fact that we had multiple project members and sponsors in COIL all interested to complement a BI 4 solution.
It is interesting to note that in order to get at the desired project results, solution optimizations need to be performed across the stack and this is where getting to work with different firms specializing in the micro-architecture and hardware, the operating system, the dB and equally developing pertinent knowledge relative to storage, network, and load balancing add the most value to establishing an ideal base for BI 4 to perform at this level.
The project requestors described the hardware as well as the subject matter expertise needed. COIL then set to work at developing an alignment with technology companies seeking to meet business objectives from participating in such project work and access to results. A great deal of knowledge brokering begins from the first day a project is proposed. We learned early on that when you build a team comprised of Hardware and Software experts that over communication is key to eliciting accurate requirements and making sure all engineers understand each other well.
This project was active in its beginning months developing the team, its primary objectives and forming the target BI 4 architecture as well identifying the right use cases and how to create and perform the necessary tests. The more details to emerge from the original project brainstorming is that it led to the understanding that the end result would benefit from tapping the SAP ecosystem and where COIL was well situated in which to cultivate awareness and interest among other project members and sponsors. We built off of relationships formed in the earlier project to quickly identify all of the required BI stakeholders and subject matter experts we would want close to the project and to introduce to COIL project members and sponsors taking an interest to connect the project to their own business objectives.
Focus in this project has been to develop the use cases and corresponding tests needed to produce results demonstrating the infrastructure and architecture configuration and optimization needed for a large scale BI 4 deployment. We are on track to produce test results as planned and pleased to know that the project has yielded other observations and data of interest to those challenged to implement and manage a BI system. There are two primary goals that the project is meant to achieve based upon the output derived from the project:
Establish customer confidence in large scale BI deployments
- Demonstrate scalability and performance with a combination of SM blade technology + RHEL6 + SBO
- Show the impact of various optimizations on throughput vs. power consumption
- Provide marketing collateral for ecosystem of services and reseller partners where project output contributes
Optimize SAP BusinessObjects Enterprise Architecture
- Provide guidance for engineering and documentation teams on enhancements that will ease deployment and system administration
- Detail recommendations for improving platform architecture by leveraging underlying middleware and hardware
- Drive product innovation by removing bottlenecks through collaborating with product team
- Provide a trove of tuning and configuration information that our enterprise customers can use
- Sustainability and energy savings
Despite some delay, the project is still on target to deliver test results and analysis needed to help secure the aforementioned goals in early Q1 2012.
At the end of July we deployed the necessary hardware contributed by Supermicro:
Four 7U chassis’ each containing 10 “Twinblades” as well as multiple terabytes of local storage and 1GB Ethernet switching per chassis. Each Twinblade features a single motherboard with 2 Intel Westmere processors and 48GB of RAM (for a total of 80 hosts). Super micro has also supplied a separate 2U 48 core (Intel) Super server with 16GB or RAM and multiple GB of local storage used to support the requirements of the Sybase ASE 15.7 dB. The rack also features an F5 BigIP Appliance providing all load balancing and wan optimization.
There is also Cisco Nexus Top of Rack Switch providing interconnection to our Cisco Nexus 7000 10Gb backbone. Each of the blades are allocated computational resources used by BI 4 as well as our load generation and testing harness supplied by SOASTA and the RedHat Satellite PXE server to provide centralized configurations. All of the blades and the Sybase ASE 15.7 server were installed over RedHat RHEL 6.
The rack is serviced for power by multiple smart PDUs where each is capable of providing per component energy use collected and analyzed by an OSISoft PI server. The racks are serviced by multiple 50A 3 phase breakers attached to a Starline Bus System from UEC. We have a variety of options for how we attach to and measure system power usage for each component and will use the configuration that best meets the needs of the tests performed.
We are interested to know, how did knowledge and solution derived from pursuing co-innovation with partners in the form of discrete projects, lead to new opportunity for SAP and all of the participating firms? It is our expectation that the performance and energy consumption data we collect and analyze will be output to a variety of future documents offering proof points of interest to BI 4 implementors.
What we also know is that the project demonstrates how valuable tacit knowledge is to the success of such projects and that frequent tacit knowledge exchange occurs within a large ecosystem-based project where it is possible to try and capture and document these knowledge flows and to observe how people come to value the information we are able to share. SAP and all the participating firms benefit in some fashion from the tacit knowledge shared in a project.
It may not be economical or even practical to capture all the shared knowledge, but effort should be made to extract the most meaningful knowledge to come from the knowledge exchanges occurring within the ecosystems-based team running a project. It is the goal of the COIL PAL team to look for ways to help our different project teams to capture and share tacit knowledge. For COIL, we see discovering how to do this well as a way to educate and support other project teams in COIL for how they can do the same.
What the team has learned along the way has been rich and knowledge flows occur in a well distributed form to all active project participants. The emails originate from across the entire project team which indicates that commitment and interest may be uniform. It is generally clear to everyone on the team what is of interest and importance and you simply know that everyone wants to part of the Monsta!
From a project management perspective we must continuously balance the technical information flows with the persistent interest from the business side to cultivate content useful to meet sales and marketing goals. Harvesting content from the project suitable to those managing sales and marketing objectives is largely at the mercy of how well the project executes to the time frame targeted for performing the work and the quality of the test results. We are pleased that the project could deliver beyond this to capture and share some specific tacit knowledge of value to future implementers of like systems before the project itself is completed.
Sharing Project Information
This team has already introduced the project to those interested in the things learned along the way. We did this by staging some network lounge sessions at TechEd Las Vegas where we shared information about the architecture we are using and the things we learned in just building out the system needed to conduct the testing, that we thought systems and dB administrators or other implementers would find of interest.
We are well aware that considerable knowledge gets shared in all COIL projects, but in the case of Monsta we made a strong effort to capture and to document tacit knowledge flows. We chose to simply look for what we could capture that would be of value versus trying to optimize how we did it all at once. The question remains if every project team can do it the same way each time and if it can be done spanning several domains of expertise.
The work the team is performing now is focusing much higher in the stack now in its testing, but what we did previously to make the current work even possible; drew out a lot of useful information. This first wave of work we performed and this is what we shared with session attendees this fall at TechEd Las Vegas.
Our main topic considered the complexities surrounding the overall architecture, system and network design, sizing, app, middleware and OS configuration as well as an understanding of the scale out requirements across the stack that are non-trivial. As an example we shared the things learned establishing a network boot configuration for the target test platform essential upon recognizing how often this project would require change management spanning 80+ machines.
We led with describing our use of the RedHat Satellite Server software to manage a PXE central boot system used to effectively provision, patch and monitor this deployment. From having SAP and RedHat engineers sitting side by side in COIL, we observed that the Satellite setup was not simple or always straight forward.
Our ability to install Satellite required lots of prerequisites and knowledge at the system level, DNS, routing, firewall, NTP, etc. COIL and RedHat project team members worked side by side to do the Satellite installation and configuration and captured what they learned. During the installation they ran into issues but through the collaboration rigor of the COIL program, Subject Matter Experts were more easily located within Red Hat to help solve all remaining configuration problems.
The knowledge we captured and can now share may in fact be formally collected and available from sources elsewhere, but this project made it possible to observe and capture information about Satellite that we could share that is not always found by sys admins armed only with the an installation guide. We equally shared similar tuning tips we uncovered in configuring an ASE and IQ dB for this BI4 implementation and looking for a revised view of these elements at the end of the project.
We had a large crowd at one session and it was very apparent how interested people are to understand how to successfully implement BI. They wanted the information needed that is never in the manual. This single large response to the session at least suggests that the “hands on” configuration information is of value and that there is a salacious appetite for it.
The follow on questions we were getting indicates a good sense for what types of tacit knowledge capture we should be targeting that might reveal even more useful configuration tips and best practices. One thing was certain in that while the large scale implementation was important, smaller deployments are very real too. The same actors are looking for such tuning tips for lesser scale deployments too.
A lot of super information was captured during the first phase of the project between our BIP developers and our colleagues with Sybase responsible for ASE and IQ. The colleagues on both sides benefited from the opportunity to work together through COIL which netted a more thorough understanding of technology on both sides. For BIP, The test strategy of ‘building out’ the system that was used to achieve our test goal worked. Starting the test using a smaller number of users while monitoring and analyzing the resource usage and performance as they gradually increased load as opposed to immediately ‘jumping’ to thousands of users was well considered as this helps to determine when to add more ASE engines and CMS instances, as well as, helping to quickly narrow down various bottlenecks when they emerge.
There were so many comments made in the different reports representing useful insights and providing rich context for the content that will find its way into future white papers and tech briefs. As the project evolves through its 3 phases we are keen to keep capturing this level of detailed knowledge and to see things learned in the first phase help to improve doing things better in the later phases.
Given the length of time this project has taken it was seen as a positive thing for this team to publish some early content as well as perhaps becoming surprised to discover how much knowledge exchange occurs and how easy it is to miss it. We look forward to refreshing all of the information captured to date and using it to enrich the final analysis and reports once the performance results are complete.