Using a Software Proof of Concept (POC) or Pilot Installation for Change Control Automation ROI Validation

[Former Member]

You’re probably familiar with KPIs, or Key Performance Indicators, which use system performance metrics to judge success. In part one of this series, we suggested ways to measure ROI after implementation. Those data points were an IT form of KPI, having to do with user satisfaction, risk management, problem elimination and so forth.

Benefits of a POC or Pilot Implementation

But those KPIs measure results after implementation. A small-scale pilot test before you plunge into full implementation will help avoid unpleasant surprises and validate both known and unknown ROI data points. Often best achieved with a POC installation, a pilot lets you test the waters. Not unlike putting changes through QAS before moving them into production. A carefully planned POC can also serve as a precursor to a major roll out. A new software roll out will get better results if you run a pilot before committing to an overall implementation.

In an SAP change control automation software context, this may mean simply running the software over, for example, the CRM system to test processes, validate strategies and give early feedback on specifics like emergency processes, documentation, security, migration, approval, special object management, validation rules, and object collision management issues.

Training is very important to any final roll-out. Your pilot testers should use the same documents as planned for roll-out. They can note unexpected problems. Does the documentation miss x, y or z? A pilot will let you correct such problems ahead of time.

Not every change control solution allows for piloting, so make sure you don’t get locked into a “big bang” style, all-or-nothing roll-out (of course Rev-Trac easily accommodates small pilot implementations to ensure final results will be as expected).

When it comes to testing your planned ROI indicators, a pilot can help identify unpredicted issues enabling a tuning of data points to deliver more meaningful metrics, and possibly higher value metrics to better add to your ROI proof points.

Create success criteria so the pilot team knows what to look for and a plan to capture variances for review and correction. Keep your criteria small in scope to allow your pilot users to work efficiently without high overhead of reporting of the criteria.

Team size will depend on project complexity, size of your company, locations if you have a geographically distributed team – consider all affected IT teams as stake holders in the pilot. Also, take into consideration the impact to the pilot team members if they are also part of a large project, they may become overwhelmed!

Schedule frequent meetings to get feedback on what works and what doesn’t. An SAP change control automation solution should be easily adoptable enterprise wide, so it’s important to listen if your pilot group gets frustrated with any part of the process. At the end, everyone on the pilot team should agree that they can roll out the new processes system-wide successfully.

Publish the results of the pilot – both positive and negative. Credibility of your work pays off in spades when you are transparent about your results and how you have adjusted your processes. In addition, senior management expect you to be transparent and authentic when delivering your final business case. Particularly if the POC is part of your business case development process.

The result can be a smooth, trouble-free enterprise wide implementation that does what you expect, solves problems without creating new ones, and produces well-tuned ROI data to judge project performance.

There’s another benefit – your pilot testers can become a super user team, acting as internal experts during final roll-outs. They can advise the full IT team and act from knowledge if the larger implementation needs further tuning.

Some closing thoughts

Here are a few points to bear in mind as you plan how to use the pilot test to help measure project success:

• Have you established a cross-functional team of “super users” to assist in the final SAP change control solution roll-out?
• What additional data points can they suggest for ROI metrics?
• What is their feedback on potential trouble points when scaling up to full enterprise size?
• Do potential trouble points reflect pre-implementation problems?
• Is there a way to use trouble points as “baseline points” during later ROI measurement?

By the way, If you are still struggling to find meaningful change control automation data points here are a few to start with:

• Mean time to problem resolution
• Mean time for change to go from approved from development to delivery into production
• Percentage of requested change actually being delivered with each release
• Number of emergency changes approved
• Number of post release go-live issues reported

Next month, look out for the final post from the ROI series where we will be discussing how to leverage your data-gathering during and after go-live.