Automating UNX Adoption – Bulk Conversion, Report Migration and Validation

[laura_vega]

Introduction

In my last post Automating UNX Adoption – Overview, I described briefly the features of APOS Migrator for Web Intelligence Alan Golding from APOS showed me – how it automates scanning, universe conversion, Web Intelligence repointing, validation and committing your changes. I want to describe the processes of that solution a little more here.

One of the big points that Alan made was that every customer’s migration will be different and the APOS tool is designed to take care of such different scenarios.

System Inventory

As I said in my previous post, the APOS tool’s scanning and custom selection capabilities stood out for me. Here’s how Alan described them:

We have different search options. You can search through the folder structure – public folders, personal folders, user inboxes, etc. You can select multiple options; you can drill down to sub-folders – all depending on how you want to manage your project. Once the search is configured, you click Add to List and Migrator will search for all the reports in the specified folders and display them in the interface.

Now you have a list of reports showing report name, location, owner, when it was created, when it was last modified, universe information and more.

This is the simplest scenario, but what if you have a large number of reports connected to a universe and you want to discover which ones should be your priority? Here’s how Alan responded to that question:

We deployed Migrator successfully at a global high-tech organization that had hundreds of thousands of Web Intelligence reports to address. In such a case, it may be prudent to form an understanding of which reports the organization needs to prioritize and migrate, and which are not needed moving forward. It is an opportunity to retire technical debt associated with maintaining a vast number of unnecessary reports that may have accumulated over the years.

Here's where the integrated APOS Insight solution becomes useful. Insight is a set of automated services that run against the SAP BusinessObjects CMS and Audit databases to extract metadata about report and universe usage, structure and impact analysis, along with various other facets of the SAP BusinessObjects environment.

The information gathered using these services is recorded in the APOS Insight database, which may be running on Microsoft SQL Server, Oracle, or some other relational database.

This central repository captures the full picture of your SAP BusinessObjects environment, which provides vital assistance to an intelligent migration. APOS Insight provides some out-of-the-box reports that can be helpful in deciding which reports to migrate. For example, the report called Inactive Report shows all reports that have not been scheduled in the past 365 days, or other customizable time selection.

You can drill down even further for a list of Web Intelligence Reports that have never been run against a universe in the SAP BusinessObjects environment. Such reports may have been migrated from a previous environment, but never used in the current environment.

APOS COO Allan Pym also commented on the value of a robust system inventory:

A Telco customer based in the US with over 200,000 reports to migrate are using the Insight search functionality to determine which reports they need to migrate and which they can retire. By their calculations, they believe they can reduce the size of the migration project by about 25 %.

Bulk Universe Conversion

The APOS tool offers flexibility in the way you execute your migration. Alan Golding told me about the variety of approaches customers are taking with Web Intelligence migration:

There are options to exercise in your approach to migration. Many customers prefer to migrate universes first, so they would choose to limit their search to universes only, and the Migrator will list all universes in the environment. They can then perform a bulk conversion of these universes to UNX. Once the conversion of universes is complete, they can perform a search for Web Intelligence reports. Alternatively, the customer can search for universes and reports and convert them all in a singe operation.

Alan also noted that some customers do not want a one-to-one relationship between UNV and UNX universes. They are looking at migration as an opportunity to consolidate universes. For such customers, the APOS tool offers mapping capabilities to form correspondences between the two universe types – details here. Alan described that some customers use this universe mapping feature during the UNX upgrade process as a result of database changes that are happening at the same time, while other customers use this capability to address similar database changes but completely separate from the UNX upgrade.

Repointing Web Intelligence Reports

Here’s where automation saves you a lot of time and resources.  Realistically, automation is required for this step of the process for most organizations.  An SDK exists which gives potential for someone to build scripts for automation, but it would be very difficult to replicate the feature set of the APOS tool.  Alan Golding described how the APOS tool automates repointing:

Migrator generates copies of the Web Intelligence reports. You select the reports and Migrator shows you the universes associated with the selected reports.

You select the target universe, if you have already converted the universe to UNX. If you have not yet converted the universe, you can specify to convert the universe as part of the process, as well as a path for the newly created UNX universe.

You specify the location in which the copied reports are to be stored, click Change Universes, and the copied Web Intelligence reports are repointed to the UNX universe. You can keep them in the same folder as the original reports with new names, or create a new folder which you use to control the workflow.

This is when you can use the results of APOS Insight, previously described. You can decide to run the repointing only on the Web Intelligence reports that have been filtered using APOS Insight advanced scanning capabilities.

Validation Automation

Validation is a critical step in any migration: you have to know that the reports you have migrated will work as expected. But it is often not possible to test every report, and customers need to make decisions about how much testing is enough. How much testing gives them confidence that the migration has been successful. Also, not all reports are equal. Reports that are controls for regulatory and financial matters need more rigorous testing than other reports. Allan Pym noted that it is useful for customers to be able to determine which reports they test, and also how much of the individual selected reports is tested:

Organizations make decisions about what degree of data is being tested, because they know that if they validate a certain amount of representative data within the report, they can be reasonably assured that the remainder of the report will run as expected. There are a number of interesting options there, and it really comes down to having strong controls over the project.

The APOS tool provides flexibility for validation. It lets you validate against the Web Intelligence document, the latest instance, or the latest recurring instance. You can choose to validate as an administrator, the report owner, or as a specific user. You can validate variable types and variable values. What is more, as you can see by the Advanced Features in the screenshot below you can restrict the validation to specified columns and to the first or last portions of the data:

How granular is the validation process?
Alan Golding explained that the APOS tool “compares the number of rows, the number of columns, the values in the columns, and the data types of each column in the data flow. This validation is completed for each data provider. So this gives assurance that the new UNX is providing the same data to the Web Intelligence report as the previous UNV, which is key. However, if someone wants to go the extra step of doing deeper report level comparison testing then we can bulk automate that testing also.  This level of testing provides automated comparison of the values within Web Intelligence reports, incorporating all the calculations and report level factors, and validates that the UNX-based report is presenting the same data values as the UNV-based report.  If desired, this step brings an even higher level of certainty and governance.”

The APOS Migrator tool provides a final step in the upgrade process, which they refer to as the Commit step. Alan describes the commit step in this way:

After the validation step you then have confidence that the copied report is accurate, so you can now commit the report. Right-click the report and select Commit, and with that the Migrator finalizes the changes of the connection on the original report to the UNX. All the original schedules, instances, CUID and ID remain the same. The only difference is that Migrator has changed the universe connection on the original Web Intelligence report. As part of the commit process, there is an option to back up the original report before the change is made so that the commit process can be rolled back if any discrepancies are discovered later.

Not all customers perform the commit step. Some will simply use the copied report moving forward. They store the original report in a separate folder and decommission it eventually, while maintaining a comfort level with the process.

This seems like a great step to have in place to finalize the process, and will give extra assurance to those that oversee the process.

Process Controls

The validation capabilities of the APOS tool provide effective controls, and the tool has other features that help you manage your migration projects. For example, it has an option for high availability that lets customers process parallel projects. Alan Golding:

Organizations want to know that they can complete their migration projects in a timely manner in comparison to manual efforts or using the IDT. With larger volumes of reports, some organizations have tight timelines for these projects. The Migrator has a high-availability option that allows the customer to outsource processes to other servers to handle the processing of the migration, rather than relying on the machine on which Migrator is installed.

The APOS tool includes other process controls to synchronize and scale your migration processes. To summarize:

Project-based migration management
Customers create projects and add universes and Web Intelligence reports, then run conversion, repointing, validation and commit processes against those projects.

Advanced system scanning
The customer fills the project containers using simple or advanced system scanning performed using APOS Insight metadata scanning technology.

Process validation
Granular validation settings let customers control the amount and depth of validation processes.

Process commit
Migrator changes the connection on the original report to the UNX. All the original schedules, instances, CUID and ID remain the same.

Process monitoring / dashboard
Customers can watch processes run in real time and receive alerts when processing encounters errors.

Scalability
Asynchronous processing and the high-availability option provide high-performance, distributed processing for projects.

APOS BI Upgrade Freemium Offer

Again, I suggest you look at a Freemium offer that APOS has. Knowing what automation options are available can help you formulate your migration strategy and plan your migration. The APOS BI Upgrade Freemium Offer lets you convert universes, repoint up to 200 Web Intelligence reports, and automate bulk testing.