As regular readers will know, my team and I have completed The 1st Trial Cutover (CUUC Series part 4) and we collected a vast amount of data. We also blew our time budget of 60 hours out of 96 hours total downtime, but we recorded the areas where the overruns occurred, so now I will take you through the analysis of what we found and what mitigations we put in place.
As a veteran of these type of projects, I have done this analysis many times, and I find that it is better to structure my thinking into Constants and Variables – for example
Variables: things we can change – for example, the number of work processes
Constants: things we cannot change – for exmaple, the time a backup takes
During the Upgrade and Unicode processes data is flowing through the system at phenominal rates, it is important to benchmark these data operations/flows to provide performance analysis – thank fully most of this work is done for you and it is simply a case of working out how to use this data to give you the best analysis and edge for your CUUC process.
We have seen that the Upgrade provides a file called UPGANA, which gives a high level breakdown of the Upgrade phases and their timings. The next important data set to be analysed is the Unicode conversion – the Unicode conversion, like the Upgrade, writes a great many log files which can become quickly bewildering, the important thing is to start at the top and work your way down. In order to do this, you start with the Export_time.txt and the Import_time.txt, these files provide you with your package performance details, like Operation duration, Size, Data Throughput.
The table above shows how specific packages performed during the Unicode export and the time period they were executed in. This can be very important, as the Unicode process should be viewed as a holistic process – everything running on the system/servers has the ability to affect other things. So keep an eye on things like R3load processes accessing the same cluster table at the same time and causing performance issues. The text file is also represented as an HTML file with nice graphics, which if you are inclined you can merge together to give a graphical representation of what packages run when and for how long – I was not so inclinded and did not. These files and their contents are also provided for the Import processes.
So now that we know where to find the information and the formats that information takes, let’s dive in and see what variables exist which can be used to affect change in the Upgrade and Unicode performance as shown in the previous post (The 1st Trial Cutover (CUUC Series part 4))
From the table and the details above you can see the variables we can affect and the places where we can find some of the data which will influence how we change the variables. There is a wealth of deep dive analysis we could do, like running performance traces which will highlight the CPU, Memory, Disk I/O at every second throughout these processes – Jim Spath’s excellent Unicode blog series shows the type of deep dives you can do into data sets. Personally I am not a massive fan of doing this, unless I really have to, I find that the analysis can become a goal in itself and distract from the overall goal of getting the thing done!