This is part 3 of a three-part blog series, covering The How aspect of Newcastle University's journey to SAP S/4HANA, which focuses on the approach taken to deliver this project .
This post is broken down into the following sections: Project Planning | Project Governance | Research & Upskilling | Resourcing & Validation | The Impact of S/4HANA | Readiness Checks | Testing & Issues | Supporting Business as Usual | Go Live & Hypercare | Project Lessons Learned | What's Next?
The University has a well established annual SAP maintenance schedule. Work typically starts around the end of Oct each year, when the the latest software releases are scoped and impact assessed on our Sandbox environment, before being phased through our landscape by the end of Jan the following year. The Sandbox systems are usually recent copies from Production to enable representative testing to be carried out early in the project. This surfaces problems early and means we only commit to cutover to our live landscape if we're confident of meeting the target go live. The end-to-end project duration is around the three month mark.
With the move to S/4HANA the approach needed to be different given the increased complexity and risk due to the scale of change. We decided to perform repeated trial conversions (five in total) of our SAP ERP system using a fresh Production copy. This system was standalone and not integrated to any other systems, allowing us to focus our early project efforts specifically on the technical conversion, finance data migration, Customer Vendor Integration (CVI) conversion and employee business partner conversion. Only once these were sufficiently understood and rehearsed did we move to our Sandbox Phase, which introduced the many other SAP and non-SAP systems into the S/4HANA test cycle. Some of the relevant Simplification Items were addressed during the Trial Conversion Phase but the bulk were investigated during the Sandbox Phase. At the end of the Sandbox Phase we had a number of open issues but there was sufficient confidence to proceed to Development, which was followed in a matter of weeks by QA. User acceptance testing was initially planned to straddle the University closure for Christmas but was brought earlier, with most business units signed off by the end of Dec 2019. There is no time in the University calendar that suits all departments, with experience showing the end of Jan being the 'least worst' option for the majority of users. We went live as planned on Mon 27 Jan 2020 but had a contingency go live date secured for four weeks later at the end of Feb 2020 should we have needed it.
In addition to S/4HANA we had several other significant SAP projects running in parallel, most notably our first SuccessFactors module, Recruiting (go live Sep 2019) and the integration of HubSpot CRM (third party marketing automation platform) for inbound marketing (go live Sep 2019). Both of these projects involved integration with SAP ERP and were contending for the same resources in some teams, which was challenging at times.
We referred to the Transition to SAP S/4HANA Roadmap, particularly during the early planning stages of the project. If customers are really struggling to know how to start on their S/4HANA journey, this is one of the best resources. We certainly didn't follow the SAP Activate Methodology prescriptively but chose the tasks and activities we felt needed to be in our project scope.
An SAP Roadmap Programme Board was established at the outset of the S/4HANA project to oversee key projects with significant touch points with SAP. This has met monthly since 2018 and is comprised of senior University representatives from each of the major business units and the IT Service. A number of projects have been successfully delivered under the board's watchful eye and new projects have been added to its remit due to the passage of time. Recently the board has been renamed to the Enterprise Systems Roadmap Board, to reflect the vendor agnostic role it plays, albeit SAP projects feature significantly.
The S/4HANA project had a steering group that met monthly, reporting project highlight reports to the board. Members included managers from Finance, People Services and the IT Service. Given the significant impact of S/4HANA was weighted towards the Finance line of business, there was greater representation from this business area.
Once our engagement with SAP Premium Engagement had begun, we held periodic steering group meetings with SAP, which proved helpful to establish relationships, formulate the supporting SAP Value Assurance delivery plans and agree escalation routes. Attendees from SAP included senior executives, relationship managers and our assigned Technical Quality Manager (TQM). We had two TQMs during the project and they acted as the glue between SAP resources when seeking clarification or issue resolution.
As an early adopter of S/4HANA we were able to apply to be a member of SAP's S/4HANA Customer Care programme. This was a free service offered to SAP customers meeting eligibility criteria, who would benefit from additional support as early adopters of S/4HANA. The programme is supported by an SAP Regional Implementation Group (RIG) team for the various continents. At the start of the engagement, the RIG (EMEA) team arranged a series of virtual awareness and training sessions on various S/4HANA topics, including: Finance, Logistics, HR, CRM, Fiori, Extensibility, Architecture and the Belize Theme. These were led by domain experts and proved very informative. The sessions were recorded for future reference. We were assigned a nominated contact (SAP Enterprise Architect) who joined us on calls once a fortnight to monitor progress, answer any queries we had and to escalate issues on our behalf. This was a reassuring relationship to establish, although on several occasions some issues still took longer to resolve than we would have liked despite being well supported.
We have been a latecomer to Enterprise Support having only moved from Standard Support for our SAP maintenance contract in early 2018. itelligence UK are our service provider and have helped us to leverage value from the various value-added services that Enterprise Support offers SAP customers. We were assigned an itelligence Service Delivery manager during the project, who is one of our key contacts when we need their support or to help with SAP incident escalation.
The core project team met weekly to manage the many spinning plates and deliver status updates. The group included finance business managers and key users plus representatives from each of our SAP teams (Finance Systems, HR/Payroll/CRM Systems, Student Systems, Business Intelligence, SAP Development and SAP Basis). Towards the latter stages of the project (and into the hypercare period), a subset of this group met daily for a short stand up meeting to maintain the discussion and momentum.
And finally, task and finish style working groups were established throughout the project to focus on key deliverables such as the Customer Vendor Integration (CVI) conversion or dependent projects such as the SAP GUI 7.6 upgrade.
Few stones were left unturned in our research on S/4HANA. I won't attempt to boil the ocean and list all the research sources we used during our S/4HANA project, and I'm aware readers are likely to be interested in different aspects depending on their role, but I'd like to share a select few of the key resources I personally found useful in my role as an SAP architect which covers the full spectrum of S/4HANA related topics. Part 2 of this blog post series covered the available SAP Digital Transformation tools, which are also a key research option. There's a ton of material out there. In fact one of the problems is that there is simply too much content and a lot of repetition. I know SAP has acknowledged this and is making strides to consolidate where possible, including the tooling.
Conferences & User Groups
I consider myself very fortunate to have been afforded the opportunity by the University during my career to attend community events all around the world. Being able to triangulate the messages from SAP, partners and customers helps to obtain a comprehensive understanding of the benefits and reality of change. From an S/4HANA standpoint, we've regularly sent staff to conferences such as SAP Tech Ed to get the low down on the latest developments and user groups both nationally (e.g. hosted by UK & Ireland SAP User Group) and internationally (e.g. SAP Higher Education & Research User Group). The global, industry-specific nature of the latter event has proven to be of high value to keep a finger on pulse of our peer institutions, and a source of inspiration for many of our historical projects. Following Purdue University's go live with S/4HANA (the first university globally) we had a fruitful call with their implementation partner EPI-USE to glean experiences and lessons learned.
openSAP
Long gone are the days when SAP project team members had to travel to training centres for a week at a time to upskill. I'm a big fan of the free MOOC openSAP courses. I sometimes cannot afford the time to complete all aspects of the courses or participate to the course schedule. If I find a course of interest, I'll scope the effort to watch the videos (watching at x1.25 speed compresses the time) and usually find a typical four-week course can be completed in a morning or afternoon. We were fortunate that SAP launched the System Conversion to SAP S/4HANA course during our prepare and explore phases. We scheduled weekly meetings when interested project team members could get together and watch that week's recordings. It provided an opportunity to obtain commitment and stimulate discussion.
Here are my course recommendations, with a focus on S/4HANA system conversions:
SAP Learning Hub
When I tried searching for 'S/4HANA' in the SAP Learning Hub (Professional Edition) Catalog it returned me 2475 titles! I'm not going to call out any specific items, but if this is a resource you can access then I recommend using it. We're lucky in that we have obtained discounted licences via the UK & Ireland User Group. They have brokered a deal with SAP which has enabled us to purchase licenses for a number of our team members. We've used the Learning Journeys to compile training paths and the Learning Rooms to engage with community members on S/4HANA topics. I've been impressed with the content offered from the SAP Enterprise Support Academy, such as the Expert-Led Guided Implementation (EGI) offerings. Some of these can be consumed on demand in the form of e-books, while others are scheduled as virtual learning courses allowing participants to interact with the instructor and ask questions.
SAP Community
This is such a valuable resource, albeit at times overwhelming due to the sheer volume of content. My ruthless shortlist therefore has to be the SAP Fiori for S4HANA WIKI. This is kept up-to-date with the latest content published in the community. One of the key topics covered is Fiori Elements which I believe can offer us a lot of value for routine app development scenarios (low-code/no-code solution). We started dabbling with this a few years ago but it has really matured in the past year.
Consulting
Most of the major SAP partners have published content or provided tools to guide customers on their S/4HANA journey.
Gartner
Please note the following research documents require a Gartner subscription (we have a subscription for University staff members).
The University has a large IT department of circa 250 staff, supporting the diverse needs of students, academic and administrative staff. Within Corporate Systems, the department in which our SAP teams are based, we have around 30 SAP specialists. As Derek Prior (former Gartner Research Director) aptly states in Resulting IT's The Ultimate SAP Centre of Excellence (CoE) Guide:
I can testify to the value of having an internal SAP CoE team, especially in complex SAP system environments. That's not to say we don't rely on SAP and partners to complement our internal resource, but we use this selectively. I consider our resourcing model like an iceberg, where the (small) part sticking out of the water is the external (premium) resource we call upon, while the bulk of the work is delivered by internal resource. This means we are very much self-sufficient. Our internal teams have intimate knowledge and context that can only be obtained from extensive exposure to our systems, technologies, projects and processes.
We approached a number of SAP partners during our S/4HANA discovery phase in 2017 but it was evident at the time that they lacked the practical, project experience we sought. Ultimately, we approached SAP Premium Engagement and selected a number of their recommended SAP Value Assurance for S/4HANA services. SAP acknowledged they too were in the early stages of building their S/4HANA capability through project exposure. The decision was taken that our S/4HANA implementation would be led by our internal IT team but validated by SAP to ensure the approach was sound. In total we consumed around 150 days of SAP resource time, the majority of which was utilised to prepare and deliver the workshops delivered below. For support with our Customer Vendor Conversion (CVI) and Fiori (SAPUI5) adjustments for S/4HANA we used a small number of consultancy days from itelligence UK and Netherlands. I anticipate we will need more external resource to support phase 2 of our S/4HANA implementation, particularly the first deliverable, to ensure we obtain the necessary knowledge and skills to align with SAP best practice.
Once we briefed SAP on the work we had done to date, we secured the following Value Assurance and Enterprise Support services:
The image below highlights the key messages we wanted to land to our business user community regarding the impact of S/4HANA phase 1 go live. While all business units had to perform regression testing, the majority of mandatory process changes were confined to the finance area which was most impacted through the S/4HANA simplifications. We were able to lump in the annual SAP updates we make to all our systems, to ensure there would be no need for further testing to accommodate these before go live. Fortunately, 2018 was the first year SAP reduced the dependency between HR Sync Packs and S/4HANA stack levels so we were able to avoid the need for patching later in the project that would have impacted more than just the HR area.
While we could articulate SAP's mantra 'Run Simple' and what this meant from a business perspective for our finance processes, the message is quite different for our HR and student processes. HR processes remained largely untouched, with the exception of the Simplification Item for employee business partner conversion. Student processes were the least impacted, as while a simplified data model is provided by SAP, it is still optional unlike the simplified finance data model which is mandatory part of the migration to S/4HANA. We plan to consider the new student data model as a phase 2 activity.
Compatibility Scope is somewhat of a double-edged sword. On the one hand it gives us time to tackle the complete migration to new and improved S/4HANA technical and functional capabilities. On the other hand, however, there's the danger that customers consider the job done and ignore items hidden in that scope. We have key business functions such as Travel Management (expenses) that still sit within Compatibility Scope even in the latest S/4HANA 1909 release, which we'll need to monitor carefully and act upon.
Belize Theme on SAP GUI (for Windows)
The change with the broadest user impact was as a result of our decision to introduce the Belize Theme for SAP GUI transactions. We had received a walk through of Belize by SAP and conducted our own research, the white paper - SAP Fiori Visual Theme for Classic Applications - offering a helpful explanation of the benefits. Opinions and user preferences varied, and I was a little sceptical myself at first but willing to give it a try. Our SAP Training team conducted user research with around 10 users from different parts of the University with varying experience with SAP GUI, to seek their candid views. What surprised me was the response, that users overwhelming voted in favour of the Belize Theme versus the existing Blue Crystal Theme. There were comments that Belize looked much more like other modern systems they use, screen estate was utilised more effectively, and how text for user actions was more intuitive than icons. Not everyone likes Belize, but muscle memory comes into play here. Once our users had spent a little time working with the Belize Theme, it became familiar and a lot of the change friction dissipated. On larger monitors, the mouse travel to reach buttons on the footer has frustrated some users and is something we will need to monitor. We have found SAP and custom screens which needed some adjustments to improve the usability (e.g. moving menu items or alignment issues), with the SAP Help on Customization Guidelines proving helpful. Our direct contact with the SAP Student Lifecycle Management product team has enabled issues in that functional area to be reported and fixed promptly.
After the promising user consultation, the decision was taken to proceed with the Belize Theme. We were in the process of rolling out an upgrade to SAP GUI 7.6 in the months leading up to our S/4HANA go live. We discovered a neat feature in system behaviour. By setting the theme properties as shown below, this had the advantage that users conducting their S/4HANA testing in our test systems would see the Belize Theme, which is what we intended to go live with for S/4HANA. However, users would continue to see the old Blue Crystal Theme for business as usual, as the SAP ERP Production system didn't support SAP Fiori Features (only S/4HANA systems). This made change management easier as we only needed to roll out one change to the SAP GUI, but the theme change would be handled for us automatically.
The majority of users involved in testing S/4HANA had their PC's upgraded to SAP GUI 7.6 early in the project. Our SAP Training Team compiled an e-learning course which provided an opportunity for users to familiarise themselves with the key changes delivered with Belize.
Belize Theme on SAP Fiori & Fiori 2.0
We were running Fiori 1.0 on our SAP Gateway (MyApps) system with a custom developed theme based on Blue Crystal Theme, plus extensive customisation of the styling (using CSS rules) to match University brand guidelines. When we went live with Fiori in 2016 there was a policy of ensuring corporate applications were branded to our house style. With the move to S/4HANA, however, we took the conscious decision to revert back to SAP standard with the Belize Theme set as the user default, only changing the SAP logo to our own. We also made the Belize High Contrast Black and White theme variants available to users to support accessibility needs.
We followed the SAP guidance on migrating to Fiori 2.0, and once the work on themes was complete, turned our attention to the new enhanced Fiori features. Although we set up all the pre-requisites for Fiori Launchpad Personalisation, Enterprise Search and Notifications and got this working, we didn't switch on these features for S/4HANA phase 1 go live. This was in part due to a lack of time. There was also no great urgency, as we could always turn these features on later when we had more time to full consider the user benefit and impact. Now that Fiori 3.0 has been released with S/4HANA 1909 (see this blog post for key differences), we may well move to that first to avoid two change exercises.
Early trial conversions revealed that we would need to implement the Fiori app Manage Bank Accounts as a replacement to the SAP GUI classic transaction FI12. Although we managed to set this up, it wasn't very appealing to enable such a specific function on its own in Fiori and disjoint the user experience. We engaged with SAP and they released SAP note 2646577 - Re-enabling of Transaction Code FI12 for maintaining House Banks. We've since moved to using transaction FI12_HBANK with S/4HANA as described in SAP note 2424544 - How to Create House Bank Account in S/4 HANA but we're now considering provisioning limited access to the Fiori app.
One of the first recommended steps to take on a customer's S/4HANA journey is to run the SAP Readiness Check tool (further guidance offered in this blog post). We ran a number of analyses using the tool for different S/4HANA release/stack versions which became available during our project. The results output for our target version (S/4HANA 1709 SPS04) are shown below. This was using version 1.0 of the tool, but we're currently in the process of running the newer 2.0 version to scope the changes delivered in more recent S/4HANA releases.
A whole blog post could be written on the results of the Readiness Check. I covered the sections 'Recommended SAP Fiori Apps' and 'Business Process Analytics' (under Pathfinder) in part 2 of this blog post series, but I'll provide a summary below for the other sections of the tool.
Active Business Functions
We had 22 business functions active within software components belonging to the HR, Finance and Student Lifecycle Management areas. All of these were compatible with S/4HANA without any further action being necessary. This blog post provides guidance on the implications of changes to business functions brought about in S/4HANA.
Add-On Compatibility
We had 14 add-ons of which three related to SAP standard functionality (integration add-on 3.0 for SAP ERP HCM and SAP SuccessFactors HCM Suite, SAP Fiori 1.0 for SAP ERP HCM and SAP GRC Access Control 5.3). The other 11 related to non-SAP, third party functionality for our Accounts Payable (WNS) solution, an ABAP extension. We reached out to WNS very early during the discovery phase of our S/4HANA project. The net result was they developed an S/4HANA compliant version of their add-ons, which we implemented across our SAP ERP system landscape as a discrete project well ahead of S/4HANA.
Data Volume Management
As part of a University project called Data Retention, a significant amount of work has been undertaken to review data requirements, including retention obligations from a regulatory perspective. Some quick wins have been achieved in recent years by purging old, unnecessary data from large SAP tables. We've also implemented SAP Information Lifecycle Management (ILM), together with standard SAP archiving functionality (transaction SARA) to manage the archiving and deleting of data in our SAP ERP Production system. Periodic review and purging of old table change protocol data has also taken place. The net result is that we didn't have any significant need to perform any archiving prior to our move to S/4HANA. There is still data we would like to revisit and we plan to utilise ILM to support these efforts.
SAP S/4HANA Sizing
As mentioned earlier in this blog post, we purchased new HANA hardware and optimised the configuration prior to converting to S/4HANA. Consequently, sizing concerns were factored into the hardware procurement exercise and SAP validated our approach during the Technical Architecture and Infrastructure (TAI) workshop we held with them.
Business Warehouse Extractors
Although the Readiness Check indicated potential problems with some of the SAP standard BW extractors, the impact was relatively low. During the S/4HANA impact assessment (Sandbox Phase), significant performance problems were identified with the Employee extractor (0EMPLOYEE_ATTR). Due to a combination of time constraints and functional requirements to extend this extractor, our Business Intelligence team re-wrote it to meet our needs. SAP has since recognised the performance issue and released a correction. These adjustments were implemented on the SAP ERP landscape to be release agnostic, and ready ahead of the move to S/4HANA.
Simplification Items
The Readiness Check highlighted 84 Simplification Items that were relevant to us. We downloaded these to a spreadsheet and annotated with our preliminary investigations. We classified them by functional area impacted, assigned project workstream (with assigned leads) and figured out which teams would need to be involved or consulted. After detailed analysis, we concluded there were only 15 items with significant impact. This is where we obtained real value from our SAP Premium Engagement contract. Via our SAP Technical Quality Manager (TQM), we managed to arrange Expert on Demand consultancy to discuss the intricate details of these items. Our TQM also facilitated a number of calls with SAP to escalate the resolution of product defects.
The two standout items which incurred the most complexity and effort were:
Both items were focused upon from the first trial conversion. Through repetition and refinement of the process, together with increased understanding of the impact and a healthy dose of data cleansing, the process became easier. The employee business partner conversion proved problematic and we had two occasions when incidents were logged with SAP that went back and forth for an eternity before the issue was fully resolved, one of which took six months and over 100 ticket updates! This proved to be one of the downsides of an early adopter of S/4HANA, making planning difficult and concerning.
Funds Management
Despite not being related to the Readiness Check or Simplification Items, we decided to deactivate Funds Management (FM) during our move to S/4HANA. FM is an SAP Finance module that supports the creation and execution of budgets. The purpose of FM is to budget all revenues and expenditures for individual areas of responsibility, to control future funds transactions in accordance with the distributed budget and to stop the budget being exceeded. FM was activated as part of the implementation of SAP Finance in 1999 but was never properly utilised. We didn't feel we needed it.
During the early testing of S/4HANA, an issue was identified in the FM master data that prevented certain financial postings. Changing the master data settings of commitment item was proven to be one viable solution, but this would have required a significant maintenance exercise due to the data volumes involved. We theorised about the implications of turning off FM then tested this as part of one of our early trial S/4HANA conversions. Switching FM off resolved the S/4HANA posting issues but presented some new challenges. We identified a problem with discount splitting for student tuition fee accounts, which was resolved by introducing contract objects; this also resolved another regulatory (HESA) reporting issue. A third issue related to BW drilldown reporting on tuition fees, which was overcome using an alternative solution.
With the newfound confidence from impact assessment, we proceeded to evaluate the deactivation of FM further during repeated trial conversions. The first conversion went well, then we hit a snag. The second conversion failed (loss of leading ledger) and we didn't know why as the same procedure had been followed each time, or so we thought. Analysis revealed that we had performed financial postings after the first conversion but before FM was deactivated, but this hadn't been the case for the second conversion. We opened an SAP incident as this was a critical part of our project and the guidance on FM deactivation wasn't entirely clear. It took a while for the incident to be resolved and it required escalation via the SAP Premium Engagement team who pulled the necessary strings, resulting in an SAP correction to fix the leading ledger issue.
Custom Code Analysis
We currently have a high volume of custom ABAP code. During phase 2 of our S/4HANA project, we hope to significantly reduce the level of system modification and customisation. The Readiness Check indicated 43 SAP notes to guide our developers and analysts on the necessary code adjustments. In total we touched around 260 ABAP objects as a result of resolving issues identified from the ABAP Test Cockpit variant 'S4HANA_READINESS' and the Simplification items. Our focus was on resolution of errors (priority 1 & 2 check messages). One top tip I can recommend was our usage of abapGit to backup ABAP code changes to a GitHub repository; this enabled us to re-apply the changes after each trial S/4HANA conversion with little effort. As we had already moved to Suite on HANA for our SAP ERP system in 2016, we'd already done the heavy lifting to adjust and optimise code from a HANA functional and performance perspective (e.g. ensuring correct database sort order). Additionally, during the S/4HANA SPAU we had to adjust 551 objects, of which 247 related to the re-application of custom modifications.
For many years we have used an in-house developed solution to monitor usage statistics for our custom ABAP objects. This builds on the usage data available in SAP transaction ST03N, but extends the history up to two years. We had been utilising this data to identify obsolete code and delete it from our systems, or to mark objects as candidates for deletion subject to a further monitoring. The process had served us well but we decided to consider the capabilities offered in SAP Solution Manager for Custom Code Lifecycle Management (CCLM). Members of our development team participated in an Expert Guided Implementation (EGI) course on CCLM as a primer. We then held a two-day workshop with an SAP CCLM specialist who helped us understand the functionality better and set up a workflow to monitor and decommission (retire) obsolete code. As of Jan 2020 we will have captured a full year of usage data using the CCLM tool so can be confident in deleting flagged objects using the decommissioning cockpit.
Authorisations
We followed the S/4HANA best practice guidance to review and update changes to SAP authorisation defaults within our custom authorisation roles using transaction SU25. This was the first time we had ran this transaction since 2013, which meant there was some housekeeping to perform. We had to change 40 of the 517 roles actively used in our SAP ERP Production system. We created a further five new roles. While the role adjustment procedure followed for the S/4HANA conversion proved relatively straightforward, we relied too heavily on existing test scripts which meant we didn't catch all authorisation issues during testing.
Given the significant changes in the conversion to S/4HANA we decided to bring users testing forward much earlier in the project at the Sandbox Phase, which we normally confine to only analyst testing. By asking users to run their core business processes on a recent SAP ERP Production system copy that had been converted to S/4HANA, meant we were able to flush out the majority of issues early with sufficient time to resolve. The downside being this was additional effort on top of user acceptance testing that was still performed on the QA environment. We don't currently have any form of test automation in place, so testing is done manually and is a significant resource commitment. Securing test resource was difficult as many business users were under intense pressure due to key University activity such as Clearing. But we know there's no good time for all users and everyone did their best to fit it in.
We managed the testing using an established mechanism which uses an integrated set of spreadsheets to capture test coverage, progress and issues for each phase of testing. This was used to monitor progress during our daily stand up meetings. Teams can see issues assigned to their areas at a glance and email notifications are sent upon issue logging.
During the Sandbox phase, 194 issues were logged by analysts, compared with 39 issues on Development and 111 issues on QA. The dip during the Development phase was simply due to the short nature of this phase and the lack of quality test data in this environment, hence we moved quickly to QA to enable user acceptance testing to start.
Volume Testing
We did consider the SAP Value Assurance (Safeguarding) service called Volume Test Optimization, but there was a dependency on the customer already having a load testing tool in place (e.g. Microfocus LoadRunner or similar). We had some established load testing processes in place for our Student Self Service portal, a custom developed solution that integrates with SAP, but nothing for our three SAP user interfaces: SAP GUI, SAP Portal and SAP Gateway. So we decided volume testing would need to be carried out manually. This would be in the form of 'day in the life' and 'night in the life' tests to replicate typical Production load on our QA environment, which was built to the same specification from an architectural and hardware resource standpoint.
The 'day in the life' test was conducted a few weeks before go live. It was due to be earlier but other priorities took precedence. This involved key business users and IT staff performing a period of intensive system activity for 30 minutes, running the applications and transactions we would expect to be running immediately after S/4HANA go live. In a perfect world this would have been longer and extended into soak testing had we had test automation, but we didn't have the tools nor the capacity to do more. We used a web form to capture issues during the test. This helped to identify a few issues early and resolve them before the test ended. A few functional issues were logged but nothing of concern relating to system performance, so another milestone had been achieved on our preparation for go live.
We conducted the 'night in the life' test several times. Firstly, we extracted the run times for all the scheduled batch jobs that were running nightly on our SAP ERP Production system. We captured this data twice, once shortly before moving our Production systems to the new HANA hardware we phased in before S/4HANA go live, and once afterwards. The same batch jobs were then scheduled on our SAP S/4HANA QA system for comparison purposes. The net result was that the cumulative effect of the new hardware and move to S/4HANA showed a small performance improvement (~15% average per job). We weren't expecting a great difference as our SAP ERP Production system was already performing well, but the purpose of the benchmarking exercise was a mitigating measure to ensure no detrimental impact at go live from all the changes.
Despite S/4HANA taking the lion's share of the SAP team's resource, we had many other in-flight projects and changes moving through the landscape at the start of the project. We put out a call to action during Jul 2019 requesting that teams avoid starting any major new initiatives and to focus on pushing changes through our SAP landscape to Production wherever possible ahead of the cutover to Development in Oct 2019. There were a few developments that had been open for a long time sat on the SAP ERP QA environment which we decided to back out, ensuring a solid system state for user acceptance testing. We wanted to avoid unwanted surprises due to landscape differences.
During the run up to the Development cutover we built a full Legacy Track by cloning the existing Development and QA systems, to support the path to Production for SAP ERP changes. This was the first time since the early SAP R/3 days that we had used a Legacy Track to support a maintenance project. We contemplated whether we needed the full complement of SAP systems in our landscape but decided that compromises shouldn't be made and built clones of all systems.
In the main, Business as Usual (BAU) changes were confined to the SAP ERP system which was to be expected given this it was our largest and most critical system. The provision of the dual track landscape also enabled us to support the bedding in of changes that had recently gone live to support the integration of our SAP ERP system with HubSpot (third party marketing automation platform).
With the dual track landscape, came the requirement to ensure changes made on the Legacy Track (ERP) were correctly retrofitted to the Project Track (S/4HANA). We were conscious that Solution Manager had a number of solutions that could help us manage this requirement (e.g. Standalone Retrofit, Cross System Object Lock & Change and Release Management) but we didn't have the capacity to set this up in time. The compromise we made was to use the following SAP reports that were already available in our SAP ERP and S/4HANA systems ready for use:
We locked all users on the Legacy Development systems before they were released and access was provided on a 'need to use' basis. We used a simple web form builder tool used by the University to publish a web form used to capture Legacy change requests. After submission, requests would be discussed at our daily stand up meeting the following day and approved or rejected as appropriate. Decisions were captured in a spreadsheet exported from the web form tool for audit purposes and as a reference to grant user access to Legacy Development.
Production Cutover
We performed the final dress rehearsal in the weeks running up to go live. This was to convert a fresh copy of our SAP ERP Production system to validate our cutover plan and obtain timings for key activities. Our cutover plan was a shared Excel spreadsheet containing a detailed task list by phase, with links to team's own workbooks and the master transport list. Even with all the careful planning and effort to that point, we couldn't be certain precisely when the technical conversion and finance data migration steps would be complete and system verification would start. We had seen varying task completion times from different system conversions, partly due to variables such as data quality and server resources, but hoped Production would be slick.
As per our usual maintenance approach, all SAP systems (plus connected, dependent third party applications) entered downtime at noon on the Fri of cutover weekend (Fri 24 Jan 2020). Then we started the S/4HANA conversion and applied updates to the other SAP systems. We had secured a longer downtime window than usual, running until 09:00 on Tue 29 Jan 2020, giving us the Mon as contingency should we need it. The first period from Fri afternoon through Sat morning went as planned before we hit our first snag. Despite the SPAU process completing cleanly without issue during QA cutover during which modified SAP standard objects are adjusted, in Production the SAP notes did not adjust as expected. The workaround was to manually resolve each SAP note then perform version checking with Development and QA to ensure the integrity of the system was intact. Following SPAU, the remaining cutover steps on S/4HANA went to plan, with all the major tasks completed by end of Sat 25 Jan 2020 including the all-important finance reconciliation report which checked out as required. The only other significant issue that day was another difference in behaviour when upgrading the SAP Portal, which required application of an SAP note to overcome. Early on the morning of Sun 26 Jan 2020 we performed a comprehensive check of transports, object versions and core integration tests, to ensure the system was ready for integrity checking. This included 171 tests covering select, core and business critical processes. During Sunday testing, 14 defects were logged, investigated and resolved, with one notable exception (discussed below). This aside, we were able to release all SAP Production systems back to resume normal service on the morning of Mon 27 Jan 2020, a day ahead of schedule.
In the weeks preceding go live, we had been been attempting to mitigate one of our higher scoring risks. This related to a discrepancy in our SAP landscape whereby our SAP Gateway system (known to our users as MyApps) had been reconfigured on all test systems from Shibboleth (an identity management solution commonly used in higher education) to Microsoft Active Directory Federation Services (ADFS). In the months leading up to the S/4HANA project starting we had implemented Microsoft Azure Multi-Factor Authentication together with ADFS in a bid to strengthen system security. We ran out of time and had to park this project to pick back up again post S/4HANA go-live, hence our SAP Gateway Production system retained Shibboleth. We felt confident that, despite not having been able to test the updates to Gateway with Shibboleth, that the authentication would most likely continue to work after go live. A few weeks before go live we decided to try and re-configure our Sandbox Gateway system to revert back to Shibboleth. Despite a dogged effort, we could not get this working due to an error in the SAML message exchange. Consequently, we prepared a contingency plan. With hindsight, we wrongly assumed the problem somehow related to the reconfiguration, rather than the changes to Gateway brought in with our S/4HANA project. To our surprise at go live SAP Gateway authentication didn't work, so our contingency plan was instigated. While investigating the issue internally, we opened a 'Very High' priority incident with SAP and reached out for support via our SAP Technical Quality Manager (TQM) and assigned SAP NetWeaver Engineer for that day. Although an SAP note was identified in the area of SAML session handling, it was already applied to our Gateway systems. The SAP incident was forwarded to development support at the mothership in Walldorf. After much effort and stress, a breakthrough was reached on a call with the assigned SAP lead. SAP suggested re-configuring the SAML 2.0 configuration to change from Back-Channel Communication (multiple message exchange) to Front-Channel Communication (SAML messages are transported in a single HTTP POST). We implemented the change during the call, and confirmed the authentication was working. We suspect a bug may have been introduced with the Gateway updates but await SAP's further investigation into the root cause of the Back-Channel Communication issue. Following the call we performed testing of each of our Fiori app processes before releasing the MyApps system early afternoon on Tue 27 Jan 2020. The relief was palpable!
Post Go Live Hypercare
We decided to implement a period of hypercare from the point of S/4HANA go live for five weeks, to cover a complete monthly University business cycle, including finance quarter end and the first payroll calculation. We wanted transparency and scrutiny of change requests but didn't want to impede urgent corrections needed to resolve issues in Production. Similar to the web form we used during the dual track period, we captured change requests for review. If urgent, the agreement was managers of the functional area impacted could self approve by replying to the mailing list. This has proved to be a simple but effective process and helped share knowledge of issues across the project team.
During the week after go live, everyone was on the front foot to ensure any issues were captured and investigated quickly. We were proactively monitoring the systems, watching key KPIs (e.g. runtime errors and response times) to get a step ahead of incidents logged via our IT Service Desk where possible. At the time of writing (day 5 after go live), we've had a very busy week reviewing issues and fielding questions, but nothing reported of concern that we couldn't resolve. We do have some issues remaining from user acceptance testing to investigate further post go-live where we assessed the risk was acceptable (e.g. student fees and credit management being key).
Having just gone live with S/4HANA, we haven't yet formally captured lessons learned from the project, but Facebook's maxim "Done is better than perfect" springs to mind here. In the spirit of continuous improvement, you never stop learning and there's always room for improvement, but in a project this complex, delivering S/4HANA without detrimental business impact has been a huge achievement. The table below highlights some of the things we did well that led to the project's success and others where there there was room for improvement.
Update May 2020: following a successful period of bedding in our new S/4HANA system we eventually managed to hold a workshop to capture lessons learned from all key project stakeholders. These were captured in the following slide.
The University's Enterprise Systems Programme Board will meet during the first few months of 2020 to consider and determine the next priorities on our SAP roadmap. Research and scoping work is underway to collate all of the mandatory and known project commitments to help inform discussions. The IT Service is currently undergoing its own transformation exercise as it evolves to meet our user's needs. On the SAP side, we'll need to consider role and skills gaps (e.g. referring to Essential Roles for S/4HANA Fiori Projects, we don't currently have UX Lead). The customer will be at the centre of everything we do going forward, which hasn't been the case historically. A challenge will be balancing the many priorities, such as maintaining business continuity ('keeping the lights on'), evolving our IT Service to be service-driven, and the renovation and replacement of enterprise systems which are coming to the end of their maintenance cycles. It is certainly an exciting time to work for the University if you like change!
To help better understand the maintenance schedule for all the SAP products we run today, the SAP Product Availability Matrix was used to compile the spreadsheet shown below. We're currently evaluating a number of Enterprise Architecture tools that we hope will help to build views such as this to help with decision making. The spreadsheet is intended to serve as a helpful planning aid, to determine the sequence in which the SAP products we use today go out of maintenance. There's a lot of focus on the Dec 2025 deadline, but once we factored in that the SAP NetWeaver 7.5 Java component goes out of support after 31.12.2024 (used by our SAP Portal, Business Warehouse and Process Integration systems), combined with the timing of the University's annual maintenance window (Q1 each year), means that we will need to target completion of our SAP transition to new or successor products during the four remaining maintenance slots we have available before the end of 2024. There's no assumption that SAP will be the most appropriate solution for each of the products in scope, so we'll need to prepare business cases and options appraisals for each deliverables on our roadmap.
Focusing specifically on S/4HANA Phase 2, at a minimum we will need to upgrade our SAP Fiori Front-end Server as it goes out of maintenance at the end of this year. I'm advocating that we consider uplifting the S/4HANA major release and bundle in with our annual SAP maintenance exercise (the delta appears to be manageable). When we implemented SAP Gateway (our MyApps system) in 2016, SAP best practice was to deploy this as a standalone hub deployment and integrate with SAP back-end systems. This recommendation has now changed so it makes sense for us to collapse Gateway into S/4HANA early, given the simplification benefits this will have for Phase 2 (e.g. from both an architectural and authorisations perspective). Early thinking is we can complete or all of Phase 2 over the course of the next two annual maintenance slots.
This is part 3 of a three-part blog series:
If you have any questions on our S/4HANA implementation, please ask them below in the comments. Thanks for reading!
- Newcastle University's Journey to SAP S/4HANA - The What (part 1)
- Newcastle University's Journey to SAP S/4HANA - The Why (part 2)
- Newcastle University's Journey to SAP S/4HANA - The How (part 3)
This post is broken down into the following sections: Project Planning | Project Governance | Research & Upskilling | Resourcing & Validation | The Impact of S/4HANA | Readiness Checks | Testing & Issues | Supporting Business as Usual | Go Live & Hypercare | Project Lessons Learned | What's Next?
Project Planning
The University has a well established annual SAP maintenance schedule. Work typically starts around the end of Oct each year, when the the latest software releases are scoped and impact assessed on our Sandbox environment, before being phased through our landscape by the end of Jan the following year. The Sandbox systems are usually recent copies from Production to enable representative testing to be carried out early in the project. This surfaces problems early and means we only commit to cutover to our live landscape if we're confident of meeting the target go live. The end-to-end project duration is around the three month mark.
With the move to S/4HANA the approach needed to be different given the increased complexity and risk due to the scale of change. We decided to perform repeated trial conversions (five in total) of our SAP ERP system using a fresh Production copy. This system was standalone and not integrated to any other systems, allowing us to focus our early project efforts specifically on the technical conversion, finance data migration, Customer Vendor Integration (CVI) conversion and employee business partner conversion. Only once these were sufficiently understood and rehearsed did we move to our Sandbox Phase, which introduced the many other SAP and non-SAP systems into the S/4HANA test cycle. Some of the relevant Simplification Items were addressed during the Trial Conversion Phase but the bulk were investigated during the Sandbox Phase. At the end of the Sandbox Phase we had a number of open issues but there was sufficient confidence to proceed to Development, which was followed in a matter of weeks by QA. User acceptance testing was initially planned to straddle the University closure for Christmas but was brought earlier, with most business units signed off by the end of Dec 2019. There is no time in the University calendar that suits all departments, with experience showing the end of Jan being the 'least worst' option for the majority of users. We went live as planned on Mon 27 Jan 2020 but had a contingency go live date secured for four weeks later at the end of Feb 2020 should we have needed it.
The high-level project plan
In addition to S/4HANA we had several other significant SAP projects running in parallel, most notably our first SuccessFactors module, Recruiting (go live Sep 2019) and the integration of HubSpot CRM (third party marketing automation platform) for inbound marketing (go live Sep 2019). Both of these projects involved integration with SAP ERP and were contending for the same resources in some teams, which was challenging at times.
We referred to the Transition to SAP S/4HANA Roadmap, particularly during the early planning stages of the project. If customers are really struggling to know how to start on their S/4HANA journey, this is one of the best resources. We certainly didn't follow the SAP Activate Methodology prescriptively but chose the tasks and activities we felt needed to be in our project scope.
Project Governance
An SAP Roadmap Programme Board was established at the outset of the S/4HANA project to oversee key projects with significant touch points with SAP. This has met monthly since 2018 and is comprised of senior University representatives from each of the major business units and the IT Service. A number of projects have been successfully delivered under the board's watchful eye and new projects have been added to its remit due to the passage of time. Recently the board has been renamed to the Enterprise Systems Roadmap Board, to reflect the vendor agnostic role it plays, albeit SAP projects feature significantly.
The S/4HANA project had a steering group that met monthly, reporting project highlight reports to the board. Members included managers from Finance, People Services and the IT Service. Given the significant impact of S/4HANA was weighted towards the Finance line of business, there was greater representation from this business area.
Once our engagement with SAP Premium Engagement had begun, we held periodic steering group meetings with SAP, which proved helpful to establish relationships, formulate the supporting SAP Value Assurance delivery plans and agree escalation routes. Attendees from SAP included senior executives, relationship managers and our assigned Technical Quality Manager (TQM). We had two TQMs during the project and they acted as the glue between SAP resources when seeking clarification or issue resolution.
As an early adopter of S/4HANA we were able to apply to be a member of SAP's S/4HANA Customer Care programme. This was a free service offered to SAP customers meeting eligibility criteria, who would benefit from additional support as early adopters of S/4HANA. The programme is supported by an SAP Regional Implementation Group (RIG) team for the various continents. At the start of the engagement, the RIG (EMEA) team arranged a series of virtual awareness and training sessions on various S/4HANA topics, including: Finance, Logistics, HR, CRM, Fiori, Extensibility, Architecture and the Belize Theme. These were led by domain experts and proved very informative. The sessions were recorded for future reference. We were assigned a nominated contact (SAP Enterprise Architect) who joined us on calls once a fortnight to monitor progress, answer any queries we had and to escalate issues on our behalf. This was a reassuring relationship to establish, although on several occasions some issues still took longer to resolve than we would have liked despite being well supported.
We have been a latecomer to Enterprise Support having only moved from Standard Support for our SAP maintenance contract in early 2018. itelligence UK are our service provider and have helped us to leverage value from the various value-added services that Enterprise Support offers SAP customers. We were assigned an itelligence Service Delivery manager during the project, who is one of our key contacts when we need their support or to help with SAP incident escalation.
The core project team met weekly to manage the many spinning plates and deliver status updates. The group included finance business managers and key users plus representatives from each of our SAP teams (Finance Systems, HR/Payroll/CRM Systems, Student Systems, Business Intelligence, SAP Development and SAP Basis). Towards the latter stages of the project (and into the hypercare period), a subset of this group met daily for a short stand up meeting to maintain the discussion and momentum.
And finally, task and finish style working groups were established throughout the project to focus on key deliverables such as the Customer Vendor Integration (CVI) conversion or dependent projects such as the SAP GUI 7.6 upgrade.
Research & Upskilling
Few stones were left unturned in our research on S/4HANA. I won't attempt to boil the ocean and list all the research sources we used during our S/4HANA project, and I'm aware readers are likely to be interested in different aspects depending on their role, but I'd like to share a select few of the key resources I personally found useful in my role as an SAP architect which covers the full spectrum of S/4HANA related topics. Part 2 of this blog post series covered the available SAP Digital Transformation tools, which are also a key research option. There's a ton of material out there. In fact one of the problems is that there is simply too much content and a lot of repetition. I know SAP has acknowledged this and is making strides to consolidate where possible, including the tooling.
Conferences & User Groups
I consider myself very fortunate to have been afforded the opportunity by the University during my career to attend community events all around the world. Being able to triangulate the messages from SAP, partners and customers helps to obtain a comprehensive understanding of the benefits and reality of change. From an S/4HANA standpoint, we've regularly sent staff to conferences such as SAP Tech Ed to get the low down on the latest developments and user groups both nationally (e.g. hosted by UK & Ireland SAP User Group) and internationally (e.g. SAP Higher Education & Research User Group). The global, industry-specific nature of the latter event has proven to be of high value to keep a finger on pulse of our peer institutions, and a source of inspiration for many of our historical projects. Following Purdue University's go live with S/4HANA (the first university globally) we had a fruitful call with their implementation partner EPI-USE to glean experiences and lessons learned.
openSAP
Long gone are the days when SAP project team members had to travel to training centres for a week at a time to upskill. I'm a big fan of the free MOOC openSAP courses. I sometimes cannot afford the time to complete all aspects of the courses or participate to the course schedule. If I find a course of interest, I'll scope the effort to watch the videos (watching at x1.25 speed compresses the time) and usually find a typical four-week course can be completed in a morning or afternoon. We were fortunate that SAP launched the System Conversion to SAP S/4HANA course during our prepare and explore phases. We scheduled weekly meetings when interested project team members could get together and watch that week's recordings. It provided an opportunity to obtain commitment and stimulate discussion.
Here are my course recommendations, with a focus on S/4HANA system conversions:
- Find Your Path to SAP S/4HANA
- System Conversion to SAP S/4HANA
- Key Technical Topics in a System Conversion to SAP S/4HANA
- Key Functional Topics in a System Conversion to SAP S/4HANA
- Implementation of SAP S/4HANA
- SAP S/4HANA in a Nutshell
- Extending SAP S/4HANA with SAP HANA Cloud Platform
- Extending SAP S/4HANA Cloud and SAP S/4HANA
SAP Learning Hub
When I tried searching for 'S/4HANA' in the SAP Learning Hub (Professional Edition) Catalog it returned me 2475 titles! I'm not going to call out any specific items, but if this is a resource you can access then I recommend using it. We're lucky in that we have obtained discounted licences via the UK & Ireland User Group. They have brokered a deal with SAP which has enabled us to purchase licenses for a number of our team members. We've used the Learning Journeys to compile training paths and the Learning Rooms to engage with community members on S/4HANA topics. I've been impressed with the content offered from the SAP Enterprise Support Academy, such as the Expert-Led Guided Implementation (EGI) offerings. Some of these can be consumed on demand in the form of e-books, while others are scheduled as virtual learning courses allowing participants to interact with the instructor and ask questions.
SAP Community
This is such a valuable resource, albeit at times overwhelming due to the sheer volume of content. My ruthless shortlist therefore has to be the SAP Fiori for S4HANA WIKI. This is kept up-to-date with the latest content published in the community. One of the key topics covered is Fiori Elements which I believe can offer us a lot of value for routine app development scenarios (low-code/no-code solution). We started dabbling with this a few years ago but it has really matured in the past year.
Consulting
Most of the major SAP partners have published content or provided tools to guide customers on their S/4HANA journey.
- Resulting IT - Don't migrate to SAP S/4HANA until you've read this research
- An interesting quote from this article being:
"Only SAP customers whose ECC systems are in a complete mess are going to throw their config away and start over from scratch in greenfield fashion. For customers who spent heavily to customise their ECC solutions, and where the customisation is delivering critical business processes, brownfield migration makes total sense."
- itelligence UK - SAP S/4HANA Round Table Series
- Insightful discussion on all the key considerations.
Gartner
Please note the following research documents require a Gartner subscription (we have a subscription for University staff members).
- What Customers Need to Know When Considering a Move to S/4HANA — 2018 Update
- How Public-Sector CIOs Should Assess SAP S/4HANA
Resourcing & Validation
The University has a large IT department of circa 250 staff, supporting the diverse needs of students, academic and administrative staff. Within Corporate Systems, the department in which our SAP teams are based, we have around 30 SAP specialists. As Derek Prior (former Gartner Research Director) aptly states in Resulting IT's The Ultimate SAP Centre of Excellence (CoE) Guide:
"In our 20+ years working with SAP customers as advisors and in a research capacity, we hold one principle at the core of our values around SAP success. SAP customers who build a mature SAP Centre of Excellence achieve much greater SAP success than those who don’t."
I can testify to the value of having an internal SAP CoE team, especially in complex SAP system environments. That's not to say we don't rely on SAP and partners to complement our internal resource, but we use this selectively. I consider our resourcing model like an iceberg, where the (small) part sticking out of the water is the external (premium) resource we call upon, while the bulk of the work is delivered by internal resource. This means we are very much self-sufficient. Our internal teams have intimate knowledge and context that can only be obtained from extensive exposure to our systems, technologies, projects and processes.
We approached a number of SAP partners during our S/4HANA discovery phase in 2017 but it was evident at the time that they lacked the practical, project experience we sought. Ultimately, we approached SAP Premium Engagement and selected a number of their recommended SAP Value Assurance for S/4HANA services. SAP acknowledged they too were in the early stages of building their S/4HANA capability through project exposure. The decision was taken that our S/4HANA implementation would be led by our internal IT team but validated by SAP to ensure the approach was sound. In total we consumed around 150 days of SAP resource time, the majority of which was utilised to prepare and deliver the workshops delivered below. For support with our Customer Vendor Conversion (CVI) and Fiori (SAPUI5) adjustments for S/4HANA we used a small number of consultancy days from itelligence UK and Netherlands. I anticipate we will need more external resource to support phase 2 of our S/4HANA implementation, particularly the first deliverable, to ensure we obtain the necessary knowledge and skills to align with SAP best practice.
Once we briefed SAP on the work we had done to date, we secured the following Value Assurance and Enterprise Support services:
- Migration Planning Workshop (MPW) (3 day workshop)
- The MPW workshop covered all aspects of an S/4HANA project and was very comprehensive. Although we requested that SAP tailor this service to meet our requirements, the content delivered was mostly from their standard packaged service. The majority of the attendees were happy with the quality and outcome, though it did feel a little padded in some areas particularly as we were well primed on many of the topics and wanted to focus on issues we needed SAP's help to address. However, obtaining the necessary validation and SAP expert contacts was a key milestone achieved. In the weeks that followed, analysts followed SAP's workshop recommendations and we utilised a small number of Expert on Demand days to focus on specific issues, particularly around the outstanding Simplification Items. Topics discussed in the MPW included:
- Project Objectives
- System Conversion Process
- Mandatory Adjustments
- Custom Code
- Transition Approaches
- High-Level Planning
- Architecture & Sizing
- The MPW workshop covered all aspects of an S/4HANA project and was very comprehensive. Although we requested that SAP tailor this service to meet our requirements, the content delivered was mostly from their standard packaged service. The majority of the attendees were happy with the quality and outcome, though it did feel a little padded in some areas particularly as we were well primed on many of the topics and wanted to focus on issues we needed SAP's help to address. However, obtaining the necessary validation and SAP expert contacts was a key milestone achieved. In the weeks that followed, analysts followed SAP's workshop recommendations and we utilised a small number of Expert on Demand days to focus on specific issues, particularly around the outstanding Simplification Items. Topics discussed in the MPW included:
- Technical Architecture Infrastructure (TAI) (3 day workshop)
- During the MPW workshop, it became clear that we would need to provision a dual track SAP landscape to support Business as Usual (BAU) during the S/4HANA cutover process. We considered ways and means of accomplishing this using existing HANA hardware but it was a case of 'robbing Peter to pay Paul' as compromises would need to be made. Thankfully, a case was put forward and supported for budget to purchase additional HANA servers. We leveraged the TAI workshop to validate our thinking on how best to configure the hardware topography. The content and delivery was once again strong and we came away with a clear plan of action. SAP helped us to optimise the configuration of our existing HANA servers, particularly around the area of memory management. Shortly after the workshop, our new hardware arrived and implementation started. This supported not only the building of our Legacy Track (ERP), but also the reconfiguration of systems on our Project Track (S/4HANA) so we would be on new hardware before the S/4HANA conversion. Topics discussed in the TAI workshop included:
- HANA Sizing & Scalability
- Fiori Front End Server
- Technical Platform Options & Architecture
- High Availability, Disaster Recovery & Backup
- Dual Landscape & Change Management
- During the MPW workshop, it became clear that we would need to provision a dual track SAP landscape to support Business as Usual (BAU) during the S/4HANA cutover process. We considered ways and means of accomplishing this using existing HANA hardware but it was a case of 'robbing Peter to pay Paul' as compromises would need to be made. Thankfully, a case was put forward and supported for budget to purchase additional HANA servers. We leveraged the TAI workshop to validate our thinking on how best to configure the hardware topography. The content and delivery was once again strong and we came away with a clear plan of action. SAP helped us to optimise the configuration of our existing HANA servers, particularly around the area of memory management. Shortly after the workshop, our new hardware arrived and implementation started. This supported not only the building of our Legacy Track (ERP), but also the reconfiguration of systems on our Project Track (S/4HANA) so we would be on new hardware before the S/4HANA conversion. Topics discussed in the TAI workshop included:
- Going Live Support (on-call SAP resource)
- We had a blend of SAP specialist expertise providing cover over the go live cutover weekend, from the start of downtime on Fri 24 Jan 2020 until the systems were handed back on the afternoon of Mon 27 Jan 2020 inclusive. This included a blend of Basis technical skills (i.e. SUM, NetWeaver & HANA) plus our SAP Technical Quality Manager (TQM), who was our single point of contact for assistance.
- Continuous Quality Check (CQC) for Upgrade
- This CQC is delivered in two parts: Analysis and Verification (link to info sheet). The first part of the service (Analysis) was delivered three weeks ahead of go live. This comprised a 60 page Service Report containing a 24 item action plan with SAP recommendations to address with suggested deadlines (the majority to be addressed before go live). The actions were a mix of tuning suggestions, warnings requiring further investigation and a few false positives. It read very similar to the EarlyWatch Alert reporting we receive for our SAP Production systems on a weekly basis. Our Basis team reviewed the actions, addressed what they could in terms of resource capacity, while others have been parked to be looked at more closely in the coming weeks and months. The second part of the service (Verification) is scheduled for six weeks after go live, to check how the system has bedded in.
- Continuous Quality Check (CQC) for Going Live Support
- This CQC was delivered during the first five days after go live (link to info sheet). To help SAP focus their attention on key business critical processes, our analysts and key users identified and documented the anticipated business process activity that would be occurring immediately after go live. SAP monitored our systems each day and provided a report with observations and recommendations.
- Continuous Quality Check (CQC) for Technical Performance Optimization
- This CQC is scheduled to be delivered around 2-3 months after go live (link to info sheet), once again to verify the integrity of the system once we've conducted a few months of business operations on S/4HANA. The focus is specifically around HANA database performance and tuning.
Impact of S/4HANA
The image below highlights the key messages we wanted to land to our business user community regarding the impact of S/4HANA phase 1 go live. While all business units had to perform regression testing, the majority of mandatory process changes were confined to the finance area which was most impacted through the S/4HANA simplifications. We were able to lump in the annual SAP updates we make to all our systems, to ensure there would be no need for further testing to accommodate these before go live. Fortunately, 2018 was the first year SAP reduced the dependency between HR Sync Packs and S/4HANA stack levels so we were able to avoid the need for patching later in the project that would have impacted more than just the HR area.
Impact during phase 1 of our S/4HANA project
While we could articulate SAP's mantra 'Run Simple' and what this meant from a business perspective for our finance processes, the message is quite different for our HR and student processes. HR processes remained largely untouched, with the exception of the Simplification Item for employee business partner conversion. Student processes were the least impacted, as while a simplified data model is provided by SAP, it is still optional unlike the simplified finance data model which is mandatory part of the migration to S/4HANA. We plan to consider the new student data model as a phase 2 activity.
Compatibility Scope is somewhat of a double-edged sword. On the one hand it gives us time to tackle the complete migration to new and improved S/4HANA technical and functional capabilities. On the other hand, however, there's the danger that customers consider the job done and ignore items hidden in that scope. We have key business functions such as Travel Management (expenses) that still sit within Compatibility Scope even in the latest S/4HANA 1909 release, which we'll need to monitor carefully and act upon.
Belize Theme on SAP GUI (for Windows)
The change with the broadest user impact was as a result of our decision to introduce the Belize Theme for SAP GUI transactions. We had received a walk through of Belize by SAP and conducted our own research, the white paper - SAP Fiori Visual Theme for Classic Applications - offering a helpful explanation of the benefits. Opinions and user preferences varied, and I was a little sceptical myself at first but willing to give it a try. Our SAP Training team conducted user research with around 10 users from different parts of the University with varying experience with SAP GUI, to seek their candid views. What surprised me was the response, that users overwhelming voted in favour of the Belize Theme versus the existing Blue Crystal Theme. There were comments that Belize looked much more like other modern systems they use, screen estate was utilised more effectively, and how text for user actions was more intuitive than icons. Not everyone likes Belize, but muscle memory comes into play here. Once our users had spent a little time working with the Belize Theme, it became familiar and a lot of the change friction dissipated. On larger monitors, the mouse travel to reach buttons on the footer has frustrated some users and is something we will need to monitor. We have found SAP and custom screens which needed some adjustments to improve the usability (e.g. moving menu items or alignment issues), with the SAP Help on Customization Guidelines proving helpful. Our direct contact with the SAP Student Lifecycle Management product team has enabled issues in that functional area to be reported and fixed promptly.
Belize vs Blue Crystal on SAP GUI
After the promising user consultation, the decision was taken to proceed with the Belize Theme. We were in the process of rolling out an upgrade to SAP GUI 7.6 in the months leading up to our S/4HANA go live. We discovered a neat feature in system behaviour. By setting the theme properties as shown below, this had the advantage that users conducting their S/4HANA testing in our test systems would see the Belize Theme, which is what we intended to go live with for S/4HANA. However, users would continue to see the old Blue Crystal Theme for business as usual, as the SAP ERP Production system didn't support SAP Fiori Features (only S/4HANA systems). This made change management easier as we only needed to roll out one change to the SAP GUI, but the theme change would be handled for us automatically.
Settings to support the theme switch
The majority of users involved in testing S/4HANA had their PC's upgraded to SAP GUI 7.6 early in the project. Our SAP Training Team compiled an e-learning course which provided an opportunity for users to familiarise themselves with the key changes delivered with Belize.
Belize Theme on SAP Fiori & Fiori 2.0
We were running Fiori 1.0 on our SAP Gateway (MyApps) system with a custom developed theme based on Blue Crystal Theme, plus extensive customisation of the styling (using CSS rules) to match University brand guidelines. When we went live with Fiori in 2016 there was a policy of ensuring corporate applications were branded to our house style. With the move to S/4HANA, however, we took the conscious decision to revert back to SAP standard with the Belize Theme set as the user default, only changing the SAP logo to our own. We also made the Belize High Contrast Black and White theme variants available to users to support accessibility needs.
Custom Theme vs Belize Theme on SAP Gateway (MyApps) system
We followed the SAP guidance on migrating to Fiori 2.0, and once the work on themes was complete, turned our attention to the new enhanced Fiori features. Although we set up all the pre-requisites for Fiori Launchpad Personalisation, Enterprise Search and Notifications and got this working, we didn't switch on these features for S/4HANA phase 1 go live. This was in part due to a lack of time. There was also no great urgency, as we could always turn these features on later when we had more time to full consider the user benefit and impact. Now that Fiori 3.0 has been released with S/4HANA 1909 (see this blog post for key differences), we may well move to that first to avoid two change exercises.
Early trial conversions revealed that we would need to implement the Fiori app Manage Bank Accounts as a replacement to the SAP GUI classic transaction FI12. Although we managed to set this up, it wasn't very appealing to enable such a specific function on its own in Fiori and disjoint the user experience. We engaged with SAP and they released SAP note 2646577 - Re-enabling of Transaction Code FI12 for maintaining House Banks. We've since moved to using transaction FI12_HBANK with S/4HANA as described in SAP note 2424544 - How to Create House Bank Account in S/4 HANA but we're now considering provisioning limited access to the Fiori app.
Readiness Checks
One of the first recommended steps to take on a customer's S/4HANA journey is to run the SAP Readiness Check tool (further guidance offered in this blog post). We ran a number of analyses using the tool for different S/4HANA release/stack versions which became available during our project. The results output for our target version (S/4HANA 1709 SPS04) are shown below. This was using version 1.0 of the tool, but we're currently in the process of running the newer 2.0 version to scope the changes delivered in more recent S/4HANA releases.
Readiness Check results overview page
A whole blog post could be written on the results of the Readiness Check. I covered the sections 'Recommended SAP Fiori Apps' and 'Business Process Analytics' (under Pathfinder) in part 2 of this blog post series, but I'll provide a summary below for the other sections of the tool.
Active Business Functions
We had 22 business functions active within software components belonging to the HR, Finance and Student Lifecycle Management areas. All of these were compatible with S/4HANA without any further action being necessary. This blog post provides guidance on the implications of changes to business functions brought about in S/4HANA.
Add-On Compatibility
We had 14 add-ons of which three related to SAP standard functionality (integration add-on 3.0 for SAP ERP HCM and SAP SuccessFactors HCM Suite, SAP Fiori 1.0 for SAP ERP HCM and SAP GRC Access Control 5.3). The other 11 related to non-SAP, third party functionality for our Accounts Payable (WNS) solution, an ABAP extension. We reached out to WNS very early during the discovery phase of our S/4HANA project. The net result was they developed an S/4HANA compliant version of their add-ons, which we implemented across our SAP ERP system landscape as a discrete project well ahead of S/4HANA.
Data Volume Management
As part of a University project called Data Retention, a significant amount of work has been undertaken to review data requirements, including retention obligations from a regulatory perspective. Some quick wins have been achieved in recent years by purging old, unnecessary data from large SAP tables. We've also implemented SAP Information Lifecycle Management (ILM), together with standard SAP archiving functionality (transaction SARA) to manage the archiving and deleting of data in our SAP ERP Production system. Periodic review and purging of old table change protocol data has also taken place. The net result is that we didn't have any significant need to perform any archiving prior to our move to S/4HANA. There is still data we would like to revisit and we plan to utilise ILM to support these efforts.
SAP S/4HANA Sizing
As mentioned earlier in this blog post, we purchased new HANA hardware and optimised the configuration prior to converting to S/4HANA. Consequently, sizing concerns were factored into the hardware procurement exercise and SAP validated our approach during the Technical Architecture and Infrastructure (TAI) workshop we held with them.
Business Warehouse Extractors
Although the Readiness Check indicated potential problems with some of the SAP standard BW extractors, the impact was relatively low. During the S/4HANA impact assessment (Sandbox Phase), significant performance problems were identified with the Employee extractor (0EMPLOYEE_ATTR). Due to a combination of time constraints and functional requirements to extend this extractor, our Business Intelligence team re-wrote it to meet our needs. SAP has since recognised the performance issue and released a correction. These adjustments were implemented on the SAP ERP landscape to be release agnostic, and ready ahead of the move to S/4HANA.
Simplification Items
The Readiness Check highlighted 84 Simplification Items that were relevant to us. We downloaded these to a spreadsheet and annotated with our preliminary investigations. We classified them by functional area impacted, assigned project workstream (with assigned leads) and figured out which teams would need to be involved or consulted. After detailed analysis, we concluded there were only 15 items with significant impact. This is where we obtained real value from our SAP Premium Engagement contract. Via our SAP Technical Quality Manager (TQM), we managed to arrange Expert on Demand consultancy to discuss the intricate details of these items. Our TQM also facilitated a number of calls with SAP to escalate the resolution of product defects.
Key simplification items
The two standout items which incurred the most complexity and effort were:
- S4TWL - Business Partner Approach (CVI)
- S4TWL - Conversion of Employees to Business Partners
Both items were focused upon from the first trial conversion. Through repetition and refinement of the process, together with increased understanding of the impact and a healthy dose of data cleansing, the process became easier. The employee business partner conversion proved problematic and we had two occasions when incidents were logged with SAP that went back and forth for an eternity before the issue was fully resolved, one of which took six months and over 100 ticket updates! This proved to be one of the downsides of an early adopter of S/4HANA, making planning difficult and concerning.
Funds Management
Despite not being related to the Readiness Check or Simplification Items, we decided to deactivate Funds Management (FM) during our move to S/4HANA. FM is an SAP Finance module that supports the creation and execution of budgets. The purpose of FM is to budget all revenues and expenditures for individual areas of responsibility, to control future funds transactions in accordance with the distributed budget and to stop the budget being exceeded. FM was activated as part of the implementation of SAP Finance in 1999 but was never properly utilised. We didn't feel we needed it.
During the early testing of S/4HANA, an issue was identified in the FM master data that prevented certain financial postings. Changing the master data settings of commitment item was proven to be one viable solution, but this would have required a significant maintenance exercise due to the data volumes involved. We theorised about the implications of turning off FM then tested this as part of one of our early trial S/4HANA conversions. Switching FM off resolved the S/4HANA posting issues but presented some new challenges. We identified a problem with discount splitting for student tuition fee accounts, which was resolved by introducing contract objects; this also resolved another regulatory (HESA) reporting issue. A third issue related to BW drilldown reporting on tuition fees, which was overcome using an alternative solution.
With the newfound confidence from impact assessment, we proceeded to evaluate the deactivation of FM further during repeated trial conversions. The first conversion went well, then we hit a snag. The second conversion failed (loss of leading ledger) and we didn't know why as the same procedure had been followed each time, or so we thought. Analysis revealed that we had performed financial postings after the first conversion but before FM was deactivated, but this hadn't been the case for the second conversion. We opened an SAP incident as this was a critical part of our project and the guidance on FM deactivation wasn't entirely clear. It took a while for the incident to be resolved and it required escalation via the SAP Premium Engagement team who pulled the necessary strings, resulting in an SAP correction to fix the leading ledger issue.
Custom Code Analysis
We currently have a high volume of custom ABAP code. During phase 2 of our S/4HANA project, we hope to significantly reduce the level of system modification and customisation. The Readiness Check indicated 43 SAP notes to guide our developers and analysts on the necessary code adjustments. In total we touched around 260 ABAP objects as a result of resolving issues identified from the ABAP Test Cockpit variant 'S4HANA_READINESS' and the Simplification items. Our focus was on resolution of errors (priority 1 & 2 check messages). One top tip I can recommend was our usage of abapGit to backup ABAP code changes to a GitHub repository; this enabled us to re-apply the changes after each trial S/4HANA conversion with little effort. As we had already moved to Suite on HANA for our SAP ERP system in 2016, we'd already done the heavy lifting to adjust and optimise code from a HANA functional and performance perspective (e.g. ensuring correct database sort order). Additionally, during the S/4HANA SPAU we had to adjust 551 objects, of which 247 related to the re-application of custom modifications.
SAP note guidance for custom code
For many years we have used an in-house developed solution to monitor usage statistics for our custom ABAP objects. This builds on the usage data available in SAP transaction ST03N, but extends the history up to two years. We had been utilising this data to identify obsolete code and delete it from our systems, or to mark objects as candidates for deletion subject to a further monitoring. The process had served us well but we decided to consider the capabilities offered in SAP Solution Manager for Custom Code Lifecycle Management (CCLM). Members of our development team participated in an Expert Guided Implementation (EGI) course on CCLM as a primer. We then held a two-day workshop with an SAP CCLM specialist who helped us understand the functionality better and set up a workflow to monitor and decommission (retire) obsolete code. As of Jan 2020 we will have captured a full year of usage data using the CCLM tool so can be confident in deleting flagged objects using the decommissioning cockpit.
Authorisations
We followed the S/4HANA best practice guidance to review and update changes to SAP authorisation defaults within our custom authorisation roles using transaction SU25. This was the first time we had ran this transaction since 2013, which meant there was some housekeeping to perform. We had to change 40 of the 517 roles actively used in our SAP ERP Production system. We created a further five new roles. While the role adjustment procedure followed for the S/4HANA conversion proved relatively straightforward, we relied too heavily on existing test scripts which meant we didn't catch all authorisation issues during testing.
Testing and Issues
Given the significant changes in the conversion to S/4HANA we decided to bring users testing forward much earlier in the project at the Sandbox Phase, which we normally confine to only analyst testing. By asking users to run their core business processes on a recent SAP ERP Production system copy that had been converted to S/4HANA, meant we were able to flush out the majority of issues early with sufficient time to resolve. The downside being this was additional effort on top of user acceptance testing that was still performed on the QA environment. We don't currently have any form of test automation in place, so testing is done manually and is a significant resource commitment. Securing test resource was difficult as many business users were under intense pressure due to key University activity such as Clearing. But we know there's no good time for all users and everyone did their best to fit it in.
Test resource availability during Sandbox Phase
We managed the testing using an established mechanism which uses an integrated set of spreadsheets to capture test coverage, progress and issues for each phase of testing. This was used to monitor progress during our daily stand up meetings. Teams can see issues assigned to their areas at a glance and email notifications are sent upon issue logging.
Test spreadsheet with KPI reporting on progress and issues
During the Sandbox phase, 194 issues were logged by analysts, compared with 39 issues on Development and 111 issues on QA. The dip during the Development phase was simply due to the short nature of this phase and the lack of quality test data in this environment, hence we moved quickly to QA to enable user acceptance testing to start.
Issue throughput was steady
Volume Testing
We did consider the SAP Value Assurance (Safeguarding) service called Volume Test Optimization, but there was a dependency on the customer already having a load testing tool in place (e.g. Microfocus LoadRunner or similar). We had some established load testing processes in place for our Student Self Service portal, a custom developed solution that integrates with SAP, but nothing for our three SAP user interfaces: SAP GUI, SAP Portal and SAP Gateway. So we decided volume testing would need to be carried out manually. This would be in the form of 'day in the life' and 'night in the life' tests to replicate typical Production load on our QA environment, which was built to the same specification from an architectural and hardware resource standpoint.
The 'day in the life' test was conducted a few weeks before go live. It was due to be earlier but other priorities took precedence. This involved key business users and IT staff performing a period of intensive system activity for 30 minutes, running the applications and transactions we would expect to be running immediately after S/4HANA go live. In a perfect world this would have been longer and extended into soak testing had we had test automation, but we didn't have the tools nor the capacity to do more. We used a web form to capture issues during the test. This helped to identify a few issues early and resolve them before the test ended. A few functional issues were logged but nothing of concern relating to system performance, so another milestone had been achieved on our preparation for go live.
We conducted the 'night in the life' test several times. Firstly, we extracted the run times for all the scheduled batch jobs that were running nightly on our SAP ERP Production system. We captured this data twice, once shortly before moving our Production systems to the new HANA hardware we phased in before S/4HANA go live, and once afterwards. The same batch jobs were then scheduled on our SAP S/4HANA QA system for comparison purposes. The net result was that the cumulative effect of the new hardware and move to S/4HANA showed a small performance improvement (~15% average per job). We weren't expecting a great difference as our SAP ERP Production system was already performing well, but the purpose of the benchmarking exercise was a mitigating measure to ensure no detrimental impact at go live from all the changes.
Supporting Business as Usual
Despite S/4HANA taking the lion's share of the SAP team's resource, we had many other in-flight projects and changes moving through the landscape at the start of the project. We put out a call to action during Jul 2019 requesting that teams avoid starting any major new initiatives and to focus on pushing changes through our SAP landscape to Production wherever possible ahead of the cutover to Development in Oct 2019. There were a few developments that had been open for a long time sat on the SAP ERP QA environment which we decided to back out, ensuring a solid system state for user acceptance testing. We wanted to avoid unwanted surprises due to landscape differences.
During the run up to the Development cutover we built a full Legacy Track by cloning the existing Development and QA systems, to support the path to Production for SAP ERP changes. This was the first time since the early SAP R/3 days that we had used a Legacy Track to support a maintenance project. We contemplated whether we needed the full complement of SAP systems in our landscape but decided that compromises shouldn't be made and built clones of all systems.
The complex dual track landscape needed careful management
In the main, Business as Usual (BAU) changes were confined to the SAP ERP system which was to be expected given this it was our largest and most critical system. The provision of the dual track landscape also enabled us to support the bedding in of changes that had recently gone live to support the integration of our SAP ERP system with HubSpot (third party marketing automation platform).
With the dual track landscape, came the requirement to ensure changes made on the Legacy Track (ERP) were correctly retrofitted to the Project Track (S/4HANA). We were conscious that Solution Manager had a number of solutions that could help us manage this requirement (e.g. Standalone Retrofit, Cross System Object Lock & Change and Release Management) but we didn't have the capacity to set this up in time. The compromise we made was to use the following SAP reports that were already available in our SAP ERP and S/4HANA systems ready for use:
- /SDF/CMO_TR_CHECK
- this report predicts transport related errors before the requests are imported into a target system
- we managed to push all open SAP ERP transports through to Production before S/4HANA go live however, had this not been possible, this report would have enabled us to confirm whether issues would have arisen if the changes had been imported after go live (via cross-release check)
- /SDF/OBJ_TR_COMP
- this report compares workbench objects in two different systems
- we used this report to verify retrofitting had been done correctly by analysts and developers
- /SDF/OBJ_OVERLAP
- this report identifies the object intersection between two transport sets in the same or in different systems
- we used this report to verify retrofitting had been done correctly by analysts and developers
/SDF/OBJ_TR_COMP report example showing successful retrofit
We locked all users on the Legacy Development systems before they were released and access was provided on a 'need to use' basis. We used a simple web form builder tool used by the University to publish a web form used to capture Legacy change requests. After submission, requests would be discussed at our daily stand up meeting the following day and approved or rejected as appropriate. Decisions were captured in a spreadsheet exported from the web form tool for audit purposes and as a reference to grant user access to Legacy Development.
Go Live and Hypercare
Production Cutover
We performed the final dress rehearsal in the weeks running up to go live. This was to convert a fresh copy of our SAP ERP Production system to validate our cutover plan and obtain timings for key activities. Our cutover plan was a shared Excel spreadsheet containing a detailed task list by phase, with links to team's own workbooks and the master transport list. Even with all the careful planning and effort to that point, we couldn't be certain precisely when the technical conversion and finance data migration steps would be complete and system verification would start. We had seen varying task completion times from different system conversions, partly due to variables such as data quality and server resources, but hoped Production would be slick.
Tool for transport checking and object validation
As per our usual maintenance approach, all SAP systems (plus connected, dependent third party applications) entered downtime at noon on the Fri of cutover weekend (Fri 24 Jan 2020). Then we started the S/4HANA conversion and applied updates to the other SAP systems. We had secured a longer downtime window than usual, running until 09:00 on Tue 29 Jan 2020, giving us the Mon as contingency should we need it. The first period from Fri afternoon through Sat morning went as planned before we hit our first snag. Despite the SPAU process completing cleanly without issue during QA cutover during which modified SAP standard objects are adjusted, in Production the SAP notes did not adjust as expected. The workaround was to manually resolve each SAP note then perform version checking with Development and QA to ensure the integrity of the system was intact. Following SPAU, the remaining cutover steps on S/4HANA went to plan, with all the major tasks completed by end of Sat 25 Jan 2020 including the all-important finance reconciliation report which checked out as required. The only other significant issue that day was another difference in behaviour when upgrading the SAP Portal, which required application of an SAP note to overcome. Early on the morning of Sun 26 Jan 2020 we performed a comprehensive check of transports, object versions and core integration tests, to ensure the system was ready for integrity checking. This included 171 tests covering select, core and business critical processes. During Sunday testing, 14 defects were logged, investigated and resolved, with one notable exception (discussed below). This aside, we were able to release all SAP Production systems back to resume normal service on the morning of Mon 27 Jan 2020, a day ahead of schedule.
In the weeks preceding go live, we had been been attempting to mitigate one of our higher scoring risks. This related to a discrepancy in our SAP landscape whereby our SAP Gateway system (known to our users as MyApps) had been reconfigured on all test systems from Shibboleth (an identity management solution commonly used in higher education) to Microsoft Active Directory Federation Services (ADFS). In the months leading up to the S/4HANA project starting we had implemented Microsoft Azure Multi-Factor Authentication together with ADFS in a bid to strengthen system security. We ran out of time and had to park this project to pick back up again post S/4HANA go-live, hence our SAP Gateway Production system retained Shibboleth. We felt confident that, despite not having been able to test the updates to Gateway with Shibboleth, that the authentication would most likely continue to work after go live. A few weeks before go live we decided to try and re-configure our Sandbox Gateway system to revert back to Shibboleth. Despite a dogged effort, we could not get this working due to an error in the SAML message exchange. Consequently, we prepared a contingency plan. With hindsight, we wrongly assumed the problem somehow related to the reconfiguration, rather than the changes to Gateway brought in with our S/4HANA project. To our surprise at go live SAP Gateway authentication didn't work, so our contingency plan was instigated. While investigating the issue internally, we opened a 'Very High' priority incident with SAP and reached out for support via our SAP Technical Quality Manager (TQM) and assigned SAP NetWeaver Engineer for that day. Although an SAP note was identified in the area of SAML session handling, it was already applied to our Gateway systems. The SAP incident was forwarded to development support at the mothership in Walldorf. After much effort and stress, a breakthrough was reached on a call with the assigned SAP lead. SAP suggested re-configuring the SAML 2.0 configuration to change from Back-Channel Communication (multiple message exchange) to Front-Channel Communication (SAML messages are transported in a single HTTP POST). We implemented the change during the call, and confirmed the authentication was working. We suspect a bug may have been introduced with the Gateway updates but await SAP's further investigation into the root cause of the Back-Channel Communication issue. Following the call we performed testing of each of our Fiori app processes before releasing the MyApps system early afternoon on Tue 27 Jan 2020. The relief was palpable!
Post Go Live Hypercare
We decided to implement a period of hypercare from the point of S/4HANA go live for five weeks, to cover a complete monthly University business cycle, including finance quarter end and the first payroll calculation. We wanted transparency and scrutiny of change requests but didn't want to impede urgent corrections needed to resolve issues in Production. Similar to the web form we used during the dual track period, we captured change requests for review. If urgent, the agreement was managers of the functional area impacted could self approve by replying to the mailing list. This has proved to be a simple but effective process and helped share knowledge of issues across the project team.
Example hypercare change request email
During the week after go live, everyone was on the front foot to ensure any issues were captured and investigated quickly. We were proactively monitoring the systems, watching key KPIs (e.g. runtime errors and response times) to get a step ahead of incidents logged via our IT Service Desk where possible. At the time of writing (day 5 after go live), we've had a very busy week reviewing issues and fielding questions, but nothing reported of concern that we couldn't resolve. We do have some issues remaining from user acceptance testing to investigate further post go-live where we assessed the risk was acceptable (e.g. student fees and credit management being key).
Project Lessons Learned
Having just gone live with S/4HANA, we haven't yet formally captured lessons learned from the project, but Facebook's maxim "Done is better than perfect" springs to mind here. In the spirit of continuous improvement, you never stop learning and there's always room for improvement, but in a project this complex, delivering S/4HANA without detrimental business impact has been a huge achievement. The table below highlights some of the things we did well that led to the project's success and others where there there was room for improvement.
 |  |
|
|
Update May 2020: following a successful period of bedding in our new S/4HANA system we eventually managed to hold a workshop to capture lessons learned from all key project stakeholders. These were captured in the following slide.
S/4HANA lessons learned
What's Next?
The University's Enterprise Systems Programme Board will meet during the first few months of 2020 to consider and determine the next priorities on our SAP roadmap. Research and scoping work is underway to collate all of the mandatory and known project commitments to help inform discussions. The IT Service is currently undergoing its own transformation exercise as it evolves to meet our user's needs. On the SAP side, we'll need to consider role and skills gaps (e.g. referring to Essential Roles for S/4HANA Fiori Projects, we don't currently have UX Lead). The customer will be at the centre of everything we do going forward, which hasn't been the case historically. A challenge will be balancing the many priorities, such as maintaining business continuity ('keeping the lights on'), evolving our IT Service to be service-driven, and the renovation and replacement of enterprise systems which are coming to the end of their maintenance cycles. It is certainly an exciting time to work for the University if you like change!
To help better understand the maintenance schedule for all the SAP products we run today, the SAP Product Availability Matrix was used to compile the spreadsheet shown below. We're currently evaluating a number of Enterprise Architecture tools that we hope will help to build views such as this to help with decision making. The spreadsheet is intended to serve as a helpful planning aid, to determine the sequence in which the SAP products we use today go out of maintenance. There's a lot of focus on the Dec 2025 deadline, but once we factored in that the SAP NetWeaver 7.5 Java component goes out of support after 31.12.2024 (used by our SAP Portal, Business Warehouse and Process Integration systems), combined with the timing of the University's annual maintenance window (Q1 each year), means that we will need to target completion of our SAP transition to new or successor products during the four remaining maintenance slots we have available before the end of 2024. There's no assumption that SAP will be the most appropriate solution for each of the products in scope, so we'll need to prepare business cases and options appraisals for each deliverables on our roadmap.
Clarifying product maintenance cycles to inform planning
Focusing specifically on S/4HANA Phase 2, at a minimum we will need to upgrade our SAP Fiori Front-end Server as it goes out of maintenance at the end of this year. I'm advocating that we consider uplifting the S/4HANA major release and bundle in with our annual SAP maintenance exercise (the delta appears to be manageable). When we implemented SAP Gateway (our MyApps system) in 2016, SAP best practice was to deploy this as a standalone hub deployment and integrate with SAP back-end systems. This recommendation has now changed so it makes sense for us to collapse Gateway into S/4HANA early, given the simplification benefits this will have for Phase 2 (e.g. from both an architectural and authorisations perspective). Early thinking is we can complete or all of Phase 2 over the course of the next two annual maintenance slots.
This is part 3 of a three-part blog series:
- Newcastle University's Journey to SAP S/4HANA - The What (part 1)
- Newcastle University's Journey to SAP S/4HANA - The Why (part 2)
- Newcastle University's Journey to SAP S/4HANA - The How (part 3)
If you have any questions on our S/4HANA implementation, please ask them below in the comments. Thanks for reading!
- SAP Managed Tags:
13 Comments
