Preparing Your SAP BusinessObjects Environment for Daylight Savings Time (DST) 2014
It’s that time of year again. Daylight Savings Time will be occurring for most of us very soon. This blog serves as a friendly reminder for what you should do to prepare based on the version of SAP BusinessObjects you have deployed.
Spring 2014 DST Timeline
|
March 9 * |
Most of North America * |
|
March 30 * |
Most of Europe and UK * |
|
April 6 |
Most of Australia |
|
No DST |
Most of Asia |
Fall 2014 DST Timeline
|
November 2 |
Most of North America |
|
October 26 |
Most of Europe and UK |
|
October 5 * |
Most of Australia * |
|
No DST |
Most of Asia |
* Identifies when majority of issues would be logged with Support. IE when DST time changes forward.
Exact 2014 dates per country can be found here.
In previous years, the DST change has affected the SAP BusinessObjects Enterprise product in different ways. There are two main workflows where we have seen issues in the past.
- Report Scheduling
- Web Application Server time
The first, Report Scheduling, has been seen in multiple forms. This ranged anywhere from duplicate scheduled reports, reports running 1 hour early or late, or as we saw last year, thousands of report instances failing over and over for a single parent report.
The second, Web Application Server time, is a generic description for the UI (BI Launchpad, CMC, InfoView, etc) not displaying the correct time for the user which is logged in. In some cases, since the user is detected as being logged in an hour earlier than current time, anything they schedule could be in turn scheduled for an hour earlier than their intended time.
Both SAP Development and SAP Active Global Support teams have completed their internal testing to prepare for DST 2013 and have not found any new regressions in either of the XI3.1 or BI4.0 product lines. Just in case there were any scenarios unaccounted for, please monitor your systems on the days following the time change. Utilize the Instance Manager within the CMC to monitor for report failures and/or schedule timing issues. If you do encounter an issue, please utilize the following kbases where needed and/or create a support ticket in Service MarketPlace.
** NOTE: If using SAP BusinessObjects XI 3.1, be sure to have your environment patched up to the minimum patch level otherwise you risk running into the issues many of our customers encountered last year.
SAP BusinessObjects XI 3.1
Make sure to have your system patched to:
- XI 3.1 SP3.6 or greater
- XI 3.1 SP4 or greater
- XI 3.1 SP5 or greater
SAP BusinessObjects BI4.x
Does not appear to be affected.
Related SAP Notes:
- SAP Note# 1448881 – Multiple instances spawned after daylight savings time change
This article has the “EDST reschedule tool” attached to the note which was used during the 2011 issues faced last year to reschedule all recurring reports in bulk.
- SAP Note# 1568239 – After DST change Schedules fail in Business Objects Enterprise XI 3.1 with: Object could not be scheduled within the specified time interval
This article references the url for the Oracle Java Time Zone updater tool which can help mitigate issues where the Java Web Application Server is incorrect by an hour.
- SAP Note# 1771416 – Wrong schedule time displayed for recurring reports in Australia timezone. Not necessarily DST related but more of a general time zone issue. Fixed in XI 3.1 SP6.
Thank you for helping ensure we are proactive in 2012 and for the efforts to consoldate this feedback. I have taken the opportunity to share with as many customers as I can who may be interested as well as internally to ensure all teams are staffed and aware.
Thanks!
Chris
Our current version is XI3.1 SP3.1. It looked like we will have problems and it's not possible to apply patches overnight. Is there any workarounds? For example, can we stop the BO Services at 1:30am and start the services at 3:30 (1 hour downtime)to avoid the scheduled reporting issue?
Many thanks,
Fusen
What you can do is stop your jobservers for the night, wait for DST to occur, and then use the reschedule script in the referenced kbase to repair the affected reports (IE. daily recurrings). The reschedule script will change the time for any of the reports you select to +1 hour. Once completed, run it again for -1 hour to change it back. That small change will update the entry in the CMS database enough to prevent the issue from occuring.
Please note: SP4 and SP5 do not have this issue. Only XI3.1 SP3.5 and below. It was resolved in XI3.1 FP3.6 under ADAPT01318486
Unfortunately we saw this post too late to know that the DST bug was still present and have thousands of failed instances with the originals purged from our server. Now that DST has occurred, can we simply reschedule the reports going forward while we work on recovering purged instances? Thank you for your assistance.
Hi Dan,
Apologies on the occurence - the steps outlined in 144881 provide the reschedule script to stop the issue from occurring. Once this is done the script in KB 1568718 recurrsively will delete failed instances to elminate them from the system. Depending on how many accumulated the delete can take some time but this is the approach. At this stage I expect you raised a message if you need a hand on this but if you haven't please do so.
Thanks!
Chris
Hello,
What about deleting failed instances for customers who have an IIS deployment?
I believe there is a vbs script for them but can't see any note about it.
Thanks
Philippe
SAP Note #1448881 mentioned above, makes a reference to the following article which describes the deletion process. See the delete script attached to SAP Note# 156718.
Hi Josh,
Thanks for replying so quickly 😆
I am aware of this script but this is only for a tomcat deployment.
Customers with IIS can't run this .jsp file
I know there was a vbs script developped for them. But I can't find any official info about it.
Regards,
Philippe
This has also happened for a client I am working with and they have installed SAP BusinessObjects BI 4.0 SP02 Patch 6. So in BI 4.0, it didn't go away. I set up BI 4.0 for the client in September 2011 and in Australia, Daylight savings started first weekend of October 2011.
I've come back to the client now, June 2012 and they found an issue where some daily scheduled reports had millions of instances. When I did a search through Instance Manager to see when the failed instances for the reports occurred, they started the weekend of Daylight savings kicking in, i.e. 2nd October 2011.
I have not found any SAP KB note for this for SAP BusinessObjects BI 4.0 verison. I'm going to try the steps outlined in SAP KB note 1448881.
This is the first case I've heard of in BI4. Is there a chance the report was migrated from XI3.1? Remember, this issue reported in XI3.1 also happened the year prior. Depending on the timeframe and whether it was migrated, it might be possible to bring over an affected report from XI3.1. If you need assistance, please log a message in SMP under the component BI-BIP-ADM. If possible, update this thread with your findings so others can benefit.
Thanks for sharing.
Hi Josh,
I ended up having to incrementally delete all the failed instances from the CMS repository table CMS_INFOOBJECTS7 table of the SAP BusinessObjects BI 4.0 SP4 environment.
As this was stored within an Oracle 11G database I ran the following statement:
DELETE FROM CMS_INFOOBJECTS7
WHERE SCHEDULESTATUS=3
AND SI_INSTANCE_OBJECT=1
AND ROWNUM <100000;
COMMIT;
This was run numerous times as there were over 4 million failed instances.
The reports with failed instances were from a BOE XI 3.1 that had been upgraded to BI 4.0 with the schedules.
Hope this helps.
Excellent!
Hi Josh,
We faced this issue last year for our XI 3.1 SP 3 environment and did take SAP support for the same and used the tools to recover from the issue. One thing during that time I asked the support guy was how to avoid this issue going forward would it not be a good idea to just shutdown the instance for the time and he said yes to this. Has any one any idea around this. I dont have this in writing from SAP but do remember the discussion with them around this. Do people practice this and does this work or we have to use the 2 notes 1448881 and 1568718 even after your have brought up the instance from sleep.
Thanks
GAB
Hi Ashish,
I don't believe simply shutting down your services during the time change will prevent the issue. It's been a few years since the original problem was raised in XI3.1 SP3 but from what I remember the main issue (the creation of thousands of instances) would still happen to the daily run jobs at their next runtime, whenever you restarted the jobservers. The reasoning was due to how the CMS handles time. The CMS uses UTC time and actions are constantly converted from local server time or browser time to the equilent UTC time. So regardless if you stop the servers at midnight and start them back at 3am, the original scheduled recurring report was still scheduled to a previous time which the CMS will attempt to run whenever it starts or as soon as a jobserver is available. This process is what was handled incorrectly in certain XI3.1 SP3 versions.
I believe the safest way to "prevent" the DST scenario, other than patching, is to leave the XI3.1 system running and stop only the jobservers during the time change. After the time changes but before you start the jobservers, run the script to alter the report objects schedule time to +1 hour and then again -1 hour. The reason the script works is because it tells the CMS to update each scheduled object with the new time stamp as it should be after daylight savings time. The only reason you want to run it a 2nd time is to change the time stamps back to what they were originally.
Although the steps to repair the system are pretty full proof now, the best recommendation is to be on a patch revision which includes the fix.
Hi Joshua,
We've upgraded XI 3.1 to SP 3.6...however, on March 10'th we had multiple failed instances for one report again.
Note 1448881 had to be used again.
Please can you advise if SP 3.6 does not contain a fix for this issue...or if we need to use the Java TZ updater...
Thank you.
Xi3.1 FP3.6 does include the fix. If you experienced the same issue on March 10th, I would recommend opening a support ticket so someone can verify the files needed for the fix were updated correctly.
Hi Joshua,
Apart from the above recommendations; do we have to restart the BI 4 systems during DST change period? We have both BI 4 SP2 P5 and BODS 4.0 SP1
No, you shouldn't. The CMS periodically checks the system time and adjusts its own internal clock to match. Don't quote me on this, but I believe it syncs every 5 minutes.
I'm still trying to get the correct behaviour when using the Calendar Date Picker for a date object (With and without LOV)
In NZ, if you select 7/4/2013, it appends a time portion and defaults this to 2am.
No matter what you do, 2am is always used in the SQL, resulting in missing data from reports.
We are using BI4 SP6, any ideas would be much appreciated !
Hi Pete,
I'd suggest logging an incident via Service MarketPlace. Any additional info you can provide to the engineer will help, such as:
Ours was adding 5 hours but only in HTML mode. It didn't happen in Applet mode. Fixed in 4.1 SP1 Patch 8
FYI, I'd like to mention one other item for consideration:
If a recurring report has a scheduled runtime, *on the day of DST*, between:
- 2-2:59am on March 9 in the US
- 1-1:59am on March 30 in the UK
- or the equivalent DST time shift in other regions
then that instance will run an hour later. For example, since the US skips from 2am to 3am on March 9, a daily recurring instance set to run on March 9 at 2:30am will instead run at 3:30am. This instance may continue to run at 3:30am on subsequent days unless re-scheduled. Please re-schedule instances where needed. See Knowledge Base Article 1986781 for full details.
To be clear, the only instances that can be affected are those with a runtime that falls in the hour skipped during the Spring DST time shift (1-1:59am on March 30 in the UK, for example, since the clock there jumps from 12:59:59am straight to 2am)
BI 4.1 SP2FP3 2008R2 MST =GMT-7,GMT-6. No matter which release I have been on I have had this problem twice every year. I don't have the spawning issue but every scheduled instance is one hour off every time we have a change in daylight savings status. Not just displaying one hour off but running one hour late in this case (3/9/2014). It was not just instances between 2:00 & 3:00 on March 9th. In this case all scheduled instances were impacted. I am using the BOE supplied JRE and Tomcat 7. Could I have something wrong with a configuration. I looked in some Tomcat logs and could see those timestamps changing from GMT-7 to GMT-6. BI logs showed same behavior. I am not sure if these logs indicating time correctly is conducive to scheduled instances knowing the correct time. CMC timezone settings are Default - Local to web server. This is really frustrating. Any advice would be greatly appreciated.
Thank You
sorry - but we have BO XI 3.5.5 - and same problems occur every year, also the US dst time has impacts on our scheduled reports - and we have european DST.
4 times per year, we are fighting with wrong scheduled times ( US issue ) - and after 3 weeks we have wrong display / scheduling times when german DST happen.
Hi Sabine,
I've seen a few other customers with servers in Europe who are affected by the US DST switchover. In the other cases, we've found the root cause to be incorrect time zone configurations at the operating system level.
I'll venture to guess you're running in Unix or Linux, is that correct? If so, please see KBA 1404484 for an example of such an issue -- and if you define your TZ variable using POSIX format, this page is helpful as well.
Please feel free to reach out to me directly if I may help with this.
Thanks
-Mike
michael.richards@sap.com
Sorry – but that is what we did 3 weeks ago, using POSIX format , and we had issues when last weekend the german switchover occurred.
In cmc – preferences – the timezone was set to “default – use web server timezone” – we saw all instances ran one hour too early !
To fix that, we had to set cmc preferences to : use GMT+1:00 Berlin, and so on… then the instance times were corrected.
But the instance manager still showed wrong timestamps for completed reports.
What fixed this behaviour, was to set TZ on the Linux / Solaris servers back to
TZ = MET .
No POSIX format again – and we can use cmc preferences set to “default – use web server timezone” - with correct completion timestamps for "completed" instances.
So obviously we have to change settings when US DST comes / and additionally to change these settings back for German DST.
S.Soheili
Hi,
One of our customers experienced this issue earlier this year which led to 3.5 million failed instances of a single report being created. We then spent then next 4 days running the script (KB 1568718) to purge the failed instances from the CMS, which did eventually clear them all out.
Cheers,
Chris
3.5 Million is a lot. The default for number of instances to keep is unlimited. We set our top level limits for instance to keep at 10 or less and gave users the option to request more on specific folders if needed. This way the system cleans them up nightly rather than having to run Delete SQL on the DB. I do wish this could be set independently for the different types of instances like Failed, Success, Recurring. I don't think it should count Recurring in the limit, but it seems to, so be careful.
SAP BusinessObjects BI4.x is definitely affected.
We are running two BO 4.1 SP4 in two separate environments and have observed the problem on both environments. After going to Summertime on March 29th 2015 one report generated approx. 110.000 failed instances in a few days. The error message was "Object could not be scheduled within the specified time interval".
To avoid new failures, the problematic schedules had to be rescheduled. We chose Replace existing schedule and updated the Start Date and the End Date. Some dates in prompts also had a bad format and had to be updated.
To remove the failed instances, it was necessary to set a limit on each of the schedule owners. An overall limit for the entire report or folder did not work. We are aware, a tool is available for removing the failed instances, however since we are operating GxP environments, we are not allowed to execute any adhoc software on the servers.
I've added a new SCN blog post for DST 2016 here:
Daylight Saving Time in BI Platform - 2016