Unexpected Sync Job Retriggering in Hybris 6.7v

[amit-ghiya]

Hi Experts,

We have a job in Hybris 6.7v that checks for the eligible products to sync by checking status and later it creates a catalog sync job that picks these products. In production during the deployment the sync job got aborted due to server restart at around 4 AM, however, it got re-triggered around 7 AM and synced the whole catalog which created the mess.

This is strange since this never happened in the past 5 years of the job execution.

can the aborted sync job get triggered by the system or could this be by someone?

Initial Job Start and Failure:

• The sync job with the identifier 000J8KFJ began and produced several WARN messages about other cron jobs being in an UNKNOWN state, which indicates it was waiting for other jobs before proceeding.
• This was then followed by an ERROR indicating a communication failure with the database: Communications link failure.
• Shortly after that, another message logs that the CronJob 000J8KFJ is marked as running but will be marked as ABORTED due to a system shutdown or a hard signal. The message explicitly mentions possible causes, like a system being forcibly shut down while the job was still in a running state.

Job Restart:

• Later in the logs, the same job 000J8KFJ produces similar WARN messages, implying the job has been started again or resumed.
• The job then seems to proceed further as it begins configureFullVersionSync and ends up configuring over a million entries for synchronization.
• The logs clearly show the job 000J8KFJ restarting or resuming. The nature of hybris jobs, especially synchronization jobs, is such that they can be resumed if they were interrupted. This is so that they don't have to start from scratch and can continue from where they left off. However, how the job was resumed is a mystery.

Logs-

INFO | jvm 1 | main | 2023/09/21 08:02:51.900 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [JaloSession] There are some active local session contexts left (1) for tenant master, this might indicate there are not properly freed localsession context instances.

INFO | jvm 1 | main | 2023/09/21 08:02:53.803 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000I8NDN' (pk:9800781201909) is still in UNKNOWN state. Waiting '377' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:54.204 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IKQSQ' (pk:9819403911669) is still in UNKNOWN state. Waiting '280' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:54.504 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IML7O' (pk:9822256136693) is still in UNKNOWN state. Waiting '677' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:55.205 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IMMW0' (pk:9822327931381) is still in UNKNOWN state. Waiting '906' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:56.107 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IMOAI' (pk:9822386881013) is still in UNKNOWN state. Waiting '805' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:56.908 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IMQLS' (pk:9822485119477) is still in UNKNOWN state. Waiting '1088' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:58.010 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000INIJ4' (pk:9823671058933) is still in UNKNOWN state. Waiting '1135' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:59.112 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000INIW0' (pk:9823686328821) is still in UNKNOWN state. Waiting '475' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:02:59.613 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IS3GN' (pk:9830776242677) is still in UNKNOWN state. Waiting '1183' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:00.816 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IU4YL' (pk:9833862300149) is still in UNKNOWN state. Waiting '443' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:01.216 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IWOMO' (pk:9837886079477) is still in UNKNOWN state. Waiting '970' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:02.218 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IYFTT' (pk:9840605266421) is still in UNKNOWN state. Waiting '713' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:02.919 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IYKO5' (pk:9840809771509) is still in UNKNOWN state. Waiting '1149' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:04.121 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IZRQO' (pk:9842670207477) is still in UNKNOWN state. Waiting '888' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:05.023 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IZTA8' (pk:9842702975477) is still in UNKNOWN state. Waiting '317' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:03:05.323 | WARN [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [CatalogVersionSyncJob] Other cron job '000IZUTS' (pk:9842768511477) is still in UNKNOWN state. Waiting '862' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 08:09:59.026 | ERROR [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [FlexibleSearch] Flexible search error occured...

INFO | jvm 1 | main | 2023/09/21 08:09:59.026 | ERROR [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [Job] Caught throwable org.springframework.dao.RecoverableDataAccessException: query; SQL []; Communications link failure

INFO | jvm 1 | main | 2023/09/21 08:09:59.037 | ERROR [ampToCaProdCatSyncJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (000J8KFJ) [Job] Error saving log file Communications link failure

INFO | jvm 1 | main | 2023/09/21 08:17:09.687 | WARN [localhost-startStop-13] [CronJobManager] CronJob 000J8KFJ is marked as running on cluste

r node 403. It will be marked as ABORTED. This could happen if the system is shut down with a hard signal (e.g. CTRL-C) and a job is still in running state.

INFO | jvm 1 | main | 2023/09/21 08:17:09.687 | WARN [localhost-startStop-13] [CronJobManager] CronJob 000J8KFJ is set aborted

INFO | jvm 1 | main | 2023/09/21 08:17:09.687 | WARN [localhost-startStop-13] [CronJobManager] CronJob 000J8KFJ is marked as running on cluste

r node 403. It will be marked as ABORTED. This could happen if the system is shut down with a hard signal (e.g. CTRL-C) and a job is still in running state.

INFO | jvm 1 | main | 2023/09/21 08:17:09.687 | WARN [localhost-startStop-13] [CronJobManager] CronJob 000J8KFJ is set aborted

INFO | jvm 1 | main | 2023/09/21 11:04:54.115 | WARN [000J8KFJ::de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncJob] (000J8K

FJ) [CatalogVersionSyncJob] Other cron job '000I8NDN' (pk:9800781201909) is still in UNKNOWN state. Waiting '511' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 11:04:54.716 | WARN [000J8KFJ::de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncJob] (000J8KFJ

) [CatalogVersionSyncJob] Other cron job '000IKQSQ' (pk:9819403911669) is still in UNKNOWN state. Waiting '869' milliseconds for start

INFO | jvm 1 | main | 2023/09/21 11:15:52.726 | INFO [000J8KFJ::de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncJob] (000J8KFJ

) [CatalogVersionSyncJob] Starting configureFullVersionSync ...

INFO | jvm 1 | main | 2023/09/21 11:20:24.945 | INFO [000J8KFJ::de.hybris.platform.catalog.jalo.synchronization.CatalogVersionSyncJob] (000J8KFJ

) [CatalogVersionSyncJob] Sync 'sync ampToCaProdCatalog:Staged->Staged' (pk:8822602367476) configured 1682442 entries for job '000J8KFJ' (pk:9856119964149)

schedule medias: 1

[moshenaveh]

Welcome to the SAP Community! Thank you for visiting us to get answers to your questions.

Since you're asking a question here for the first time, I'd like to offer some friendly advice on how to get the most out of your community membership and experience.

First, please see https://community.sap.com/resources/questions-and-answers, as this resource page provides tips for preparing questions that draw responses from our members. Secondly, feel free to take our Q&A tutorial at https://developers.sap.com/tutorials/community-qa.html as well, as that will help you when submitting questions to the community.

Finally, I recommend that you include a profile picture. By personalizing your profile, you encourage readers to respond: https://developers.sap.com/tutorials/community-profile.html.

I hope you find this advice useful, and we're happy to have you as part of SAP Community!