A Lightweight Distributed Scheduled Jobs Resource ...

former_member321232 · ‎12-10-2019

In projects, it is often necessary to use scheduled jobs to do some business.

In the JAVA world, commonly used open-source distributed scheduled job frameworks are:
1. Quartz
2. Spring Batch
Quartz and Spring Batch are very powerful, however, they just need multiple database tables to support the entire business. For a simple scheduled job application, they are a bit heavy.

This article will introduce a lightweight, Redis-based resource coordination for distributed schedule jobs.

The overall architecture is as follows:

It can be seen that this solution uses Redis Cluster as a unified central resource database for multiple distributed application servers.

To demo and explain the solution, I launched three applications locally with the same code, the ports are: 8081, 8082, and 8083, and each application has three jobs: TestScheduleA, TestScheduleB, and TestScheduleN.

In each distributed application, each scheduled job defines a unique Key for its own.
Then each job will call the method to obtain the key occupation, if the occupation is successful, continue the business process.
Take "TestScheduleA" as an example:

/**

 *

 */

package com.sap.ibso;



import java.text.SimpleDateFormat;

import java.util.Calendar;

import java.util.Random;



import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.core.env.Environment;

import org.springframework.scheduling.annotation.Scheduled;

import org.springframework.stereotype.Component;



import com.sap.ibso.proxy.ScheduleResourceLockProxy;



/**

 * @author I323560 Ryan Wang

 *

 */

@Component

public class TestScheduleA {



    private Logger logger = LoggerFactory.getLogger(this.getClass());



    @Autowired

    private Environment environment;



    @Autowired

    private ScheduleResourceLockProxy resourceLockProxy;



    @Scheduled(cron = "0 0/1 * * * ?")

    public void testSchedule() throws InterruptedException {

        String uniqueIdentifier = generateUniqueIdentifier();

        this.logger.info(uniqueIdentifier);



        if (this.resourceLockProxy.occupyResource(ScheduleConstants.KEY_SCHEDULE_A, uniqueIdentifier, null)) {

            this.logger.info(uniqueIdentifier + " is succeed.");

        }

    }



    private String generateUniqueIdentifier() {

        String serverPort = this.environment.getProperty("server.port");

        Calendar current = Calendar.getInstance();

        SimpleDateFormat sdf = new SimpleDateFormat("__yyyyMMddHHmmssSSS__");

        Random random = new Random();

        StringBuilder stringBuilder = new StringBuilder();

        stringBuilder.append(serverPort).append(": ").append(ScheduleConstants.KEY_SCHEDULE_A)

                .append(sdf.format(current.getTime())).append(random.nextInt(10)).append(random.nextInt(10))

                .append(random.nextInt(10)).append(random.nextInt(10)).append(random.nextInt(10))

                .append(random.nextInt(10));

        return stringBuilder.toString();

    }



}

TestScheduleB and TestScheduleN are the same, except for the unique key.

The proxy class is the core part of this solution.

/**

 *

 */

package com.sap.ibso.proxy;



import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.data.redis.core.RedisTemplate;

import org.springframework.stereotype.Component;

import org.springframework.util.Assert;



/**

 * @author I323560 Ryan Wang

 *

 */

@Component

public class ScheduleResourceLockProxy {



    @Autowired

    private RedisTemplate<String, String> redisTemplate;



    private static final Long DEFAULT_CHECK_AFTER_SECONDS = 5L;



    /**

     * Utility for schedule job to occupy the resource.

     *

     * @param resourceKey

     * @param identifier

     * @param checkAfterSeconds

     *            Caution: check time should not be greater than the schedule job interval time.

     * @return true if the one occupied the resource

     * @throws InterruptedException

     */

    public boolean occupyResource(String resourceKey, String identifier, Long checkAfterSeconds)

            throws InterruptedException {

        Assert.hasText(resourceKey, "Resource Key should not be null or empty.");

        Assert.hasText(identifier, "Identifier should not be null or empty.");

        checkAfterSeconds = null != checkAfterSeconds ? checkAfterSeconds : DEFAULT_CHECK_AFTER_SECONDS;

        // 1. Set the resource key to the unique identifier of each scheduled job instant

        this.redisTemplate.opsForValue().set(resourceKey, identifier);



        // 2. Sleep for a short time, waiting for the same schedule job from other different servers to set the value of

        // the key with their unique identifiers.

        Thread.sleep(checkAfterSeconds * 1000);



        // 3. Returns true to the job that set the Key to Redis last, indicating that this job instance successfully

        // occupied the schedule job resource.

        Boolean succeed = false;

        String value = this.redisTemplate.opsForValue().get(resourceKey);

        if ((null != value) && value.equals(identifier)) {

            succeed = true;

            this.redisTemplate.delete(resourceKey);

        }



        return succeed;

    }



}

As described in the comments, the proxy method will first set the resource key to the unique identifier of each scheduled job instant.
Then sleeps for a short time, waiting for the other schedule job instants from other different servers to set the value of the key with their unique identifiers.
After that, it returns true to the job that set the Key to Redis last, indicating that this job instance successfully occupied the schedule job resource.

From the log file

15:55:00.000 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155500000__593365

15:55:00.001 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155500001__707944

15:55:00.000 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155500000__158089

15:55:00.001 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155500001__583014

15:55:00.002 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155500002__574083

15:55:00.003 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155500003__376776

15:55:00.003 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155500003__946282

15:55:00.007 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155500007__639775

15:55:00.025 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155500025__538749

15:55:05.464 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155500003__376776 is succeed.

15:55:05.778 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155500003__946282 is succeed.

15:55:06.371 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155500025__538749 is succeed.



15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155600015__560542

15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155600015__275560

15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155600015__341414

15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155600015__103417

15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155600015__572481

15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155600015__862513

15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155600015__134534

15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155600015__066091

15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155600015__759380

15:56:05.460 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155600015__572481 is succeed.

15:56:05.494 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155600015__759380 is succeed.

15:56:05.781 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155600015__560542 is succeed.



15:57:00.008 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155700008__118990

15:57:00.008 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155700008__742365

15:57:00.008 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155700008__645389

15:57:00.008 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155700008__492029

15:57:00.008 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155700008__834727

15:57:00.008 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155700008__916998

15:57:00.011 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155700011__020188

15:57:00.013 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155700013__034318

15:57:00.013 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155700013__971104

15:57:05.482 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155700011__020188 is succeed.

15:57:05.482 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155700013__034318 is succeed.

15:57:05.487 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155700013__971104 is succeed.



15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155800012__646195

15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155800012__460715

15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155800012__376216

15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155800012__488104

15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155800012__799671

15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155800012__562950

15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155800012__885915

15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155800012__616935

15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155800012__807803

15:58:05.472 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155800012__807803 is succeed.

15:58:05.472 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155800012__799671 is succeed.

15:58:05.757 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155800012__885915 is succeed.



15:59:00.004 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155900004__012936

15:59:00.007 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155900007__931726

15:59:00.007 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155900007__535456

15:59:00.016 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155900016__158983

15:59:00.018 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155900017__083242

15:59:00.018 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155900018__864875

15:59:00.016 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155900016__384194

15:59:00.018 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155900018__972536

15:59:00.010 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155900010__065440

15:59:05.459 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155900018__972536 is succeed.

15:59:05.487 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155900010__065440 is succeed.

15:59:05.784 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155900016__384194 is succeed.



16:00:00.004 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160000004__075066

16:00:00.004 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160000004__439137

16:00:00.006 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160000006__716388

16:00:00.006 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160000006__308428

16:00:00.007 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160000007__735314

16:00:00.011 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160000011__655069

16:00:00.011 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160000011__394457

16:00:00.007 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160000007__470683

16:00:00.029 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160000029__103247

16:00:05.450 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160000007__735314 is succeed.

16:00:05.464 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160000007__470683 is succeed.

16:00:05.511 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160000029__103247 is succeed.



16:01:00.001 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160100001__882412

16:01:00.001 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160100001__161429

16:01:00.003 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160100003__583106

16:01:00.004 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160100004__096199

16:01:00.006 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160100006__258357

16:01:00.007 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160100007__775813

16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160100023__778833

16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160100023__287820

16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160100023__520435

16:01:05.424 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160100007__775813 is succeed.

16:01:05.426 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160100004__096199 is succeed.

16:01:05.788 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160100023__520435 is succeed.



16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160200009__259304

16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160200009__459263

16:02:00.009 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160200009__665382

16:02:00.009 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160200009__367906

16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160200009__846683

16:02:00.009 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160200009__858342

16:02:00.013 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160200013__407613

16:02:00.014 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160200013__407833

16:02:00.015 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160200015__523621

16:02:05.431 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160200013__407613 is succeed.

16:02:05.448 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160200015__523621 is succeed.

16:02:05.449 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160200009__846683 is succeed.



16:03:00.002 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160300002__427318

16:03:00.002 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160300002__389814

16:03:00.002 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160300002__604872

16:03:00.009 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160300009__499131

16:03:00.010 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160300010__716448

16:03:00.010 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160300010__628931

16:03:00.011 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160300011__112429

16:03:00.012 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160300012__754256

16:03:00.012 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160300012__348712

16:03:05.773 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160300012__754256 is succeed.

16:03:06.041 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160300010__716448 is succeed.

16:03:06.074 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160300010__628931 is succeed.

You can find:
Each job of each server was successfully started, and the actual execution of the business was performed only once, which successfully avoided the situation where multiple servers performed the same job at the same time.

For applications that require high concurrency, a large number of jobs, and high performance, it is recommended to use open-source frameworks such as Spring Batch and Quartz.

For simple business scenarios, this solution would be a fast way to implement distributed scheduled jobs.

A Lightweight Distributed Scheduled Jobs Resource Coordination Solution

Get Started with the ABAP Development Tools for SAP NetWeaver

Become an ABAP in Eclipse Feature Explorer and earn the Explorer Badge

Six kinds of debugging tips to find the source code where the message is raised