Skip to Content
Technical Articles

A Lightweight Distributed Scheduled Jobs Resource Coordination Solution

In projects, it is often necessary to use scheduled jobs to do some business.

In the JAVA world, commonly used open-source distributed scheduled job frameworks are:
1. Quartz
2. Spring Batch
Quartz and Spring Batch are very powerful, however, they just need multiple database tables to support the entire business. For a simple scheduled job application, they are a bit heavy.

 

This article will introduce a lightweight, Redis-based resource coordination for distributed schedule jobs.

The overall architecture is as follows:

It can be seen that this solution uses Redis Cluster as a unified central resource database for multiple distributed application servers.

To demo and explain the solution, I launched three applications locally with the same code, the ports are: 8081, 8082, and 8083, and each application has three jobs: TestScheduleA, TestScheduleB, and TestScheduleN.

In each distributed application, each scheduled job defines a unique Key for its own.
Then each job will call the method to obtain the key occupation, if the occupation is successful, continue the business process.
Take “TestScheduleA” as an example:

/**
 *
 */
package com.sap.ibso;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Random;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.env.Environment;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import com.sap.ibso.proxy.ScheduleResourceLockProxy;

/**
 * @author I323560 Ryan Wang
 *
 */
@Component
public class TestScheduleA {

    private Logger logger = LoggerFactory.getLogger(this.getClass());

    @Autowired
    private Environment environment;

    @Autowired
    private ScheduleResourceLockProxy resourceLockProxy;

    @Scheduled(cron = "0 0/1 * * * ?")
    public void testSchedule() throws InterruptedException {
        String uniqueIdentifier = generateUniqueIdentifier();
        this.logger.info(uniqueIdentifier);

        if (this.resourceLockProxy.occupyResource(ScheduleConstants.KEY_SCHEDULE_A, uniqueIdentifier, null)) {
            this.logger.info(uniqueIdentifier + " is succeed.");
        }
    }

    private String generateUniqueIdentifier() {
        String serverPort = this.environment.getProperty("server.port");
        Calendar current = Calendar.getInstance();
        SimpleDateFormat sdf = new SimpleDateFormat("__yyyyMMddHHmmssSSS__");
        Random random = new Random();
        StringBuilder stringBuilder = new StringBuilder();
        stringBuilder.append(serverPort).append(": ").append(ScheduleConstants.KEY_SCHEDULE_A)
                .append(sdf.format(current.getTime())).append(random.nextInt(10)).append(random.nextInt(10))
                .append(random.nextInt(10)).append(random.nextInt(10)).append(random.nextInt(10))
                .append(random.nextInt(10));
        return stringBuilder.toString();
    }

}

TestScheduleB and TestScheduleN are the same, except for the unique key.

The proxy class is the core part of this solution.

/**
 *
 */
package com.sap.ibso.proxy;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;
import org.springframework.util.Assert;

/**
 * @author I323560 Ryan Wang
 *
 */
@Component
public class ScheduleResourceLockProxy {

    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    private static final Long DEFAULT_CHECK_AFTER_SECONDS = 5L;

    /**
     * Utility for schedule job to occupy the resource.
     *
     * @param resourceKey
     * @param identifier
     * @param checkAfterSeconds
     *            Caution: check time should not be greater than the schedule job interval time.
     * @return true if the one occupied the resource
     * @throws InterruptedException
     */
    public boolean occupyResource(String resourceKey, String identifier, Long checkAfterSeconds)
            throws InterruptedException {
        Assert.hasText(resourceKey, "Resource Key should not be null or empty.");
        Assert.hasText(identifier, "Identifier should not be null or empty.");
        checkAfterSeconds = null != checkAfterSeconds ? checkAfterSeconds : DEFAULT_CHECK_AFTER_SECONDS;
        // 1. Set the resource key to the unique identifier of each scheduled job instant
        this.redisTemplate.opsForValue().set(resourceKey, identifier);

        // 2. Sleep for a short time, waiting for the same schedule job from other different servers to set the value of
        // the key with their unique identifiers.
        Thread.sleep(checkAfterSeconds * 1000);

        // 3. Returns true to the job that set the Key to Redis last, indicating that this job instance successfully
        // occupied the schedule job resource.
        Boolean succeed = false;
        String value = this.redisTemplate.opsForValue().get(resourceKey);
        if ((null != value) && value.equals(identifier)) {
            succeed = true;
            this.redisTemplate.delete(resourceKey);
        }

        return succeed;
    }

}

As described in the comments, the proxy method will first set the resource key to the unique identifier of each scheduled job instant.
Then sleeps for a short time, waiting for the other schedule job instants from other different servers to set the value of the key with their unique identifiers.
After that, it returns true to the job that set the Key to Redis last, indicating that this job instance successfully occupied the schedule job resource.

From the log file

15:55:00.000 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155500000__593365
15:55:00.001 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155500001__707944
15:55:00.000 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155500000__158089
15:55:00.001 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155500001__583014
15:55:00.002 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155500002__574083
15:55:00.003 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155500003__376776
15:55:00.003 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155500003__946282
15:55:00.007 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155500007__639775
15:55:00.025 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155500025__538749
15:55:05.464 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155500003__376776 is succeed.
15:55:05.778 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155500003__946282 is succeed.
15:55:06.371 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155500025__538749 is succeed.

15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155600015__560542
15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155600015__275560
15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155600015__341414
15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155600015__103417
15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155600015__572481
15:56:00.015 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155600015__862513
15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155600015__134534
15:56:00.015 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155600015__066091
15:56:00.015 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155600015__759380
15:56:05.460 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155600015__572481 is succeed.
15:56:05.494 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155600015__759380 is succeed.
15:56:05.781 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155600015__560542 is succeed.

15:57:00.008 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155700008__118990
15:57:00.008 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155700008__742365
15:57:00.008 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155700008__645389
15:57:00.008 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155700008__492029
15:57:00.008 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155700008__834727
15:57:00.008 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155700008__916998
15:57:00.011 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155700011__020188
15:57:00.013 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155700013__034318
15:57:00.013 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155700013__971104
15:57:05.482 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155700011__020188 is succeed.
15:57:05.482 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155700013__034318 is succeed.
15:57:05.487 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155700013__971104 is succeed.

15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155800012__646195
15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155800012__460715
15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155800012__376216
15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155800012__488104
15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155800012__799671
15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155800012__562950
15:58:00.012 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155800012__885915
15:58:00.012 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155800012__616935
15:58:00.012 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155800012__807803
15:58:05.472 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155800012__807803 is succeed.
15:58:05.472 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155800012__799671 is succeed.
15:58:05.757 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155800012__885915 is succeed.

15:59:00.004 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209155900004__012936
15:59:00.007 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209155900007__931726
15:59:00.007 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209155900007__535456
15:59:00.016 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209155900016__158983
15:59:00.018 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209155900017__083242
15:59:00.018 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209155900018__864875
15:59:00.016 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155900016__384194
15:59:00.018 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155900018__972536
15:59:00.010 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155900010__065440
15:59:05.459 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209155900018__972536 is succeed.
15:59:05.487 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209155900010__065440 is succeed.
15:59:05.784 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209155900016__384194 is succeed.

16:00:00.004 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160000004__075066
16:00:00.004 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160000004__439137
16:00:00.006 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160000006__716388
16:00:00.006 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160000006__308428
16:00:00.007 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160000007__735314
16:00:00.011 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160000011__655069
16:00:00.011 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160000011__394457
16:00:00.007 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160000007__470683
16:00:00.029 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160000029__103247
16:00:05.450 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160000007__735314 is succeed.
16:00:05.464 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160000007__470683 is succeed.
16:00:05.511 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160000029__103247 is succeed.

16:01:00.001 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160100001__882412
16:01:00.001 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160100001__161429
16:01:00.003 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160100003__583106
16:01:00.004 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160100004__096199
16:01:00.006 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160100006__258357
16:01:00.007 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160100007__775813
16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160100023__778833
16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160100023__287820
16:01:00.023 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160100023__520435
16:01:05.424 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160100007__775813 is succeed.
16:01:05.426 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160100004__096199 is succeed.
16:01:05.788 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160100023__520435 is succeed.

16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160200009__259304
16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160200009__459263
16:02:00.009 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160200009__665382
16:02:00.009 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160200009__367906
16:02:00.009 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160200009__846683
16:02:00.009 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160200009__858342
16:02:00.013 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160200013__407613
16:02:00.014 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160200013__407833
16:02:00.015 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160200015__523621
16:02:05.431 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160200013__407613 is succeed.
16:02:05.448 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160200015__523621 is succeed.
16:02:05.449 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160200009__846683 is succeed.

16:03:00.002 INFO  com.sap.ibso.TestScheduleB - 8082: Schedule_B__20191209160300002__427318
16:03:00.002 INFO  com.sap.ibso.TestScheduleN - 8082: Schedule_N__20191209160300002__389814
16:03:00.002 INFO  com.sap.ibso.TestScheduleA - 8083: Schedule_A__20191209160300002__604872
16:03:00.009 INFO  com.sap.ibso.TestScheduleA - 8082: Schedule_A__20191209160300009__499131
16:03:00.010 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160300010__716448
16:03:00.010 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160300010__628931
16:03:00.011 INFO  com.sap.ibso.TestScheduleB - 8081: Schedule_B__20191209160300011__112429
16:03:00.012 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160300012__754256
16:03:00.012 INFO  com.sap.ibso.TestScheduleN - 8081: Schedule_N__20191209160300012__348712
16:03:05.773 INFO  com.sap.ibso.TestScheduleB - 8083: Schedule_B__20191209160300012__754256 is succeed.
16:03:06.041 INFO  com.sap.ibso.TestScheduleA - 8081: Schedule_A__20191209160300010__716448 is succeed.
16:03:06.074 INFO  com.sap.ibso.TestScheduleN - 8083: Schedule_N__20191209160300010__628931 is succeed.

You can find:
Each job of each server was successfully started, and the actual execution of the business was performed only once, which successfully avoided the situation where multiple servers performed the same job at the same time.

For applications that require high concurrency, a large number of jobs, and high performance, it is recommended to use open-source frameworks such as Spring Batch and Quartz.

For simple business scenarios, this solution would be a fast way to implement distributed scheduled jobs.

1 Comment
You must be Logged on to comment or reply to a post.