Scheduling periodic jobs dynamically at runtime using Spring & Quartz

Big Picture & Problem: Cron jobs are a type of automated task scheduler that execute tasks at scheduled time which is fixed and needs to be provided before application starts up.

Many times we come across situations where we need to dynamically schedule tasks that needs to be executed periodically.

In this blog, we will be building a monitoring tool that can track uptime of any web
site. The name of the tool is UMT. (Uptime Monitoring Tool)

Solution: For scheduling tasks we can use simple Spring Scheduler or frameworks like quartz. Spring Scheduler is simpler and more lightweight, making it suitable for smaller applications with simpler scheduling requirements. As in our case we need to periodically check for website health at regular interval we need quartz because it helps in more complex scheduling, job persistence, clustering, or job chaining features. It will help us in scaling our application as per need.

Keeping that basic requirement in our mind, let us come up with a basic database design that may look like this –

Entity_Tables

We have a check table to store the information like URL to be checked along with the periodic frequency for it to be performed. We also have a health table to store the health status against a check that is getting executed.

Configuration: To configure quartz in your spring application you need to add the following dependency in your build.gradle or pom.xml file.

    // https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-quartz
    implementation 'org.springframework.boot:spring-boot-starter-quartz'
    // https://mvnrepository.com/artifact/com.mchange/c3p0
    implementation 'com.mchange:c3p0:0.9.5.5'

and in you main/resources/application-properties file add

spring.quartz.job-store-type=jdbc
spring.quartz.jdbc.initialize-schema=always

This will create the quartz schema in the database.
We will provide the quartz properties using quartz.properties file.

#Quartz
org.quartz.scheduler.instanceName = SampleJobScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.idleWaitTime = 10000
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 4
org.quartz.threadPool.threadPriority = 5
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.isClustered = false
org.quartz.jobStore.maxMisfiresToHandleAtATime = 10
org.quartz.jobStore.useProperties = true

#quartz mysql database connection
org.quartz.jobStore.dataSource = mySql
org.quartz.dataSource.mySql.driver = com.mysql.cj.jdbc.Driver
org.quartz.dataSource.mySql.URL=jdbc:mysql://localhost:3306/cmt-db
org.quartz.dataSource.mySql.user=cmt-user
org.quartz.dataSource.mySql.password=cmt-pass
org.quartz.dataSource.mySql.maxConnections = 10
org.quartz.dataSource.mySql.validationQuery=select 0 from dual
#org.quartz.dataSource.mySql.maxIdleTime = 60

Make sure you do the necessary changes like database name, user, password etc in the. above code. We have used a mysql database for our use case.

Now we will create the configuration class files for SchedulerFactoryBean and AutowiringSpringBeanJobFactory,

@Configuration
public class QuartzConfig {
    @Autowired
    ApplicationContext applicationContext;

    @Autowired
    JobFactory jobFactory;


    @Bean
    public JobFactory jobFactory()
    {
        AutowiringSpringBeanJobFactory jobFactory = new AutowiringSpringBeanJobFactory();
        jobFactory.setApplicationContext(applicationContext);
        return jobFactory;
    }

    @Bean
    public SchedulerFactoryBean schedulerFactoryBean() throws IOException {
        SchedulerFactoryBean schedulerFactory = new SchedulerFactoryBean();
        schedulerFactory.setQuartzProperties(quartzProperties());
        schedulerFactory.setWaitForJobsToCompleteOnShutdown(true);
        schedulerFactory.setAutoStartup(true);
        schedulerFactory.setJobFactory(jobFactory);
        return schedulerFactory;
    }

    public Properties quartzProperties() throws IOException {
        PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
        propertiesFactoryBean.setLocation(new ClassPathResource("/quartz.properties"));
        propertiesFactoryBean.afterPropertiesSet();
        return propertiesFactoryBean.getObject();
    }

}

public class AutowiringSpringBeanJobFactory extends SpringBeanJobFactory
        implements ApplicationContextAware{

    AutowireCapableBeanFactory beanFactory;

    @Override
    public void setApplicationContext(final ApplicationContext context) {
        beanFactory = context.getAutowireCapableBeanFactory();
    }

    @Override
    protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception {
        final Object job = super.createJobInstance(bundle);
        beanFactory.autowireBean(job);
        return job;
    }
}

To schedule jobs periodically during runtime, we will create a scheduling utility function:

@Component
public class ScheduleUtility {
    @Autowired
    private QuartzConfig quartzConfig;

    public void schedule (CheckModel checkModel) throws ParseException, SchedulerException {
        //creating job detail instance
        String id = String.valueOf(checkModel.getId());
        try{
        JobDetail jobDetail = JobBuilder.newJob(CheckJob.class).withIdentity(id).build();

        // adding jobdatamap to jobdetail
        jobDetail.getJobDataMap().put("id", id);

        // currently trying with seconds only
        Trigger trigger = new CronTriggerImpl(checkModel.getCheck_name(), checkModel.getCheck_name(),generateCronExpression("0/"+checkModel.getFrequency(), "*", "*", "?", "*", "*","*"));

        Scheduler scheduler = quartzConfig.schedulerFactoryBean().getScheduler();
        scheduler.scheduleJob(jobDetail, trigger);
        scheduler.start();
    } catch (IOException | SchedulerException e){
        // scheduling failed
        e.printStackTrace();
    }
    }
    private static String generateCronExpression(final String seconds, final String minutes, final String hours,
                                                 final String dayOfMonth,
                                                 final String month, final String dayOfWeek, final String year)
    {
        return String.format("%1$s %2$s %3$s %4$s %5$s %6$s %7$s", seconds, minutes, hours, dayOfMonth, month, dayOfWeek, year);
    }
}

This function schedules the jobs to be executed based on the cron expression. In job details we pass the checkId.
Now to execute the schedule job in the service instance we will implement our job class like –

public class CheckJob implements Job {
    private static final Logger log = LoggerFactory.getLogger(CheckJob.class);

    @Autowired
    private CheckRepository checkRepository;

    @Autowired
    HealthService healthService;

    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {

        /* Get check id recorded by scheduler during scheduling */
        JobDataMap dataMap = context.getJobDetail().getJobDataMap();

        String checkId = dataMap.getString("id");

        log.info("Executing job for check id {}", checkId);

        /* Get check from database by id */
        Long id = Long.parseLong(checkId);
        Optional<CheckModel> checkModel = checkRepository.findById(id);

        /* update check detail in database */
        CheckModel check = checkModel.get();
        Integer code  = CheckHealthUtil.check(check.getUrl());
        log.info("Health is: {}",code);
        if (code == HttpURLConnection.HTTP_NOT_FOUND) {
            healthService.updateHealthStatus(check.getId(), false,Integer.parseInt(check.getFrequency()),check.getUnit().toString());
        } else {
            healthService.updateHealthStatus(check.getId(), true,Integer.parseInt(check.getFrequency()),check.getUnit().toString());
        }

     

    }
}

We fetch the url of the check with the help of the id and perform the health checkup for that particular website. To check a website health we can write a simple utility function :

public class CheckHealthUtil {
    public static Integer check(String checkUrl){
        // perform health check
        URL url  = null;
        try {
            url = new URL(checkUrl);
            HttpURLConnection huc = (HttpURLConnection) url.openConnection();
            huc.setRequestMethod("HEAD");
            int responseCode = huc.getResponseCode();
            return responseCode;
        } catch (MalformedURLException e) {
            throw new RuntimeException(e);
        }  catch (ProtocolException e) {
            throw new RuntimeException(e);
        }  catch (IOException e) {
            throw new RuntimeException(e);
        }
    }
}

Results :

Now upon running the application, when we save a check, a periodic trigger will get saved in quartz cron triggers table and Cron job will be executed as per requirement.

health check table

Thus with the help of Quartz we have designed to support distributed scheduling, which allows multiple nodes in a cluster to share job schedules and workloads. The clustering feature called JobStore, which manages the scheduling data and coordinates the job execution across the nodes in the cluster. In a distributed environment, each node runs an instance of the Quartz Scheduler, but only one node acts as the “scheduler leader” that manages the job scheduling and distribution. The other nodes act as “scheduler followers” that receive job assignments and execute them.

To learn more about the quartz–

• Reference to official documentation
• Ask questions about SAP BTP, Cloud Foundry environment and follow (https://answers.sap.com/tags/73555000100800000287)
• Read other SAP BTP, Cloud Foundry environment blog posts and follow (https://blogs.sap.com/tags/73555000100800000287/)

Please share your thoughts about this blog in the comment section.

Please follow my profile for future posts.