Open in app

Sign up

Sign in

Write

Sign up

Sign in

New Paradigm on Scheduling (Rocketry)

Mikael Koli
ITNEXT

--

There are some scheduling frameworks available:

  • Cron
  • Airflow
  • Windows Scheduler
  • APScheduler

and then there is Rocketry. As the author of Rocketry, I get every now and then questions on why one should use Rocketry over the others and what’s so special in it. Well, there are a lot of features in Rocketry but one stands over the others: the unique scheduling paradigm.

In the alternative frameworks, the task triggers are embedded with the schedulers. You specify a certain time of day and the scheduler checks whether the time is reached and then runs the task (if it has been reached). One might think that this is the obvious way to do it but there is a limitation with this approach: what if you wanted to run a task daily but only when a specific file exists, or daily but only after another task? In some frameworks, there are workarounds for these but often they are not natively supported and they lead to quite ugly code that is harder to maintain. Airflow, for example, implements a cron-like scheduler, task pipelines and sensors for customization but all of these are separate concepts, there are some limitations and one cannot arbitrarily combine these.

Rocketry’s scheduler, on the other hand, does not care about the current time and it only checks whether the scheduling statement is true. If it is true, then the task is run, and if not, then it isn’t. It is the statement’s responsibility to check the current time or anything else needed to verify the status of the statement. In other words, you don’t tell Rocketry’s scheduler a time when to run the task, you just give it a statement and tell it to run the task when the statement is true. This statement can be something like the current time is between 11:00 and 12:00. This statement is true if the current time is inside this range, and false otherwise.

Using this approach, Rocketry’s scheduling can support arbitrary scheduling needs regardless of the complexity. The statements can also be combined with a simple logic (AND, OR and NOT). This approach enables us to have the exact same scheduling options as with Cron or any other scheduler but we can also combine them effortlessly with other restrictions such as a specific file must also exist or another task must have first run or create arbitrarily complex scheduling logic.

Before we go more into the scheduling in Rocketry, I’ll briefly introduce the framework in case it is not familiar.

Introduction to Rocketry

Rocketry is a modern scheduling framework written in Python. It suits well for quick automation problems but also for bigger applications. It can be run stand-alone or integrated to other applications such as FastAPI. Rocketry’s focus is on clean code, productivity and customization.

Its core features include:

  • Condition mechanics
  • Task parallelization/concurrency (async, threads and processes)
  • Task parametrization
  • Modifiable runtime sessions

Here is a minimal example:

We first created the Rocketry application, then we created a simple task that runs once a day and then we started the application. In this article, we don’t go deeper into the other functionalities of the framework except for the scheduling but you can read more about the framework from the documentation.

Relevant links:

Rocketry’s Scheduling

As mentioned, the logic of how the tasks are launched can be anything. A statement, or a condition as called in the framework, is simply an instance which has observe method that evaluates the state of the condition returning true or false depending on time, some state or the task itself. Conditions can also be combined with basic boolean logic and it is also trivial to create your own custom conditions but there are a lot of built-in conditions you can use. We go through some basic scheduling needs including:

  • Running a task at specific times
  • Pipelining tasks (run a task after another task has run)
  • Combining conditions (using logic)
  • Custom conditions

There are several layers of abstraction in the condition mechanics but in this article, we focus on the condition API as that is useful for most cases.

Time-based Scheduling

The time-based conditions in Rocketry can be divided into three categories:

  • Time delta
  • Fixed time
  • Cron

Time delta scheduling means that we simply specify a length of time and the task is run every time that amount of time has passed from the previous run. Here are some examples:

Then there are fixed time conditions. This is different from the time delta in a way that fixed time conditions are anchored to our clock and calendar: a day starts from 00:00 and ends at 24:00. Here are some simple examples of such:

To be more specific, these are conditions that are true if the task has not run during the given period and the current time is in the period. In principle, these could be two separate conditions but they are combined for simplicity.

These can be further constrained using before, after, between and on. For example, if a task is set to run before 11:00 (11 a.m.), it means the task is allowed to start only if the current time is between 00:00 and 11:00. And if a task is set to run after 11:00, it means the task is allowed to start between 11:00 and 24:00. The operation on in Rocketry is a time period of one subunit of the given range: one day of week for weekly, one hour for daily, a minute for hourly etc. Ie. the statement on Monday translates to from Monday at 00:00 to Tuesday at 00:00.

Here are some examples:

You can also use these methods with minutely, hourly and monthly exactly the same way.

Due to how the condition and related time period mechanisms work, Rocketry also has full support for Cron-like scheduling in case you are more familiar with it. Just pass the cron string to the cron condition:

See more how to create the cron string: https://crontab.guru.

You may wonder how one can form the same cron scheduling with the fixed time conditions. Fixed time conditions can be further split into two categories: conditions that are linked with the task and conditions that simply check whether the current time is in the given period. Previously we used the former but the latter can be used exactly the same with time_of_minute, time_of_hour, time_of_day, day_of_week and day_of_month.

Here is an example using the cron condition and an equivalent with fixed conditions:

Note that we used the logical operation AND (&) in this example.

Pipelining Tasks

There are also pipelining conditions in Rocketry. Task pipelining means that we run one task after another task has finished. Pipelining conditions are conditions that are true when the task has not run after another specified task has finished. Here is an example:

There are also after_fail and after_finish if you wish to run the task after another task has failed or run the task after another task has finished with any status. There are also conditions such as after_any_success and after_all_success so that one can have multiple prerequisite tasks without typing after_success many times over.

You can also pipeline the output-input arguments: the return value of a task can be fed as the input arguments to another task. In this article, we focus on the scheduling but you can read more about that from the documentation.

Custom Conditions

Now that we know the basic conditions, how does one make their own? Well, that’s super simple:

This task runs constantly (as True or False equals True). You can add more logic using the logical operations:

We already covered the AND (&) operator. The symbol “|” is an OR operator and the symbol “~” is a NOT operator. In the example above we made a pointless custom condition but you can easily create a condition that checks:

  • Whether a specific file exists
  • Whether a database table has data
  • Whether your computer has internet access
  • Whether a given website can be accessed
  • Etc.

Closing Words

I hope you liked the article and found the framework useful for your problems. The scheduling is the core part of the framework but there are a lot of other features as well in the framework and you can read more about them from the documentation.

If you liked the framework, consider telling about it to your colleagues and friends and leave it a star on Github. The project is free and open source and it is developed with a passion. The project is also open for contributions, whether they are big or small.

Here are some other links you might find interesting as you got this far:

--

--

Mikael Koli
ITNEXT
Follow

Written by Mikael Koli

175 Followers
Writer for

Programming + Finance + Analytics. Developer in a trading room and author of Rocketry (modern automation back-end)

Follow