How to Write Safe Timers With OutSystems
One of my favorite features of the OutSystems platform are the timers. Instead of writing code and then using an external tool to schedule and control it, I have a one-stop shop for building scheduled tasks. However, just sitting down and slamming code together for your timers is going to cause you some headaches in the case of failures at runtime. Let’s look at how to write timers that won’t get you in trouble.
What’s the Problem?
You may be wondering “why does this matter? OutSystems automatically retries a failed timer, it isn’t a big deal!”
For some applications, that may be fine! But imagine that you are working on a system that looks at invoices every night and tries to charge customers for each invoice due. Depending on how you wrote your code, you could end up re-charging the failed invoices multiple times, or worse, re-charging the successful invoices multiple times. Yikes! Or, perhaps you are importing data. Re-importing the same data will be a performance sink at best and create duplicate data at worst. Not good.
Track Success/Failure per Record and per Job
Each time you run the timer, you should create a record that tracks the “job” being performed (“importing data” or “billing customers” are examples), and another record for each record you are processing (row of data imported from the spreadsheet, invoice that you are trying to bill, and so on).
Data you definitely need to track is:
- Start date/time
- Stop date/time
- Status (success, completed with no errors, completed with errors, failed)
- Any output or error messages
Exception Handling
Each record should be processed in an action by itself. That action should return a success/failure flag and any needed messages and handle exceptions inside itself. The main timer should simply call that action, handle the output (changing the job’s status as-appropriate), and do a CommitTransaction after each record is processed. This way, all status information about the job and record is always saved. If the job needs to be re-run, you will be able to filter out the records that have already been processed. You will also not be broken out of the loop if an exception occurs, and continue processing the remaining records. Be aware that committing transactions has a performance hit associated with it, and in some cases you may be better off only committing every ten or hundred (or whatever) records, assuming that re-trying work is allowable.
Of course, if that logic does not make sense for your application, do a commit transaction after the failure (to record the record history) and throw an exception to break out of the loop and go into the timer’s exception handler (to mark the job as failed and exit out).
Don’t Retry Automatically
Now that you are tracking the status of of your jobs, the next step to getting safe timers is to stop them from retrying automatically. You do this in the OutSystems Service Center. Set the “Timer Execution Attempts” setting to 1.
If You Need to Retry…
But sometimes you really do want to retry in case of a failure! At the end of your timer, if there are errors, and in your exception handler, you can query your job status history. If there are only a few recent failures, call the wake action for your timer to have it retry. You will want to configure a sane and safe setting for how many times it should retry. For example, if there are no more than five failed jobs in the last 15 minutes, then retry.
At the Beginning of the Timer
Now that we have a ton of data giving us the history of our timer, we can do a neat trick: query and double check the failure history to see if this timer really should try to run right now. If the last execution was a success, then ignore other recent failures, since the problem seems to be resolved. This ensure that if a schedule is trying to retry the timer, we do not keep repeating the same failures over and over again.
Next, when you query your work (data to import, invoices to process, etc.), you can filter out the ones that are tied to failures. As a result, you do not retry records that are going to fail. This is especially important when working with money! The last thing you want is for a problem or a bug to cause you to re-try payments. Even if a payment will fail a second time, it is bad to keep retrying it.
Wrapping it up
OutSystems has given us a super-useful tool for running code on a scheduled basis. But, without good coding practices, we can create some big problems. Following these steps will ensure data integrity and proper application of business logic.
J.Ja
