Quartz.NET meets Design Patterns

This is the third in a series of posts.

In the last post I showed you how I set up some tests for my implementation of retries with Quartz.NET. I repeatedly hinted at some neat tricks to make things more convenient so here they are.

Quartz.NET requires that your jobs only throw JobExecutionExceptions (as explained at the very bottom of this page). There are reasons why this restriction makes sense but I don’t want to litter my business logic with repetitions of the exact same exception handling code. I think that’s what DRY is all about. I don’t want to force all of my jobs to inherit from a specific base class either. At least not for the single purpose of following Quartz.NET’s rules for exception handling.

But by applying a decorator to my job classes I can fix that once and for all.

public class EnsureJobExecutionExceptionDecorator : IJob
{
  private readonly IJob inner;
  public EnsureJobExecutionExceptionDecorator(IJob inner)
  {
    this.inner = inner;
  }
  public void Execute(IJobExecutionContext context)
  {
    try
    {
      this.inner.Execute(context);
    }
    catch (JobExecutionException)
    {
      throw;
    }
    catch (Exception cause)
    {
      throw new JobExecutionException(cause);
    }
  }
}

JobExecutionExceptions are simply rethrown. Which allows you to throw them in your job if you have to tweak what to tell the scheduler. All other exceptions become InnerExceptions of a new JobExecutionException. Done. Now that was easy.

But how do I ensure that each time Quartz.NET instantiates a job the decorator is in place?

By replacing the scheduler’s default IJobFactory with something more advanced. For my playground I derived from the PropertySettingJobFactory base class and use Unity to create my jobs.

private sealed class UnityJobFactory : PropertySettingJobFactory
{
  private readonly IUnityContainer container;
  public UnityJobFactory(IUnityContainer container)
  {
    this.ThrowIfPropertyNotFound = false;
    this.WarnIfPropertyNotFound = true;
    this.container = container;
  }
  public override IJob NewJob(
    TriggerFiredBundle bundle,
    IScheduler scheduler)
  {
    Type jobType = bundle.JobDetail.JobType;
    IJob job = (IJob)this.container.Resolve(jobType);
    JobDataMap data = new JobDataMap();
    data.PutAll(scheduler.Context);
    data.PutAll(bundle.JobDetail.JobDataMap);
    data.PutAll(bundle.Trigger.JobDataMap);
    this.SetObjectProperties((object)job, data);
    return job;
  }
  public override void ReturnJob(IJob job)
  {
    this.container.Teardown(job);
  }
}

And then its a simple matter to configure Unity to wrap every job it creates with the EnsureJobExecutionExceptionDecorator. Not that hard is it?

And finally there is the code snippet that unfreezes my test thread when I’m done.

public class UnfreezeWhenJobShouldNotRunAgain : IRetryStrategy
{
  private readonly IRetryStrategy inner;
  private readonly ManualResetEvent reset;
  public UnfreezeWhenJobShouldNotRunAgain(
    IRetryStrategy inner,
    ManualResetEvent reset)
  {
    this.inner = inner;
    this.reset = reset;
  }
  public bool ShouldRetry(IJobExecutionContext context)
  {
    bool shouldRetry = this.inner.ShouldRetry(context);
    if (!shouldRetry)
    {
      this.reset.Set();
    }
    return shouldRetry;
  }
  public ITrigger GetTrigger(IJobExecutionContext context)
  {
    return this.inner.GetTrigger(context);
  }
}

Yet another decorator. Whenever the RetryJobListener from the first post of this series queries the IRetryStrategy wether a job should be run again the decorator checks for “yes” or “no”. And in case of a “no” it will set the ManualResetEvent and allow the test thread to continue.

So we have decorators, an abstract factory and dependency injection here. And all of that in less than 200 lines of code. All pieces short and to the point but by combining them you can build mighty powerful solutions that are still clean and easy to understand.

I hope you enjoyed the series and come back for another read. See you soon!

Advertisements

Testing Retries in Quartz.NET

This is the second in a series of posts.

In the last post I showed you the moving parts for persistent retries with Quartz.NET. Now it’s time to bring them all together.

With a little trick it is quite easy to run the actual Quartz.NET scheduler in a unit test. My current testing framework of choice is xUnit. So that’s what I will use in my code.

[Fact]
public void Should_Try_3_Times_And_Then_Give_Up()
{
  ISchedulerFactory factory = new StdSchedulerFactory();
  IScheduler scheduler = factory.GetScheduler();

  AlwaysFails alwaysFails = new AlwaysFails();
  IJob decoratedJob = new EnsureJobExecutionExceptionDecorator(alwaysFails);
  var jobFactory = new Mock<IJobFactory>();
  jobFactory
    .Setup(
      jf => jf.NewJob(It.IsAny<TriggerFiredBundle>(), It.IsAny<IScheduler>()))
    .Returns(decoratedJob);

  ManualResetEvent reset = new ManualResetEvent(false);
  IRetrySettings settings = new InMemoryRetrySettings
    {
      BackoffBaseInterval = 250.Milliseconds(),
      MaxRetries = 2
    };
  IRetryStrategy sut = new ExponentialBackoffRetryStrategy(settings);
  IRetryStrategy retryStrategy = new UnfreezeWhenJobShouldNotRunAgain(sut, reset);
  IJobListener retryListener = new RetryJobListener(retryStrategy);

  scheduler.ListenerManager.AddJobListener(
    retryListener,
    GroupMatcher<JobKey>.AnyGroup());
  scheduler.JobFactory = jobFactory.Object;

  ITrigger trigger = TriggerBuilder
    .Create()
    .StartNow()
    .WithSimpleSchedule(
      x =>
      {
        x.WithIntervalInSeconds(1);
        x.WithRepeatCount(0);
      })
    .WithIdentity("always", "fails")
    .Build();

  IJobDetail job = JobBuilder
    .Create<AlwaysFails>()
    .WithIdentity("always", "fails")
    .Build();

  scheduler.ScheduleJob(job, trigger);
  scheduler.Start();
  scheduler.ResumeAll();

  reset.WaitOne(15.Seconds());

  Assert.Equal(3, alwaysFails.Counter);
}

We start by setting up the SchedulerFactory and use it to get us a live Quartz.NET scheduler.

For this test I want to be able to tell that scheduler how it should create instances of my job class. I usually use a DI container to implement that factory but for the purpose of this test I use a mock object created with Moq.

Whenever the scheduler asks for an instance of the AlwaysFails job the factory returns a canned instance I created as part of my test setup. I explain the EnsureJobExecutionExceptionDecorator in another post.

Next I plug the actual RetryJobListener together and set it up to listen to all jobs using a catch-all matcher.

Then I configure my job to run immediately. Note that you can instruct the scheduler to run your job repeatedly but that is not what I want here. Under ideal circumstances the job should run once and be done with it. Only in case of failure do I want to run it again. And as another note: You can set a flag on the JobExecutionException that will cause the scheduler to retry immediately. But again: Not what I want.

When you start the scheduler it will create a background thread and do all its work there. The test method will just continue and exit before anything meaningful happens. Thus I need to freeze the test execution thread for a little while. This is what the ManualResetEvent is used for. To allow the test to continue as soon as the retry strategy says we are done trying I use another decorator. This one wraps the retry strategy. Please see the aforementioned post for further details.

In this test I use a large part of the Quartz.NET infrastructure. It’s not quite a system test but close enough for me at the moment. It can be run along with my unit tests because everything is done in-memory and it really doesn’t take that long to finish.

In the final post of this series we will make an excursion into the realm of composable software systems. Stay tuned!

Retries with Quartz.NET

This is the first in a series of posts.

Whenever you have a task that might take a while to complete it is usually a good idea to run it in the background and not block your application.

What makes up “a while” and how far “in the background” you should run your tasks is usually up for discussion.

The scenario here is an ASP.NET web application that needs to perform tasks and might fail because some objects that need to be modified are locked. If a task hits such a lock it should be run again later after waiting a little. And again after waiting some more. And again. And then again sometime in the middle of the night when it is rather improbable that the objects are still locked. And then give up eventually sending a notification to a human to resolve the problem.

For in process retries Microsoft p&p offers the Transient Fault Handling Application Block (infos on MSDN and CodePlex). But I needed a persistent retry strategy that would kick in if the application goes down for one reason or another.

There are a lot of things to keep in mind when running recurring tasks inside ASP.NET and a lot of really good reasons why you don’t want to write your own scheduler. Luckily there are quite a few alternatives to that.

I decided to solve my problem using Quartz.NET. It is free. It is battle hardened. It is widely used so the search engine of my choice will be able to find a substantial amount of blog posts and questions on popular Q&A sites. And last but not least: I wanted to take a look at it for quite some time.

I don’t want to give an introduction to Quartz.NET here. Their website does a fairly good job (no pun intended) at that.

So I will jump right into the thick of things and show you my approach to how to implement retries using Quartz.NET.

After some web search I decided to use this post as a starting point but make some adjustments along the way.

To implement custom tasks (or jobs as Quartz calls them) you have to implement the IJob interface. It doesn’t get any more straightforward. Implement your logic inside the single method of the interface and you are done. The only thing to keep in mind is that your job must only throw exceptions of a specific type. I outline my solution to that problem in another post of this series.

To be able to test my solution I created a job that would always fail. That turned out to be really simple.

public class AlwaysFails : IJob
{
  private int counter;
  public AlwaysFails()
  {
    this.counter = 0;
  }
  public int Counter
  {
    get { return this.counter; }
  }
  public void Execute(IJobExecutionContext context)
  {
    this.counter++;
    throw new NotImplementedException();
  }
}

For every call to the Execute()-method I increase a counter by one and throw an exception. Quartz requires that you only throw JobExecutionException inside your jobs. There is a simple solution to that problem that allows me to ignore this requirement here.

Now to the interesting part. As mentioned by the post linked above a custom IJobListener is the best place to put your retry logic. But I don’t want to hard-wire the logic that calculates whether the job should run again or when the next attempt should be made into the handler itself.

The listener only looks at whether the job failed. Then it asks a retry strategy (we will come to that in a moment) if the job should run again. And if the answer is yes it asks the strategy when the next attempt should be made and reschedules the job accordingly.

public class RetryJobListener : JobListenerSupport
{
  private readonly IRetryStrategy retryStrategy;
  public RetryJobListener(IRetryStrategy retryStrategy)
  {
    this.retryStrategy = retryStrategy;
  }
  public override string Name { get { return "Retry"; } }
  public override void JobWasExecuted(IJobExecutionContext context, JobExecutionException jobException)
  {
    if (JobFailed(jobException) && this.retryStrategy.ShouldRetry(context))
    {
      ITrigger trigger = this.retryStrategy.GetTrigger(context);
      bool unscheduled = context.Scheduler.UnscheduleJob(context.Trigger.Key);
      DateTimeOffset nextRunAt = context.Scheduler.ScheduleJob(context.JobDetail, trigger);
    }
  }
  public override void JobToBeExecuted(IJobExecutionContext context)
  {
  }
  private static bool JobFailed(JobExecutionException jobException)
  {
    return jobException != null;
  }
}

Nothing fancy so far. The retry strategy that contains the logic to determine whether or not the job should run again consists of the simple interface IRetryStrategy

public interface IRetryStrategy
{
  bool ShouldRetry(IJobExecutionContext context);
  ITrigger GetTrigger(IJobExecutionContext context);
}

and (as of now) one implementation for an exponential back-off strategy.

public class ExponentialBackoffRetryStrategy : IRetryStrategy
{
  private const string Retries = "Retries";
  private readonly IRetrySettings settings;
  public ExponentialBackoffRetryStrategy(IRetrySettings settings)
  {
    this.settings = settings;
  }
  public bool ShouldRetry(IJobExecutionContext context)
  {
    int retries = GetAlreadyPerformedRetries(context);
    return retries < this.settings.MaxRetries;
  }
  public ITrigger GetTrigger(IJobExecutionContext context)
  {
    int retries = GetAlreadyPerformedRetries(context);
    long factor = (long)Math.Pow(2, retries);
    TimeSpan backoff = new TimeSpan(this.settings.BackoffBaseInterval.Ticks * factor);

    ITrigger trigger = TriggerBuilder.Create()
      .StartAt(DateTimeOffset.UtcNow + backoff)
      .WithSimpleSchedule(x => x.WithRepeatCount(0))
      .WithIdentity(context.Trigger.Key)
      .ForJob(context.JobDetail)
      .Build();

    context.JobDetail.JobDataMap[Retries] = ++retries;
    return trigger;
  }
  private static int GetAlreadyPerformedRetries(IJobExecutionContext context)
  {
    int retries = 0;
    object o;
    if (context.JobDetail.JobDataMap.TryGetValue(Retries, out o) && o is int)
    {
      retries = (int)o;
    }
    return retries;
  }
}

Like in the example we use the JobDataMap of the job to store the retry counter. That map is automatically persisted (if you configured a persistent IJobStore) between runs so we won’t loose that counter. We can configure how many retries should be performed using the IRetrySettings. There is an in-memory implementation and one that hooks up with my playground’s configuration system. The details don’t matter here so I will just show you the interface declaration.

public interface IRetrySettings
{
  int MaxRetries { get; }
  TimeSpan BackoffBaseInterval { get; }
}

Until we hit the upper boundary of MaxRetries we calculate the time the job should wait and create a new trigger for the next run using Quartz’ fluent TriggerBuilder. Don’t forget to increase the counter for the retries!

Well and that’s the implementation part. The next post will show you how to bring it all together for a test run.

Quartz and other gems

Originally I intended to write a post about retries using the Quartz.NET scheduler.

But the post ended up being far too long so I decided to break it up into a short series.

  1. Retries with Quartz.NET
  2. Testing Retries with Quartz.NET
  3. Quartz.NET meets Design Patterns