Pipes and Filters

Pipes and filters (or just pipeline) is another common pattern. Oren Eini and Jeremy Likness both have very interesting posts about it on their respective blogs.

While Jeremy’s post aims at the slick creation of pipelines, Oren talks more about the pattern itself and how to implement it in a specific manner (using input and output values of Type IEnumerable<T>).

Some interesting argument came up in the comments to Oren’s post. “Why not use LINQ instead of your custom pipeline (framework)?”

I won’t repeat all the pros and cons here. Better read the posts yourself, they are worth your time!

My opinion on the topic: LINQ is a great tool. But I believe it’s neither the only solution for chains of queries and transformations, nor is it the best in all cases.

I will pick up one point from the “pro LINQ” point of view. “With LINQ you can have different input and output types.”

Sure you can. But who says you can’t do that with pipes and filters just as easily?

We define an abstract base class Filter<TIn, TOut>

public abstract class Filter<TIn, TOut>
{
  public abstract IEnumerable<TOut> Process(IEnumerable<TIn> input);
  public Filter<TIn, TNext> Pipe<TNext>(Filter<TOut, TNext> next)
  {
    return new Pipe<TIn, TOut, TNext>(this, next);
  }
}

and a derived class Pipe<TIn, T, TOut>

public class Pipe<TIn, T, TOut> : Filter<TIn, TOut>
{
  private readonly Filter<TIn, T> source;
  private readonly Filter<T, TOut> destination;
  public Pipe(Filter<TIn, T> source, Filter<T, TOut> destination)
  {
    this.source = source;
    this.destination = destination;
  }
  public override sealed IEnumerable<TOut> Process(IEnumerable<TIn> input)
  {
    var x = this.source.Process(input);
    var result = this.destination.Process(x);
    return result;
  }
}

With these two as a base we can easily chain filters with different input and output types.

We can also fine-tune the ends of the pipeline a bit so that the code is a nicer read. With a small extension method and a just as small dummy class we can use any enumerable as the starting point for defining a pipeline.

public static Filter<TIn, TOut> Pipe<TIn, TOut>(this IEnumerable<TIn> enumerable, Filter<TIn, TOut> filter)
{
  return new PipelineStartDummy<TIn, TOut>(enumerable, filter);
}
private class PipelineStartDummy<TIn, TOut> : Filter<TIn, TOut>
{
  private readonly IEnumerable<TIn> enumerable;
  private readonly Filter<TIn, TOut> filter;
  public PipelineStartDummy(IEnumerable<TIn> enumerable, Filter<TIn, TOut> filter)
  {
    this.enumerable = enumerable;
    this.filter = filter;
  }
  public override IEnumerable<TOut> Process(IEnumerable<TIn> input)
  {
    return this.filter.Process(this.enumerable);
  }
}

If we don’t care what comes out of the pipeline and just want to start processing values from the source we can use another extension method that encapsulates Oren’s enumerator magic.

public static void Start<TIn, TOut>(this Filter<TIn, TOut> filter)
{
  var enumerable = filter.Process(null);
  var enumerator = enumerable.GetEnumerator();
  while (enumerator.MoveNext()) { }
}

And now we put it all together and get the following:

new Numbers(3).Pipe(new Square()).Pipe(new Printer()).Start();

Numbers just returns integer values between 1 and the constructor parameter. Square squares all input values and the Printer writes the input to the console. The call to Start() starts the processing.

While it is not the most impressive example implementation of the pipes and filters pattern I believe that it demonstrates how powerful and flexible the pattern can be. And the code is still readable and very explicit about what you are doing. With just a few lines of code you have a composable, easy to understand solution where you can recombine filters in different orders to change the behavior of the pipeline. And you can do just about anything inside of those filters (think validation or enriching the values traversing through the pipeline with data from services or persistent storage). You can even change the type of the values you are processing between steps. This is yet another case of “Like it a lot!”

Specification Pattern

The Specification Pattern. Yet another one I find useful in daily business. But a few enhancements can make it even more useful and a lot nicer to handle.

1, 2, many!

The CompositeSpecification is most often implemented with two fields for spec1 and spec2 or left and right. The AndSpecification and OrSpecifications evaluate these fields with their respective operator.

public class Or<T> : CompositeSpecification<T>
{
  // ...

  public override bool IsSatisfiedBy(T candidate)
  {
    return spec1.IsSatisfiedBy(candidate) || spec2.IsSatisfiedBy(candidate);
  }
}

This can result in a degenerated tree structure when you link many specifications with the same operator (e.g. a.Or(b).Or(c)...Or(n)).

To avoid this I decided to change the behavior of the composite. Instead of two fields it uses a list of children. If you try to link two specifications the code checks wether either one of them is of the same type as the composite and adds the other composites children instead of the composite itself to the list.

Sounds more complicated than it is. Let’s see some code.

public abstract class CompositeSpecification<T> : Specification<T>
{
  private readonly List<Specification<T>> children;
  public CompositeSpecification()
  {
    this.children = new List<Specification<T>>();
  }
  public IEnumerable<Specification<T>> Children
  {
    get { return children; }
  }
  public int Count
  {
    get { return this.children.Count; }
  }
  protected void Add(Specification<T> specification)
  {
    this.children.Add(specification);
  }
  protected void AddRange(IEnumerable<Specification<T>> specifications)
  {
    this.children.AddRange(specifications);
  }
}

public class Or<T> : CompositeSpecification<T>
{
  public Or(Specification<T> specification, Specification<T> other)
  {
    this.Include(specification);
    this.Include(other);
  }
  public override string Description
  {
    get { return " || "; }
  }
  public override bool IsSatisfiedBy(T candidate)
  {
    foreach (Specification<T> specification in this.Children)
    {
      if (specification.IsSatisfiedBy(candidate))
      {
        return true;
      }
    }
    return false;
  }
  private void Include(Specification<T> specification)
  {
    Or<T> or = specification as Or<T>;
    if (or != null)
    {
      this.AddRange(or.Children);
    }
    else
    {
      this.Add(specification);
    }
  }
}

And this is how two specifications can be linked with an Or operator.

public abstract class Specification<T>
{
  // ...
  
  public abstract bool IsSatisfiedBy(T candidate);
  public Specification<T> Or(Specification<T> other)
  {
    return new Or<T>(this, other);
  }
}

In the end this will put all successive Or’s in a single list, which makes finding the right specification in the tree a lot easier.

What do we have?

Generating a human readable representation of the specification tree can be tedious but is often beneficial if you need to see “what you have”. The easiest way to traverse a tree structure is a Visitor. The same pattern is used by Microsoft in their expression trees.

We add an Accept method and a property for the description of the specification to the base classes

public abstract class Specification<T>
{
  // ...
  public abstract string Description { get; }
  public virtual void Accept(SpecificationVisitor<T> visitor)
  {
    visitor.Visit(this);
  }
}

public abstract class CompositeSpecification<T> : Specification<T>
{
  // ...
  public override void Accept(SpecificationVisitor<T> visitor)
  {
    visitor.Visit(this);
  }
}

public class Or<T> : CompositeSpecification<T>
{
  // ...
  public override string Description
  {
    get { return " || "; }
  }
}

Define a base class for the SpecificationVisitor

public abstract class SpecificationVisitor<T>
{
  public abstract void Visit(Specification<T> specification);
  public abstract void Visit(CompositeSpecification<T> composite);
}

And the implementation of a PrettyPrinter becomes as simple as that

public class PrettyPrinter<T> : SpecificationVisitor<T>
{
  private readonly StringBuilder sb;
  public PrettyPrinter()
  {
    this.sb = new StringBuilder(250);
  }
  public override void Visit(Specification<T> specification)
  {
    this.sb.Append(specification.Description);
  }
  public override void Visit(CompositeSpecification<T> composite)
  {
    this.sb.Append("(");
    foreach (Specification<T> child in composite.Children)
    {
      child.Accept(this);
      this.sb.Append(composite.Description);
    }
    int l = composite.Description.Length;
    this.sb.Remove(sb.Length - l, l);
    this.sb.Append(")");
  }
  public override string ToString()
  {
    return this.sb.ToString();
  }
}

And this gives you a nice and friendly printout of your graph

var spec = new AlwaysFalse().Or(new AlwaysTrue().And(new Odd())).Or(new AlwaysTrue());
var printer = new PrettyPrinter<T>();
spec.Accept(printer);
string friendly = printer.ToString(); // (false || (true && Odd) || true)

First time I tried to attach some zipped source code and found out that WordPress won’t let me… Get the source code here (project TecX.Common folder Specifications and the test suite that puts them to use in TecX.Common.Test).

Rebuttal: Composite Pattern

Oren Eini (aka Ayende Rahien) has an interesting series on his blog where he reviews the GoF Design Patterns and how they apply (or don’t) to modern software development.

As part of this series he also reviews the Composite Pattern and finishes with the following advice:

Recommendation: If your entire dataset can easily fit in memory, and it make sense, go ahead. Most of the time, you probably should stay away.

I find his conclusion somewhat limited as he only seems to take data storage/structures into account. But what about code that “does” things?

If you consider the following interface:

public interface IWriter
{
  void Write(Foo foo);
}

there may be a lot of implementations that write somewhere. Like the local file system. A database. A WCF service or any number of other targets. And what if you want to write to several of those targets and not only one? Do you want to change your code to handle a List<IWriter>? Everywhere? No? Me neither. Instead I prefer to leave my existing code that knows how to write to a single target as is and introduce a composite writer that does the job of writing to multiple targets for me.

public class CompositeWriter : IWriter
{
  private readonly List<IWriter> writers = new List<IWriter>();
  public void Write(Foo foo)
  {
    foreach(var writer in this.writers)
    {
      writer.Write(foo);
    }
  }
  public void Add(IWriter writer)
  {
    this.writers.Add(writer);
  }
}

The sample ignores validation or error handling (Does the composite ignore errors in its targets and just continues? Do the targets have to handle the errors themselves etc.). It also ignores the possible complexity of the Foo it has to write. This is where I agree with Oren. If this structure is large and you need to duplicate it for every Write() you are screwed.

I also ignore the delay that might be introduced by writing to an arbitrarily large set of targets. But you can also handle that inside the composite. Maybe you introduce timeouts for calling a single writer. Or you write a decorator that calls a writer on a background thread if that fits your needs. You can make your solution as complex or simple as you have to. The fact is: Your original code won’t know about that complexity. It just calls an implementation of your IWriter interface. I think this is what SRP is about.

You isolate the callers from the possibly complex task of calling out to multiple targets. Your composite is responsible for dealing with that complexity. And that is it’s only responsibility.

The Builder Pattern

A couple of years ago Jan van Ryjswyck published some articles on the Builder Pattern.

In his series he describes the pattern in the context of test data creation

• Test Data Builder vs. Object Mother
• Test Data Builder
• Test Data Builders Refined

While it is absolutely true that this pattern is very useful when you have to repeatedly create test data with small variations it does not stop there!

Frameworks like Enterprise Library or AutoFac use the same pattern to create their configuration settings.

IConfigurationSource configSource = new DictionaryConfigurationSource();
ConfigurationSourceBuilder builder = new ConfigurationSourceBuilder();

builder.ConfigureLogging()
  .LogToCategoryNamed("DataAccess")
  .SendTo.RollingFile("MyListener")
    .FormatWith(
      new FormatterBuilder()
        .TextFormatterNamed("MyFormatter")
        .UsingTemplate("{timestamp} {message}"));

builder.UpdateConfigurationWithReplace(configSource);
IUnityContainer container = new UnityContainer();
IContainerConfigurator configurator = new UnityContainerConfigurator(container);
EnterpriseLibraryContainer.ConfigureContainer(configurator, configSource);

But it still doesn’t stop there! If you are developing a component that others will have to use, builders are a way to use Dependency Injection in your code but neither enforce the usage of a specific DI container nor the usage of the DI Pattern at all on these consumers.

Mark Seemann, author of Dependency Injection in .NET, outlines how this can be done in his answer to a question on StackOverflow.

You can encapsulate meaningful default values in your builders whitout poluting your API with various constructor overloads. This allows novice consumers to use your components easily while experts can swap the defaults for custom implementations if neccessary or even use a DI container to wire them up and bypass the builders.