Pipes and Filters

Pipes and filters (or just pipeline) is another common pattern. Oren Eini and Jeremy Likness both have very interesting posts about it on their respective blogs.

While Jeremy’s post aims at the slick creation of pipelines, Oren talks more about the pattern itself and how to implement it in a specific manner (using input and output values of Type IEnumerable<T>).

Some interesting argument came up in the comments to Oren’s post. “Why not use LINQ instead of your custom pipeline (framework)?”

I won’t repeat all the pros and cons here. Better read the posts yourself, they are worth your time!

My opinion on the topic: LINQ is a great tool. But I believe it’s neither the only solution for chains of queries and transformations, nor is it the best in all cases.

I will pick up one point from the “pro LINQ” point of view. “With LINQ you can have different input and output types.”

Sure you can. But who says you can’t do that with pipes and filters just as easily?

We define an abstract base class Filter<TIn, TOut>

public abstract class Filter<TIn, TOut>
{
  public abstract IEnumerable<TOut> Process(IEnumerable<TIn> input);
  public Filter<TIn, TNext> Pipe<TNext>(Filter<TOut, TNext> next)
  {
    return new Pipe<TIn, TOut, TNext>(this, next);
  }
}

and a derived class Pipe<TIn, T, TOut>

public class Pipe<TIn, T, TOut> : Filter<TIn, TOut>
{
  private readonly Filter<TIn, T> source;
  private readonly Filter<T, TOut> destination;
  public Pipe(Filter<TIn, T> source, Filter<T, TOut> destination)
  {
    this.source = source;
    this.destination = destination;
  }
  public override sealed IEnumerable<TOut> Process(IEnumerable<TIn> input)
  {
    var x = this.source.Process(input);
    var result = this.destination.Process(x);
    return result;
  }
}

With these two as a base we can easily chain filters with different input and output types.

We can also fine-tune the ends of the pipeline a bit so that the code is a nicer read. With a small extension method and a just as small dummy class we can use any enumerable as the starting point for defining a pipeline.

public static Filter<TIn, TOut> Pipe<TIn, TOut>(this IEnumerable<TIn> enumerable, Filter<TIn, TOut> filter)
{
  return new PipelineStartDummy<TIn, TOut>(enumerable, filter);
}
private class PipelineStartDummy<TIn, TOut> : Filter<TIn, TOut>
{
  private readonly IEnumerable<TIn> enumerable;
  private readonly Filter<TIn, TOut> filter;
  public PipelineStartDummy(IEnumerable<TIn> enumerable, Filter<TIn, TOut> filter)
  {
    this.enumerable = enumerable;
    this.filter = filter;
  }
  public override IEnumerable<TOut> Process(IEnumerable<TIn> input)
  {
    return this.filter.Process(this.enumerable);
  }
}

If we don’t care what comes out of the pipeline and just want to start processing values from the source we can use another extension method that encapsulates Oren’s enumerator magic.

public static void Start<TIn, TOut>(this Filter<TIn, TOut> filter)
{
  var enumerable = filter.Process(null);
  var enumerator = enumerable.GetEnumerator();
  while (enumerator.MoveNext()) { }
}

And now we put it all together and get the following:

new Numbers(3).Pipe(new Square()).Pipe(new Printer()).Start();

Numbers just returns integer values between 1 and the constructor parameter. Square squares all input values and the Printer writes the input to the console. The call to Start() starts the processing.

While it is not the most impressive example implementation of the pipes and filters pattern I believe that it demonstrates how powerful and flexible the pattern can be. And the code is still readable and very explicit about what you are doing. With just a few lines of code you have a composable, easy to understand solution where you can recombine filters in different orders to change the behavior of the pipeline. And you can do just about anything inside of those filters (think validation or enriching the values traversing through the pipeline with data from services or persistent storage). You can even change the type of the values you are processing between steps. This is yet another case of “Like it a lot!”

Advertisements

2 Responses to Pipes and Filters

  1. PipesAndFiltersDesignPatternNewbie says:

    Thanks for your blog on the Pipes and Filters Design Pattern. I would really be interested in seeing code for the actual concrete classes used in your example. At the very least, the Numbers, Square and Printer classes mentioned in your example.

    However, in my case, ideally I’d like to see it with more business classes that would say decrypt the request class sent in (say GetMemberRequest, which is comprised of the following: Firstname, Lastname, DOB, Gender), validate it (for example, validate the name is alpha-numeric), process the request (for example, call database storproc to get Member properties), transform it (say the (internal) MemberIn class has Addresses as an array of Address, but (external) MemberOut class has Addresses as a generic list of Address), and finally encrypt it for sending out to the client. The implementations can be trivial, i’m more interested in the syntax used to denote the different input and output types in the concrete classes.

    • weberse says:

      I hope that WordPress won’t garble the code too much. A very basic outline of what you want to do would look something like this:

      Filter filter = new DecryptFilter()
        .Pipe(new ProcessMemberPropertiesFilter())
        .Pipe(new TransformFilter())
        .Pipe(new EncryptFilter());
      
      public class EncryptFilter : Filter
      {
        public override IEnumerable Process(IEnumerable members)
        {
          foreach (MemberOut member in members)
          {
            yield return new GetMemberResponse
            {
              EncryptedMember = this.Encrypt(member)
            };
          }
        }
        private string Encrypt(MemberOut member)
        {
          return string.Empty;
        }
      }
      
      public class TransformFilter : Filter
      {
        public override IEnumerable Process(IEnumerable members)
        {
          foreach (MemberIn member in members)
          {
            yield return new MemberOut
            {
              FirstName = member.FirstName,
              Addresses = new List<Address>(member.Addresses)
            };
          }
        }
      }
      
      public class ProcessMemberPropertiesFilter : Filter
      {
        public override IEnumerable Process(IEnumerable input)
        {
          foreach (MemberIn member in input)
          {
            member.Addresses = this.GetAddressesFromDatabase(member.FirstName);
            yield return member;
          }
        }
        private Address[] GetAddressesFromDatabase(string firstName)
        {
          return new Address[0];
        }
      }
      
      public class DecryptFilter : Filter
      {
        public override IEnumerable Process(IEnumerable requests)
        {
          foreach (GetMemberRequest request in requests)
          {
            yield return new MemberIn { FirstName = Decrypt(request.FirstName) };
          }
        }
        private string Decrypt(string encrypted)
        {
          return string.Empty;
        }
      }
      
      public class GetMemberRequest
      {
        public string FirstName { get; set; }
      }
      public class Address
      {
      }
      public class MemberIn
      {
        public string FirstName { get; set; }
        public Address[] Addresses { get; set; }
      }
      public class MemberOut
      {
        public string FirstName { get; set; }
        public List<Address> Addresses { get; set; }
      }
      public class GetMemberResponse
      {
        public string EncryptedMember { get; set; }
      }
      

      There is really no magic involved. LINQ to objects does things in a very similar way.

      Hope that helps.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: