Testing custom StyleCop rules

StyleCop is the preferred way to enforce consistency in your project’s coding style. Code guidelines in printed or electronic form are always inferior to automatic validation of said guidelines as part of your build process. You get compiler warnings or errors with (hopefully) helpful and descriptive explanations of why what you did is a bad idea. StyleCop even contains a ReSharper plugin that adds quick-fixes for many rule violations.

But StyleCop can do more than just notify you that someone forgot an empty line after a closing curly bracket. As a proof of concept I wrote rules that should prevent faulty re-throws (catch-blocks in the form of catch(Exception ex) { throw ex; }) that swallow the StackTrace or constructors with more than 4 parameters (which is considered a code smell that indicates a component has too many responsibilities).

The point of this post however is not to show you how to write such rules but how you can make sure that they really work by testing them as part of your regular unit testing process. If you are looking for samples on how to write custom rules check out this post or use the search engine of your choice. There are lots of samples.

Posts like this one explain how to setup tests for custom StyleCop rules using the Visual Studio test-runner and VS’ Test View. In my opinion that thing gives you the worst possible user-experience. It makes running your tests as convenient as reading a paper about quantum physics. ReSharper or TestDriven.Net are way more easy to handle. But the given examples don’t work properly with those tools or alternative test-runners (e.g. NUnit or xUnit). I love my tools and I hate to be forced to use something inferior by circumstances that I can influence. So I set out to remove that particular bump in the road.

I did not want to fumble with files and how and where to deploy them for test-runs with different test frameworks or tools. Instead I wanted to make these files embedded resources that are just there when I run my tests. I started with the sample mentioned above and made several changes to it.

1. Use the StyleCopObjectConsole instead of the StyleCopConsole.
2. Create a class derived from SourceCode that handles loading code from embedded resources (I called mine ResourceBasedSourceCode. Samples for that can be found in the StyleCop source code. Search for StringBasedSourceCode).
3. The StyleCopObjectConsole requires an instance of ObjectBasedEnvironment which in turn requires a delegate that creates SourceCode instances for given file names. Make that delegate create instances of your ResourceBasedSourceCode.
4. Hook up the handlers for output and violations to the events of StyleCopObjectConsole.Core instead of the events of the console object.

Lets have a look at the source code of the tests for which I use xUnit.

public class CodeQualityAnalyzerFixture
{
  private readonly StyleCopObjectConsole console;
  private readonly ICollection output;
  private readonly ICollection violations;
  private readonly CodeProject codeProject;

  public CodeQualityAnalyzerFixture()
  {
    this.violations = new List<string>();
    this.output = new List<string>();
    ObjectBasedEnvironment environment = new ObjectBasedEnvironment(ResourceBasedSourceCode.Create, GetProjectSettings);
    ICollection<string> addinPaths = new[] { "." };
    this.console = new StyleCopObjectConsole(environment, null, addinPaths, true);
    CsParser parser = new CsParser();
    parser.FileTypes.Add("CS");
    this.console.Core.Environment.AddParser(parser);
    this.console.Core.ViolationEncountered += (s, e) =&gt; this.violations.Add(e.Violation);
    this.console.Core.OutputGenerated += (s, e) =&gt; this.output.Add(e.Output);
    this.codeProject = new CodeProject(0, null, new Configuration(null));
  }

  [Fact]
  public void Should_Flag_TooManyCtorParameters()
  {
    this.AnalyzeCodeWithAssertion("TooManyConstructorArguments.cs", 1);
  }

  [Fact]
  public void Should_Flag_IncorrectRethrow()
  {
    this.AnalyzeCodeWithAssertion("IncorrectRethrow.cs", 1);
  }

  private Settings GetProjectSettings(string path, bool readOnly)
  {
    return null;
  }

  private void AnalyzeCodeWithAssertion(string codeFileName, int expectedViolations)
  {
    this.AddSourceCode(codeFileName);
    this.StartAnalysis();
    this.WriteViolationsToConsole();
    this.WriteOutputToConsole();
    Assert.Equal(expectedViolations, this.violations.Count);
  }

  private void WriteOutputToConsole()
  {
    foreach (var o in this.output)
    {
      Console.WriteLine(o);
    }
  }

  private void WriteViolationsToConsole()
  {
    foreach (var violation in this.violations)
    {
      Console.WriteLine(violation.Message);
    }
  }

  private void AddSourceCode(string resourceFileName)
  {
    bool sourceCodeSuccessfullyLoaded = this.console.Core.Environment.AddSourceCode(this.codeProject, resourceFileName, null);
    if (sourceCodeSuccessfullyLoaded == false)
    {
      throw new InvalidOperationException(string.Format("Source file '{0}' could not be added.", resourceFileName));
    }
  }

  private void StartAnalysis()
  {
    IList projects = new[] { this.codeProject };
    this.console.Start(projects);
  }
}

What does it do? Lets start with the test setup in the constructor.

1. Initialize the collections for the output generated by StyleCop.
2. Instantiate the ObjectBasedEnvironment providing it with the required delegates to create SourceCode and Settings. The SourceCodeFactory delegate is a static method of my ResourceBasedSourceCode class so we will come to that in a moment. The ProjectSettingsFactory delegate always returns null.
3. Define a path to the add-in under test (which is the assembly that contains our custom rules). So the dot stands for the “current working folder” where said add-in will be copied to by the build by default.
4. Create the console object and add a CsParser to the console. As the name indicates it is responsible for parsing C# code files. Don’t forget to register the file extension for C# code files when you create the parser or it just won’t work. The extension name is case-sensitive. Use upper-case.
5. Register the eventhandlers that will write output and rule violations to the respective collections that will be evaluated after StyleCop is done processing your code files.
6. Create a new CodePoject. The code files you want analyzed will be added to that project during your tests.

The test methods are simple. They call the AnalyzeCodeWithAssertion method with the name of the file you want to analyze and the number of violations you expect. This is the most basic validation possible. You can make it arbitrarily complex if you want to (e.g. by asserting the type of the expected violation or message texts or …).

Note that I made my ResourceBasedSourceCode class handle incomplete file names. I didn’t want to have to deal with all the namespaces that are added as prefixes to embedded resources so I decided to go with “EndsWith” semantics. As long as the name of the resource ends with the file name provided as a parameter it works just fine.

The AnalyzeCodeWithAssertion method is responsible for

1. Adding the source code to the project that will be analyzed.
2. Running the actual analysis using StyleCop.
3. Writing the output generated during the analysis to the console so you can explore it at leisure after the test is run.
4. Asserting your expectations.

Nothing spectacular there. Now lets have a look at the code for the ResourceBasedSourceCode.

public class ResourceBasedSourceCode : SourceCode
{
  private readonly string _ResourceName;
  public ResourceBasedSourceCode(CodeProject project, SourceParser parser, string resourceName)
    : this(project, parser, resourceName, Assembly.GetExecutingAssembly())
  {
  }
  public ResourceBasedSourceCode(CodeProject project, SourceParser parser, string resourceName, Assembly assembly)
    : base(project, parser)
  {
    Guard.AssertNotEmpty(resourceName, "resourceName");
    Guard.AssertNotNull(assembly, "assembly");
    string fullResourceName = assembly.GetManifestResourceNames()
	  .FirstOrDefault(rn => rn.EndsWith(resourceName, StringComparison.OrdinalIgnoreCase));
	  
    if (string.IsNullOrEmpty(fullResourceName))
    {
      throw new ArgumentException(string.Format("No resource has a name that ends with '{0}'", resourceName), "resourceName");
    }
    _ResourceName = fullResourceName;
  }

  public static SourceCode Create(string path, CodeProject project, SourceParser parser, object context)
  {
    return new ResourceBasedSourceCode(project, parser, path);
  }

  public override TextReader Read()
  {
    Stream stream = this.GetType().Assembly.GetManifestResourceStream(_ResourceName);
    return new StreamReader(stream);
  }

  public override bool Exists
  {
    get { return true; }
  }

  public override string Name
  {
    get { return _ResourceName; }
  }

  public override string Path
  {
    get { return _ResourceName; }
  }

  public override DateTime TimeStamp
  {
    get { return DateTime.MinValue; }
  }

  public override string Type
  {
    get { return "cs"; }
  }
}

There is very little magic involved in this class. In the constructor we try to find an embedded resource matching the name that is provided as a parameter and store the full name of the resource in a field. If that step doesn’t fail we can safely assume that we can open a Stream to read that resource later on. The file extension is hard-coded to C# files (property Type). Lower-case for the extension name is OK here. Don’t ask, I have no clue why the developers decided to make that different from the behavior of the CsParser. The Create method is a factory method with the signature of the SourceCodeFactory delegate as required by the ObjectBasedEnvironment. Everything else is implemented in the most basic way to make the analysis pass.

Now just add a folder that contains the code files to your test project (I called it “Resources”). Make sure to set the files’ Build Action to “Embedded Resource”. And you are ready to roll.

That’s all the infrastructure you need. Doesn’t look that complicated, does it? If you are interested in the complete source code you can find that here. The project names are TecX.Build and TecX.Build.Test. Have fun!