Smoke test for batches

The trend is to create applications which are based on all little services working together, have real time processing and rely on communication between them trough some kind of service bus. Never the less there are many organisations that struggle with big monolithic applications and rely heavily on batch  driven processing every night for several hours. Given that people aren’t very thrilled to monitor the batches during the night and when a problem arises what should they do, can it wait for tomorrow should I call a developer in the middle of the night. Organisations develop some kind of release schedule with stand by roosters and only call the release a success after the entire batch schedule has run.

To improve the release time, stand by hours and feedback cycle there must be an understanding of what already happens during development:

• Run a huge amount of unit test every time the code is checked in to the repository
• Deploy to an environment every time the CI build succeeds
• Run automated UI tests to cover the parts that can’t be hit by unit tests
• Manually test the requested changes
• Deploy to a production like environment
• Run regression test + validate the performance
• Deploy to production

Leaving you to only test the parts that change between the different environments like your configuration files. Those tests are called Smoke Testing and are a set of basic cheap tests to verify that the deployment was successful and that all the aspects to run real tests are available and set up before proceeding with the actual testing.

I am not a big fan of writing code inside a program with the sole purpose is to test and not add any value to the product. But in perspective to the scenario above it’s a small price to pay to reduce the overhead on releases. To validate the settings inside your config file without your batch to run for hours, you must run the application with a command line parameter like “-smoke” and route it to your test framework instead of running your batch activities. The generic test of visual studio enables you to do so just create a new UnitTestProject and add a Generic test.

During the smoke test you can create your own summary result file to add to your build deployment pipeline you can do this by running the xsd.exe commmand.

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Xml\Schemas>xsd /classes SummaryResult.xsd /language:cs /outputdir:c:\temp
Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 4.6.1055.0]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'c:\temp\SummaryResult.cs'.

From your application you can use the SummaryResult.cs to give feedback to your pipeline run the Visual Studio Test and publish your testresults.

However this will only work when you have Visual Studio Enterprise. In any other On Premise situation you have to find a another way.

Instead of using the SummaryResults.cs you need vstst.cs so we can use the publish test results directly.

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Xml\Schemas>xsd /classes vstst.xsd /language:cs /outputdir:c:\temp
Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 4.6.1055.0]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'c:\temp\vstst.cs'.

To get the vstst.cs working in your program you need to remove the public override object[] Items.

public partial class CodedWebTestElementType : BaseTestType
{

    //private object[] itemsField;

    ///// <remarks/>
    //[System.Xml.Serialization.XmlElementAttribute("IncludedWebTests", typeof(CodedWebTestElementTypeIncludedWebTests))]
    //[System.Xml.Serialization.XmlElementAttribute("WebTestClass", typeof(CodedWebTestElementTypeWebTestClass))]
    //public override object[] Items
    //{
    //    get
    //    {
    //        return this.itemsField;
    //    }
    //    set
    //    {
    //        this.itemsField = value;
    //    }
    //}
}

public partial class GenericTestType : BaseTestType
{

    //private object[] itemsField;

    ///// <remarks/>
    //[System.Xml.Serialization.XmlElementAttribute("Command", typeof(GenericTestTypeCommand))]
    //[System.Xml.Serialization.XmlElementAttribute("SummaryXmlFile", typeof(GenericTestTypeSummaryXmlFile))]
    //public override object[] Items
    //{
    //    get
    //    {
    //        return this.itemsField;
    //    }
    //    set
    //    {
    //        this.itemsField = value;
    //    }
    //}
}

Now to make your file work with the Publish Testresult task you have to do the following at runtime.

            var x = new XmlSerializer(testRun.GetType());
            var xmlnsEmpty = new XmlSerializerNamespaces();
            xmlnsEmpty.Add("x", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");
            xmlnsEmpty.Add("x1", "http://www.w3.org/2001/XMLSchema-instance");
            xmlnsEmpty.Add("x2", "http://www.w3.org/2001/XMLSchema"); 
            using (var fs = new FileStream(filePath, FileMode.Create, FileAccess.Write))
            {
                x.Serialize(fs, testRun, xmlnsEmpty);
                fs.Flush();
            }

            //Replace namespaces because on the publish test results to TFS, TFS doesnt accept the namespaces and we couldn't suppress them during serialization
            string text = File.ReadAllText(filePath);
            text = text.Replace("x:", "");
            text = text.Replace("x1:", "");
            text = text.Replace("x2:", "");
            text = text.Replace("xmlns:x=\"http://microsoft.com/schemas/VisualStudio/TeamTest/2010\"", "xmlns=\"http://microsoft.com/schemas/VisualStudio/TeamTest/2010\"");
            text = text.Replace("xmlns:x1=\"http://www.w3.org/2001/XMLSchema-instance\"", "");
            text = text.Replace("xmlns:x2=\"http://www.w3.org/2001/XMLSchema\"", "");
            File.WriteAllText(filePath, text);

To run it for you can use a direct command shell or use a remote powershell session to get your credentials in line with your tested application.

C:\Temp>MyTestApplication -smoke

The file you make can now be used in the Publish Testresult task and the results will be added to your pipeline for each environment you choose.

In this zip you will find the modified vstst.cs with a base class to generate the desired resultfile and a example of a config being tested.

I hope this post will give some new insights in how to overcome long lingering releases and provide the evidence that after all your unit, component testing also your environment variables are correct.