The Ministry of Nesting & Testing

Unit testing was introduced to the OO world by Kent Beck, in his seminal work on SUnit, the Smalltalk unit testing framework. Other languages have introduced their own unit testing frameworks following SUnit’s lead.

Tangent: Unit testing was part of the overall introduction of extreme programming/agile development, which is just one of the major trends Smalltalk has brought to the world. Along with refactoring (which we all know can’t be done without static types, which is why it was invented in a dynamically typed language), IDEs, reflective OO APIs, GUI builders, pop-up menus and bitmapped GUIs in general. Smalltalk is the veritable Prometheus of OO, and its destiny seems not dissimilar.

Newspeak started out with an adaptation of SUnit, NSUnit, which is what you’ll find in the public release. It has a rather nice Hopscotch based GUI integrated into the IDE, but we always felt we could improve upon it.

Minitest is our revised unit testing framework, which we’ve been using since, oh, mid-2010 or so. Minitest takes the opportunity to rationalize the way we structure unit tests and takes advantage of Newspeak’s support for nesting to make things simpler and easier to use.

Minitest was designed by Vassili Bykov, and the examples below are shamelessly lifted from the superb documentation Vassili wrote for the Minitest class.

In Minitest, you define a testing module, that is designed to test a particular interface (not a particular implementation). To run tests, one needs to feed the testing module with the particular implementation(s) that one wishes to test. A test configuration module does just that. Newspeak naturally enforces this separation of interface and implementation.

The testing module class’s factory typically takes three arguments: the Newspeak platform, the testing framework (a Minitest instance) and a factory for the object under test.

class ListTesting usingPlatform: platform minitest: minitest listClass: listClass = (
 |     
 private TestContext = minitest TestContext.     
 private List = listClass. 
|
)(
      class ListTests = TestContext ( 
      | list = List new. |  
     ) (     
          testAddition = (         
             list add: 1.         
             assert: (list includes: 1)     
          )     
          testRemoval = (         
             list add: 1; remove: 1.         
             deny: (list includes: 1)     
          )  
    ) : ( TEST_CONTEXT = () ) 
)

The example shows a hypothetical (and rather simplistic) module definition for testing lists. I’m sure all readers of this blog are fluent in Newspeak, but just in case, the module definition has a factory method that takes the 3 parameters mentioned above: platform (the Newspeak platform, from which all kinds of generally useful libraries might be obtained), minitest (an instance of Minitest, naturally) and listClass, a factory that will produce lists for us to test.

Nested inside the testing module is a test context (aka test fixture) class ListTests, inside of which you write your tests. Test methods are identified by the convention that their names begin with test. Each test will be executed in a test context; that is, for each test method being run, Minitest will instantiate a fresh ListTests object. That is why ListTests is called a test context - it provides a context for a single test.

It is common to define test context classes like ListTests as subclasses of the class TestContext defined by the Minitest framework, but that is not essential. TestContext provides useful methods like deny:, so it is convenient to use it. However, what identifies ListTests as a test context is the marker class method TEST_CONTEXT.

Minitest will do its work by examining the nested classes of the test module and seeing which are test contexts (that is, which have a class method named TEST_CONTEXT). For each test context tc, Minitest will list all its test methods (the ones with names beginning with test) and for each of those, it will instantiate tc and call the selected method on it, gathering data on success or failure.

Minitest does away with concepts like TestResource. that are typically used to hold data for tests.

In the simple case above, the data for the test gets created by the instance initializer of ListTests . However, what if the data for the test needs to be shared among multiple tests (say, because it is expensive to create)?

As an example, suppose we want to test a compiler, and setting up the compiler is relatively costly.

class CompilerTesting usingPlatform: platform
                   minitest: minitest
                   compilerClass: compilerClass = ( 
| Compiler = compilerClass. |  )
(
    class CompilerHolder = (     
      | compiler = Compiler configuredInAParticularWay. |    
 )( 
    class StatementsTests ( ...) (....): ( TEST_CONTEXT = ()) 
  )
)

Minitest leverages Newspeak’s nested structure in these cases. A test context (StatementTests above) does not have to be a direct nested class of the test module. Instead, we can nest it more deeply inside another nested class (CompilerHolder). That nested class will serve to hold any state that we want to share among multiple tests - in our case, an instance of the compiler, which it will create and store as part of its initialization.

As you can see there is no need for a special setUp method or a test resource class. Newspeak’s nesting structure and built-in instance initializers take care of all that. If the shared resource is just an object in memory, then it will also be disposed of via garbage collection after the test is run. Of course, some resources cannot be just garbage collected. In that case, one should define a method named cleanUp in the test context class.

As mentioned in the beginning of the post, we need a test configuration to run the tests, as the test module definition is always parametric with respect to any implementation that we would actually test.

A test configuration module is defined by a top level class with the factory method

packageTestsUsing: ideNamespace
The factory takes a namespace object that should provide access to the testing module declaration and to any concrete classes or objects we want to test. This arrangement is very similar to how we package applications from within the IDE.

    
class ListTestingConfiguration packageTestsUsing: ideNamespace = (   
|     
private ListTesting = ideNamespace ListTesting.
private Collections = ideNamespace Collections.  
|  )( ‘required’
    testModulesUsingPlatform: platform minitest: minitest = (       
     ^{ ListTesting usingPlatform: platform
                    minitest: minitest
                    listClass: (Collections usingPlatform: platform) LinkedList.      
      }
 )
)

The method testModulesUsingPlatform:minitest: must be provided by the configuration. It will be called by Minitest to produce a set of testing modules, each of which will be processed by the framework as outlined above (i.e., searched for test contexts to be run). In the example, only one test module is returned, but if we wanted to process multiple List implementations (say ArrayList as well as LinkedList) we could write:

    
class ListTestingConfiguration packageTestsUsing: ideNamespace = (   
|     
private ListTesting = ideNamespace ListTesting.
private Collections = ideNamespace Collections. 
|  )( ‘required’
    testModulesUsingPlatform: platform minitest: minitest = ( 
     | collections = Collections using: platform. |      
     ^{ ListTesting usingPlatform: platform
                    minitest: minitest
                    listClass: collections LinkedList.         
        ListTesting usingPlatform: platform
                    minitest: minitest
                    listClass: collections ArrayList.     
      }
 )
)

The IDE recognizes test configurations based on the name of the factory method - that is, a class with a class method packageTestsUsing: is considered a test configuration, and the IDE will provide a run tests link in the (upper right hand corner) class browser in that case, as shown in the screenshot below (click on it to enlarge).



Clicking on the link will call the packageTestsUsing: method on the class with an argument representing the IDE’s namespace, and feed the results into Minitest.



This is all you need to know to use Minitest. Actually, it’s considerably more than what you need to know, as I’ve also explained how a bit about how the framework goes about its business.

It is worth noting how Minitest cleanly breaks down the multiple roles an SUnit TestCase has. The definition of a set of tests is done by a test context. The actual configuration is done a test configuration. And the actual command to run a specific test (the thing that should be called TestCase) is not the user’s concern anymore - the test framework handles it but need not expose it. In SUnit these three roles are conjoined. Perhaps this is why I never really felt comfortable with SUnit.

Likewise, no need to worry over test resources and special set up methods. The net result is a framework that is very easy to use and simple to understand.

It’s intriguing to note that one could actually structure a Java unit testing framework this way; we rely on introspection, interfaces and nested (inner) classes. However, it is not natural to do so in Java. Nested classes in Java are usually (and often rightly) regarded a trap to be avoided. A design like Minitest is much more likely to crop up in a setting where nesting is idiomatic, like Newspeak. Language influences thought - or lack of thought, as the case may be.