Package csb :: Package test

[frames] | no frames]

# Source Code for Package csb.test

      1  """
      2  This is a top level package, hosting the entire CSB test framework. It is divided
      3  into several major parts:
      4
      5      - test cases, located under csb.test.cases
      6      - test data, in C{/csb/test/data} (not a package)
      7      - test console, in C{/csb/test/app.py}
      8
      9  This module, csb.test, contains all the glue-code functions, classes and
     10  decorators you would need in order to write tests for CSB.
     11
     12      1. Configuration and Tree
     13
     14         L{Config<csb.test.Config>} is a common config object shared between CSB
     15         tests. Each config instance contains properties like:
     16
     17              - data: the data folder, automatically discovered and loaded in
     18                csb.test.Config.DATA at module import time
     19              - temp: a default temp folder, which test cases can use
     20
     21         Each L{Config<csb.test.Config>} provides a convenient way to retrieve
     22         files from C{/csb/test/data}. Be sure to check out L{Config.getTestFile}
     23         and L{Config.getPickle}. In case you need a temp file, use
     24         L{Config.getTempStream} or have a look at L{csb.io.TempFile} and
     25         L{csb.io.TempFolder}.
     26
     27         All test data files should be placed in the C{data} folder. All test
     28         modules must be placed in the root package: csb.test.cases. There is
     29         a strict naming convention for test modules: the name of a test module
     30         should be the same as the name of the CSB API package it tests. For
     31         example, if you are writing tests for C{csb/bio/io/__init__.py}, the
     32         test module must be C{csb/test/cases/bio/io/__init__.py}. C{csb.test.cases}
     33         is the root package of all test modules in CSB.
     34
     35      2. Writing Tests
     36
     37         Writing a test is easy. All you need is to import csb.test and then
     38         create your own test cases, derived from L{csb.test.Case}:
     39
     40             >>> import csb.test
     41             >>> @csb.test.unit
     42                 class TestSomeClass(csb.test.Case):
     43                     def setUp(self):
     44                         super(TestSomeClass, self).setUp()
     45                         # do something with self.config here...
     46
     47         In this way your test case instance is automatically equipped with a
     48         reference to the test config, so your test method can be:
     49
     50             >>> @csb.test.unit
     51                 class TestSomeClass(csb.test.Case):
     52                     def testSomeMethod(self):
     53                         myDataFile = self.config.getTestFile('some.file')
     54                         self.assert...
     55
     56         The "unit" decorator marks a test case as a collection of unit tests.
     57         All possibilities are: L{csb.test.unit}, L{csb.test.functional}, L{csb.test.custom},
     58         and L{csb.test.regression}.
     59
     60         Writing custom (a.k.a. "data", "slow", "dynamic") tests is a little bit
     61         more work. Custom tests must be functions, not classes. Basically a
     62         custom test is a function, which builds a unittest.TestSuite instance
     63         and then returns it when called without arguments.
     64
     65         Regression tests are usually created in response to reported bugs. Therefore,
     66         the best practice is to mark each test method with its relevant bug ID:
     67
     68             >>> @csb.test.regression
     69                 class SomeClassRegressions(csb.test.Case)
     70                     def testSomeFeature(self)
     71                     \"""
     72                     @see: [CSB 000XXXX]
     73                     \"""
     74                     # regression test body...
     75
     76      3. Style Guide:
     77
     78         - name test case packages as already described
     79         - group tests in csb.test.Case-s and name them properly
     80         - prefix test methods with "test", like "testParser" - very important
     81         - use camelCase for methods and variables. This applies to all the
     82           code under csb.test (including test) and does not apply to the rest
     83           of the library!
     84         - for functional tests it's okay to define just one test method: runTest
     85         - for unit tests you should create more specific test names, for example:
     86           "testParseFile" - a unit test for some method called "parse_file"
     87         - use csb.test decorators to mark tests as unit, functional, regression, etc.
     88         - make every test module executable::
     89
     90             if __name__ == '__main__':
     91                 csb.test.Console()   # Discovers and runs all test cases in the module
     92
     93      4. Test Execution
     94
     95         Test discovery is handled by C{test builders} and a test runner
     96         C{app}. Test builders are subclasses of L{AbstractTestBuilder}.
     97         For every test type (unit, functional, regression, custom) there is a
     98         corresponding test builder. L{AnyTestBuilder} is a special builder which
     99         scans for unit, regression and functional tests at the same time.
    100
    101         Test builder classes inherit the following test discovery methods:
    102
    103             - C{loadTests} - load tests from a test namespace. Wildcard
    104               namespaces are handled by C{loadAllTests}
    105             - C{loadAllTests} - load tests from the given namespace, and
    106               from all sub-packages (recursive)
    107             - C{loadFromFile} - load tests from an absolute file name
    108             - C{loadMultipleTests} - calls C{loadTests} for a list of
    109               namespaces and combines all loaded tests in a single suite
    110
    111         Each of those return test suite objects, which can be directly executed
    112         with python's unittest runner.
    113
    114         Much simpler way to execute a test suite is to use our test app
    115         (C{csb/test/app.py}), which is simply an instance of L{csb.test.Console}::
    116
    117             $ python csb/test/app.py --help
    118
    119         The app has two main arguments:
    120
    121             - test type - tells the app which TestBuilder to use for test dicsovery
    122               ("any" triggers L{AnyTestBuilder}, "unit" - L{UnitTestBuilder}, etc.)
    123             - test namespaces - a list of "dotted" test modules, for example::
    124
    125                  csb.test.cases.bio.io.*   # io and sub-packages
    126                  csb.test.cases.bio.utils  # only utils
    127                  .                         # current module
    128
    129         In addition to running the app from the command line, you can run it
    130         also programmatically by instantiating L{csb.test.Console}. You can
    131         construct a test console object by passing a list of test namespace(s)
    132         and a test builder class to the Console's constructor.
    133
    134
    135      5. Commit Policies
    136
    137         Follow these guidelines when making changes to the repository:
    138
    139             - B{no bugs in "trunk"}: after fixing a bug or implementing a new
    140               feature, make sure at least the default test set passes by running
    141               the test console without any arguments. This is equivalent to:
    142               app.py -t any "csb.test.cases.*". (If no test case from this set covers
    143               the affected code, create a test case first, as described in the other
    144               policies)
    145
    146             - B{no recurrent issues}: when a bug is found, first write a regression
    147               test with a proper "@see: BugID" tag in the docstring. Run the test
    148               to make sure it fails. After fixing the bug, run the test again before
    149               you commit, as required by the previous policy
    150
    151             - B{test all new features}: there should be a test case for every new feature
    152               we implement. One possible approach is to write a test case first and
    153               make sure it fails; when the new feature is ready, run the test again
    154               to make sure it passes
    155
    156  @warning: for compatibility reasons do NOT import and use the unittest module
    157            directly. Always import unittest from csb.test, which is guaranteed
    158            to be python 2.7+ compatible.
    159  """
    160  import os
    161  import sys
    162  import imp
    163  import types
    164  import time
    165  import tempfile
    166  import traceback
    167  import argparse
    168
    169  import csb.io
    170  import csb.core
    171
    172  try:
    173      from unittest import skip, skipIf
    174      import unittest
    175  except ImportError:
    176      import unittest2 as unittest
    177
    178  from abc import ABCMeta, abstractproperty
    179
    180
    181 -class Attributes(object):
    182
    183      UNIT       = '__CSBUnitTest__'
    184      CUSTOM     = '__CSBCustomTest__'
    185      FUNCTIONAL = '__CSBFunctionalTest__'
    186      REGRESSION = '__CSBRegressionTest__'
    187
    188 -class Config(object):
    189      """
    190      General CSB Test Config. Config instances contain the following properties:
    191
    192          - data - path to the CSB Test Data directory. Default is L{Config.DATA}
    193          - temp - path to the system's temp directory. Default is L{Config.TEMP}
    194          - config - the L{Config} class
    195      """
    196
    197      DATA = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
    198      """
    199      @cvar: path to the default test data directory: <install dir>/csb/test/data
    200      """
    201      GENERATED_DATA = DATA
    202      """
    203      @cvar: path to the default data directory for generated test files
    204      """
    205      TEMP = os.path.abspath(tempfile.gettempdir())
    206      """
    207      @cvar: path to the default system's temp directory
    208      """
    209
    210      @staticmethod
    211 -    def setDefaultDataRoot(path):
    212          """
    213          Override the default L{Config.DATA} with a new data root directory.
    214
    215          @param path: full directory path
    216          @type path: str
    217          """
    218          if not os.path.isdir(path):
    219              raise IOError('Path not found: {0}'.format(path))
    220
    221          Config.DATA = os.path.abspath(path)
    222
    223      @staticmethod
    224 -    def setDefaultGeneratedDataRoot(path):
    225          """
    226          Override the default L{Config.GENERATED_DATA} with a new data root directory.
    227
    228          @param path: full directory path
    229          @type path: str
    230          """
    231          if not os.path.isdir(path):
    232              raise IOError('Path not found: {0}'.format(path))
    233
    234          Config.GENERATED_DATA = os.path.abspath(path)
    235
    236      @property
    237 -    def data(self):
    238          """
    239          Test data directory
    240          @rtype: str
    241          """
    242          return Config.DATA
    243
    244      @property
    245 -    def generatedData(self):
    246          """
    247          Test data directory for generated files
    248          @rtype: str
    249          """
    250          return Config.GENERATED_DATA
    251
    252      @property
    253 -    def temp(self):
    254          """
    255          Test temp directory
    256          @rtype: str
    257          """
    258          return Config.TEMP
    259
    260 -    def getTestFile(self, fileName, subDir=''):
    261          """
    262          Search for C{fileName} in the L{Config.DATA} directory. If not found,
    263          try also L{Config.GENERATED_DATA} (if different).
    264
    265          @param fileName: the name of a test file to retrieve
    266          @type fileName: str
    267          @param subDir: scan a sub-directory of L{Config.DATA}
    268          @type subDir: str
    269
    270          @return: full path to C{fileName}
    271          @rtype: str
    272
    273          @raise IOError: if no such file is found
    274          """
    275          for data in [self.data, self.generatedData]:
    276              file = os.path.join(data, subDir, fileName)
    277
    278              if os.path.isfile(file):
    279                  return file
    280
    281          raise IOError('Test file not found: {0}'.format(fileName))
    282
    283 -    def getPickle(self, fileName, subDir=''):
    284          """
    285          Same as C{self.getTestFile}, but try to unpickle the the file
    286          and return the unpickled object. Pickles are usually stored in
    287          L{Config.GENERATED_DATA}.
    288
    289          @param fileName: the name of a test file to retrieve
    290          @type fileName: str
    291          @param subDir: scan a sub-directory of L{Config.DATA}
    292          @type subDir: str
    293
    294          @rtype: object
    295          """
    296          file = self.getTestFile(fileName, subDir)
    297          return csb.io.Pickle.load(open(file, 'rb'))
    298
    299 -    def getContent(self, fileName, subDir=''):
    300          """
    301          Same as C{self.getTestFile}, but also read and return the contents of
    302          the file.
    303
    304          @param fileName: the name of a test file to retrieve
    305          @type fileName: str
    306          @param subDir: scan a sub-directory of L{Config.DATA}
    307          @type subDir: str
    308
    309          @rtype: str
    310          """
    311          with open(self.getTestFile(fileName, subDir)) as f:
    312              return f.read()
    313
    314 -    def getTempStream(self, mode='t'):
    315          """
    316          Return a temporary file stream::
    317
    318              with self.getTempStream() as tmp:
    319                  tmp.write(something)
    320                  tmp.flush()
    321                  file_name = tmp.name
    322
    323          @param mode: file open mode (text, binary), default=t
    324          @type mode: str
    325          @rtype: file stream
    326          """
    327          return csb.io.TempFile(mode=mode)
    328
    329 -    def ensureDataConsistency(self):
    330          """
    331          Try to deserialize some pickled data files. Call L{Config.updateDataFiles}
    332          if the pickles appeared incompatible with the current interpreter.
    333          """
    334          try:
    335              self.getPickle('1nz9.model1.pickle')
    336          except:
    337              self.updateDataFiles()
    338
    339 -    def updateDataFiles(self):
    340          """
    341          Refresh the pickled structures in csb/test/data. This might be needed when
    342          the internal representation of some classes has changed.
    343          """
    344          from csb.io import Pickle
    345          from csb.bio.io.wwpdb import RegularStructureParser
    346          from csb.bio.structure import Ensemble, ChemElements
    347
    348          parser = RegularStructureParser(self.getTestFile('1nz9.pdb'))
    349          model1 = parser.parse_structure(model=1)
    350          model2 = parser.parse_structure(model=2)
    351
    352          ensemble = Ensemble()
    353          ensemble.models.append(model1)
    354          ensemble.models.append(model2)
    355          Pickle.dump(ensemble, open(os.path.join(self.generatedData, '1nz9.full.pickle'), 'wb'))
    356
    357          mse = model1.chains['A'].find(164)
    358          mse.label = 'MSE'
    359          mse.atoms['SD']._element = ChemElements.Se
    360          mse.atoms['SD']._full_name = 'SE  '
    361          Pickle.dump(model1, open(os.path.join(self.generatedData, '1nz9.model1.pickle'), 'wb'))
    362
    363 -class Case(unittest.TestCase):
    364      """
    365      Base class, defining a CSB Test Case. Provides a default implementation
    366      of C{unittest.TestCase.setUp} which grabs a reference to a L{Config}.
    367      """
    368
    369      @property
    370 -    def config(self):
    371          """
    372          Test config instance
    373          @rtype: L{Config}
    374          """
    375          return self.__config
    376
    377 -    def setUp(self):
    378          """
    379          Provide a reference to the CSB Test Config in the C{self.config} property.
    380          """
    381          self.__config = Config()
    382          assert hasattr(self.config, 'data'), 'The CSB Test Config must contain the data directory'
    383          assert self.config.data, 'The CSB Test Config must contain the data directory'
    384
    385 -    def reRaise(self, addArgs=()):
    386          """
    387          Re-raise the last exception with its full traceback, but modify the
    388          argument list with C{addArgs} and the original stack trace.
    389
    390          @param addArgs: additional arguments to append to the exception
    391          @type addArgs: tuple
    392          """
    393          klass, ex, _tb = sys.exc_info()
    394          ex.args = list(ex.args) + list(addArgs) + [''.join(traceback.format_exc())]
    395
    396          raise klass(ex.args)
    397
    398 -    def assertAlmostEqual(self, first, second, places=None, msg=None, delta=None):
    399
    400          if first == second:
    401              return
    402          if delta is not None and places is not None:
    403              raise TypeError("specify delta or places not both")
    404
    405          if delta is not None:
    406
    407              if abs(first - second) <= delta:
    408                  return
    409
    410              m = '{0} != {1} within {2} delta'.format(first, second, delta)
    411              msg = self._formatMessage(msg, m)
    412
    413              raise self.failureException(msg)
    414
    415          else:
    416              if places is None:
    417                  places = 7
    418
    419              return super(Case, self).assertAlmostEqual(first, second, places=places, msg=msg)
    420
    421 -    def assertFasterThan(self, duration, callable, *args, **kargs):
    422          """
    423          Fail if it took more than C{duration} seconds to invoke C{callable}.
    424
    425          @param duration: maximum amount of seconds allowed
    426          @type duration: float
    427          """
    428
    429          start = time.time()
    430          callable(*args, **kargs)
    431          execution = time.time() - start
    432
    433          if execution > duration:
    434              self.fail('{0}s is slower than {1}s)'.format(execution, duration))
    435
    436      @classmethod
    437 -    def execute(cls):
    438          """
    439          Run this test case.
    440          """
    441          suite = unittest.TestLoader().loadTestsFromTestCase(cls)
    442          runner = unittest.TextTestRunner()
    443
    444          return runner.run(suite)
    445
    446 -class InvalidNamespaceError(NameError, ImportError):
    447      pass
    448
    449 -class AbstractTestBuilder(object):
    450      """
    451      This is a base class, defining a test loader which exposes the C{loadTests}
    452      method.
    453
    454      Subclasses must override the C{labels} abstract property, which controls
    455      what kind of test cases are loaded by the test builder.
    456      """
    457
    458      __metaclass__ = ABCMeta
    459
    460      @abstractproperty
    461 -    def labels(self):
    462          pass
    463
    464 -    def loadFromFile(self, file):
    465          """
    466          Load L{csb.test.Case}s from a module file.
    467
    468          @param file: test module file name
    469          @type file: str
    470
    471          @return: a C{unittest.TestSuite} ready for the test runner
    472          @rtype: C{unittest.TestSuite}
    473          """
    474          mod = self._loadSource(file)
    475          suite = unittest.TestLoader().loadTestsFromModule(mod)
    476          return unittest.TestSuite(self._filter(suite))
    477
    478 -    def loadTests(self, namespace):
    479          """
    480          Load L{csb.test.Case}s from the given CSB C{namespace}. If the namespace
    481          ends with a wildcard, tests from sub-packages will be loaded as well.
    482          If the namespace is '__main__' or '.', tests are loaded from __main__.
    483
    484          @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
    485                            load tests from '/csb/test/cases/bio/__init__.py'
    486          @type namespace: str
    487
    488          @return: a C{unittest.TestSuite} ready for the test runner
    489          @rtype: C{unittest.TestSuite}
    490          """
    491          if namespace.strip() == '.*':
    492              namespace = '__main__.*'
    493          elif namespace.strip() == '.':
    494              namespace = '__main__'
    495
    496          if namespace.endswith('.*'):
    497              return self.loadAllTests(namespace[:-2])
    498          else:
    499              loader = unittest.TestLoader()
    500              tests = loader.loadTestsFromName(namespace)
    501              return unittest.TestSuite(self._filter(tests))
    502
    503 -    def loadMultipleTests(self, namespaces):
    504          """
    505          Load L{csb.test.Case}s from a list of given CSB C{namespaces}.
    506
    507          @param namespaces: a list of test module namespaces, e.g.
    508                             ('csb.test.cases.bio', 'csb.test.cases.bio.io') will
    509                             load tests from '/csb/test/cases/bio.py' and
    510                             '/csb/test/cases/bio/io.py'
    511          @type namespaces: tuple of str
    512
    513          @return: a C{unittest.TestSuite} ready for the test runner
    514          @rtype: C{unittest.TestSuite}
    515          """
    516          if not csb.core.iterable(namespaces):
    517              raise TypeError(namespaces)
    518
    519          return unittest.TestSuite(self.loadTests(n) for n in namespaces)
    520
    521 -    def loadAllTests(self, namespace, extension='.py'):
    522          """
    523          Load L{csb.test.Case}s recursively from the given CSB C{namespace} and
    524          all of its sub-packages. Same as::
    525
    526              builder.loadTests('namespace.*')
    527
    528          @param namespace: test module namespace, e.g. 'csb.test.cases.bio' will
    529                            load tests from /csb/test/cases/bio/*'
    530          @type namespace: str
    531
    532          @return: a C{unittest.TestSuite} ready for the test runner
    533          @rtype: C{unittest.TestSuite}
    534          """
    535          suites = []
    536
    537          try:
    538              base = __import__(namespace, level=0, fromlist=['']).__file__
    539          except ImportError:
    540              raise InvalidNamespaceError('Namespapce {0} is not importable'.format(namespace))
    541
    542          if os.path.splitext(os.path.basename(base))[0] != '__init__':
    543              suites.append(self.loadTests(namespace))
    544
    545          else:
    546
    547              for entry in os.walk(os.path.dirname(base)):
    548
    549                  for item in entry[2]:
    550                      file = os.path.join(entry[0], item)
    551                      if extension and item.endswith(extension):
    552                          suites.append(self.loadFromFile(file))
    553
    554          return unittest.TestSuite(suites)
    555
    556 -    def _loadSource(self, path):
    557          """
    558          Import and return the Python module identified by C{path}.
    559
    560          @note: Module objects behave as singletons. If you import two different
    561                 modules and give them the same name in imp.load_source(mn), this
    562                 counts for a redefinition of the module originally named mn, which
    563                 is basically the same as reload(mn). Therefore, you need to ensure
    564                 that for every call to imp.load_source(mn, src.py) the mn parameter
    565                 is a string that uniquely identifies the source file src.py.
    566          """
    567          name = os.path.splitext(os.path.abspath(path))[0]
    568          name = name.replace('.', '-').rstrip('__init__').strip(os.path.sep)
    569
    570          return imp.load_source(name, path)
    571
    572 -    def _recurse(self, obj):
    573          """
    574          Extract test cases recursively from a test C{obj} container.
    575          """
    576          cases = []
    577          if isinstance(obj, unittest.TestSuite) or csb.core.iterable(obj):
    578              for item in obj:
    579                  cases.extend(self._recurse(item))
    580          else:
    581              cases.append(obj)
    582          return cases
    583
    584 -    def _filter(self, tests):
    585          """
    586          Filter a list of objects using C{self.labels}.
    587          """
    588          filtered = []
    589
    590          for test in self._recurse(tests):
    591              for label in self.labels:
    592                  if hasattr(test, label) and getattr(test, label) is True:
    593                      filtered.append(test)
    594
    595          return filtered
    596
    597 -class AnyTestBuilder(AbstractTestBuilder):
    598      """
    599      Build a test suite of cases, marked as either unit, functional or regression
    600      tests. For detailed documentation see L{AbstractTestBuilder}.
    601      """
    602      @property
    603 -    def labels(self):
    604          return [Attributes.UNIT, Attributes.FUNCTIONAL, Attributes.REGRESSION]
    605
    606 -class UnitTestBuilder(AbstractTestBuilder):
    607      """
    608      Build a test suite of cases, marked as unit tests.
    609      For detailed documentation see L{AbstractTestBuilder}.
    610      """
    611      @property
    612 -    def labels(self):
    613          return [Attributes.UNIT]
    614
    615 -class FunctionalTestBuilder(AbstractTestBuilder):
    616      """
    617      Build a test suite of cases, marked as functional tests.
    618      For detailed documentation see L{AbstractTestBuilder}.
    619      """
    620      @property
    621 -    def labels(self):
    622          return [Attributes.FUNCTIONAL]
    623
    624 -class RegressionTestBuilder(AbstractTestBuilder):
    625      """
    626      Build a test suite of cases, marked as regression tests.
    627      For detailed documentation see L{AbstractTestBuilder}.
    628      """
    629      @property
    630 -    def labels(self):
    631          return [Attributes.REGRESSION]
    632
    633 -class CustomTestBuilder(AbstractTestBuilder):
    634      """
    635      Build a test suite of cases, marked as custom tests. CustomTestBuilder will
    636      search for functions, marked with the 'custom' test decorator, which return
    637      a dynamically built C{unittest.TestSuite} object when called without
    638      parameters. This is convenient when doing data-related tests, e.g.
    639      instantiating a single type of a test case many times iteratively, for
    640      each entry in a database.
    641
    642      For detailed documentation see L{AbstractTestBuilder}.
    643      """
    644      @property
    645 -    def labels(self):
    646          return [Attributes.CUSTOM]
    647
    648 -    def loadFromFile(self, file):
    649
    650          mod = self._loadSource(file)
    651          suites = self._inspect(mod)
    652
    653          return unittest.TestSuite(suites)
    654
    655 -    def loadTests(self, namespace):
    656
    657          if namespace.strip() == '.*':
    658              namespace = '__main__.*'
    659          elif namespace.strip() == '.':
    660              namespace = '__main__'
    661
    662          if namespace.endswith('.*'):
    663              return self.loadAllTests(namespace[:-2])
    664          else:
    665              try:
    666                  mod = __import__(namespace, fromlist=[''])
    667              except ImportError:
    668                  raise InvalidNamespaceError('Namespace {0} is not importable'.format(namespace))
    669              suites = self._inspect(mod)
    670              return unittest.TestSuite(suites)
    671
    672 -    def _inspect(self, module):
    673
    674          objects = map(lambda n: getattr(module, n), dir(module))
    675          return self._filter(objects)
    676
    677 -    def _filter(self, factories):
    678          """
    679          Filter a list of objects using C{self.labels}.
    680          """
    681          filtered = []
    682
    683          for obj in factories:
    684              for label in self.labels:
    685                  if hasattr(obj, label) and getattr(obj, label) is True:
    686                      suite = obj()
    687                      if not isinstance(suite, unittest.TestSuite):
    688                          raise ValueError('Custom test function {0} must return a '
    689                                           'unittest.TestSuite, not {1}'.format(obj.__name__, type(suite)))
    690                      filtered.append(suite)
    691
    692          return filtered
    693
    694 -def unit(klass):
    695      """
    696      A class decorator, used to label unit test cases.
    697
    698      @param klass: a C{unittest.TestCase} class type
    699      @type klass: type
    700      """
    701      if not isinstance(klass, type):
    702          raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
    703
    704      setattr(klass, Attributes.UNIT, True)
    705      return klass
    706
    707 -def functional(klass):
    708      """
    709      A class decorator, used to label functional test cases.
    710
    711      @param klass: a C{unittest.TestCase} class type
    712      @type klass: type
    713      """
    714      if not isinstance(klass, type):
    715          raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
    716
    717      setattr(klass, Attributes.FUNCTIONAL, True)
    718      return klass
    719
    720 -def regression(klass):
    721      """
    722      A class decorator, used to label regression test cases.
    723
    724      @param klass: a C{unittest.TestCase} class type
    725      @type klass: type
    726      """
    727      if not isinstance(klass, type):
    728          raise TypeError("Can't apply class decorator on {0}".format(type(klass)))
    729
    730      setattr(klass, Attributes.REGRESSION, True)
    731      return klass
    732
    733 -def custom(function):
    734      """
    735      A function decorator, used to mark functions which build custom (dynamic)
    736      test suites when called.
    737
    738      @param function: a callable object, which returns a dynamically compiled
    739                       C{unittest.TestSuite}
    740      @type function: callable
    741      """
    742      if isinstance(function, type):
    743          raise TypeError("Can't apply function decorator on a class")
    744      elif not hasattr(function, '__call__'):
    745          raise TypeError("Can't apply function decorator on non-callable {0}".format(type(function)))
    746
    747      setattr(function, Attributes.CUSTOM, True)
    748      return function
    749
    750 -def skip(reason, condition=None):
    751      """
    752      Mark a test case or method for skipping.
    753
    754      @param reason: message
    755      @type reason: str
    756      @param condition: skip only if the specified condition is True
    757      @type condition: bool/expression
    758      """
    759      if isinstance(reason, types.FunctionType):
    760          raise TypeError('skip: no reason specified')
    761
    762      if condition is None:
    763          return unittest.skip(reason)
    764      else:
    765          return unittest.skipIf(condition, reason)
    766
    767 -class Console(object):
    768      """
    769      Build and run all tests of the specified namespace and kind.
    770
    771      @param namespace: a dotted name, which specifies the test module
    772                        (see L{csb.test.AbstractTestBuilder.loadTests})
    773      @type namespace: str
    774      @param builder: test builder to use
    775      @type builder: any L{csb.test.AbstractTestBuilder} subclass
    776      @param verbosity: verbosity level for C{unittest.TestRunner}
    777      @type verbosity: int
    778      @param update: if True, refresh all pickles in csb/test/data
    779      @type update: bool
    780      @param generated_data: where to cache generated test files (directory)
    781      @type generated_data: str
    782      """
    783
    784      BUILDERS = {'unit': UnitTestBuilder, 'functional': FunctionalTestBuilder,
    785                  'custom': CustomTestBuilder, 'any': AnyTestBuilder,
    786                  'regression': RegressionTestBuilder}
    787
    788
    789 -    def __init__(self, namespace=('__main__',), builder=AnyTestBuilder, verbosity=1,
    790                   update=False, generated_data=Config.GENERATED_DATA, argv=None):
    791
    792          if not argv:
    793              argv = sys.argv
    794
    795          self._namespace = None
    796          self._builder = None
    797          self._verbosity = 1
    798          self._update = False
    799          self._gendata = str(generated_data)
    800          self._program = os.path.basename(argv[0])
    801
    802          self.namespace = namespace
    803          self.builder = builder
    804          self.verbosity = verbosity
    805          self.update = update
    806          self.generated_data = generated_data
    807
    808          self.parseArguments(argv[1:])
    809          self.run()
    810
    811      @property
    812 -    def namespace(self):
    813          return self._namespace
    814      @namespace.setter
    815 -    def namespace(self, value):
    816          if csb.core.iterable(value):
    817              self._namespace = list(value)
    818          else:
    819              self._namespace = [value]
    820
    821      @property
    822 -    def builder(self):
    823          return self._builder
    824      @builder.setter
    825 -    def builder(self, value):
    826          self._builder = value
    827
    828      @property
    829 -    def verbosity(self):
    830          return self._verbosity
    831      @verbosity.setter
    832 -    def verbosity(self, value):
    833          self._verbosity = value
    834
    835      @property
    836 -    def builders(self):
    837          return ', '.join(Console.BUILDERS)
    838
    839      @property
    840 -    def program(self):
    841          return self._program
    842
    843      @property
    844 -    def update(self):
    845          return self._update
    846      @update.setter
    847 -    def update(self, value):
    848          self._update = bool(value)
    849
    850      @property
    851 -    def generated_data(self):
    852          return self._gendata
    853      @generated_data.setter
    854 -    def generated_data(self, value):
    855          self._gendata = os.path.abspath(value)
    856
    857 -    def run(self):
    858
    859          Config.setDefaultGeneratedDataRoot(self.generated_data)
    860
    861          if self.update:
    862              Config().updateDataFiles()
    863          else:
    864              Config().ensureDataConsistency()
    865
    866          builder = self.builder()
    867          suite = builder.loadMultipleTests(self.namespace)
    868
    869          runner = unittest.TextTestRunner(verbosity=self.verbosity)
    870          runner.run(suite)
    871
    872 -    def parseArguments(self, argv):
    873
    874          parser = argparse.ArgumentParser(prog=self.program, description="CSB Test Runner Console.")
    875
    876          parser.add_argument("-t", "--type", type=str, default="any", choices=list(Console.BUILDERS),
    877                              help="Type of tests to load from each namespace (default=any)")
    878          parser.add_argument("-v", "--verbosity", type=int, default=1,
    879                              help="Verbosity level passed to unittest.TextTestRunner (default=1).")
    880          parser.add_argument("-u", "--update-files", default=False, action="store_true",
    881                              help="Force update of the test pickles in " + Config.GENERATED_DATA)
    882          parser.add_argument("-g", "--generated-resources", type=str, default=Config.GENERATED_DATA,
    883                              help="Generate, store and load additional test resources in this directory"
    884                              " (default=" + Config.GENERATED_DATA + ")")
    885
    886          parser.add_argument("namespaces", nargs='*',
    887                              help="""An optional list of CSB test dotted namespaces, from which to
    888                                      load tests. '__main__' and '.' are interpreted as the
    889                                      current module. If a namespace ends with an asterisk
    890                                      '.*', all sub-packages will be scanned as well.
    891
    892                                      Examples:
    893                                          "csb.test.cases.bio.*"
    894                                          "csb.test.cases.bio.io" "csb.test.cases.bio.utils"
    895                                          ".")""")
    896
    897          args = parser.parse_args(argv)
    898
    899          self.builder = Console.BUILDERS[args.type]
    900          self.verbosity = args.verbosity
    901          self.update = args.update_files
    902          self.generated_data = args.generated_resources
    903
    904          if args.namespaces:
    905              self.namespace = args.namespaces
    906
    907
    908  if __name__ == '__main__':
    909
    910      Console()
    911