Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#26198 closed Bug (invalid)

partition_suite() removing non duplicate tests

Reported by: Luke Scott Owned by: nobody
Component: Testing framework Version: 1.8
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

In django 1.8 partition_suite() in django.test.runner is only including my behave tests from the first app.

e.g. ./manage.py test foo bar baz
will test all unit tests for apps foo, bar and baz, but will only test behave tests for app foo.

I am using django-behave==0.1.5 and behave==1.2.5 with django==1.8.9

This setup was working fine - tested in django==1.7.11 and was not a problem - upgrading to 1.8 is when the problem began.

All tests appear to be passed correctly to reorder_suite() but in partition_suite() only behave tests of the first app listed are being added to be tested.

In the 'django.tests.runner' module this appears to be due to reorder_suite() using an OrderedSet and subsequent '<django_behave.runner.DjangoBehaveTestCase testMethod=runTest>' objects not getting added to this set in partition_suite(). To confirm - the subsequent objects that should be added do have different object IDs and are in fact different tests.

def partition_suite(suite, classes, bins, reverse=False):
    suite_class = type(suite)
    if reverse:
        suite = reversed(tuple(suite))
    for test in suite:
        if isinstance(test, suite_class):
            partition_suite(test, classes, bins, reverse=reverse)
        else:
            for i in range(len(classes)):
                if isinstance(test, classes[i]):
                    bins[i].add(test)  # subsequent behave test objects not being added here
                    break
            else:
                bins[-1].add(test)

Looking at this commit:
​https://github.com/django/django/commit/2ca0870b672413cf5bc6e441bda58839f902f5b0
In partition_suite() I changed:

  bins = [OrderedSet() for i in range(class_count + 1)]

back to

  bins = [suite_class() for i in range(class_count + 1)]

and I changed

  bins[i].add(test)

to

  bins[i].addTest(test)

I retested using my test suite for my project and all the tests ran as expected.

Not sure what the further implications of changing this are with regards to preventing duplicated tests from being added to the suite.

Thanks

Change History (3)

comment:1 by Simon Charette, 9 years ago

Resolution: → invalid
Status: new → closed

Hi lukeaus,

Thanks for your report but it looks like this is bug in django-behave.

Python set (and Django's OrderedSet) use the __hash__ method of objects that are added to them to identify them uniquely.

In the case of unittest.TestCase instance this method returns the type of the testcase and the name of the the test method:

def __hash__(self):
    return hash((type(self), self._testMethodName))

It looks like django-behave adds multiple instance of DjangoBehaveTestCase to its suite but all of them have runTest as test method name resulting in the same hash() output and making them indistinguishable.

From a quick look the DjangoBehaveTestCase.__hash__() method should be adjusted to also include self.features_dir:

def __hash__(self):
    return hash((type(self), self._testMethodName, self.features_dir))

comment:2 by Luke Scott, 9 years ago

Thanks charettes - I appreciate your feedback.
I will follow this up on django-behave

Thanks

Luke

comment:3 by Luke Scott, 9 years ago

That worked - thanks charettes. I have raised a PR over on django-behave.

Note: See TracTickets for help on using tickets.
Back to Top