Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#34010 closed Bug (fixed)

Parallel tests don't work with spawn.

Reported by: Kevin Renskers Owned by: Adam Johnson
Component: Testing framework Version: 4.1
Severity: Release blocker Keywords:
Cc: Carlton Gibson, David Smith Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I've upgraded to Django 4.1.1 and parallel tests don't work, but now with a different error as what I got in 4.1 (see #33891).

$ ./manage.py test --parallel
Found 153 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/queues.py", line 367, in get
    return _ForkingPickler.loads(res)
  File "/Users/kevin/Workspace/cn-django/criticalnotes/auth/test_models.py", line 5, in <module>
    from criticalnotes.auth.models import User
  File "/Users/kevin/Workspace/cn-django/criticalnotes/auth/models.py", line 10, in <module>
    class User(models.Model):
  File "/Users/kevin/Workspace/cn-django/env/lib/python3.10/site-packages/django/db/models/base.py", line 127, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/Users/kevin/Workspace/cn-django/env/lib/python3.10/site-packages/django/apps/registry.py", line 260, in get_containing_app_config
    self.check_apps_ready()
  File "/Users/kevin/Workspace/cn-django/env/lib/python3.10/site-packages/django/apps/registry.py", line 138, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

(That error then repeat 26 times in my case).

If I run the tests just with ./manage.py test, everything works fine.

Attachments (1)

parallel_tests.zip (6.6 KB ) - added by Carlton Gibson 2 years ago.
Minimal reproduce. Install Django and manage.py test --parallel on macOS (and presumably Windows)

Download all attachments as: .zip

Change History (16)

comment:1 by Mariusz Felisiak, 2 years ago

Cc: Carlton Gibson added

I'm confused you ā€‹confirmed that d4db417c8e6935222d198925dd4c5ac1ca737909 works for you. Can you provide a sample project?

in reply to:  1 comment:2 by Kevin Renskers, 2 years ago

Replying to Mariusz Felisiak:

I'm confused you ā€‹confirmed that d4db417c8e6935222d198925dd4c5ac1ca737909 works for you. Can you provide a sample project?

I'm not sure why that seemed to work. Maybe I made a mistake updating Django to the commit with the fix, or maybe I ran the tests without the parallel flag šŸ¤”

I can email you a closed source project, if it won't be shared with other people. Just let me know where to send the project.

comment:3 by Mariusz Felisiak, 2 years ago

Resolution: ā†’ needsinfo
Status: new ā†’ closed
Summary: Parallel tests don't work ā†’ Parallel tests don't work with spawn.

We also confirmed that #33891 is fixed so it must be something different. You can send a sample project to me or Carlton to emails from GitHub profiles. Thanks.

in reply to:  3 comment:4 by Kevin Renskers, 2 years ago

Replying to Mariusz Felisiak:

We also confirmed that #33891 is fixed so it must be something different. You can send a sample project to me or Carlton to emails from GitHub profiles. Thanks.

Yep I sent a project to Carlton on the 15th. Not sure why this ticket got closed though, since I have a reproducible problem running parallel tests?

comment:5 by Mariusz Felisiak, 2 years ago

Yep I sent a project to Carlton on the 15th. Not sure why this ticket got closed though, since I have a reproducible problem running parallel tests?

Don't worry, we will reopen the ticket as soon as Carlton confirms it's an issue in Django. Django Con Europe starts today so it might take a little longer but it's on the radar.

comment:6 by Mariusz Felisiak, 2 years ago

Resolution: needsinfo
Status: closed ā†’ new

comment:7 by Carlton Gibson, 2 years ago

Yes, confirming that. Iā€™m currently reducing the example project to work out exactly what the issue is. Thanks.

comment:8 by Carlton Gibson, 2 years ago

Cc: David Smith added
Severity: Normal ā†’ Release blocker
Triage Stage: Unreviewed ā†’ Accepted

OK, thanks for the report Kevin. We're not setting the ParallelTestSuite.process_setup and process_setup_args correctly, instead being the _process_setup_stub no-op. We'll need an implementation for that.

(If your ā€” i.e. my ā€” test project doesn't come out of partition_suite_by_case() the right way you don't trigger this. I shall upload a minimal project later.)

This was an issue in the original patch 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc, rather than the other referenced issues.

by Carlton Gibson, 2 years ago

Attachment: parallel_tests.zip added

Minimal reproduce. Install Django and manage.py test --parallel on macOS (and presumably Windows)

comment:9 by Carlton Gibson, 2 years ago

Has patch: set
Owner: changed from nobody to Carlton Gibson
Status: new ā†’ assigned

comment:10 by Carlton Gibson, 2 years ago

Owner: changed from Carlton Gibson to Adam Johnson

comment:11 by Mariusz Felisiak, 2 years ago

Triage Stage: Accepted ā†’ Ready for checkin

comment:12 by Mariusz Felisiak <felisiak.mariusz@ā€¦>, 2 years ago

Resolution: ā†’ fixed
Status: assigned ā†’ closed

In 4a910f3d:

Fixed #34010 -- Made parallel tests using spawn set up Django.

Bug in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

Thanks Kevin Renskers for the report.

comment:13 by Mariusz Felisiak <felisiak.mariusz@ā€¦>, 2 years ago

In 0f5b11ec:

Refs #34010 -- Made --debug-mode work for parallel tests using spawn.

Bug in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

Thanks Kevin Renskers for the report.

comment:14 by Mariusz Felisiak <felisiak.mariusz@ā€¦>, 2 years ago

In 5630a6ca:

[4.1.x] Fixed #34010 -- Made parallel tests using spawn set up Django.

Bug in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

Thanks Kevin Renskers for the report.

Backport of 4a910f3de35338df7d4fcd8b7729fd1e31edaad0 from main

comment:15 by Mariusz Felisiak <felisiak.mariusz@ā€¦>, 2 years ago

In b826b388:

[4.1.x] Refs #34010 -- Made --debug-mode work for parallel tests using spawn.

Bug in 3b3f38b3b09b0f2373e51406ecb8c9c45d36aebc.

Thanks Kevin Renskers for the report.

Backport of 0f5b11eca0ba199501941fa244b276aaa10353c8 from main

Note: See TracTickets for help on using tickets.
Back to Top