django-admin runserver mostly does not work on Windows

[Christian Ullrich]

I frequently encounter the problem that running "django-admin runserver" on Windows fails:

<some-venv>\scripts\python.exe: can't open file '<some-venv>\test\Scripts\django-admin': [Errno 2] No such file or directory

The command works if run with --noreload.

There are probably other conditions that must be met for it to fail, or work. I *think* it makes a difference, for example, if Django was installed with pip or setuptools. The .exe "console scripts " are different between the two; one type has a bit of readable Python at the end, the other does not.

What I do know is that the problem is either caused or enabled by the generated console scripts (the .exe wrappers in venv/Scripts) stripping the .exe extension from their own sys.argv[0] when passing that to Python to run. If I sabotage this by changing the regex in the file itself (where present), or by adding the extension again in utils/autoreload.py before running the command line, it works fine.

To be honest, I have no idea whether this is a bug in Django, or pip, or setuptools, or distutils, or something else entirely. I have spent the last two days trying to figure out what exactly goes wrong, and where, with little success. I'm reporting it here because Django is the only place I could find where it is possible to kludge the code into working. I'll be happy to take my business elsewhere if someone can point the way.

[Carlton Gibson]

I can't reproduce an error here. venv (and virtualenv and ...) and pip (and ...) all seem to work correctly. Unless you can provide a reproduce I don't think there's anything we can say.

<some-venv>\scripts\python.exe: can't open file '<some-venv>\test\Scripts\django-admin': [Errno 2] No such file or directory

I would expect both python and django-admin to be in <some-venv>\Scripts.
The .exe. generally isn't needed.
If your venv is active you can run django-admin directly, e.g. > django-admin help

All I can think is to recommend checking that you're on up to date versions and such, but day to day this works fine normally... Sorry I can't say more.
You may need to see TicketClosingReasons/UseSupportChannels.

[Christian Ullrich]

Here you go.

C:\Daten>py -m venv test

C:\Daten>test\Scripts\activate

(test) C:\Daten>python -c "import sys; print(sys.version)"
3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)]

(test) C:\Daten>python -m pip install -U pip setuptools Django
Looking in indexes: https://nexus/repository/PyPI-mirror/simple
Collecting pip
  Downloading https://nexus/repository/PyPI-mirror/packages/pip/20.1.1.post1/pip-20.1.1.post1-py2.py3-none-any.whl (1.6MB)
     |████████████████████████████████| 1.6MB 656kB/s
Collecting setuptools
  Downloading https://nexus/repository/PyPI-mirror/packages/setuptools/47.3.1/setuptools-47.3.1-py3-none-any.whl (582kB)
     |████████████████████████████████| 583kB 726kB/s
Collecting Django
  Downloading https://nexus/repository/PyPI-mirror/packages/django/3.0.7/Django-3.0.7-py3-none-any.whl (7.5MB)
     |████████████████████████████████| 7.5MB 726kB/s
Collecting asgiref~=3.2 (from Django)
  Downloading https://nexus/repository/PyPI-mirror/packages/asgiref/3.2.9/asgiref-3.2.9-py3-none-any.whl
Collecting pytz (from Django)
  Downloading https://nexus/repository/PyPI-mirror/packages/pytz/2020.1/pytz-2020.1-py2.py3-none-any.whl (510kB)
     |████████████████████████████████| 512kB 726kB/s
Collecting sqlparse>=0.2.2 (from Django)
  Downloading https://nexus/repository/PyPI-mirror/packages/sqlparse/0.3.1/sqlparse-0.3.1-py2.py3-none-any.whl (40kB)
     |████████████████████████████████| 40kB 871kB/s
Installing collected packages: pip, setuptools, asgiref, pytz, sqlparse, Django
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed Django-3.0.7 asgiref-3.2.9 pip-20.1.1.post1 pytz-2020.1 setuptools-47.3.1 sqlparse-0.3.1

(test) C:\Daten\dtest>pip list
Package    Version
---------- ------------
asgiref    3.2.9
Django     3.0.7
pip        20.1.1.post1
pytz       2020.1
setuptools 47.3.1
sqlparse   0.3.1

pip 20.1.1.post1 is 20.1.1 minus the information disclosure. For our purposes it's the same.

(test) C:\Daten>openssl.exe sha256 test\Scripts\django-admin.exe
SHA256(test\Scripts\django-admin.exe)= 24a24fefaaa27e069474985353cceacc9cb4128a35f50e99af487768d4255c90

No idea if they are identical between different installations.

(test) C:\Daten>django-admin startproject dtest

(test) C:\Daten>set PYTHONPATH=C:\Daten\dtest

(test) C:\Daten>set DJANGO_SETTINGS_MODULE=dtest.settings

(test) C:\Daten>django-admin

Type 'django-admin help <subcommand>' for help on a specific subcommand.

Available subcommands:

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    migrate
    runserver
    sendtestemail
    shell
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver
Note that only Django core commands are listed as settings are not properly configured (error: No module named 'dtest').

(test) C:\Daten>cd dtest

(test) C:\Daten\dtest>django-admin runserver
C:\Program Files\Python38\python.exe: can't open file 'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory

Note the missing extension.

(test) C:\Daten\dtest>django-admin runserver --noreload
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
June 17, 2020 - 08:58:46
Django version 3.0.7, using settings 'dtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
^C

Works with --noreload.

Now let's see what it is actually trying to execute.

(test) C:\Daten\dtest>vi ..\test\Lib\site-packages\django\utils\autoreload.py

Before line 230 (the subprocess.run call in restart_with_reloader()), insert "print(args)".

(test) C:\Daten\dtest>django-admin runserver
['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts\\django-admin', 'runserver']
C:\Program Files\Python38\python.exe: can't open file 'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory

This cannot possibly work.

(test) C:\Daten\dtest>dir ..\test\scripts
 Volume in drive C is Windows
 Volume Serial Number is 8EC9-7F66

 Directory of C:\Daten\test\scripts

2020-06-17  08:54    <DIR>          .
2020-06-17  08:54    <DIR>          ..
2020-06-17  08:53             2.273 activate
2020-06-17  08:53               950 activate.bat
2020-06-17  08:53            18.454 Activate.ps1
2020-06-17  08:53               368 deactivate.bat
2020-06-17  08:54           106.388 django-admin.exe
2020-06-17  08:54               142 django-admin.py
2020-06-17  08:54           106.355 easy_install-3.8.exe
2020-06-17  08:54           106.355 easy_install.exe
2020-06-17  08:54           106.346 pip.exe
2020-06-17  08:54           106.346 pip3.8.exe
2020-06-17  08:54           106.346 pip3.exe
2020-06-17  08:53           532.040 python.exe
2020-06-17  08:53           531.016 pythonw.exe
2020-06-17  08:54           106.341 sqlformat.exe
2020-06-17  08:54    <DIR>          __pycache__
              14 File(s)      1.829.720 bytes
               3 Dir(s)  [...] bytes free

(test) C:\Daten\dtest>vi -b ..\test\scripts\django-admin.exe

Edit the Python block at the end to not match and remove .exe anymore. I replaced "\.exe" with "\.rxe" in the regex.

(test) C:\Daten\dtest>django-admin runserver
['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts\\django-admin.exe', 'runserver']
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
June 17, 2020 - 09:07:47
Django version 3.0.7, using settings 'dtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
^C

With the extension left alone, this works now, too.

The .exe. generally isn't needed.

Not if it is the command name. Here it is an argument on python's command line, though, and python does not guess about missing extensions.

[Carlton Gibson]

OK, good yes, I see it. Thanks for the follow-up.

Looks like the auto-reloaded should indeed map back to an actually existing file.
Fancy making a PR for that?

(Wondering if Windows would let us drop the exe extensions totally as we did in POSIX-land...)

In the meantime, manage.py works as expected.

[Christian Ullrich]

The best solution that I can come up with is adding this in the else branch of get_child_arguments():

    if sys.platform == 'win32':
        for i in range(1, len(args)):
            if args[i].lower().endswith("scripts\\django-admin"):
                # setuptools style; setup.py develop or pip install -e
                if os.path.exists(args[i] + "-script.py"):
                    args[i] = args[i] + "-script.py"
                # pip style
                elif os.path.exists(args[i] + ".exe"):
                    args[i] = args[i] + ".exe"

I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support, but if you think I should make the attempt, please let me know.

The two branches exist because there are (at least) two types of these .exe "scripts". If generated by pip install, they have ~100 KiB, contain a small block of Python at the end, and can be run directly by the interpreter ("python.exe django-admin.exe" works in that Python finds the embedded script). However, if they are generated by setuptools, they do not have the Python block, are only ~70 KiB, and can not be run as scripts. Instead, there is the "...-script.py" companion script whose name they derive from their own and that they run via sys.executable .

Then there is the case of "python -m django runserver", about whose existence I just found out in get_child_arguments(), and whether that may need special treatment, too.

Not to mention that the processes end up stacked five deep. Yes, it is a dev situation so it doesn't matter much, but a better fix would still be to reinvent autoreloading entirely, again, and somehow get by without the monitoring process and $RUN_MAIN. (Getting rid of the bouncing around between the venv and system pythons would be nice, too, but that's another department.)

[Carlton Gibson]

I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support...

😀 We can't do that I'm afraid. (I'm just warming to it as it happens...)

I see Tom has cc-d himself. Let's see if he has any thoughts... ;)

[Tom Forbes]

Replying to Carlton Gibson:

I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support...

😀 We can't do that I'm afraid. (I'm just warming to it as it happens...)

I see Tom has cc-d himself. Let's see if he has any thoughts... ;)

My autoreloader-sense was tingling! I wonder if this has always been the case - the previous implementation was pretty much the same, but I see that Werkzeug has a partial workaround in their reloader: ​https://github.com/pallets/werkzeug/blob/fc999a6262c847ad185ea3ffe0cc6f2e915a867a/src/werkzeug/_reloader.py#L69-L93

I think we can take inspiration (read: shamelessly copy parts) their implementation as it seems to make sense, but I'd be interested to know why they don't handle -script.py companions.

Lets be a bit generic here rather than hard-code anything to do with django-admin.py. If sys.argv[0] doesn't exist we check if -script.py exists, falling back to looking for .exe on Windows only. If we cannot find it we explode with a reasonable error message (what that is I cannot say!).

I'll work on a patch, and i would appreciate it if someone on Windows could test it? :D

[Tom Forbes]

I think this might fix it: ​https://github.com/orf/django/commit/cb200453065ef0dec7ed97cb09eab02bc99342c1

Christian, any chance you could test with my branch? pip install https://github.com/orf/django/archive/31716-got-99-problems-but-exe-aint-one.zip

[Christian Ullrich]

Yep, works; both with pip and setuptools .exes.

[Tom Forbes]

Thanks Christian!

PR: ​https://github.com/django/django/pull/13077

[GitHub <noreply@…>]

In 8a902b7:

Fixed #31716 -- Fixed detection of console scripts in autoreloader on Windows.

[Mariusz Felisiak <felisiak.mariusz@…>]

In ac7f7ea:

[3.1.x] Fixed #31716 -- Fixed detection of console scripts in autoreloader on Windows.

Backport of 8a902b7ee622ada258d15fb122092c1f02b82698 from master