The 'runserver' command doesn't work when run from an installed script on Windows

[Joël Larose]

My manage.py is as follows:

#!/usr/bin/env python
import os
import sys


def django_manage():
    """Function implementation of python manage.py"""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<project_package>.settings.dev")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    django_manage()

In my pyproject.toml I have:

[project.scripts]
    "djm" = "<project_package>.manage:django_manage"

In Windows this generates a djm.exe file.

This allows me to save a few keystrokes when issuing commands from CLI. And it works for most of the django commands. The only exception I've encountered so far is with the runserver command.
It gives:

<project_path>\venv\Scripts\python.exe: Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)

After much debugging and tracing, I found where the issue lies. The problem is in the get_child_arguments function in django/utils/autoreload.py. When you flip the first two if-elif blocks, everything works. That is, the check for not py_script.exists() needs to come before the check for getattr(__main__, "__spec__", None) is not None.

I'm not sure if this creates different problems, but it certainly fixes the one I was having.

[Mariusz Felisiak]

Replying to Joël Larose:

It gives:

<project_path>\venv\Scripts\python.exe: Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)

After much debugging and tracing, I found where the issue lies. The problem is in the get_child_arguments function in django/utils/autoreload.py. When you flip the first two if-elif blocks, everything works. That is, the check for not py_script.exists() needs to come before the check for getattr(__main__, "__spec__", None) is not None.

Can you provide the full stacktrace? I'm not sure how swapping these branches can make a difference as the first one is protected against None __spec__.

[Joël Larose]

There's no stack trace produced. I even tried wrapping my code in try-catch block, and it doesn't trap anything. I had to import pdb and trace it to discover where the problem lies.

Execution with the original code

Using the exe:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> djm runserver
C:\Users\jplarose\Projects\green-rosewood\art-django\venv\Scripts\python.exe: Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)
(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

Using python ...py:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> python .\greenrosewood_art\manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 21, 2023 - 00:11:37
Django version 4.2.4, using settings 'greenrosewood_art.site.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

Using python -m:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> python -m greenrosewood_art.manage runserver  
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 21, 2023 - 00:14:36
Django version 4.2.4, using settings 'greenrosewood_art.site.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

Execution with fixed code (i.e. if blocks flipped):

Using the exe:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> djm runserver                                 
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 21, 2023 - 00:17:46
Django version 4.2.4, using settings 'greenrosewood_art.site.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

Using python ...py:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> python .\greenrosewood_art\manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 21, 2023 - 00:19:58
Django version 4.2.4, using settings 'greenrosewood_art.site.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

Using python -m:

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> python -m greenrosewood_art.manage runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 21, 2023 - 00:19:09
Django version 4.2.4, using settings 'greenrosewood_art.site.settings.dev'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

(venv) PS C:\Users\jplarose\Projects\green-rosewood\art-django> 

[Sarah Boyce]

Hi Joël, thank you for this report!
Just looking at what changes might be related and found these: #31716 #32314. As not many of us have a Windows machine, can I be cheeky and ask for you to check if this was working on a previous version of Django (to rule out if this is a regression)? I would check 3.0 and 3.1. If it's never been working then maybe a small sample project would also help to make sure I get the file structure correct 💜

I'm still trying to wrap my head around it but it confuses me that we're inside the if of getattr(__main__, "__spec__", None) is not None and yet the error you seem to be seeing is ValueError: __main__.__spec__ is None 🤔

[Joël Larose]

I created a minimal Django project with a pyproject.toml file to create the custom script exe. I set the various versions of Django as optional dependencies to make it easy to switch between the versions without having to edit the file each time.

Here's a summary of the results:

• Django 3.0:
  • C:\Python\v311\python.exe: can't open file 'C:\\Users\\jplarose\\Projects\\django-windows-mvp\\venv\\Scripts\\djm': [Errno 2] No such file or directory
• Django 3.1 and 3.2:
  • Works properly
• Django 4.0, 4.1, and 4.2:
  • C:\Users\jplarose\Projects\django-windows-mvp\venv\Scripts\python.exe: Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)

[Joël Larose]

One observation is that the system exits here:

def run_with_reloader(main_func, *args, **kwargs):
    signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
    try:
        if os.environ.get(DJANGO_AUTORELOAD_ENV) == "true":
            reloader = get_reloader()
            logger.info(
                "Watching for file changes with %s", reloader.__class__.__name__
            )
            start_django(reloader, main_func, *args, **kwargs)
        else:
            exit_code = restart_with_reloader()
            sys.exit(exit_code)        #  <---  Exits here
    except KeyboardInterrupt:
        pass

I added a pdb.set_trace() call in main(), here's a trace that might help:

(venv) PS C:\Users\jplarose\Projects\django-windows-mvp> djm runserver
> c:\users\jplarose\projects\django-windows-mvp\django_windows_mvp\manage.py(10)main()
-> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_windows_mvp.settings')
(Pdb) b django/utils/autoreload.py:674 
Breakpoint 1 at c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\utils\autoreload.py:674       
(Pdb) c
C:\Users\jplarose\Projects\django-windows-mvp\venv\Scripts\python.exe: Error while finding module specification for '__main__' (ValueError: __main__.__spec__ is None)
> c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\utils\autoreload.py(674)run_with_reloader() 
-> sys.exit(exit_code)
(Pdb) where
  <frozen runpy>(198)_run_module_as_main()
  <frozen runpy>(88)_run_code()
  c:\users\jplarose\projects\django-windows-mvp\venv\scripts\djm.exe\__main__.py(7)<module>()
-> sys.exit(main())
  c:\users\jplarose\projects\django-windows-mvp\django_windows_mvp\manage.py(19)main()
-> execute_from_command_line(sys.argv)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\__init__.py(442)execute_from_command_line()
-> utility.execute()
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\__init__.py(436)execute()   
-> self.fetch_command(subcommand).run_from_argv(self.argv)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\base.py(412)run_from_argv() 
-> self.execute(*args, **cmd_options)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\commands\runserver.py(74)execute()
-> super().execute(*args, **options)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\base.py(458)execute()       
-> output = self.handle(*args, **options)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\commands\runserver.py(111)handle()
-> self.run(**options)
  c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\core\management\commands\runserver.py(118)run()
-> autoreload.run_with_reloader(self.inner_run, **options)
> c:\users\jplarose\projects\django-windows-mvp\venv\lib\site-packages\django\utils\autoreload.py(674)run_with_reloader() 
-> sys.exit(exit_code)
(Pdb)

[Sarah Boyce]

Thank you for the extra input Joël! I really appreciate it
I suspect this might be a regression introduced in #32669. Accepted the ticket 👍

I have a draft PR: ​https://github.com/django/django/pull/17203
Hoping for some testing to confirm whether this works (will mark as "Patch need improvement" until we're happy it's working as expected).

[Joël Larose]

Thanks Sarah! Your code fixes the issue.

[Sarah Boyce]

Thank you for testing!

[Mariusz Felisiak <felisiak.mariusz@…>]

In f6ed2c3:

Fixed #34787 -- Fixed autoreloader crash when run from installed script on Windows.