manage.py: difficult to run in background (and crashes when input isn't a terminal)

[dstndstn@…]

This is with Django 1.3, and seems to be different than previous releases.

When I try to run manage.py in the background (python manage.py > django.log 2>&1 &) the process stalls.

When I try to pipe /dev/null to stdin it crashes as shown below.

thanks!

# python manage.py --version
1.3

# python manage.py runserver < /dev/null
Validating models...

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_manager(settings)
  File "/usr/local/django-1.3/lib/python/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/local/django-1.3/lib/python/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/django-1.3/lib/python/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/django-1.3/lib/python/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/local/django-1.3/lib/python/django/core/management/commands/runserver.py", line 67, in handle
    self.run(*args, **options)
  File "/usr/local/django-1.3/lib/python/django/core/management/commands/runserver.py", line 76, in run
0 errors found
    autoreload.main(self.inner_run, args, options)
  File "/usr/local/django-1.3/lib/python/django/utils/autoreload.py", line 131, in main
Django version 1.3, using settings 'astrometry_gsoc_challenge.settings'
Development server is running at http://**************/
Quit the server with CONTROL-C.
    reloader(main_func, args, kwargs)
  File "/usr/local/django-1.3/lib/python/django/utils/autoreload.py", line 104, in python_reloader
    reloader_thread()
  File "/usr/local/django-1.3/lib/python/django/utils/autoreload.py", line 83, in reloader_thread
    ensure_echo_on()
  File "/usr/local/django-1.3/lib/python/django/utils/autoreload.py", line 77, in ensure_echo_on
    attr_list = termios.tcgetattr(fd)
termios.error: (25, 'Inappropriate ioctl for device')

[Jacob]

I'm curious - why are you trying to run the dev server in the background? And why pipe /dev/null to stdin?

We probably need to fix this, but I'm hesitant because it makes running the devserver as a "real" server easier, and we kinda try to prevent that.

[dstndstn@…]

It used to work in previous releases :)

A standard use case for me is to ssh to a remote server, run the devserver in the background while editing the code in the foreground. This bug makes that difficult to do.

Piping /dev/null to stdin was just an attempted hack to make it continue running in the background.

If I run it normally and then try to put it in the background with "ctrl-Z bg" it stalls whenever the code is modified and has to be brought to the foreground before it resumes operation. That means leaving my editor after every edit, bringing the devserver to the foreground for a moment, putting it back in the background and then returning to the editor. That is not easier than just ctrl-Cing the devserver, which means that this bug cancels out the usefulness of the [extremely useful!] auto-reload functionality, at least when the devserver is used in this way.

thanks!
--dustin

[Luke Plant]

A standard use case for me is to ssh to a remote server, run the devserver in the background while editing the code in the foreground

Terminal multiplexers like tmux and screen will allow you to do this. Obviously this is only going to work if they are installed or can be installed. Another option is to simply have two remote sessions.

[dstndstn@…]

Yes, I am a big screen enthusiast (its 'multiuser' mode, combined with skype, is a great way to do remote pair-coding). But IMO it's just way more convenient to be able to run the devserver and an editor in the same bash session.

[Karen Tracey]

This was likely caused by r15883. The exception with piping to /dev/null has already been fixed on trunk and 1.3.X. The suspending on reload it seems can be fixed by ignoring SIGTTOU/SIGTTIN (or maybe just one of those, I don't have time to experiment right now) though I have no idea what side-effects that has. Or if anyone else has a better suggestion for fixing this or the original problem fixed by r15883 feel free to chime in. I'd prefer to prevent the original problem, but since I don't know what causes terminal echo to get turned off on reload, I don't know how to prevent it.

[Karen Tracey]

So apparently termios.tcsetattr sends a SIGTTOU signal when the process has been put in the background, which by default causes the process to be stopped. Attached patch ignores SIGTTOU for the duration of tcsetattr. It has the virtue of seeming to work, the only weirdness I notice is that the first command typed after starting manage.py runserver in the background is double-echoed. But I'd really rather fix whatever is causing terminal echo to be turned off, if anyone has a clue how to do that. This other approach of ensuring it is on is getting uglier and uglier.

[anonymous]

For the sake of other people who are willng to do brutal hacks, the following change to autoreload.ensure_echo_on() makes this problem go away
on supervisord:

    try:
        attr_list = termios.tcgetattr(fd)
    except termios.error:
        return

[John Anderson]

I use supervisord for my development environment so I can launch rabbitmq, devserver, celery, redis, etc all at once and now I can't because of this bug :)

[Karen Tracey]

Replying to sontek:

I use supervisord for my development environment so I can launch rabbitmq, devserver, celery, redis, etc all at once and now I can't because of this bug :)

The original description of this bug doesn't mention supervisord, and actually describes two problems, so I'm not entirely such which problem, exactly, afflicts the supervisord case. Based on the comment above yours, which indicates that catching/ignoring termios.error fixes the supervisord problem, I assumed the supervisord issue was the termios error in the traceback above. As I tried to note in comment 5, that problem already has a fix in place in the Django code on trunk (r15911), 1.2.X (r15912) and 1.3.X (r16029). So, it's broken in 1.3 release, and I'm sorry for that, but I can't do anything about it at this point. There are fixes available if you care to use one of those other options, or patch your own copy of Django.

If that already-existing fix does not fix the uspervisord problem, then it would help if someone would spell out exactly what the problem is under supervisord.

Is it the process stalling issue? If yes, does the patch attached to this ticket fix (or not fix) the issue under supervisord?

Is it something else entirely?

I don't use supervisord and it appears non-trivial to set up to try this case out, so it would help if the people who are seeing the problem could provide a little more information so that we can get a fix in place for the next 1.3.X release.

[Brillgen Developers]

I can confirm that the brutal hack described above works for me as well...not sure of its bigger implications

[Karen Tracey]

Replying to brillgen:

I can confirm that the brutal hack described above works for me as well...not sure of its bigger implications

But is that "brutal hack" described above even necessary with current trunk, 1.3.X, or 1.2.X code levels? The fix put in (r15911, r15912, r16029) should avoid the need for that hack since it avoids calling termios if stdin is not a tty. So far as I can see there's no need for the brutal hack since that problem was already fixed in current code levels...if that is not the case someone please spell out what exactly you are doing to still see a problem current code levels.

To my mind this ticket is open to deal with the stalling issue, which was not addressed by the already-in-place fix. Nobody has commented on that patch -- are you saying that you don't see the stalling in whatever setup you are using (which is...?)?

[Luke Plant]

Changing state to indicate that we cannot progress with this unless we get the feedback Karen asked for.

[dstndstn@…]

Original bug-reporter here.

In 1.3.X I am still getting stalls:

$ python -c "import django; print django.__file__"
/usr/local/django-1.3.X/django/__init__.py
$ svn info /usr/local/django-1.3.X/ | grep 'URL\|^Rev'
URL: http://code.djangoproject.com/svn/django/branches/releases/1.3.X
Revision: 16305
$ python manage.py runserver &
$ jobs
[1]+  Stopped                 python manage.py runserver

[anonymous]

Oh, but it now appears that piping /dev/null to stdin no longer causes a crash!

$ python manage.py runserver < /dev/null &
### starts up
$ touch settings.py
### successfully reloads

and this works:

$ python manage.py runserver < /dev/null > man.log &

Thank-you!

[Karen Tracey]

Re-opening to fix the stalling issue. Since no one has commented with any ideas for a better approach, I'll use the previously-posted patch.

[Karen Tracey]

In [16326]:

Fix #15880: Prevent "stalling" when running dev server in background by ignoring SIGTTOU for the duration of tcsetattr.

[Karen Tracey]

In [16327]:

[1.3.X] Fix #15880: Prevent "stalling" when running dev server in background by ignoring SIGTTOU for the duration of tcsetattr.

Backport of [16326] from trunk.