django 1.7 + inotify breaks autoreload of runserver with vim

[Thomas Capricelli]

Quick summary : since i've migrated to django-1.7, autoreload is broken on my systems. I've used django since 0.96 and i have never had any problem with this great feature.
I've turned around the problem until today by removing pyinotify : that fixed the problem for me. But of course i need pyinotify for other things...

Today I've spent time testing more thoroughly, here are the results. My system is fairly standard : linux, gentoo, python 2.7, and vim-7.4.488.

I patched django/utils/autoreload.py to add some debug to get more information.

*) when pyinotify is not installed, USE_INOTIFY is indeed False and everything is working as expected and as before.

*) when pyinotify is installed, USE_INOTIFY is True, and the file i'm testing with (main/views/gestion.py) is indeed added to the inotify watch list (around line 185 of autoreload.py). I need to load a page for this to happen, but this is expected because the autoreload stuff only consider those modules loaded by the python interpreter iiuc.

In this setup, if i use CLI and does "touch main/views/gestion.py", the autoreload is triggered (displayed in the output of runserver).

With the same setup, if i open the file with vim and just do ":w", which saves the file again, nothing is triggered. I tried doing an actual modification and save (in case vim or inotify of even django optimizes the stuff by not reloading a file whose content hasn't changed), but the behaviour is the same.

I can confirm the file was not reloaded by changing something in the file and testing from my browser.

I think to remember that vim doesn't just overwrite the file but instead writes the content to a new file and then relinks the files, which might be the reason inotify doesn't notice the change. But i can't guarantee this.

I know of so many people using django+vim, that i'm very surprised nobody was hit by this before, so maybe I'm not understanding something obvious and the reason is somewhere else ..

I'm "orzel" on irc, very often available, don't hesitate to ping me (or here) if you need some more information/test.

[Thomas Capricelli]

For completness, inotify is compiled in my kernel, and works perfectly with other software using it. The "touch" test described above kinda confirms this.

[Claude Paroz]

It may be that we should also monitor the pyinotify.IN_CLOSE_WRITE event. Could you try to add it in the inotify_code_changed function and see if it makes a difference?

[Thomas Capricelli]

It doesn't work. I've read ​http://stackoverflow.com/questions/13312794/inotify-dont-treat-vim-editting-as-a-modification-event as well, and i agree it should. But nope :-(

I even tried putting all the flags on :

mask = (

pyinotify.IN_MODIFY |
pyinotify.IN_ATTRIB |
pyinotify.IN_CLOSE_WRITE |
pyinotify.IN_CLOSE_NOWRITE |
pyinotify.IN_MOVED_FROM |
pyinotify.IN_MOVED_TO |
pyinotify.IN_CREATE |
pyinotify.IN_DELETE |
pyinotify.IN_DELETE_SELF |
pyinotify.IN_MOVE_SELF |
pyinotify.IN_ACCESS |
pyinotify.IN_OPEN

)

that's actually 4095 !

[Thomas Capricelli]

I did some more test and i'm getting confused. Using only mask = ( pyinotify.IN_CREATE), i have it triggered if i actually modify the file and save. Wont trigger if i don't change the file before writting.

[Thomas Capricelli]

I'm using the "backup" option in ~/.vimrc, which might explain both the problem and why the problem is not as widespread as i thought it would. It seems that in this case, vim just moves the file to the backup location.
I tested without this option though, and the behaviour is the same. I'm lost :-(

[Claude Paroz]

#24875 was a dup (where adding IN_DELETE_SELF and IN_MOVE_SELF events was suggested).

[Chris Bainbridge]

The patch I provided in bug #24875 fixed this issue for me. Required flags:

pyinotify.IN_CREATE | pyinotify.IN_DELETE_SELF | pyinotify.IN_MOVE_SELF

@orzel Try the following test code, run as python test dir_to_watch then start vim and edit/save a file in the test directory. The notifier loop will print whatever inotify events occur:

import sys
import pyinotify

wm = pyinotify.WatchManager()
wm.add_watch(sys.argv[1], pyinotify.ALL_EVENTS)
notifier = pyinotify.Notifier(wm)
notifier.loop()

Django does not watch the whole directory, but instead generates a list of paths to watch based on the current python imports. This means there are some cases where files are not initially watched - basically any python file that is dynamically loaded at runtime, such as a view specified by a string literal in urls.py. These files will be watched after the corresponding web page has been loaded for the first time.

You can see the current list of paths being watched by adding a print statement to django/utils/autoreload.py:inotify_code_changed()

[Thomas Capricelli]

I had tried adding IN_DELETE_SELF and IN_MOVE_SELF since the beginning, but this doesn't work here. I actually still have those locally added.

[Tim Graham]

I'll commit the patch and leave the ticket open since it appears there are still some unresolved cases.

[Tim Graham <timograham@…>]

In e5cfa394:

Refs #23882 -- Added detection for moved files when using inotify polling

Commit 15f82c7 ("used pyinotify as change detection system when
available") introduced a regression where editing a file in vim with
default settings (writebackup=auto) no longer causes the dev server
to be restarted. On a write, vim moves the monitored file to a backup
path and then creates a new file in the original. The new file is not
monitored as it has a different inode. Fixed this by also watching for
inotify events IN_DELETE_SELF and IN_MOVE_SELF.

[Tim Graham <timograham@…>]

In b252e0f:

[1.8.x] Refs #23882 -- Added detection for moved files when using inotify polling

Commit 15f82c7 ("used pyinotify as change detection system when
available") introduced a regression where editing a file in vim with
default settings (writebackup=auto) no longer causes the dev server
to be restarted. On a write, vim moves the monitored file to a backup
path and then creates a new file in the original. The new file is not
monitored as it has a different inode. Fixed this by also watching for
inotify events IN_DELETE_SELF and IN_MOVE_SELF.

Backport of e5cfa394d79b6ab1b412bd3b30c6a433b415d56b from master

[Tim Graham]

I guess we can close these. If anyone still has problems, please open a new ticket with details.