Image upload in Windows changes filename to lowercase

[Rob van der Linde]

When using an ImageField and uploading an image via admin, say the filename was Test.jpg, uploading it in Linux will save the file as Test.jpg on the disk, and the reference in the database is also Test.jpg, things work normally as expected here.

However, when uploading the same file in Windows, the filename is saved on the disk as test.jpg (all lower case), but in the database is still referenced as Test.jpg. Since the filesystem in Windows is not case sensitive, this is not really much of an issue immediately. If you however, then copy your Django based site back to a Linux based machine, you will get a missing image, as it is expected to find the file "Test.jpg" in the filesystem, however the file is "test.jpg". In this case, this can become a bit of a problem.

I am not sure if the problem is related to Django or Python on Windows.

[anonymous]

As a note, Windows is case sensitive on file creation, but not after that. Django is (correctly) doing to-lower to test for file existence, but I agree with the issue that either:

• the name in the database should be lowercased to match
• the file name when actually written should honor the original casing

[Chris Beaven]

Writing with the original casing will be more difficult (this is done by django.utils._os.safe_join) so just normalizing the case seems the easiest course of action.

Easiest solution seems to be using os.path.normcase in the get_directory_name and get_filename methods of FileField.

[Soviut]

I can confirm that this issue exists in 1.0 since its the release I'm suffering from it.

[Karen Tracey]

#9442 was a dup, it also has a suggested fix.

[dawidjoubert]

Replying to SmileyChris:
Where does this put us with a fix?

[Karen Tracey]

If you want to move the process along, the next step would be posting a patch that fixes the issue, passes all existing tests, and contains a test that fails on the current code (it's OK if it fails only on Windows -- but it can't rely on a particular configuration like a networked drive) and succeeds with the fix.

[Ramiro Morales]

File with new tests added to the django test suite that fail

[Ramiro Morales]

I think that, contrarily to what SmileyChris suggests, it is worth going for the real and correct solution to this problem as the reporter of #9442 suggests rather than adding a workaround.

By correct solution I mean making sure Django uses the same path (case-wise) both for:

• the file written to disk
• and the path stored in the database

when storing a FileField/ImageField in all supported platforms.

I've attached a patch (8593-regression-tests.diff) that add tests to the test suite that show how currently, when running on Windows (whose file systems are case-insensitive but case-preserving), Django is lowercasing the path used to create the file in the file system.

These new tests shouldn't fail on other platfoms (i've tested also on Linux).

[Ramiro Morales]

Patch that solves the issue, also includes the tests already uploaded in 8593-regression-tests.diff

[Ramiro Morales]

8593-r9832.diff implements a fix for this issue by modifying Django's django.utils._os.safe_join to not call os.path.normcase() under Windows, the modification has been implemented in a way that doesn't break the security safeguards that are the raison d'être of such function.

As safe_join() is also used by the template loaders, the case-insensitive-filesystems-specific template regression tests have been also modified (AFAICS these tests as implemented aren't really useful because they aren't using a template file path as written to the file system when comparing paths but rather two Django maintained in-memory representations of such path.)

Net effect of the path is that now Django doesn't use a path in the file system that differs from the path stored in the DB for uploaded files, irrespective of platform. This should help when moving projects and deployments form platform to platform.

[Ramiro Morales]

Same patch but with two comments edited to be more accurate

[Ramiro Morales]

Updated patch so it applies after file_storage tests switch from doctests to unit tests in r10707

[Armin Ronacher]

The case is more complex. On OS X the IO system performs unicode normalization for the filename after the file was stored. This is fine until you mount the volume on a linux machine or access it over the network. I'm not sure what the fix is, OS X normalizes in the IO system somehow, it also affects IO access on NFS mounts and similar volumes. You can't really find out how the normalization works from inside Python.

[Adam Nelson]

Reflecting @mitsuhiko's comment by marking that the patch needs improvement.

[Chris Beaven]

For what it's worth, I agree with ramiro that this should be fixed to preserve the case.

I've got a branch with the a new patch here: ​https://github.com/SmileyChris/django/compare/master...8593-safejoin-lowercase-on-windows

I'm ignoring mitsuhiko's comment, as it has nothing to do with image uploads on windows.

[srj55]

Successfully tested the patch from @SmileyChris on Win7

[srj55]

Successfully tested the patch from @SmileyChris on Win7

[Chris Beaven]

In [16267]:

Fixes #8593 -- better handling of safe_join case sensitivity on windows. Thanks for the initial patch, ramiro.

[Jacob]

Milestone 1.4 deleted