Switch to a Python 3-compatible imaging library

[Aymeric Augustin]

Originally reported by Marijonas Petrauskas on the django-developers mailing-list

---

Why isn't Pillow the recommended Django image library yet? PIL has
been unmaintained for almost 3 years and has a number of annoying bugs
(e.g. fails to open some valid JPEG files, maybe has some security
issues as well). Pillow, on the other hand, is a backwards-compatible
community-maintained fork, which has most of those issues fixed and
will even support Python 3 soon.

I think this change would involve (1) running the test suite with
Pillow, (2) updating the documentation and (3) updating the ImageField
warning shown when PIL is not installed.

---

Marking as a release blocker because of our goal to fully support Python 3 in 1.6.

I don't know if Pillow is the best choice, we should review the alternatives in all cases.

[Claude Paroz]

I must say that I monitor Pillow for some time, waiting eagerly for its Python 3 support. At my current knowledge, it's the best target for a PIL replacement. Takes this as a +1 from me.

[static]

I have had many problems using PIL and solved it by swiching to Pillow. Take this as a +1 too.

[Florian Apolloner]

+1 for Pillow, even if it doesn't have Python 3 support (yet).

[aclark]

Pillow's Python 3 support will land in version 2.0. At which point, anyone needing < Python 2.6 should use Pillow < 2.0. I'm not sure if old Python versions are a concern for Django, but I thought I'd mention it.

[Carl Meyer]

Django 1.5 already has Python 2.6 as minimum version, so Python < 2.6 is not an issue.

[Mike Scott]

I'm happy to spend some time getting things up to speed here and submit a patch, however is it felt better to wait for 2.0? As that would be a better integration point? (Considering Docs etc)

[Claude Paroz]

Yes, I think we should wait for version 2 (Python 3 support) before actually recommending Pillow. But nothing prevents working on a patch now, if you don't mind maintaining the patch (i.e. periodic rebasing) until the time of merging.

[Claude Paroz]

I have successfully ran PIL tests with Pillow 2.0.0 on Python 3.2, except for the _imaging import in from PIL import Image, _imaging.
_imaging cannot be imported from Pillow. This was introduced in [4016d5264a] (#7727) to workaround a PyPy issue. Is this PyPy issue still valid? If yes, is there an other test that we can make which would be both Pillow and PyPy compatible?

[Russell Keith-Magee]

@claudep IIRC, the logic here is that Image is importable under PyPy (because it's pure python), but _imaging isn't (because it's a C module). What is being done here is a way of identifying PyPy by functionality -- if you try to use ImageField under PyPy, an exception will be raised because the _imaging module doesn't exist..

I have no idea what the analog is under Pillow. If Pillow can be executed under PyPy, the issue doesn't exist at all (which gives us another reason to switch to Pillow). If Pillow doesn't work under PyPy, then we'll need a way to identify that at runtime.

[Claude Paroz]

From ​https://bitbucket.org/pypy/compatibility/wiki/Home:
The Python Imaging Library adds image processing capabilities to your Python interpreter. Since it is not maintained, use the pillow fork instead. Pillow is pypy compatible.

See also ​https://github.com/python-imaging/Pillow/issues/67

[aclark]

IIUC you should be able to revert 4016d5264ab049c33c845c5337ce2a3e50754268 since (AFAIK) as of Pillow 2.0.0 we fully support PyPy. If this is not the case, please add a comment to: ​https://github.com/django/django/commit/4016d5264ab049c33c845c5337ce2a3e50754268

[Claude Paroz]

I don't think we will require Pillow >2.0.0, IMHO we should still support the legacy PIL for Django on Python 2.

[Aymeric Augustin]

To sum up,

1) We must make the following decisions:

1.a) Will we deprecate support for PIL?

On one hand, this would make testing easier and simplify import code — unlike PIL, Pillow is importable from a predictable location.

On the other hand, there isn't much value in forcing this change upon everyone. Since Pillow and PIL can't coexist, it would be a problem for projects that depend on PIL internals (eg. import _imaging), and even more for projects that depend on libraries that depend on PIL internals.

Finally, I haven't found any way to determine if the installed library is PIL or Pillow. We'd need one to show deprecation warnings when PIL is installed.

1.b) Will we keep the error reporting shim introduced in #7727?

Even if we deprecate support for PIL, that'll take two releases, and we need a solution in the mean time to allow using support PIL (CPython only) and Pillow (CPython or PyPy).

Unless we find a way to tell PIL from Pillow, it seems better to remove this check and allow using Pillow on PyPy, even at the cost of less-than-helpful error messages when attempting to use PIL with PyPy. When you're using PyPy, you're supposed to know what you are doing...

(NB: PIL/Image.py appears to contain code to display a nice error when _imaging isn't available, both in PIL and in Pillow. Does #7727 predate this code?)

2) We must make the following changes:

2.a) Modify the documentation to point to Pillow instead of PIL.

2.b) Modify the ImportError messages.

2.c) If the answer to 1.b is "no", revert [4016d526].

2.d) Describe the changes in the release notes. Specifically, mention that it's safer to uninstall PIL before installing Pillow (recommended by Alex Clark on django-developers).

---

My vote is -0 for 1.a and -0 for 1.b. I'd like to hear more opinions before making a final decision.

[daniellindsley]

My vote is +0 for deprecating the PIL (in that Pillow becomes the "blessed" version), but I think detecting which is installed & using shims to gloss what little Django uses of either is a very doable task. By my count, the PIL/_imaging is only used in 4 files in Django (plus lots of test files) & none of the usages are particularly advanced.

I'm also +0 on the error reporting shim, in that I think we need to be able to inform the user that their install may be mis-configured. The existing comment (& the detection going on) will have to change. Again, I'm of the belief we can shim over that & have some ideas on detection.

The only thing I'd add to the plan you've laid out is """2.a.1) Create any necessary shims to aid Django in detecting which, if any, are installed & to aid in providing useful error messages about the installation."""

[Aymeric Augustin]

That works for me, I'm happy to review a patch along these lines.

[daniellindsley]

I think I have pretty much everything implemented, at least to start with. PR is at ​https://github.com/django/django/pull/1059. Feedback would be appreciated.

[daniellindsley]

@aaugustin - One thing I missed was 2.d. I'm not actually sure where that documentation ought to go. Thoughts?

[Aymeric Augustin]

For 2.d, I was thinking of a paragraph in the release notes, probably in the backwards-incompatible changes section.

[Daniel Lindsley <daniel@…>]

In 33793f7c3edd8ff144ff2e9434367267c20af26a:

Fixed #19934 - Use of Pillow is now preferred over PIL.

This starts the deprecation period for PIL (support to end in 1.8).

[chrismedrela]

I'm working on my GSoC project and I need to be sure what kind of exceptions may be raised while importing django.utils.image. Is it possible to end with an ImportError? There is one import statement which is not wrapped in try-except clause ​https://github.com/django/django/commit/33793f7c3edd8ff144ff2e9434367267c20af26a#L3R133. If it may fail, we should wrap it in try-except blocks and reraise as an ImproprelyConfigured exception. If not, then adding a comment why it can't happen would clarify the situation.

For me it looks like it can't happen. If line 95 or 99 succeed then respectively line 131 or 133 must succeed too; otherwise an ImproperlyConfigured exception is being raised (line 102). Nonetheless, I'm not sure and I would like to hear your opinion.

[daniellindsley]

@chrismedrela - I'm not aware of a way that import can fail without a completely broken Pillow/PIL install (missing Python modules, not just a missing C compiler). It was also directly moved from ​https://github.com/django/django/commit/33793f7c3edd8ff144ff2e9434367267c20af26a#L0L38, so it would've been failing that way before. AFAICT, that shouldn't ever fail, and if it does, we should be raising an ImportError so the user knows what's wrong.

[chrismedrela]

@daniellindsley - Thank you for your answer. I created a pull request clarifying the situation ​https://github.com/django/django/pull/1349.

[Christopher Medrela <chris.medrela@…>]

In b4c61c2665c0aa06dcdc4f823be03f23b9696408:

Added clarification comments to django.utils.image; refs #19934

[Tim Graham <timograham@…>]

In b82a2c41387648f81387b1e03371331db3630269:

Merge pull request #1349 from chrismedrela/ticket19934-comment

Added clarification comments to django.utils.image; refs #19934

[Simon Charette <charette.s@…>]

In b9590a69357917a3dbf2c7234774803bd43767ef:

Correctly format missing Pillow/PIL exceptions messages. refs #19934

[Simon Charette <charette.s@…>]

In e7a6eaf5fee9c5854e0d2d86f21132ac4a1415f6:

[1.6.x] Correctly format missing Pillow/PIL exceptions messages. refs #19934

Backport of b9590a6935 from master.

[Andrew Godwin <andrew@…>]

In e9b703f5a5e5de68a28abd2e6651e6100b0c6b49:

Correctly format missing Pillow/PIL exceptions messages. refs #19934

[Tim Graham <timograham@…>]

In 4965a774074780f3e4858bcc975476f71edf2c2c:

Removed PIL compatability layer per deprecation timeline.

refs #19934.