[patch] HttpResponse should treat headers case-insensitively

[Adrian Holovaty]

(Submitted by Bryan O'Sullivan, who had problems with Trac's spam filter.)

The HttpResponse implementation uses a normal Python dict to store
headers. This makes it necessary to have every component that sets
headers in a response agree on the exact case to use for every header
name. I found this problem while trying to glue a WSGI app into a
Django app, only to discover that one believes that a header should be
named "content-type", while the other prefers "Content-Type". This
results in a HTTP response that contains both headers, leading to
undefined (and currently bad) results on the client side.

Probably the HttpResponse class should use a dict-like object that
preserves, but ignores, case, so that "Foo" and "fOo" will map to the
same item.

[Chris Beaven]

Here's a (2.3 compatible) case insensitive dict I whipped up

[Chris Beaven]

Here's the patch

[Chris Beaven]

Patch doesn't contain case_insensitive_dict.py, so you'll need to put that in django.utils yourself.

[Bastian Kleineidam <calvin@…>]

The second part of your patch modifying has_header() is not backwards compatible, and could break existing applications if they used a plain dictionary for header values

If the patch is applied, this should be noted in the documentation.

[Chris Beaven]

more backwardsly-compatible

[Chris Beaven]

Good point, Bastian. This patch should keep you happier :)

[Jacob]

Looks like a great idea. Can we get some unit tests for both the new data structure and the HttpResponse behavior? Oh, and a revised patch that moved CaseInsensitiveDict into django.utils.datastructures would be nice.

[Chris Beaven]

Tests and included CaseInsensitiveDict

[Chris Beaven]

Here you go, Jacob.

I haven't included any tests for the HttpResponse behaviour specifically. If you check out the code, it's just doing a basic property(get, set) to keep the identical behaviour as before. That, and I didn't know exactly what tests you'd want to run - I haven't changed any external behaviour apart from the case insensitivity which is being 100% done in the CaseInsensitiveDict.

[Jacob]

Awesome; thanks. I'll check this in later tonight.

[Malcolm Tredinnick]

There's a lot of processing overhead going on in the CaseInsensitiveDict class just to preserve the case of the original key. I can't think of a reason when this is really needed (we will document that the headers are case-insensitive so that people won't be surprised when headers.keys() returns all lower-case). Am I missing some obvious use-case beyond "it might be nice"?

If there isn't a strong reason for keeping the original key case, I'd like to replace CaseInsensitiveDict with something like ​this Python cookbook recipe (I'll use one of the versions from later in the comments, since they fix problems with the earlier version). That code just looks a bit easier to read.

I'll move this to "design decision needed" for the time being. Chris, if I'm missing a good reason why you have that functionality, feel free to point it out. I'm willing to admit to being dense sometimes. No need to go to all the trouble of updating the patch if we can just make the switch: I'll just drop in the replacement and some tests when I do the commit.

[Chris Beaven]

You are right, preserving case it probably doesn't matter... in which case, yea there's probably a simpler way to do it. I was just blindly following Adrian's suggestion to "preserve":

Probably the HttpResponse? class should use a dict-like object that preserves, but ignores, case, so that "Foo" and "fOo" will map to the same item.

[Philippe Raoult]

refactoring using a basic python dict with lowercase keys

[Philippe Raoult]

The new patch has less overhead and has_header() should be much faster than before. ​http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 says that field names are case-insensitive so there's no need at all to preserve case.

[Jacob]

(In [6212]) Fixed #2970: made HttpResponse headers case-insensitive. Thanks to SmileyChris for the original patch and PhiR for the final one.

[calvin@…]

In my experience changing header names to lower-case will break some clients or servers (let alone bad programmed proxies).

To follow the reference, only key lookup must be case insensitive. Otherwise the application should preserve the original header value. So please consider reverting the __setitem__ change and the changed "Content-Type" case.

[Malcolm Tredinnick]

(In [6546]) Changed the way we handle HTTP headers internally so that they appear
case-insensitive, but the original case is preserved for output. This increases the chances of working with non-compliant clients without changing the external interface. Fixed #2970.