Big base64-encoded files uploading

[anthony@…]

I'm trying to upload image file using ajax & browser FileReader.readAsDataURL API. In most cases it fails with MultiPartParserError: Could not decode base64 data: Error('Incorrect padding',).
When django cut of header from 'multipart/form-data' part with encoded file data, it does not check length of the remaining data and pass it to decoder. Length of passed data is (FileUploadHandler chunk size) - (removed header length) which is not divides by 4 in most cases.
It is reproduced by testcase I attached. It is based on standard Django test_base64_upload test but reads 512Kb from /dev/urandom instead of short string.

[Claude Paroz]

Confirmed. Do you have any idea about the cleaner way to solve this?

Some ideas:

1. force the stream to be a multiple of 4 by "ungetting" some bytes from the stream after we stripped off the headers
2. only decode the base64-encoded file at the end of the upload (using the file_complete upload handler method)

[Claude Paroz]

Test case as a patch

[Sir Anthony]

Not sure. I tried to make a patch by modifying MultiPartParser, where it must be processed in my opinion (variant 1), but was confused a little with streams there. I did not spent much time with this part of Django previously and do not realize full request processing routine in details yet. But I think, second variant is less acceptable because it will be returning to data processing after finishing and will require doubling of code (using low-level calls in higher-level abstractions) what will complicate the structure of handlers.

[Claude Paroz]

Decoding base64 by multiple of 4

[Claude Paroz]

Just attached a patch that seems to work (testing welcome). My only worry is about memory management when doing chunk += over_chunk. Hopefully the Python implementation is smart and do not copy too much stuff in memory.

[Claude Paroz]

Added blocker flag, as currently 3 of 4 base64 file uploads are doomed to fail.

[johannesl]

Corrections for stable/1.5.x branch

[johannesl]

Regarding memory usage, this should be good enough for release. After 1.5, it might be possible to use base64.encode together with StringIO, or even adjusting read-sizes in streaming code.. I doubt the complexity it adds will be worth the performance gains though.

[Claude Paroz <claude@…>]

In 2a67374b51c5705d5c64a5d7117ad8552e98b4bb:

Fixed #19036 -- Fixed base64 uploads decoding

Thanks anthony at adsorbtion.org for the report, and johannesl for
bringing the patch up-to-date.

[Claude Paroz <claude@…>]

In fc379b48655a365f0dd51880a1f68a15811f903a:

[1.5.x] Fixed #19036 -- Fixed base64 uploads decoding

Thanks anthony at adsorbtion.org for the report, and johannesl for
bringing the patch up-to-date.
Backport of 2a67374b5 from master.

[m.zak@…]

This bug is present also in Django 1.4.13 which is mentioned on Django webpage as still supported.

[Tim Graham]

The 1.4 branch is only receiving security fixes at this time.