File.chunks contains potentially expensive size operation

[Peter Sagerson]

django.core.files.base.File's chunk method uses the file's size to determine how many chunks to return. Retrieving the size of a file is very fast for simple files, but may be very expensive for compressed files and other storage mechanisms. And in this case, it's completely unnecessary. Replacing the size-based loop with a simple check for the end of the stream has fewer dependencies and avoids a potentially expensive operation.

To cite one practical example, if the file is stored as a compressed file on disk, the current implementation will result in either decompressing the entire file twice or caching the entire decompressed file in memory. Removing the size dependency avoids both.

[Adam Nelson]

We need a design decision on this - the patch looks fine.

[Adam Nelson]

Also this ticket stating that the chunk size isn't always honored at #12157

[Ian Lewis]

+1 for removing the size() check in chunks(). It tends to break things when wrapping non-file-system file-like-objects such as StringIO()

[Ian Lewis]

However, that said, this patch might pose a problem for file-like-objects that should not be over-read socket connection. The patch probably should honor the file size if it exists.

[Erin Kelly]

See also #13901.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Claude Paroz]

When fixing #15644, the chunks implementation has been changed to not depend on size any more (r17871). It remains to be seen if comment:5 (over-read socket connection) might be true, triggering new bug reports. Real use-cases welcome...