Opened 17 years ago

Closed 16 years ago

#4924 closed (fixed)

Loading of compressed fixtures

Reported by: lars.yencken@… Owned by: Jeremy Dunck
Component: Core (Serialization) Version: dev
Severity: Keywords: compression, fixtures, feature
Cc: semente@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Fixtures contain a lot of redundancy (in xml or json format), so they compress well. This patch adds detection and loading of bzip2 or gzip compressed fixtures, since these formats are supported natively in Python. It also contains some test cases.

An example of the utility of this patch: in my project, an initial_data.json fixture was around 4Mb and bzip2'ed down to ~150Kb, a much more reasonable size for storing in a repository. I'm sure there are plenty of projects where this is useful.

The tests pass, except for one which fails in the same way the existing fixture test fails.

Attachments (5)

compression.diff (6.5 KB ) - added by Lars Yencken <lars.yencken@…> 17 years ago.
Patch minus binaries
fixture5.json (405 bytes ) - added by Lars Yencken <lars.yencken@…> 17 years ago.
Fixture 5 (tests)
fixture4.json (416 bytes ) - added by Lars Yencken <lars.yencken@…> 17 years ago.
Fixture 4 (should be bzip2'ed)
compression.2.diff (6.8 KB ) - added by Lars Yencken <lars.yencken@…> 17 years ago.
Updated patch
compressed-fixture.tar.gz (3.7 KB ) - added by Jeremy Dunck 16 years ago.
Patch and new files for compressed fixture support.

Download all attachments as: .zip

Change History (19)

comment:1 by lars.yencken@…, 17 years ago

Sorry, can't add patch or tests. I'm being spam filtered =(

comment:2 by anonymous, 17 years ago

Has patch: unset

by Lars Yencken <lars.yencken@…>, 17 years ago

Attachment: compression.diff added

Patch minus binaries

by Lars Yencken <lars.yencken@…>, 17 years ago

Attachment: fixture5.json added

Fixture 5 (tests)

by Lars Yencken <lars.yencken@…>, 17 years ago

Attachment: fixture4.json added

Fixture 4 (should be bzip2'ed)

comment:3 by Lars Yencken <lars.yencken@…>, 17 years ago

Has patch: set

Ok, got past the spam filter now. There's a trac bug which won't let me attach bzip2'ed files. In the tests/modeltests/fixtures/fixtures folder, we add:

fixture4.json.bz2
fixture5.json
fixture5.json.bz2

Fixture 5 is just a copy of fixture 1, to ensure an error gets raised if there is ambiguity as to which fixture file is meant.

comment:4 by Simon G. <dev@…>, 17 years ago

Triage Stage: UnreviewedReady for checkin

I've got the tests ready to go, but can't attach them either (keep hitting this trac bug), I'll email them on if a core wants them.

by Lars Yencken <lars.yencken@…>, 17 years ago

Attachment: compression.2.diff added

Updated patch

comment:5 by Lars Yencken <lars.yencken@…>, 17 years ago

I noticed a few small problems, and updated the patch to fix them. The problems were:

  • It still checked for all the different compression types, even if a unique filename was specified. So fixture5.json.bz2 and fixture5.json would clash, even if you specifically asked for fixture5.json.bz2.
  • The test case omitted checking for data which gets loaded from the initial_data fixture.
  • The coding style used alternatingCaps for names, instead of using_underscores.

Have a look at the new one, which should be better.

Thanks,
Lars

comment:6 by Russell Keith-Magee, 17 years ago

Triage Stage: Ready for checkinAccepted
  1. Patch needs to be updated following the management.py refactor.
  2. Patch should handle zip files in addition to bz2 and gz.
  3. More extensive tests are required, including:
    1. Need to test explicit compression scheme naming, not just discovery.
    2. Need to check the edge cases where you name a compression scheme, but not a fixture format.
  4. Documentation of the new capabilities are required.

comment:7 by Guilherme M. Gondim (semente) <semente@…>, 17 years ago

Cc: semente@… added

(only subscribing)

comment:8 by mcroydon, 17 years ago

Owner: changed from nobody to mcroydon

I'll see what I can do about russellm's suggestions.

comment:9 by mcroydon, 17 years ago

Need more caffeine, dropping this for the time being.

comment:10 by mcroydon, 17 years ago

Owner: changed from mcroydon to nobody

comment:11 by Russell Keith-Magee, 17 years ago

Keywords: feature added
Needs documentation: set
Needs tests: set
Patch needs improvement: set

by Jeremy Dunck, 16 years ago

Attachment: compressed-fixture.tar.gz added

Patch and new files for compressed fixture support.

comment:12 by Jeremy Dunck, 16 years ago

Needs documentation: unset
Needs tests: unset
Owner: changed from nobody to Jeremy Dunck
Patch needs improvement: unset
Status: newassigned

I've uploaded a new patch addressing the concerns russellm raised in his 09/13/07 comment.

comment:13 by Jeremy Dunck, 16 years ago

Triage Stage: AcceptedReady for checkin

comment:14 by Jacob, 16 years ago

Resolution: fixed
Status: assignedclosed

(In [9527]) Fixed #4924: added support for loading compressed fixtures. Thanks to Lars Yencken and Jeremy Dunck.

Note: See TracTickets for help on using tickets.
Back to Top