Run i18n tests on disposable FS tree instead of source code

[Ramiro Morales]

Currently, tests that exercise translatable literal extraction to .po files by makemessages (located on test_extraction.py) and test that exercise compilation of .po files to .mo files by compilemessages (test_compilation.py) are run under the test/i18n test source code tree creating subdirectories and files under the different work projects/applications located there.

Proposal is to copy the relevant filesystem tree to a tempdir as provided by runtests.py + tempfile.mkdtemp() as part of every test case setup.

This has the following advantages:

• Avoid weird bogus errors and failures caused by stray directories left by previous runs/debugging sessions.
• Allows us to solve the fact that currently tests on test_compilation.py aren't marked as needing seralized execution and are actually being run in parallel. This, for instance, causes a couple of consecutive runs of the i18n test on my Ubuntu 1.4.04 system to report misleading failures, most probably because of race conditions.
• Ensure actual isolation among test cases
• Allow each of these test cases to be actually run in in parallel by our existing machinery. This allows us to be able to stop relying on django.test.testcases.SerializeMixin

These changes possibly aren't visible on our CI infrastructure as every job there always use a clean tree but can be of help to those running the test suite and/or working on i18n tests.

Other tests (tests.py, test_percents.py) don't have write access to disk so they are unaffected.

[Ramiro Morales]

​PR

[Tim Graham]

Comment for improvement on the PR.

[Aymeric Augustin]

I did that in a bunch of other places when I implemented test parallelization. To test it, I just made the entire source tree readonly, ran the tests and saw what broke.

I gave up for i18n tests because it required too extensive changes. It's great to see that you're picking up the flag! Indeed, once you've managed to isolate each test in its own temporary directory, you can remove SerializeMixin, and the tests will be faster.

While you're there you could check how much I/O these tests perform. I/O may be a bottleneck even with a SSD e.g. if each test copies 50 files where 5 would suffice. IIRC the total runtime of the i18n tests was about 4 seconds with a SSD, so there isn't a huge lot to gain and I didn't spend much time optimizing it. Over a few thousands runs, improvements would compound, though.

[Ramiro Morales]

Replying to aaugustin:

Indeed, once you've managed to isolate each test in its own temporary directory, you can remove SerializeMixin, and the tests will be faster.

Exactly, that's one of the included changes.

While you're there you could check how much I/O these tests perform. I/O may be a bottleneck even with a SSD e.g. if each test copies 50 files where 5 would suffice. IIRC the total runtime of the i18n tests was about 4 seconds with a SSD, so there isn't a huge lot to gain and I didn't spend much time optimizing it. Over a few thousands runs, improvements would compound, though.

Will do some testing. No SDD here but will see what conclussion can be made.

Thanks!

[Ramiro Morales <cramm0@…>]

In faeeb84e:

Fixed #26677 -- Converted some i18n tests to use disposable FS tree.

This allows makemessages/compilemessages tests in test_extraction.py
and test_compilation.py to actually run isolated from each other
(unaffected by stray FS objects left by cleanup actions failures, debug
sessions, etc.) and to take advantage of the parallel tests execution
feature like most of the Django test suite.

test_percents.py gets slightly refactored to not inherit from the new
machinery which sets up every test case to copy and run under a
temporary tree.

[Ramiro Morales <cramm0@…>]

In bb7bb37:

Refs #26677 -- Simplified i18n test cleanups.

The fact that we aren't dealing with the Django source tree anymore
allows us to drop several tearDown()/addCleanup() calls that were
concerned with removing apiece files/dirs/symlinks created by test
cases, as we are covered by the removal of the parent temporary tree
anyways.

Thanks Tim Graham for advice and review.

[Tim Graham <timograham@…>]

In c1bd4679:

[1.10.x] Fixed #26677 -- Converted some i18n tests to use disposable FS tree.

This allows makemessages/compilemessages tests in test_extraction.py
and test_compilation.py to actually run isolated from each other
(unaffected by stray FS objects left by cleanup actions failures, debug
sessions, etc.) and to take advantage of the parallel tests execution
feature like most of the Django test suite.

test_percents.py gets slightly refactored to not inherit from the new
machinery which sets up every test case to copy and run under a
temporary tree.

Backport of faeeb84edfebecf5a5f40df9ef816e5f1cd457c6 from master

[Tim Graham <timograham@…>]

In a079d5ce:

[1.10.x] Refs #26677 -- Simplified i18n test cleanups.

The fact that we aren't dealing with the Django source tree anymore
allows us to drop several tearDown()/addCleanup() calls that were
concerned with removing apiece files/dirs/symlinks created by test
cases, as we are covered by the removal of the parent temporary tree
anyways.

Thanks Tim Graham for advice and review.

Backport of bb7bb379e8cd91a91336946829519d64e919a1d2 from master

[Tim Graham]

The commits were backported to 1.10.x at the request of #27526.