test_msgfmt_error_including_non_ascii fails on Mac OS Catalina, localized msgfmt binary.

[Nilo César Teixeira]

FAIL: test_msgfmt_error_including_non_ascii (i18n.test_compilation.CompilationErrorHandling)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/teixeira/Downloads/Pessoal/etudes/etudes_py/django_source/tests/i18n/test_compilation.py", line 204, in test_msgfmt_error_including_non_ascii
    self.assertIn("' cannot start a field name", stderr.getvalue())
AssertionError: "' cannot start a field name" not found in 'Execution of msgfmt failed: /private/var/folders/sj/kll9fdms3c52l7m19_f_zn980000gn/T/django_al6dgjzh/i18n_cam5a1xi/commands/locale/ko/LC_MESSAGES/django.po:24: "msgstr" não é um formato de string Python brace válido, ao contrário de "msgid". Motivo: Na diretiva número 0, "�" não pode iniciar um nome de campo.\nmsgfmt: encontrado 1 erro fatal\n'

The assertIn fails because the msgfmt output is in pt-BR. The export LANG=C environment setting on this test doesn't help changing the binary msgfmt output to English.

[Nilo César Teixeira]

I found a fix!

env.update({'LC_ALL': 'C'})

This worked!

[Carlton Gibson]

Hi.

I'm going to close this as Invalid because I'm pretty sure it's a system configuration issue.

Happy to help if I can…

In a new shell try the locale command. You basically want something claiming to be UTF-8:

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

You want pt-br right? So putting export LC_ALL=pt_BR.UTF-8 in your shell startup script may/should solve your issues.

[Claude Paroz]

Carlton, I'm not sure you got the idea of the ticket. The problem is that we are checking an English error message content in the Django test CompilationErrorHandling.test_msgfmt_error_including_non_ascii, and we try to force unlocalize message by env.update({'LANG': 'C'}) which looks like is not sufficient in certain systems.

[Carlton Gibson]

OK, let's have another look, thanks Claude.

...which looks like is not sufficient in certain systems.

Not sure how to proceed.
I can't reproduce without more details on macOS Big Sur.
I'm afraid I don't have Catalina available (but this test never failed for me over the last few years.)

It still looks like a locale issue 🤔 @niloct what output do you get there?

[Nilo César Teixeira]

~/Downloads/Pessoal/etudes/etudes_py/django_source/tests $ locale
LANG="pt_BR.UTF-8"
LC_COLLATE="pt_BR.UTF-8"
LC_CTYPE="pt_BR.UTF-8"
LC_MESSAGES="pt_BR.UTF-8"
LC_MONETARY="pt_BR.UTF-8"
LC_NUMERIC="pt_BR.UTF-8"
LC_TIME="pt_BR.UTF-8"
LC_ALL="pt_BR.UTF-8"

For some reason LANG isn't enough to reset locale settings, but LC_ALL was.

If I change the environment override on the aforementioned test to LC_ALL instead of LANG, the test passes.

[Carlton Gibson]

Super, thanks for the extra info.

$ LC_ALL="pt_BR.UTF-8" ./runtests.py i18n

Is sufficient to cause the failure. So the env update will fix it, as you say.

diff --git a/tests/i18n/test_compilation.py b/tests/i18n/test_compilation.py
index 791c1d4f15..915d65e6e1 100644
--- a/tests/i18n/test_compilation.py
+++ b/tests/i18n/test_compilation.py
@@ -193,7 +193,7 @@ class CompilationErrorHandling(MessageCompilationTests):
         # po file contains invalid msgstr content (triggers non-ascii error content).
         # Make sure the output of msgfmt is unaffected by the current locale.
         env = os.environ.copy()
-        env.update({'LANG': 'C'})
+        env.update({'LANG': 'C', 'LC_ALL': 'C'})
         with mock.patch('django.core.management.utils.run', lambda *args, **kwargs: run(*args, env=env, **kwargs)):
             cmd = MakeMessagesCommand()
             if cmd.gettext_version < (0, 18, 3):

Would you like to make a PR @niloct?

[Carlton Gibson]

Checking the ​locale man page, LC_ALL if present trumps all else, so the LANG setting can be replaced in the test, rather than added to, I think.

[Nilo César Teixeira]

Replying to Carlton Gibson:

Checking the ​locale man page, LC_ALL if present trumps all else, so the LANG setting can be replaced in the test, rather than added to, I think.

Hi Carlton!

Yes I saw that man. Awesome that you concur :)

Sorry for the delay in response, I wasn't notified because I hadn't updated my profile with my e-mail, despite the big yellow box shouting this to me!

Regarding the pull request, I'm just starting reading through the docs (finished the toast contribution), would I have to change any docs in the pr or just the test with a comment about it ?

Thanks!

[Nilo César Teixeira]

Well, I did it I guess.

Please take a look. It's my first contribution to Django :)

​https://github.com/django/django/pull/14426

[Carlton Gibson]

Super, thanks Nilo — Welcome aboard! :) — I'll take a look.

[Carlton Gibson <carlton.gibson@…>]

In 0d67481:

Fixed #32762 -- Fixed locale reset in compilemessages test.

Reset the LC_ALL override value in the test environment to ensure that locale
values the calling environment are not used.