Opened 11 years ago

Closed 10 years ago

Last modified 10 years ago

#22880 closed Cleanup/optimization (fixed)

UnicodeDecode error when running `makemessages` (python 3.4.1)

Reported by: Vidir Valberg Gudmundsson Owned by: nobody
Component: Documentation Version: 1.7-beta-2
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Made this model for testing:

from django.db import models
from django.utils.translation import ugettext_lazy as _


class Foo(models.Model):
    bar = models.CharField(
        verbose_name=_('bar'),
        max_length=255
    )

django-admin makemessages -l da creates the following django.po file:

$ cat locale/da/LC_MESSAGES/django.po
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2014-06-22 19:49+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

#: models.py:7
msgid "bar"
msgstr ""

Changing the last line (doing the translation of "bar") to this:

msgstr "æøå"

Results in this error when running django-admin makemessages -l da again:

processing locale da
Traceback (most recent call last):
  File "./../manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/base.py", line 532, in handle
    return self.handle_noargs(**options)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 295, in handle_noargs
    self.write_po_file(potfile, locale)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 397, in write_po_file
    msgs, errors, status = popen_wrapper(args)
  File "/Users/valberg/.virtualenvs/django17/lib/python3.4/site-packages/django/core/management/utils.py", line 25, in popen_wrapper
    output, errors = p.communicate()
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 1658, in _communicate
    self.stdout.encoding)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 887, in _translate_newlines
    data = data.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 663: ordinal not in range(128)

I'm not sure whether this is a django error or a python one.

Change History (8)

comment:1 by Claude Paroz, 11 years ago

Did you try with other versions of Django?

comment:2 by Vidir Valberg Gudmundsson, 11 years ago

Just did, and it also happens with 1.6.5 and 1.5.8. With the almost the same traceback.

1.6.5:

$ django-admin makemessages -l da
processing locale da
Traceback (most recent call last):
  File "./../manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 279, in handle_noargs
    self.write_po_file(potfile, locale)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 364, in write_po_file
    msgs, errors, status = popen_wrapper(args)
  File "/Users/valberg/.virtualenvs/djang16/lib/python3.4/site-packages/django/core/management/utils.py", line 25, in popen_wrapper
    output, errors = p.communicate()
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 1658, in _communicate
    self.stdout.encoding)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 887, in _translate_newlines
    data = data.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 663: ordinal not in range(128)

1.5.8:

$ django-admin makemessages --traceback -l da
processing language da
Traceback (most recent call last):
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/base.py", line 255, in execute
    output = self.handle(*args, **options)
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/base.py", line 385, in handle
    return self.handle_noargs(**options)
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 396, in handle_noargs
    symlinks, ignore_patterns, no_wrap, no_location, no_obsolete, self.stdout)
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 335, in make_messages
    not invoked_for_django, wrap, location, no_obsolete)
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 221, in write_po_file
    (wrap, location, pofile, potfile))
  File "/Users/valberg/.virtualenvs/django15/lib/python3.4/site-packages/django/core/management/commands/makemessages.py", line 47, in _popen
    output, errors = p.communicate()
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 959, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 1658, in _communicate
    self.stdout.encoding)
  File "/Users/valberg/.pyenv/versions/3.4.1/lib/python3.4/subprocess.py", line 887, in _translate_newlines
    data = data.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 663: ordinal not in range(128)
Last edited 11 years ago by Vidir Valberg Gudmundsson (previous) (diff)

comment:3 by Vidir Valberg Gudmundsson, 11 years ago

It looks as if my locale was wrong somehow:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Setting LANG to "da_DK.UTF-8" does the trick as far as I can tell.

$ export LANG="da_DK.UTF-8"
$ locale
LANG="da_DK.UTF-8"
LC_COLLATE="da_DK.UTF-8"
LC_CTYPE="da_DK.UTF-8"
LC_MESSAGES="da_DK.UTF-8"
LC_MONETARY="da_DK.UTF-8"
LC_NUMERIC="da_DK.UTF-8"
LC_TIME="da_DK.UTF-8"
LC_ALL="da_DK.UTF-8"
$ django-admin makemessages --traceback -l da
processing language da

comment:4 by Vidir Valberg Gudmundsson, 11 years ago

Component: Core (Management commands) → Documentation
Has patch: set
Type: Uncategorized → Cleanup/optimization

PR: ​https://github.com/django/django/pull/2841

comment:5 by Claude Paroz, 10 years ago

After the night :-), here's how I would have written it: ​https://github.com/django/django/pull/2842

comment:6 by Tim Graham, 10 years ago

Triage Stage: Unreviewed → Accepted

It's not really a Django specific issue, but I guess it wouldn't hurt to mention as we already have a topic guide on unicode.

comment:7 by Claude Paroz <claude@…>, 10 years ago

Resolution: → fixed
Status: new → closed

In 460ec09d2ed26f5862aabeaa480bcb713d5f331f:

Fixed #22880 -- Added FAQ entry about UnicodeDecodeError

Thanks Víðir Valberg Guðmundsson for the report and
Tim Graham for the review.

comment:8 by Claude Paroz <claude@…>, 10 years ago

In 198ce3073c763ab17cb5e8d012d64d8ea756282d:

[1.7.x] Fixed #22880 -- Added FAQ entry about UnicodeDecodeError

Thanks Víðir Valberg Guðmundsson for the report and
Tim Graham for the review.
Backport of 460ec09d2e from master.

Note: See TracTickets for help on using tickets.
Back to Top