Document redirecting dumpdata output may result in incorrect encoding in Windows PowerShell

[OBu]

When I "dumpdata" my database on my Windows 8.1 machine and load the dumped file with "loaddata", I get encoding errors.

Opening the dumped file with notepad++, setting the encoding to UTF-8 and saving it again helps.

​A django-users thread describes the problem in detail so it seems I'm not alone...

It might not be a Django problem but a windows pipeline-issue (I was using the powershell), but it is a bit embarrassing.

[Baptiste Mispelon]

Hi,

Are you able to try to do dumpdata with both Python 2 and Python 3? Does it make any difference?

Thanks.

[OBu]

Sorry, I don't. I was using python 3.5.1 and django 1.9.7

I just tried dir > test.txt in my windows command line, just to check the encoding of the resulting file, and notepad++ tells me it is UTF-8.

When I do the same in Windows Powershell, encoding is UCE-2 BE BOM

I assume this is exactly what happened to me, so it's not really a Django issue but a problem resulting from the character encoding in Windwos Powershell.

I'm not sure whether there is a way to fix this other then using the --output option (which results in a correctly encoded UTF-8 file no matter which shell was used), but maybe this problem should be mentioned in the docs since the --output option is new and most help web sites are using > .

[Tim Graham]

I'm not knowledgeable about Windows to confirm, but assuming some can confirm it, a documentation note seems okay.

[Carlton Gibson]

#32439 was closed as a duplicate, with dumpdata failing due to the target file not explicitly setting the encoding.

Likely the solution here is to add to the ​Windows How-To that the default encoding can be set UTF8 now:

Go to Settings there's a "Use Unicode UTF-8 for worldwide language support", box in "Language" - "Administrative Language Settings" - "Change system locale" - "Region Settings".

[GitHub <noreply@…>]

In 0456d3e4:

Fixed #26721 -- Doc'd setting UTF-8 on Windows.

[Mariusz Felisiak <felisiak.mariusz@…>]

In d6b6eda4:

[3.2.x] Fixed #26721 -- Doc'd setting UTF-8 on Windows.

Backport of 0456d3e42795481a186db05719300691fe2a1029 from main