XML deserializer strips leading whitespace on loaddata

[Joseph-django@…]

If an object instance has a character field and the value of that field starts with the tab character, loaddata removes that tab character when the loaded fixture is in XML format.

Note that the XML dump data does not strip this leading tab character. Also note that both the JSON dump and load data preserve the tab character.

I have not tested this with other whitespace characters. This can be easily reproduced by creating a simple model:

class Foobar(models.model)
  name = models.CharField(max_length=20)

And then creating an instance of that model (e.g, in Django Shell) with a name value of, e.g, "\tBaz" and then using the manage.py dumpdata with --format=xml.

Once the fixture has been generated, remove the existing instance (either by deleting it, or flushing the app data, or your preferred method) and then using manage.py loaddata to load the fixture. Note the instance's name no longer contains the tab character.

[Martin Matusiak]

I can reproduce it.

[Martin Matusiak]

Trying with a few special characters:

Foobar.objects.create(name=' bar')
Foobar.objects.create(name='\abar')
Foobar.objects.create(name='\bbar')
Foobar.objects.create(name='\fbar')
Foobar.objects.create(name='\nbar')
Foobar.objects.create(name='\rbar')
Foobar.objects.create(name='\tbar')
Foobar.objects.create(name='\vbar')

XML pretty printer corrupts this completely:

    <object pk="9" model="app.foobar">
        <field type="CharField" name="name"> bar</field>
    </object>
    <object pk="10" model="app.foobar">
        <field type="CharField" name="name">bar</field>
    </object>
    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>
    </object>
    <object pk="12" model="app.foobar">
        <field type="CharField" name="name">
                                            bar</field>
    </object>
    <object pk="13" model="app.foobar">
        <field type="CharField" name="name">
bar</field>
    </object>
    <object pk="14" model="app.foobar">
bar</field>eld type="CharField" name="name">
    </object>
    <object pk="15" model="app.foobar">
        <field type="CharField" name="name">    bar</field>
    </object>
    <object pk="16" model="app.foobar">
        <field type="CharField" name="name">
                                            bar</field>
    </object>

In terse mode it's more likely to be correct when loaded again, but clearly this needs fixing:

<object pk="9" model="app.foobar"><field type="CharField" name="name"> bar</field></object><object pk="10" model="app.foobar"><field type="CharField" name="name">bar</field></object><object pk="11" model="app.foobar"><field type="CharField" name="name"bar</field></object><object pk="12" model="app.foobar"><field type="CharField" name="name">
                               bar</field></object><object pk="13" model="app.foobar"><field type="CharField" name="name">
bar</field></object><object pk="15" model="app.foobar"><field type="CharField" name="name">     bar</field></object><object pk="16" model="app.foobar"><field type="CharField" name="name">
                                                     bar</field></object>

So that's dumpdata at fault, not loaddata.

[Martin Matusiak]

Another experiment: hand edit the xml dump file using html escapes (1) to see if loaddata will load it correctly. No, also doesn't work.

So:
Problem 1: The xml serializers (SimplerXMLGenerator/pulldom) do not round trip these characters.
Problem 2: Even if they did a tab character would be stripped due to:

value = field.to_python(getInnerText(field_node).strip())

core.serializers.xml_serliazer.py:214

(1) ​http://mail-archives.apache.org/mod_mbox/xmlgraphics-fop-users/200406.mbox/%3C40C5E61C.9040304@hotmail.com%3E

[Daniele Procida]

Replying to numerodix:

    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>
    </object>> 

}}}

Wow. If that's what you're getting in the dumped text file, that's remarkable.

[Martin Matusiak]

What we could do is try to wrap any special characters in a CDATA section. So the xml would look like this:

<object pk="2" model="app.foobar"><field type="CharField" name="name"><![CDATA[\t]]>bar</field></object>

The deserializer then gives us the tab character escaped:

u'\\tbar'

So we'd have to strip the escape.

But this feels very ad-hoc and messy.

[jarshwah]

Shouldn't all straight up text (CharField/TextField) be wrapped in a CDATA though? Then there's no need to look for special characters at all. Are there any downsides to wrapping all text values in a CDATA? Unsure why the deserialiser would escape the tab though (I haven't looked into it), but that is sort of a separate - related - issue.

[Martin Matusiak]

@smeaton, I think it probably should, yes. The downside would be that you're adding 12 bytes to every value even though most strings would not need it.

One could optimize that by wrapping only strings that need to be wrapped, according to a character range or something like that.

This serializer is also used by the syndication feed framework btw. It would be nice to fix the problem in both places at once.