ePub documentation not valid

[mabdullah]

There is an error in the generated epub that prevents it from opening. Running the epub though the validator also reports an error.
​http://validator.idpf.org/

[mabdullah]

DJango 1.5.2 EPub validation failure.

[Tim Graham]

Retriggering the build at ​https://readthedocs.org/builds/django/ seems to have fixed this.

[michal@…]

Validation for lastest ePub (1.7a2) at ​http://validator.idpf.org/ fails.
Import to Google Books fails.
File can not be open in some Android eBook readers.

[Tim Graham]

I've checked both the version from readthedocs and a locally generated one using make epub and I see hundreds of validation warnings in both cases.

[afuna]

There are 3000+ validation warnings/errors generated by epubcheck. The good news is that these fall into only a dozen or so categories:

1) 1 times ERROR: value of attribute "href" is invalid; must be a URI
2) 1 times ERROR: attribute "id" not allowed here; expected attribute "charset", "defer", "src" or "xml:space"
3) 2 times ERROR: element "form" not allowed anywhere; expected the element end-tag, text or element "a", "abbr", "acronym", "address", "applet", "b", "bdo", "big", "blockquote", "br", "cite", "code", "del", "dfn", "div", "dl", "em", "h1", "h2", "h3", "h4", "h5", "h6", "hr", "i", "iframe", "img", "ins", "kbd", "map", "noscript", "ns:svg", "object", "ol", "p", "pre", "q", "samp", "script", "small", "span", "strong", "sub", "sup", "table", "tt", "ul" or "var" (with xmlns:ns="http://www.w3.org/2000/svg")
4) 2 times ERROR: attribute "start" not allowed here; expected attribute "dir", "id", "lang", "style", "title" or "xml:lang"
5) 3 times ERROR: hyperlink to non-standard resource <...> of type <...>
6) 3 times ERROR: element "dc:date" not allowed anywhere; expected the element end-tag or text
7) 7 times ERROR: <...>: referenced resource missing in the package.
8) 678 times ERROR: value of attribute "id" is invalid; must be an XML name without colons
9) 2400 times ERROR: <...>: fragment identifier is not defined in <...>
10) 5 times HINT: Link attribute with no value
11) 1 times WARNING: use of non-registered URI scheme type in href: ​irc://irc.freenode.net/django-dev
12) 4 times WARNING: hyperlink to resource outside spine <...>
13) 8 times WARNING: use of non-registered URI scheme type in href: ​irc://irc.freenode.net/django
----
3115 total

[Daniel Roschka]

I looked a bit into this issue and it seems to be mostly related to the template in use. So by further customizing the template we should be able to fix these warnings.

The djangodocs-epub theme inherits from epub which inherits from basic. Interestingly most of these issues are already present in upstreams epub theme, as it inherits from basic which is just an HTML-theme and doesn't know about epub at all.

Simply extending templates from basic and overwriting blocks doesn't work for all warnings, as some of them are outside of blocks and even if they are in blocks it means mostly replicating the existing logic of these blocks, so the cleanest solution seems to be to build an epub theme, which doesn't inherit from any other theme. That would also allow us to make better use of epub-specific features, but of course is no quick and easy undertaking.

[Claude Paroz]

​https://github.com/sphinx-doc/sphinx/issues/5070 (issue with internal anchors) has just been fixed. After the 1.7.6 sphinx release is out and deployed on readthedocs, we may check the document validity again.

[Katie McLaughlin]

readthedocs.org urls have moved since this ticket was filed, new URL: ​https://readthedocs.org/projects/django/downloads/

Tested the downloads for latest, stable, 2.2.x and 3.0x

They appear to load in ebook reviews (calibre, apple books)

The validator ​http://validator.idpf.org/ does still return errors, but they aren't breaking errors (from what I can tell):

6 x toc.ncx - Error while parsing file: different playOrder values for navPoint/navTarget/pageTarget that refer to same target
72 x *.svg - External entities are not allowed in EPUB v3 documents.
2639 x *.xhtml - Fragment identifier is not defined.

Two minor svg errors that might be important:

_images/django_unittest_classes_hierarchy.svg - Error while parsing file: element "dc:date" not allowed anywhere; expected the element end-tag or text
_images/triage_process.svg Error while parsing file: element "dc:date" not allowed anywhere; expected the element end-tag or text

Vote to close

[Carlton Gibson]

SVG

The SVG diagrams were generated using OmniGraffle. Latest versions still generate the invalid SVG. I've filed a bug report with them, which I'd guess they'd address, so we can update the files as-and-when.

In the meantime, the images look to view without problem. (In Apple Books, the background gradient on the unit test classes diagram is not rendered entirely perfectly, but it is so in Firefox and Safari, so I suspect that is Apple Books...)

Fragment Identifier

Re the two-thousand-odd Fragment identifier is not defined. errors, I think those should be (cough) straight-forward enough to fix.

The issue is in cross links:

genindex.xhtml:979 has:

        <li><a href="ref/templates/builtins.xhtml#std-templatefilter-addslashes">template filter</a>

But the destination is ref/templates/builtins.xhtml:1220, which has:

<span id="std:templatefilter-addslashes">...

std:templatefilter-addslashes is not std-templatefilter-addslashes.

Same with testing/topics/advanced.html:639 pointing to ref/settings.xhtml:553. There std-setting-TEST_SERIALIZE is not std:setting-TEST_SERIALIZE.

It looks like a different rule is just being applied to generate the url anchor fragments. (A fix there may just be finding an error in sphinx and reporting it.)
If we can solve that, then the vast majority of the output is eliminated.

From there, I suspect the remaining errors won't look so intimidating.

I'm going to mark this Easy pickings. It may take some detective work to pin down exactly the issue for the fragment identifiers (Is it something we can fix or is it a bug in Sphinx?) but I think it would be a good first issue if someone is willing to pick it up.

For me, downloading EPUBCheck to run locally was very helpful: ​https://github.com/w3c/epubcheck/releases
(It has a good README in the download but, $ java -jar epubcheck.jar django.epub was the essence of it.)

Vote to close

Yes, I kind of see that: the EPUB does work. But it's not optimal. The errors are leading to a lack of cross links (at least). If we could clear them up, I'd guess it would improve presentation... If someone is prepared to have a look, I think it's would be nice to have. (The TOC is not super helpful as presented in Apple Books at least. There's too much info there. I wonder if section content pages are possible... anyhow, for later.)

(For reference, I ran EPUB check on an EPUB from a quite large technical book publisher and they had 3 "resource isn't defined" type errors and then 250-ish Content file contains script which is not supported in EPUB v2 which I guess is progressive enhancement.)

Update: the relevant Django files, if it is our issue, rather that Sphinx's would likely be django/docs/_ext/djangodocs.py and/or django/docs/conf.py.

[Carlton Gibson]

Digging with Diane Chen at DjangoCon sprints, the Fragment identifier errors at least look to be some form of ​https://github.com/sphinx-doc/sphinx/issues/5070, which was in theory fixed over a year ago, so we've just asked for more info there. Hopefully we can get a resolution.

[Carlton Gibson]

OK, so progress. After ​https://github.com/sphinx-doc/sphinx/pull/6734 (testing on Sphinx's 2.0 branch until it's released) we're down from nearly 3000 errors to ≈825.

Check finished with errors
Messages: 0 fatals / 814 errors / 5 warnings / 5 infos

There were a few Sphinx errors on make epub:

WARNING: duplicated ToC entry found: howto/error-reporting.xhtml
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#email-reports
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#server-errors
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#errors
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#filtering-error-reports
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#filtering-sensitive-information
WARNING: duplicated ToC entry found: howto/error-reporting.xhtml#custom-error-reports
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml#serving-static-files-in-production
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml#serving-the-site-and-your-static-files-from-the-same-server
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml#serving-static-files-from-a-dedicated-server
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml#serving-static-files-from-a-cloud-service-or-cdn
WARNING: duplicated ToC entry found: howto/static-files/deployment.xhtml#learn-more

I'm sure if they were cleared up it would grab a bunch.

Then the remaining errors look to fall into a few buckets.

One in particular:

ERROR(RSC-005): ../django/docs/_build/epub/Django.epub/_modules/django/contrib/auth.xhtml(76,47): Error while parsing file: element "div" not allowed here; expected the element end-tag, text, element "a", "abbr", "area", "audio", "b", "bdi", "bdo", "br", "button", "canvas", "cite", "code", "data", "datalist", "del", "dfn", "em", "embed", "epub:switch", "i", "iframe", "img", "input", "ins", "kbd", "label", "link", "map", "mark", "meta", "meter", "ns1:math", "ns2:svg", "object", "output", "picture", "progress", "q", "ruby", "s", "samp", "script", "select", "small", "span", "strong", "sub", "sup", "template", "textarea", "time", "u", "var", "video" or "wbr" (with xmlns:ns1="http://www.w3.org/1998/Math/MathML" xmlns:ns2="http://www.w3.org/2000/svg") or an element from another namespace

If we could pin that down, it would be several hundred resolved.

(I still think this is a good ticket first ticket for someone wanting to get involved.)

[Fredrik Malmfors]

I'll give this a try.

[Fredrik Malmfors]

ToC - Duplicate section: Error reporting

[Fredrik Malmfors]

This will be a long read, but in short, I managed to cut the error count from 84 down to 1 (patch provided). While digging I also found some interesting problems that are just as relevant to the html documentation, as they are to the epub.

As Carlton suggested, I’ve been using Sphinx 2.0 branch (instead of any official release, the 2.2.2 version released yesterday still gives the errors in the thousands) along with Epubcheck version 4.2.2.

Initially, Epubcheck returned 84 errors, which is already significantly less than before. The error(s) Carlton mentioned

ERROR(RSC-005): ../django/docs/_build/epub/Django.epub/_modules/django/contrib/auth.xhtml(76,47): Error while parsing file:

is no longer present, and seems to have been resolved on Sphinx part.

Anyway, here I’ll go through the remaining errors one by one:

Error reporting and Deploying Static Files

• 3 x ERROR(RSC-005): django/docs/_build/epub/Django.epub/content.opf(621,32): Error while parsing file: Itemref refers to the same manifest entry as a previous itemref
• 6 x ERROR(RSC-005): django/docs/_build/epub/Django.epub/toc.ncx(1949,52): Error while parsing file: different playOrder values for navPoint/navTarget/pageTarget that refer to same target

Problem
These errors occurs from the chapters Error reporting and Deploying Static Files, appearing twice in the ToC. This is true for the html docs aswell, not just the epub. In essence, the same chapters can be found both under “ How-to guides“, and under “How-to guides / Deploying Django”. This is seen clearly in the screenshot below (from the website):

This violates the ToC correlation with the reading order. It also leads to problems in the epub since (if going through reading order) only the first reference is being rendered, and the second one is missing.

Fix
I removed the second reference. I beleive Error reporting and Deploying Static Files can stand on their own outside of deployment. Also, the deployment checklist already refers to both of these at least once. Please let me know if you have another suggestion.

SVGs (temporarily fixed)

• 2 * ERROR(OPF-073): django/docs/_build/epub/Django.epub/_images/triage_process.svg(2,98): External identifiers must not appear in the document type declaration.
• 71 * ERROR(HTM-003): django/docs/_build/epub/Django.epub/_images/triage_process.svg(60,31): External entities are not allowed in EPUB v3 documents. External entity declaration found: %svg-model.mod.

Problem
These errors are caused by the <!DOCTYPE> tag in SVGs. Neither sketch nor illustrator includes !DOCTYPE nowadays in SVGs. It also seems like SVG 2.0 does not care about !DOCTYPE at all.

Fix
I have removed the <!DOCTYPE> in both SVGs for now. Hopefully !DOCTYPE won’t be included in exports from future versions of OmniGirrafe.

Duplicate cover-page in spine (Unresolved)

• 1 * ERROR(RSC-005): django/docs/_build/epub/Django.epub/content.opf(980,43): Error while parsing file: Itemref refers to the same manifest entry as a previous itemref

Problem
The epub build contains a file content.opf, where items are added to the manifest. There is also is a <spine>, with references to the items in the manifest.
The item corresponding to the cover-page, epub-1, gets added (again) to the end of the spine where it should not. This breaks EPUB 3.2 rules since items must only be refered to once.

The first entry is controlled by the

epub_cover = ('', 'epub-cover.html')

in conf.py. This one is nessecary for the epub-cover to be displayed at all.

Why it get’s added a second time to the spine at the end, I haven’t yet found an explaination for.

Fix
None. This does not seem to be a critical issue though.

Unnecessary reference to search page

• 1 * ERROR(RSC-007): django/docs/_build/epub/Django.epub/intro/whatsnext.xhtml(34,62): Referenced resource 'search.xhtml' could not be found in the EPUB.

Problem
In intro/whatsnext.html, we have the following lines:

• so finding what you need can sometimes be tricky. A few good places to start
• are the :ref:search and the :ref:genindex.

The :ref:search creates a link straight to the search page.

HTML
When pressing the link we come to an empty page with the text “No search query given”. This is because the search.html only displays the results of the given search from the search bar.

Epub
When klicking the ‘search’ link, nothing happens. Even more reason to remove it.

Fix
I updated the text to instead refer to buildin search functionality.

ToC ordering (proposed solutions / maybe)

1.

• WARNING(NAV-011): django/docs/_build/epub/Django.epub/nav.xhtml(442,81): 'toc' nav must be in reading order; link target 'topics/class-based-views/index.xhtml#basic-examples' is before the previous link's target in spine order.
• INFO(INF-001): django/docs/_build/epub/Django.epub/nav.xhtml(442,81): The previous rule is under review and its severity may change in a future release. See the discussion at ​https://github.com/w3c/publ-epub-revision/issues/1283

The reason for this warning is that the ToC refers to the sections of topics/class-based-views/index AFTER the included content tree. This makes for a slightly counter intuitive ToC.

This is likely not an issue if reading everyting on the index page before going to the other pages, but it makes the ToC feel counter intuitive. And, following the ToC ordering indicates that you should read the mentioned chapters, and THEN go back again to the index file.

2.

• WARNING(NAV-011): django/docs/_build/epub/Django.epub/nav.xhtml(1377,77): 'toc' nav must be in reading order; link target 'ref/class-based-views/index.xhtml#specification' is before the previous link's target in spine order.
• INFO(INF-001): django/docs/_build/epub/Django.epub/nav.xhtml(1377,77): The previous rule is under review and its severity may change in a future release. See the discussion at ​https://github.com/w3c/publ-epub-revision/issues/1283

Similar to the one above, except in /ref.

3.

WARNING(NAV-011): django/docs/_build/epub/Django.epub/nav.xhtml(1444,59): 'toc' nav must be in reading order; link target 'ref/contrib/index.xhtml#admin' is before the previous link's target in spine order.
INFO(INF-001): django/docs/_build/epub/Django.epub/nav.xhtml(1444,59): The previous rule is under review and its severity may change in a future release. See the discussion at ​https://github.com/w3c/publ-epub-revision/issues/1283

This sections includes first a local ToC to all the contribs, then, still in the index file, a breif description of each. This might make sense when reading, but it makes for a duplicate looking ToC.

Proposed solutions for these warnings:

1. Move the inline sections in the index file to to its own file, and add it to the local ToC. This would solve the ToC issues, though it would require rearranging documentation.
2. Find a way to exclude the inline sections from the ToC. This would make the warnings go away, and would have a ToC that corresponds to reading order. Though, I have not yet found a clean way to do so.

ToC depth issue in iBooks

A last note on the ToC issue in iBooks. iBooks (at least on macOS) can only deal with a ToC depth of 2. It’s also not collapsable. This gives a ToC that is both far to long, and is also not consistent, since sections on the same level can appear on different levels in iBooks and vice versa.

By setting

epub_tocdepth = 2

instead of 3 in conf.py, the ToC in iBooks is both consistent and of good length.

The problem with this approach is the loss of detail in Calibre, that can handle a ToC depth of 3, and where levels are collapsable. I’m leaving this for now, since not everyone uses iBooks. I’m not sure about which epub-readers are most common.

[Fredrik Malmfors]

​PR

[Claude Paroz]

That Easy pickings box is an insult to your awesome work :-)

[Carlton Gibson]

Hi Fredrik.

That Easy pickings box is an insult to your awesome work :-)

Absolutely. Great work. (I ticked Easy pickings... since I thought it an accessible ticket, for someone prepared to do a bit of investigating — perhaps we need a re-name, anyhow...)

TOC depth: Whilst iBooks is popular, I'm reluctant to settle for epub_tocdepth = 2 — it seems like something an ebook reader should handle better. (Perhaps we could create a separate version... Or we may have to fold in the end... — Let's leave this one for just now.)

SVG: I don't think we can just delete the DOCTYPE. As far as I can tell, browsers support SVG 1.1, rather than 2.0.

• ​https://en.wikipedia.org/wiki/Comparison_of_layout_engines_%28Scalable_Vector_Graphics%29
• ​https://caniuse.com/#feat=svg
• ​https://developer.mozilla.org/en-US/docs/Web/SVG/SVG_2_support_in_Mozilla

Removing the doctype lets the validation pass as XML, but that doesn't thereby make them valid SVG. Take one of the errors:

<g id="Canevas_1: Calque 1">

An id attribute cannot be multiple tokens, and 1 can't be a valid id. This isn't correct because we remove the doctype, even though it's valid XML.

I've pinged Omni again. I was hoping for a quick fix, but it might be that's not happening, in which case a few manual edits might be needed here (but probably aren't essential/the best use of human time).

I'll merge the other two commits from your PR as Refs <THIS TICKET>..., then we can look at if there are any other changes we can make (you have a few points not yet addressed).

Beyond that, it looks like we're waiting for Sphinx to update, so we might be able to close this one (taking any last danglers as separate tickets)?

Thanks for the super investigation work! And Welcome Aboard ⛵️

[Carlton Gibson <carlton.gibson@…>]

In 98188cb3:

Refs #20935 -- Removed duplicate ToC references.

These sections are being referred to already from other locations. Having multiple references to the same location from the ToC is ambiguous, and causes errors in the EPUB build.

[Carlton Gibson <carlton.gibson@…>]

In c3ee42d6:

Refs #20935 -- Removed inappropriate crosslink to docs search results page.

The link to the search page does not work in the Epub, since there is no such thing. In the online docs, the link to the search page displays a “No search query given” error, since its purpose is to display results from the search bar. Alone, without a query, it's just empty.

[Carlton Gibson]

OK, left over from comment:14 is:

• Ordering of CBV docs
• Duplication of contib packages in the ToC.

[David Smith]

Replying to Carlton Gibson:

OK, left over from comment:14 is:

• Ordering of CBV docs
• Duplication of contib packages in the ToC.

I've taken a look at the CBV docs. Currently the order is

• sub page 1
• sub page 2
• sub page 3
• heading on current page 1
• heading on current page 2

I've not been able to find a way to re-order the TOC so headings on the current page are first.

One solution could be to use ::rubric and :style: to format headings on the CBV index so they look like a heading but are not included in the TOC.

Another solution would be to add the content on the current CBV index page to a sub page. Could call 'class based view quick start guide', as there is already a sub page called introduction.

Fredrik - are you still working on this?

[David Smith]

I've started a pull request which fixes 'nav-011' errors. I've achieved this through changing some of the subheadings to :rubric: and styling them as headings. If this pull request is merged this leaves the following errors.

{'HTM-003': 72, 'OPF-029': 4, 'PKG-021': 4, 'OPF-073': 2}

HTM-003 & OPF-073 : SVG errors mentioned above. The other two errors (OPF-029 & PKG-021) are new and relate to 4 images which seem to be corrupted.

These errors seem a little odd. On github they seem fine (​https://github.com/django/django/blob/master/docs/_theme/djangodocs/static/docicons-behindscenes.png) but when I view them locally they don't work.

_static/docicons-behindscenes.png
_static/docicons-note.png
_static/docicons-philosophy.png
_static/docicons-warning.png

[Carlton Gibson]

The other two errors (OPF-029 & PKG-021) are new and relate to 4 images which seem to be corrupted.

These files seem OK in my checkout. 🤔

[Carlton Gibson]

OK, decision time. Thanks for your follow-up work here David.

Testing with Sphinx==2.4.3, we're down to just the SVG errors, and these TOC ordering warnings.

• The SVG we can leave for now. There are no reports of actual bad rendering. SVG v2 allows removing the DTD, from the SVGs which causes the errors. We could do that manually but, without any real reports of rendering issues, I'm inclined to wait it out.

• The order is more tricky: Looking at the PR, we can't really use the rubric approach as it removes elements that we want in the TOC.

Another solution would be to add the content on the current CBV index page to a sub page. Could call 'class based view quick start guide', as there is already a sub page called introduction.

I think this would be step back too. The current page is good. It would be replaced with a empty page with just the TOC there... — I don't see that's a win.

The rule is under review:

The previous rule is under review and its severity may change in a future release. See the discussion at ​https://github.com/w3c/publ-epub-revision/issues/1283

It may become a SHOULD. I've left a comment supporting that, and providing a link back here so folks can see the lengths we've gone to.

At this point I'm going to suggest this ticket is fixed, and I want to thank everyone who had put time into it. These validity issues often take a lot of effort. They don't always seem important. But they ensure content is accessible across the widest range of devices, to the widest range of people. That's noble work.

[Carlton Gibson]

Some time later: The ​EPUB spec will indeed be updated to make the spine-order rule a `SHOULD`, so we'll be compliant there. 💃