Switch form rendering to template engine

[Johannes Maron]

We recently switched widgets to use the template engine and template that can be overwritten.
That is currently not the case for forms themselves. More precisely django.forms.BaseForm.as_table and it's siblings.
They use string modulation, that they pass on to the _html_output, which generates the context.

I would propose to change the behavior the following way:
Have _html_output produce a context and don't actually perform a render.
have each as_XYZ method get the context and perform a template based render.

This allows to overwrite or add a new as_XY method and inject something into the context or change the template.

This is also an opportunity to simplify the code. Since templates to render attributes already exist.

[Johannes Maron]

I drafted something, to add more context, to what I am proposing: ​https://github.com/django/django/pull/12133/files

[Simon Charette]

I think that it makes sense. It'd be great if you could do a bit of research on #15667 to figure if this was simply overlooked or if there was a reason for not doing it in the first place.

I think there's also a ticket out there to allow Form.__str__() to use as_p or as_ul instead of as_table by default. If we were using template based rendering we could simply define a django/forms/default.html that {% include "django/forms/table.html" %}, use it in .__str__(), and document that the former must be overridden to achieve the desired behaviour.

[Johannes Maron]

Yes, I'll take some time next week to do some research.
The default idea, is great. That would make it even more versatile, love it!

[Johannes Maron]

With that change I'd also like to drop [error_class]​https://docs.djangoproject.com/en/2.2/ref/forms/api/#customizing-the-error-list-format support, since this would be a template too. However, this is a documented API, though not widely used. I will work around it for now, but I'd really like to remove it.

[Johannes Maron]

Ok, I have a working patch now. It does not change tests, or removes much code to prove its backwards compatible.
However, I'd like to introduce some changes, to improve the resulting architecture and output.

1. Change the position of hidden inputs. Currently, they are injected at a certain index. I'd like to simply amend them. It would greatly simplify the templates. Since they are hidden, it shouldn't change browser rendering.
2. I'd like to deprecate the now unused django.forms.BaseForm._html_output. Yes, it is private, but we have tests running against it and it's referenced in some tickets. This gives me the impression is more commonly used than we'd like.
3. I'd like to deprecate django.forms.boundfield.BoundField.label_tag. The label can be generated with a template with a simple include. I believe this to be the better implementation.
4. Errors I would not change. I am not 100% happy with the design, however, it is based on templates now, which makes adding changes simple.

I haven't written any documentation yet, this will be another monster task. I'd like to get some feedback on the code side first, to avoid back and forth.

[Carlton Gibson]

I'm looking at the PR now, but I'll put this initial comment here, so it doesn't get lost:

Change the position of hidden inputs. Currently, they are injected at a certain index. I'd like to simply amend them. It would greatly simplify the templates. Since they are hidden, it shouldn't change browser rendering.

From experience with Crispy Forms, I'm doubtful that we can just move hidden fields to the end. Rendering, yes, would be unaffected. But the resultant data-dict would not be. With multi-value fields, as first thought, order does matter. (I can't count the number of times we've seen issues resolving around exact rendered field order on Crispy Forms...)

Open to be shown wrong here but... 😬

[Carlton Gibson]

OK, scrub that last comment: existing rendering is already placing hidden fields last, so data-dict is unaffected too.

[Johannes Maron]

Yes, I can confirm this is the case. I actually prefer the new HTML output, since we don't just amend hidden fields within the parent of another field.

So all this needs is a thorough review ;) Can't review it myself, but I am trading :P

[Carlton Gibson]

I'm going to uncheck the Needs Documentation to put this back in the review queue.

[David Smith]

​New PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In 91e8b95d:

Refs #31026 -- Moved Template tests to separate class.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 4ca508a6:

Refs #31026 -- Added extra form render tests.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 456466d:

Fixed #31026 -- Switched form rendering to template engine.

Thanks Carlton Gibson, Keryn Knight, Mariusz Felisiak, and Nick Pope
for reviews.

Co-authored-by: Johannes Hoppe <info@…>

[GitHub <noreply@…>]

In 881a479:

Refs #31026 -- Fixed forms_tests if Jinja2 is not installed.

[Mariusz Felisiak <felisiak.mariusz@…>]

In bc1fa8eb:

[4.0.x] Refs #31026 -- Fixed forms_tests if Jinja2 is not installed.

Backport of 881a4799114fccefbc0f56c6524110ede2682e16 from main

[Mariusz Felisiak <felisiak.mariusz@…>]

In 9be36f8:

Refs #31026 -- Improved BoundField.label_tag() docs.

[Mariusz Felisiak <felisiak.mariusz@…>]

In fc325a19:

[4.0.x] Refs #31026 -- Improved BoundField.label_tag() docs.

Backport of 9be36f8044c3bdfe5d1819d4b3b62bee64a039e3 from main

[Mariusz Felisiak <felisiak.mariusz@…>]

In 78f062f6:

Refs #31026 -- Updated TemplatesSetting docs to refer to forms.

[Mariusz Felisiak <felisiak.mariusz@…>]

In a0e01b00:

[4.0.x] Refs #31026 -- Updated TemplatesSetting docs to refer to forms.

Backport of 78f062f63e7dea09c219fd1310d43950817f4c78 from main

[GitHub <noreply@…>]

In 03a64881:

Refs #31026 -- Changed @jinja2_tests imports to be relative.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 3f5d43ce:

[4.0.x] Refs #31026 -- Changed @jinja2_tests imports to be relative.

Backport of 03a648811615cb623affc2d79dccd4b05919319e from main

[Mariusz Felisiak <felisiak.mariusz@…>]

In 182d25e:

Refs #31026 -- Removed BaseForm._html_output() per deprecation timeline.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 31878b4d:

Refs #31026 -- Removed ability to return string when rendering ErrorDict/ErrorList.

Per deprecation timeline.

[GitHub <noreply@…>]

In f1697ec:

Refs #31026 -- Simplified BaseForm.get_context().

bf.errors returns an ErrorList. Access this directly and avoid creating
a new instance in BaseForm.get_context()

Calling str() on the ErrorList can also be deferred to when the
variable used in the template.