Unable to create template tag which behaves similar to {% verbatim %}

[Jacob Rief]

Currently it is impossible to create custom template tags which behave similar to {% verbatim %} .
The reason is in Lexer.create_token(). There, the class member self.verbatim is set for template blocks in "verbatim" state. It is impossible to turn on that state from outside, ie. a template tag.

Fixing this, would be as simple as changing if block_content[:9] in ('verbatim', 'verbatim ') to if block_content.startswith('verbatim') or to if block_content[:9] in ('verbatim', 'verbatim ', 'verbatim_').
Then all template tags beginning with verbatim..., would start in "verbatim" state. I don't assume this will break any existing code; who starts the name of a templatetag with 'verbatim...' if not for that purpose?

Background information why this is useful:
Templates syntax for Django and AngularJS is very similar, and with some caveats it is possible to reuse a Django template for rendering in AngularJS. I therefore attempted to add a context sensitive variation of the verbatim tag to this app ​https://github.com/jrief/django-angular, but was hindered by this issue.

BTW: This part of the Django code did not change from 1.6 up to master.

[None]

I am creating template tag and need get code between {% mytag %}...{% endmytag %}.
I read code and saw that similar template tag "verbatim" is hardcode using in Lexer.
This is bad idea because nobody can't create template like "verbatim".
Please, improve this thing and make django template system more convenient and flexible.

[Jacob Rief]

BTW: In the meantime I found another solution for the described problem.

[Tim Graham]

I'm open to a solution, although I'm not sure if the change proposed in the description is ideal.

[Atul Bhouraskar]

The Lexer.create_token() method can be greatly simplified by removing the hard coded logic for self.verbatim and instead saving the full token_string in the Token for tags.

Currently the Lexer.create_token() logic returns tokens of type TokenType.TEXT if it encounters a {% verbatim %} tag. This change removes this logic, ie. tokens are created as normal, instead a simple Parser.parse_verbatim() method can be implemented to return verbatim text in the parsing stage. This can be called by any tag instead of calling Parser.parse().

I've created a quick pull request ​https://github.com/django/django/pull/14686 with the above implemented - all template tests pass.

This approach fully removes the 'special case' for the built in verbatim tag and allows creating custom tags that need similar behaviour.
The simplified Lexer.create_token() should now be easier to maintain and probably also faster than the existing code (though I haven't profiled this).

[Atul Bhouraskar]

​PR

[Chris Jerdonek]

Currently the Lexer.create_token() logic returns tokens of type TokenType.TEXT if it encounters a {% verbatim %} tag. This change removes this logic, ie. tokens are created as normal,

It seems like making this change would make it harder to fix #23424 and might even be incompatible with fixing it. The reason is that changing things in this way would require tokens to be parsed before finding the end of the verbatim tag, whereas fixing that bug would require looking for the end of the verbatim tag before attempting to parse tokens in between. Thus, I would suggest fixing the bug in #23424 (and adding test cases) before reworking the implementation into a way that might be incompatible with fixing it.

[Atul Bhouraskar]

Thanks Chris, I wasn't aware of #23424.

In fact that ticket would be much simpler to fix using this approach - I've updated my pull request to make a possible fix for #23424 possible (and added tests that expose the bug).

I've also created a proof of concept fix for #23424 based of this fix - ​https://github.com/django/django/pull/14786 - all tests pass in that one.

[Chris Jerdonek]

In fact that ticket would be much simpler to fix using this approach

It seems more complicated to me, IMO, and a lot less efficient because you need to be re-parsing multiple times. With the other approach, it would just be one pass.

[Chris Jerdonek]

If the use cases for this ticket were spelled out in more detail, I feel like it would be easier to discuss other ways of solving it. Currently, it's not clear what requirements a solution needs to satisfy. It seems like there should be a solution that preserves the "one pass" nature of parsing. For example, maybe there can be a way to register verbatim tags so that the list consulted in create_token() will no longer be hard-coded.

[Mariusz Felisiak]

I agree with Chris that re-parsing multiple times is not an acceptable solution. IMO we should take a step back and try to fix #23424 without considering ​PR14686, then we can rethink solution for this ticket.