= Custom Form Fields =

This page will document my efforts to create a custom form field for my application.  Hopefully someone will find it useful, contribute, correct my mistakes or post some pointers.

==  The Goal ==

As I'd like my database driven app to be driven by a database and not a flat file I made sure that my model was properly normalised.  One model deals with recording information about applications and services and so needs to deal with ports numbers and IP protocols.  Since many ports could have many applications I've got a model like this:

{{{
#!python
class Port(meta.model):
    number = meta.IntegerField()
    protocol = meta.CharField(maxlength=4)

class Application(meta.model):
    name = meta.CharField(maxlength=50)
    version = meta.IntegerField()
    ports = meta.ManyToManyField(Port)

}}}

The 'ports' table is prepopulated with all possible combinations of number+protocol and as such weighs in at around 130000 rows.  Each request for an add or change manipulator therefore takes quite a while (as all the rows are read in) and is not very usable.

What I'd like is something similar to the raw admin id but without having to enter ids.  I'd like the user to be able to enter values like so: 80:tcp, 443:tcp, icmp.

Now what I could do is just add a text box to the form, process it in the view and then populate the M2M field myself but that feels like cheating.  Plus there's a few other models where something like this but with more functionality would prove useful.

== Tagging ==

A quick Google made me think that someone had beaten me to it.  I found this post: [http://feh.holsman.net/articles/2005/10/24/django-and-many2many-tables Django and M2M tables] which points to an example of adding tagging to a model.  This seems to provide the answer but it doesn't quite fit my needs, mainly because the whole choices list is still being pulled in.

== Findings ==

After much messing about I've found that things are a lot simpler than first appeared.  In order to validate the input I've subclassed TextField and added a new validation method to its validator list.  I created a new render method which simply turns the list into something more presentable.  I then created a new convert_post_data method which takes '80:tcp, 443:tcp' or whatever and turns it into a list of ids from the joined table.

This now appears to work correctly for adding but not editing.  This is due to the render method getting passed the user input or a list of IDs depending on what manipulator gets called.  I can modify the user's data to change it to a list of IDs but this would need doing in the view and doesn't feel 'neat' enough.

=== Updates ===

2006-02-20-16:20-ish

Looks like do_html2python is our friend here.  In line with the [wiki:NewAdminChanges new admin documentation] we need to make sure that do_html2python is called regardless of whether there are any errors.  In doing so we can get our list of IDs in a 'neat' manner.  The only problem?  What if we've got validation errors? Trying to convert the data fails when there are errors.  Which means we can't convert.  Which means that render doesn't get its nice list of IDs anymore.  Stumped.

2006-02-24-23:14-ish

While watching Firefly I've come up with this:

{{{
#!python
class FormPortField(formfields.TextField):

    requires_data_list = True

    def __init__(self, **kw):
        formfields.TextField.__init__(self, **kw)
        self.validator_list.append(self.isValidList)

    def isValidList(self, field_data, all_data):
        for datum in field_data:
            for port in datum.split(','):
                if port.lower().strip() != 'icmp':
                    try:
                        no, proto = port.split(':')
                    except ValueError:
                        raise validators.ValidationError, _("Ports should be specified in the form 'number:protocol'.  E.g. '80:tcp'")
                    try:
                        if int(no) < 1 or int(no) > 65355:
                            raise validators.ValidationError, _("Port number must be between 1 and 65355")
                    except ValueError:
                        raise validators.ValidationError, _("Port number must be between 1 and 65355")
                    if proto.lower().strip() not in ['tcp', 'udp']:
                        raise validators.ValidationError, _("Protocol must be either 'TCP' or 'UDP'")

    def render(self, data):
        if data is None:
            data = ''
        if isinstance(data, unicode):
            data = data.encode(DEFAULT_CHARSET)
        tmp = []
        for datum in data:
            try:
                port = ports.get_object(id__exact=int(datum))
                if port.number:
                    tmp.append("%s:%s" % (port.number, port.protocol))
                else:
                    tmp.append("%s" % (port.protocol, ))
            except ValueError:
                # By catching this error we now assume that it's invalid data entered by the user and
                # so should simply be returned in the 'raw'
                return '<input type="text" id="%s" class="v%s%s" name="%s" size="%s" value="%s" />' % \
                (self.get_id(), self.__class__.__name__, self.is_required and ' required' or '',
        data = ', '.join(tmp)
        return '<input type="text" id="%s" class="v%s%s" name="%s" size="%s" value="%s" />' % \
            (self.get_id(), self.__class__.__name__, self.is_required and ' required' or '',
            self.field_name, 40, escape(data))

    def convert_post_data(self, new_data):
        name = self.get_member_name()
        if new_data.has_key(self.field_name):
            d = new_data.getlist(self.field_name)
            try:
                self.isValidList(d, new_data)
            except validators.ValidationError:
                return
            portlist=[]
            for datum in d:
                for n in datum.split(','):
                    n = n.strip()
                    if n:
                        if n.lower() == 'icmp':
                            portlist.append(ports.get_object(protocol__exact="icmp").id)
                        else:
                            port, proto = n.split(':')
                            port_ref = ports.get_object(number__exact=int(port), protocol__exact=proto.lower())
                            portlist.append(port_ref.id)
            new_data.setlist( name, portlist)
}}}

Wow.  What a mess.  We end up validating the data three times over.  What a pain.  But it works.

=== Update 15.03.2006 ===

After much searching and fiddling and wondering I've decided that I went about this all wrong.  Custom form fields are not the answer to my problem, manipulators are.  This is the code I'm now using:

{{{
#!python
class ApplicationAddManipulator(applications.AddManipulator):

    def __init__(self):
        applications.AddManipulator.__init__(self)
        newfields = []
        for field in self.fields:
            if field.field_name == 'ports':
                field = formfields.TextField(field_name='ports', length=30, maxlength=200, validator_list=[self.isValidPortList])
            newfields.append(field)
        self.fields = newfields

    def isValidPortList(self, field_data, all_data):
        for port in field_data.split(','):
            if port.lower().strip() != 'icmp':
                try:
                    no, proto = port.split(':')
                except ValueError:
                    raise validators.ValidationError, _("Ports should be specified in the form 'number:protocol'.  E.g. '80:tcp'")
                try:
                    if int(no) < 1 or int(no) > 65355:
                        raise validators.ValidationError, _("Port number must be between 1 and 65355")
                except ValueError:
                    raise validators.ValidationError, _("Port number must be between 1 and 65355")
                if proto.lower().strip() not in ['tcp', 'udp']:
                    raise validators.ValidationError, _("Protocol must be either 'TCP' or 'UDP'")

    def save(self, new_data):
        portIds = []
        if new_data['ports'] and new_data['ports'].strip().lower():
            for port in new_data['ports'].split(','):
                if port.strip().lower() == 'icmp':
                    portIds.append(ports.get_object(protocol__exact='icmp').id)
                else:
                    no, proto = port.strip().lower().split(':')
                    portIds.append(ports.get_object(protocol__exact=proto, number__exact=int(no)).id)
            new_data['ports'] = portIds
        return applications.AddManipulator.save(self, new_data)
}}}

Here's what the code actually does.  The new custom manipulator subclasses the standard AddManipulator for the Application model.  When a new instance is created (i.e. the __init__ method is called) then the AddManipulator's fields are copied and the 'ports' multiple-select box replaced with a standard text field.  The field has a custom validator attached to it and the save method is overridden to convert the user's input into a valid list of ports.

Why is this better?  Validation is only done where appropriate - in the 'isValidPortList' function and nowhere else.

Are there still issues?  Yep.  I still don't like the calling of 'do_html2python' when data could be invalid.  This really doesn't appear to make sense to me.  A lot of the manipulator code will change when Magic Removal hits trunk.  To what extent this will correct my concerns remains to be seen.