== The new Options API proposal

As of my 2014 Summer of Code project, my second deliverable is a refactored working implementation of the Options API.
The Options API is at the core of Django, it enables introspection of Django Models with the rest of the system. This includes lookups, queries, forms, admin to understand the capabilities of every model. The Options API is hidden under the _meta attribute of each model class.
Options has always been a private API, but Django developers have always been using it in their projects in a non-official way. This is obviously very dangerous because, as there are no official endpoints, Options could change breaking other people's implementation. Options did not have any unit-tests, but the entire system uses it and relies on it to work correctly.
My Summer of Code project is all about understanding and refactoring Options to make it a testable and official API that Django and any other developer can use.

=== Current state of the API
I now have a working and tested implementation of Options, I have managed to simplify 20+ functions and reduce them to 2 main endpoints, that are the main API. Because Options needs to be very fast, I necessarily had to add some accessors on Options for the most common calls (although both endpoints are cached, we can increase speed by avoiding function calls). Each accessor is a cached property and is computed, using the new API, on first access.

For this reason, I am planning to release in attached PR:
 - Unit tests for the new Meta API
 - The new Meta API
 - The implementation of the new API throughout django and django.contrib

=== Concepts


==== Field types
There are 5 main types of fields:
 
===== Data fields
A data field is any field that has an entry on the database, for example a CharField, BooleanField, a ForeignKey


{{{
class Person(models.Model):
    # DATA field
    data_abstract = models.CharField(max_length=10)
}}}


===== M2M fields
A M2M field that is defined on the current model

{{{
class Person(models.Model):
    # M2M fields
    friends = models.ManyToManyField('self', related_name='friends', symmetrical=True)
}}}


===== Related Object
A Related Object is a one-to-many relation from another model (such as a ForeignKey) that points to the current model
{{{
class City(models.Model):
    name = models.CharField(max_length=100)

class Person(models.Model):
    # M2M fields
    city = models.ForeignKey(City)
}}}
In this case, City has a related object from Person (as you can access person_set)

===== Related M2M
A Related M2M is a M2M relation from another model that points to the current model
{{{
class City(models.Model):
    name = models.CharField(max_length=100)

class Person(models.Model):
    # M2M fields
    cities_lived_in = models.ManyToManyField(City)
}}}
 
In this case, City has a related m2m from Person

===== Virtual
Virtual fields do not necessarily have an entry on the database, they are "Django fields" such as a GenericRelation
{{{
class Person(models.Model):
    content_type = models.ForeignKey(ContentType, related_name='+')
    object_id_ = models.PositiveIntegerField()
    item = GenericForeignKey('content_type', 'object_id')
}}}
GenericForeignKey uses content_type and object_id to keep track of what model type and id is set by item, but item itself does not have a concrete presence on the database.
In this case, item is a virtual field.

==== Field options
There are 5 properties that each field can have:

===== Local
A local field is one that is defined on the queries model and is not derived from inheritance.
Fields from models that directly inherit from abstract models or proxy classes are still local 

{{{
class Person(models.Model):
  name = models.CharField(max_length=50)

class Londoner(Person):
  overdraft = models.DecimalField()
}}}

Londoner has two fields (name and overdraft) but only one local field (overdraft)

===== Hidden
Hidden fields are only referred to related objects and related m2m. When a relational model (such as ManyToManyField, or ForeignKey) specifies a related_name that starts with a "+", it tells Django to not create a reverse relation.
{{{
class City(models.Model):
    name = models.CharField(max_length=100)

class Person(models.Model):
    city = models.ForeignKey(City, related_name='+')
}}}

In this case, City has a related hidden object from Person (as you can't access person_set)

===== Concrete
Concrete fields are fields that have a column

===== Proxied relations
Proxied relations are when concrete models inherit all related from their proxies.  

{{{
class Person(models.Model):
    pass

class ProxyPerson(Person):
    class Meta:
        proxy = True

class RelationToProxy(models.Model):
     proxy_person = models.ForeignKey(ProxyPerson)
}}}

In this case, Person has no related objects, but it has 1 proxied related object from RelationToProxy.

=== The new API
The new API is composed of 2 main functions: get_fields, and get_field.

===== get_fields

{{{
    def get_fields(self, m2m=False, data=True, related_m2m=False, related_objects=False, virtual=False,
                       include_parents=True, include_non_concrete=True, include_hidden=False, include_proxy=False, export_map=False):
}}}

get_fields takes a set of flags as parameters, and returns a tuple of field instances that match those parameters. All possible combinations of
options are possible here, although some will have no effect (such as include_proxy combined with data or m2m by itself).
get_fields is internally cached for speed and a recursive function that collects fields from each parent of the model.
An example of every (sane) combination of flags will be available in the model_meta test suite that I will ship with the new API.
The 'export_map' key is only used internally (by get_field) and is not part of the public API. 'export_map=True' will return an OrderedDict with fields
as keys and a tuple of strings as values. While the keys map exactly to the same output as 'export_map=False', the tuple of values will contain all
possible lookup names for that field. This is used to build a fast lookup table for get_field and to avoid re-iterating over every field to pull
out every possible name.

{{{
    >>> User._meta.get_fields() # Only data by default
    (<django.db.models.fields.AutoField: id>,
     <django.db.models.fields.CharField: password>,
     <django.db.models.fields.DateTimeField: last_login>,
     <django.db.models.fields.BooleanField: is_superuser>,
     <django.db.models.fields.CharField: username>,
     <django.db.models.fields.CharField: first_name>,
     <django.db.models.fields.CharField: last_name>,
     <django.db.models.fields.EmailField: email>,
     <django.db.models.fields.BooleanField: is_staff>,
     <django.db.models.fields.BooleanField: is_active>,
     <django.db.models.fields.DateTimeField: date_joined>)

    >>> User._meta.get_fields(data=False, related_objects=True) # only related_objects
    (<RelatedObject: admin:logentry related to user>,)

    >>> User._meta.get_fields(data=False, related_objects=True
                                  include_hidden=True) # only related_objects including hidden
    (<RelatedObject: auth:user_groups related to user>,
     <RelatedObject: auth:user_user_permissions related to user>,
     <RelatedObject: admin:logentry related to user>)
}}}

===== get_field

{{{
    def get_field(self, field_name, m2m=True, data=True, related_m2m=False, related_objects=False, virtual=False)
}}}

'get_field' returns a field_instance from a given field name. field_name can be anything from name, attname and related_query name.
get_field is recursive by default and does not include any hidden or proxied relations. There has still not been any reason to add these
and they can be derived from 'get_fields'.
If a given name is not found, it will raise a FieldDoesNotExist error.
'get_field' is internally cached and gets all field information from 'get_fields' internally.

NOTE: There is an inconsistency between the defaults of get_field and get_fields. 'get_fields' by default enables only data fields
while 'get_field' by default enables data and m2m. This is because of backwards-compatibility issues (get_field already existed).

{{{
    >>> User._meta.get_field('username') # A data field
    <django.db.models.fields.CharField: username>

    >>> User._meta.get_field('logentry', related_objects=True) # A related object
    <RelatedObject: admin:logentry related to user>

    >>> LogEntry._meta.get_field('user') # ForeignKey can be queried by field name
    <django.db.models.fields.related.ForeignKey: user>
    >>> LogEntry._meta.get_field('user_id') # .. and also by database column name
    <django.db.models.fields.related.ForeignKey: user>

    >>> User._meta.get_field('does_not_exist') # A non existent field
    *** FieldDoesNotExist: User has no field named 'does_not_exist'
}}}


=== The Decision Process

Since I started my Summer of Code project, this API has gone through several designs, and has now finalised onto the one shown above.
The API has gone through many transformations. Each decision has gone through my mentor, with whom I have weekly meetings (Russell).

==== Using bitfields as flags

get_field and get_fields were originally designed to work with bits. The main choice for this decision was because there were many options and,
in order to avoid providing multiple flags, it would be better to provide bits.
The original API for bits is:

{{{
    DATA = 0b00001
    M2M = 0b00010
    RELATED_OBJECTS = 0b00100
    RELATED_M2M = 0b01000
    VIRTUAL = 0b10000

    # Aggregates
    NON_RELATED_FIELDS = DATA | M2M | VIRTUAL
    ALL = RELATED_M2M | RELATED_OBJECTS | M2M | DATA | VIRTUAL

    NONE = 0b0000
    LOCAL_ONLY = 0b0001
    CONCRETE = 0b0010
    INCLUDE_HIDDEN = 0b0100
    INCLUDE_PROXY = 0b1000

    def get_fields(types, opts)
}}}

There are numerous reasons why we backed away from this design:
1) There is always a need to import flags from models/options, this can bring to circular dependencies
2) Importing flags all the time can also be a nuinsance
2) Importing flags is not Pythonic at all

The decision taken was to port 'get_field' and 'get_fields' to flags.
A port of the old implementation lies here if you are interested: https://github.com/PirosB3/django/blob/soc2014_meta_refactor_upgrade/django/db/models/options.py

==== Removed direct, m2m, model
In the previous API, it was a common pattern to return model, direct (bool), m2m (bool). I soon realized that not only
these three paramenters can be easily derived from a field_instance, but there were very few places that actually used some of
the attributes (there is only 1 place where m2m is used).

The decision taken was to drop direct, m2m, model in the return type and only keep field_instance. All the rest will be derived.

==== Removed all calls "with_model"
As said previously, it is redundant to include any model as this can be derived.

==== Removed the need of multiple maps
The previous implementation relied on many different cache maps internally. This is somewhat necessary, but tends to increase bug-risk
when cache-expiry happens. For this reason, my implementation relies only on 2 cache tables, and I have added a specific function to do
cache expiry (called _expire_cache) that will wipe out all memory.
The downsides if this aspect is that we cache a bit more naively (there are less layers of caching) but benchmarks show this does not
decrease performance.