ArrayField doesn't support .update() and F() objects

[ris]

On a model along the lines of

class ModelA ( models.Model ):
    name = TextField ()
    name_array = ArrayField ( models.TextField () )

If you perform

>>> ModelA.objects.update ( name_array = [ F ( "name" ) ] )

You end up with

>>> ModelA.objects.all ()[0].name_array
[u'F(name)']

I see what you've done there django, but that's not quite what I meant.

[Tim Graham]

I am not sure that F expressions can be supported like that, but if not, it might be worth documenting the restriction and/or throwing a better error message rather than silently coercing to string.

[Marc Tamlyn]

The silent coercion to string is normal behaviour for Django actually in many cases - we actually would like ModelA.objects.update(name_array=[1, 2]) to work (as it would if it were a CharField.

Supporting F objects like that is a very nice idea, but I'm not sure how possible it is. If we do implement it, it should work for Hstore and Json as well. In these cases there are more complex options - for example you can update just a single key of the Hstore without affecting the rest, which would be a very useful addition.

A simpler option could be to have a distinct API for it - something like ModelA.objects.update(name_array=FArray(0, F(name)) which would generate an update statement like UPDATE app_modela SET name_array[0]=F(name). This API could also be a bit clever, and also support ModelA.objects.update(name_array=FArray([F('name')])). We would then have similar FJSON and FHstore objects. I'm pretty sure these could be supported without changes to the core ORM by making them look sufficiently like F objects.

[Adam Johnson]

I have written a similar separate F API for the the comma-separated fields in Django-MySQL - see ​http://django-mysql.readthedocs.org/en/latest/model_fields.html#listf-expressions . I made a ListF class for doing single atomic add/remove operations on either end of the list, which is about all that can be done with MySQL's comma-separated-list parsing capabilities.

It is obviously much less involved than Postgres' arrays, but maybe it will inspire. I think add/remove operations are quite useful to consider on top of the 'set element at position X to Y' and 'set array field to [Y]' operations.

[Josh Smeaton]

This seems like a very similar problem to how F() and Sum() evolved separately when they could have been the same with a little (lot..) more work. Marc, could you require that the inputs to Array/HStore be expressions (with an appropriate _parse_expression() to wrap basic values in Value())? If you provide support for expressions only, then getting basic values in comes fairly naturally too.

I haven't looked into the implementation of these fields though, so I don't know how feasible it would be to support.

[Johannes Maron]

There is more broken here:

ModelA.objects.update(name_array=F("name_array") + 'some item')

This should result in:

...
SET "myapp_modela"."name_arry" = array_append("myapp_modela"."name_arry", 'some item')
...

not in:

...
SET "myapp_modela"."name_arry" = ("myapp_modela"."name_arry" + 'some item')
...

[Frederik Creemers]

I'm not that familiar with the internals, so I don't know if my use case helps flesh out ideas for how to implement this, but here's what I have:

I have a score_weights list, which is an array of floating point numbers.Every User has a "scores" field, containing an array with the same number of items. I'd like to sort users by a weighted sum of these scores, with the weights coming from this score_weights list. In other words, the users are ordered by the dot product of their scores array and the score_weights array.

order_expression = -1 * sum([
    F("scores__%s" % i) * score_weights[i]
    for i in xrange(len(crit.score_weights))
])

Currently, this raises this error:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 824, in execute_sql
    sql, params = self.as_sql()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 369, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 47, in pre_sql_setup
    order_by = self.get_order_by()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 308, in get_order_by
    self.query, allow_joins=True, reuse=None)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 209, in resolve_expression
    for expr in c.get_source_expressions()
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 404, in resolve_expression
    c.rhs = c.rhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 403, in resolve_expression
    c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 403, in resolve_expression
    c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 403, in resolve_expression
    c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 403, in resolve_expression
    c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 404, in resolve_expression
    c.rhs = c.rhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 403, in resolve_expression
    c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/expressions.py", line 463, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1462, in resolve_ref
    self.get_initial_alias(), reuse)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1402, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1370, in names_to_path
    " not permitted." % (names[pos + 1], name))
FieldError: Cannot resolve keyword u'0' into field. Join on 'scores' not permitted.

[Baptiste Mispelon]

Duplicate of #26355

(even though this ticket is older, the other one has a pull request which is why I'm closing this one)