Model.validate_unique method doesn't take in account multi-db

[Tetsuya Morimoto]

Model.validate_unique() method has 2 methods, _perform_unique_checks() and _perform_date_checks(), which use _default_manager for queryset. But, they are not considered with multi-db, that queryset is as follows.

qs = model_class._default_manager.filter(**lookup_kwargs)

I encountered a problem when ModelForm.is_valid() method is called since BaseModelForm calls full_clean() -> post_clean() -> validate_unique(). I could add/change/delete data with multi-db in AdminSite by setting arbitrary database with "using" keyword. However, the validation uniqueness check is invalid if default database has the same data.

# see also
​http://docs.djangoproject.com/en/dev/topics/forms/modelforms/#the-is-valid-method-and-errors
​http://docs.djangoproject.com/en/dev/ref/models/instances/#validating-objects

I only tested default table operation in AdminSite applying attached patch.
It works for me.

[Tetsuya Morimoto]

_default_manager "using" multi-db

[Adam Nelson]

This looks like a problem on the current milestone:1.2 and therefore is too late for milestone:1.3

[Karen Tracey]

Replying to adamnelson:

This looks like a problem on the current milestone:1.2 and therefore is too late for milestone:1.3

? We're not to the point of pushing bug-fixes out of 1.3 yet, I don't think. Features, yes, are no longer appropriate for the 1.3 milestone. But the description and the word "problem" there make this sound like a bug, not a feature. If it's been around since 1.2 it wouldn't be classified as a blocking bug, but it can still be in the milestone for 1.3. (Which does not guarantee it will get fixed for 1.3, but still it's not correct at this point to say a fix for this could not be put it for 1.3)

[Adam Nelson]

@kmtracey ok, can you mark it as approved at least? It seems a bit late to be committing a patch with no tests and no approval for an issue that currently affects milestone:1.2 but I'm happy to defer if somebody wants to move it forward.

[Ramiro Morales]

Tests (as an additon to the Djago test suite) that don't fail

[Ramiro Morales]

I've added tests (against trunk) that exercise the unique and unique_for_(date,month,year) field options plus the Meta.unique_together option by saving identical model instances to different databases. Saving is successful so I can't reproduce this at the ORM level.

I'm going to close this ticket as works for me. If you can describe better your use case (bonus points if they are a modification to the attached tests) your scenario, please reopen. I suspect this might be a problem in another part of the stack (admin app?) instead, or maybe I forgot to test some condition?.

[Tetsuya Morimoto]

Hi ramiro, thank you for reviewing and making tests code.
Though my explanation was not good, this problem occur only when a Model.validate_unique method is called. So, I fixed your test code to call validate_unique method. By running my test code, you can see ValidationError.

I think what the save method was successed makes you wonder. The reason why the save mehtod is considered with "using" parameter for multi-db.

[Tetsuya Morimoto]

Tests (as an additon to the Django test suite) that occur ValidationError

[Ramiro Morales]

Thanks for showing my tests were wrong. I didn't take in account Model.save() doesn't do validation by default.

[Ramiro Morales]

Patch fifing this issue by t2y plus fixed tests

[Ramiro Morales]

New version of the patch with fixes suggested by Alex

[Tetsuya Morimoto]

Replying to ramiro:

I tested my application(AdminSite) which found this problem originally after it applied Alex's patch. It's OK. Thank you Alex.

[Ramiro Morales]

Russell suggested to avoid modifying ._state.db so test were changed so the fist model is saved to the 'other' DB and the second one to the 'default' DB:

class MultiDbUniqueTests(TestCase):
    multi_db = True

    def test_unique_multi_db_isolation(self):
        "Attempt to save two model instances that don't pass Field.unique checks to different DBs."
        UniqueFieldsModel(unique_charfield='Hello world', unique_integerfield=42, non_unique_field=3).save(using='other')
        b = UniqueFieldsModel(unique_charfield='Hello world', unique_integerfield=42, non_unique_field=3)
        try:
            b.full_clean()
        except ValidationError:
            self.fail("unique field validation shouldn't erroneosuly trigger when the identical model instances are on different databases.")
        else:
            b.save()
            self.assertEqual(UniqueFieldsModel.objects.using('default').count(), 1)
            self.assertEqual(UniqueFieldsModel.objects.using('other').count(), 1)

    def test_unique_together_multi_db_isolation(self):
        "Attempt to save two model instances that don't pass Meta.unique_together checks to different DBs."
        UniqueTogetherModel(cfield='Hello world', ifield=42, efield='user@example.org').save(using='other')
        b = UniqueTogetherModel(cfield='Hello world', ifield=42, efield='user@example.org')
        try:
            b.full_clean()
        except ValidationError:
            self.fail("Meta.unique_together validation shouldn't erroneosuly trigger when the identical model instances are on different databases.")
        else:
            b.save()
            self.assertEqual(UniqueTogetherModel.objects.using('default').count(), 1)
            self.assertEqual(UniqueTogetherModel.objects.using('other').count(), 1)

    def test_unique_for_x_multi_db_isolation(self):
        "Attempt to save two model instances that don't pass Field.unique_for checks to different DBs."
        today = datetime.date.today()
        now = datetime.datetime.now()
        UniqueForDateModel(start_date=today, end_date=now, count=314, order=21, name='Foo').save(using='other')
        b = UniqueForDateModel(start_date=today, end_date=now, count=314, order=21, name='Foo')
        try:
            b.full_clean()
        except ValidationError:
            self.fail("Meta.unique_for_* validation shouldn't erroneosuly trigger when the identical model instances are on different databases.")
        else:
            b.save()
            self.assertEqual(UniqueForDateModel.objects.using('default').count(), 1)
            self.assertEqual(UniqueForDateModel.objects.using('other').count(), 1)

Problem is: In this case these tests don't fail if the changes to django/db/models/base.py are undone.

So it seems there is some deeper problem here.

[Ramiro Morales]

New tests

[Ramiro Morales]

15130.3-tests.diff contains new more granular tests (new model for tests, separated the unique_for_{date,month,year} tests). It runs ten new tests; five of them by saving first an instance to the 'default' database and then trying to save an identical instance to the 'other' DB, and the other five in the inverse order.

Theoretically all of them should fail before applying a fix for this ticket, but only the five tests with the 'default', 'other' order do.

[Jacob]

Milestone 1.3 deleted

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Tim Graham]

#20301 is a duplicate.

[Tim Graham]

#25514 is another duplicate.

[Anssi Kääriäinen]

Hmmh, I think the core problem is that .validate() doesn't take a using argument, while some of the checks are dependent on the used database. I don't think we can fix this without adding the using argument.

If we do that, then ModelForms should likely also have .is_valid(using=...) keyword argument. The next question is should the using argument be used for ModelChoiceField queryset checks, too? I say yes, with the exception that if the queryset has an explicit using argument, then honor that.

We can start with adding using argument to .validate() only, and then continue to further changes if wanted.

Any opinions if we should go forward with this?

[Tim Graham]

Did you see the solution proposed in attachment:15130.2.diff​? That seems on the right track to me. I don't see why the programmer should have to specify the database when we already have database routers to figure out where the model will be saved?

[Anssi Kääriäinen]

Yes, that would be a step in the right direction. But using the write router is just a guess. There is an explicit way to tell which database to use when saving, so there should also be an explicit way to tell Django which database to use when validating.

The solution I see as the right one is:

  1) Use explicitly given database for validation
  2) Use write router's database for validation
  3) Use the default database for validation

And currently, if I'm not mistake, we have this:

  1) Use read router's database for validation
  2) Use the default database for validation
(here 2 is actually implemented by the router for reading)

So, just changing Django to use write router instead of read router as done in the attached patch would be an improvement, but not a complete solution.

[Anssi Kääriäinen]

As an addition - maybe we don't even want the complete solution. Because if we add using to model.validate(), we kind of have to add it to ModelForm.is_clean(), too. And then we have to decide what we do with ModelChoiceField(Author.objects.all()), shouldn't it use the specified database for validation, too?

But it seems usages where you need to validate the same model against multiple databases really aren't that common. So, how much effort should we spend on fixing this, instead of for example trying to add composite foreign key support?

[Michel Perez]

Is this already fixed?, I'm still having this problem using Django==1.10.3 and Python 3.5

[Simon Charette]

Replying to Michel Perez:

Is this already fixed?, I'm still having this problem using Django==1.10.3 and Python 3.5

This has not been fixed yet.

[François Granade]

I got badly bitten by this bug in #33912, and I think that there are two issues, related but not the same, here -

1) the current behavior (validating against the default database for object that live in another database) does not make sense
2) it should be be possible to define against what DB the validation happens.

To me, the first issue is a very serious bug, and the second a useful feature.

What about fixing the first one first ? For that, the patch 15130.2.diff above would work.

What would be needed for this patch to be accepted ? I can clean it up

[François Granade]

Proposing a PR