Fix refreshing of related fields when one field is deferred

[Adam Johnson]

In #35044, we fixed the issue where loading a deferred field would clear related objects.

But it left a bug where refreshing all fields explicitly with refresh_from_db() would no longer clear related objects *if* the model instance had any deferred fields.

This would work on Django 5.0 but is broken on 5.1:

book = Book.objects.defer('title').get(pk=1)
assert book.author is not None  # fetch related object
Author.objects.all().delete()

book.refresh_from_db()
assert book.author is None  # should have been reset by the refresh

Fix incoming.

[Sarah Boyce]

I've not been able to replicate following the instructions on Django 5.1.3