Opened 2 years ago

Last modified 2 years ago

#33899 closed Bug

migrations.RemoveField causes OperationalError "no such column" upon migration when db_index=True — at Version 6

Reported by: cessor Owned by: nobody
Component: Migrations Version: 4.1
Severity: Release blocker Keywords: Migration Sqlite
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by cessor)

Description

I encountered the following error with django 4.1 in my Gitlab CI/CD Pipeline. When I bumped django versions from 4.0.7 to 4.1. my pipeline broke during the testing stage; specifically during db migrations. I have not changed any other source code.

Steps to reproduce

Minimal example attached. Run make green to see that it works with 4.0.7, run make red to see that it does not work with 4.1. It will build and exercise a docker container which installs all dependencies in isolation and sets up an example django app and run migrations.

Manual steps:

  1. Install django 4.1
  2. Create a new project
  3. Create an app
  4. Install app in project
  5. Create a model
  6. Add field on model, set db_index=True
  7. Make migrations: $ python manage.py makemigrations
  8. Remove field from model
  9. Make migrations: $ python manage.py makemigrations
  10. Apply migrations: $ python manage.py migrate

The migration fails with the following error (for an app called web, with a model called Entity with a field called attribute for example):

Running migrations:
Applying contenttypes.0001_initial... OK
...
Applying sessions.0001_initial... OK
Applying web.0001_initial... OK
Applying web.0002_remove_entity_attribute...Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
 return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 357, in execute
 return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: error in index web_entity_attribute_d22c3fcb after drop column: no such column: attribute

Details


The above steps create a set of migrations where at the end a RemoveField migration is produced. Applying this migration fails for fields which had db_index=True. The example I attached uses a SlugField where db_index defaults to True, setting this parameter to False will apply the migration without this error.

I reproduced the error with the following field types: TextField, IntegerField, SlugField, CharField, URLField

Change History (8)

by cessor, 2 years ago

Attachment: django-bug-33899.tgz added

Minimal Example

comment:1 by cessor, 2 years ago

Description: modified (diff)

comment:2 by Mariusz Felisiak, 2 years ago

I'm really puzzled. I cannot reproduce this issue manually, but make red crashes.

in reply to:  2 comment:3 by cessor, 2 years ago

Replying to Mariusz Felisiak:

I'm really puzzled. I cannot reproduce this issue manually, but make red crashes.

Thank you for giving this a try. I investigated this some further and realized that my manual description is not precise enough to reproduce the error. The example I attached uses a SlugField, but the error does not occur with other field types, such as CharFields. I noticed that SlugFields set db_index=True by default (See Link 1), and could reproduce the bug with other fields when setting db_index=True. I will change the bug description accordingly.

Links:

by fizaashraf37, 2 years ago

Attachment: migrations_33899.zip added

Regression test of the bug reported

comment:4 by fizaashraf37, 2 years ago

Easy pickings: set
Has patch: set
Needs tests: set
Triage Stage: UnreviewedAccepted

The issue is valid. I have written the regression test of the exact test case. And found the first bad commit as below:

3702819227fd0cdd9b581cd99e11d1561d51cbeb is the first bad commit
commit 3702819227fd0cdd9b581cd99e11d1561d51cbeb
Author: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Date:   Fri Feb 11 22:21:58 2022 +0100

    Refs #32502 -- Avoided table rebuild when removing fields on SQLite 3.35.5+.

    ALTER TABLE ... DROP COLUMN was introduced in SQLite 3.35+ however
    a data corruption issue was fixed in SQLite 3.35.5.

 django/db/backends/sqlite3/features.py |  2 ++
 django/db/backends/sqlite3/schema.py   | 10 ++++++++++
 tests/schema/tests.py                  | 18 ++++++++++++++++++
 3 files changed, 30 insertions(+)
bisect found first bad commit

comment:5 by cessor, 2 years ago

Description: modified (diff)
Summary: migrations.RemoveField causes OperationalError "no such column" upon migrationmigrations.RemoveField causes OperationalError "no such column" upon migration when db_index=True

comment:6 by cessor, 2 years ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.
Back to Top