Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#32424 closed Uncategorized (wontfix)

in-memory test database does not work with multiprocessing.apply_async

Reported by: Tata Krushna Chaitanya Owned by: Arkaprabha Chakraborty
Component: Database layer (models, ORM) Version: 3.0
Severity: Normal Keywords: sqlite3, threading, tests
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hi,
I am using Django 3.0 (On Ubuntu 18.04 installed using pip3). The below example works only with threading but not with multiprocessing.apply_async. Here is my sample, Looking at the Django code I have made sure that all requirements are met, I am using py3.6 and the sqlite3 version is also met.

$ python3 -c "from sqlite3 import dbapi2 as Database; print
(Database.__name__, Database.sqlite_version_info)"

sqlite3.dbapi2 (3, 22, 0)
import os

os.environ["DJANGO_SETTINGS_MODULE"] = "settings"
import datetime, threading
from multiprocessing import Pool

# django  stuff
import django

django.setup()
from polls.models import *
from django.core.mail import mail_admins
from django.test.utils import *
from django.db import connection

def create_object():
    print("Creating Poll")
    p = Question()
    p.question_text = "What's up doc ?"
    p.pub_date = datetime.date.today()
    p.save()
    print("Poll object saved. Id: %d" % p.id)


MODULE = 1

if __name__ == "__main__":
    setup_test_environment()
    old_db_name = "db.sqlite3"
    new_db_name = connection.creation.create_test_db(verbosity=1)
    print("New DATABASE:", new_db_name)
    if MODULE == 0:
        t = threading.Thread(target=create_object)
        t.start()
        t.join()
    elif MODULE == 1:
        with Pool(2) as p:
            p.apply_async(create_object)
    else:
        create_object()
    obj = Question.objects.get()
    print(obj.question_text)
    teardown_test_environment()
    connection.creation.destroy_test_db(old_db_name)

Change History (4)

comment:1 by Tata Krushna Chaitanya, 4 years ago

Similar to https://code.djangoproject.com/ticket/12118 but with multiprocess.apply_async instead of normal threads.

comment:2 by Arkaprabha Chakraborty, 4 years ago

Owner: changed from nobody to Arkaprabha Chakraborty
Status: newassigned

comment:3 by Simon Charette, 4 years ago

Resolution: wontfix
Status: assignedclosed

I fear like this won't be possible to do per the in-memory nature of such databases.

It's one thing to serialize queries between multiple threads in a single process but it's another one to coordinate it across multiple processes that need to share the same memory.

Closing as wontfix as nothing seems to indicate this is even possible and Django's core doesn't have any use for it.

comment:4 by Tata Krushna Chaitanya, 4 years ago

Thanks, Simon. It's a bit obscure use case (Django + multiprocessing) but albiet an important one as the test will fail, currently, I am working around with making the DB updates manually from the tests, we can revisit if there are more users for this.

Note: See TracTickets for help on using tickets.
Back to Top