"manage.py shell" on a fresh project raises RuntimeWarning about naive datetime, if IPython is installed

[Carl Meyer]

IPython uses SQLite internally to store shell session history. At the beginning and end of every shell session, it issues a SQLite query to get/save the history. For some reason this query includes a (timezone-naive) datetime.

Django's SQLite DB backend registers a global datetime adapter function which issues a warning in case it receives a naive datetime when timezone support is activated.

In combination, this means that on a fresh startproject if you fire up manage.py shell without making a single change to settings, if IPython is installed you'll get "RuntimeWarning: SQLite received a naive datetime (2013-02-04 17:14:13.070091) while time zone support is active." on shell startup and exit. That's an ugly experience for a newcomer to Django - it _looks_ to them as if Django is shipping a broken setup by default.

Probably this is Django's fault for registering a potentially-noisy process-global SQLite adapter, and we should consider whether to just remove that warning in the adapter. The model layer will already warn about naive datetimes, so this warning is only needed for raw SQL.

[Simon Charette]

I've also been experiencing this issue.

[Rohan Jain]

I made an attempt to temporarily hide the warning, shouldn't be much harmful. ​https://github.com/crodjer/django/commit/8b24546f2176d47a3443184728a71b2807d2a168

Quoting the commit message:

It is more of an ugly hack, required because of django injecting
global callback in a database api.

Ideally, instead of registering adapter for datetime.datetime
(django.db.backends.sqlite3.base.py Line 76), django should do it
for a pseudo wrapper over datetime.datetime and use it everywhere,
so that other applications and libraries don't have to worry about
the behaviour being infected.

Also, while trying to debug this, I came across another similar issue (#19789) which involved a DeprecationWarning in IPython. It is probably irrelevant but still decided to link it here.

[Preston Holmes]

​https://github.com/django/django/pull/1404

[Aymeric Augustin]

I disagree with the approach in PR #1404 because it introduces a data corruption bug.

[Aymeric Augustin]

Hiding the warning appears to be the least awful solution at this point. crodjer's patch probably does the job.

[Preston Holmes]

Looking back at that PR, not sure what I was thinking other than maybe I had badly misread Carl's suggestion of removing the warning, as "removing the adapter" - other than that, I can't explain why I was taking that approach.

So to be clear, the main edge case we don't have an solution for, is raw sql issued by user's Django related code, that may use naive datetimes - as these will not generate any warnings as they would in the standard shell?

I think this is worth documenting/warning somehow - but is uncommon enough it shouldn't be too noisy verbose.

[Sylvain_]

This warning is still present in:

$ python --version
Python 2.7.5+

$ python -c "import django; print(django.get_version())"
1.5.4

I'm a python newcomer and I was looking a way to respect $PYTHONSTARTUP. Following #6682 I also installed IPython.

both ipython, and django came from ubuntu packages

$ python-django   1.5.4-1ubuntu1    High-level Python web development framework


$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 13.10
Release:	13.10
Codename:	saucy

this gives:

$ python manage.py shell
/usr/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py:53: RuntimeWarning: SQLite received a naive datetime (2014-01-26 18:40:37.242040) while time zone support is active.
  RuntimeWarning)

/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/embed.py:239: DeprecationWarning: With-statements now directly support multiple context managers
  with nested(self.builtin_trap, self.display_trap):

Python 2.7.5+ (default, Sep 19 2013, 13:49:51) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 

I found a workaround here: ​http://stackoverflow.com/questions/11708821/django-ipython-sqlite-complains-about-naive-datetime

But this fixes only the sqlite warning.

$ ipython profile create
[ProfileCreate] Generating default config file: u'/home/sylvain/.config/ipython/profile_default/ipython_config.py'

and added those lines

import warnings
import exceptions
warnings.filterwarnings("ignore", category=exceptions.RuntimeWarning, module='django.db.backends.sqlite3.base', lineno=53)

rereading the ​tutorial
I found a better way (ipython removed):

export PYTHONSTARTUP=$PWD/_pythonrc
export DJANGO_SETTINGS_MODULE=mysite.settings
$ cat _pythonrc
# Import the model classes we just wrote.
from polls.models import Poll, Choice
from django.utils import timezone

$ python
Python 2.7.5+ (default, Sep 19 2013, 13:49:51) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> Poll.objects.all()
[<Poll: What's up?>]

[Carl Meyer]

I took a look at applying crodjer's patch, but there are two issues I see with it:

1) It only solves half the problem. IPython sends a naive datetime to SQLite both at session start and session end. Crodjer's patch is careful to restore the warnings after session start, but there is still a warning raised at session end.

2) In the latest (1.0+) versions of IPython, the public embed API no longer gives us separate "init" and "start" methods, just a single start_ipython method that does not return until the shell session is ended. This means that we can no longer disable the warning, initialize IPython, and then re-enable the warning before the user actually begins their shell interaction.

We can of course solve both of those problems by simply ignoring the warnings before starting IPython and never restoring them. But this means that someone using raw SQL via the IPython shell could save naive datetimes to their database without getting a warning, which re-opens a (small) window to the data-corruption bug that worried Aymeric above.

It seems that some users are ​fixing this for themselves by unconditionally disabling the warning in settings.py, which is much worse, because it means they could get silent data corruption anywhere in their code at any time, not only when using the IPython shell. So being purist about our response to this may end up putting at least some of our users in an even worse situation with regard to possible data corruption (though of course they are making an explicit choice to put themselves in that situation).

Another ugly alternative (of many ugly alternatives) is that we could fix this in at least recent IPython versions by actually monkeypatching IPython itself to send timezone-aware, rather than naive, datetimes at session start and end. This of course ties us to internal IPython implementation details with no backwards-compatibility guarantees.

In summary: I have no good suggestions for fixing this. I think it is Django who is being a badly-behaved library here by registering process-global adapters in the SQLite library (and/or SQLite is a bad library for exposing this adapter registration as a process-global API rather than a per-connection API), but it seems we are caught between a rock and a hard place.

[Aymeric Augustin]

The landscape has changed quite a bit after e9103402c0fa873aea58a6a11dba510cd308cb84.

I'm working on simplifying conversions of datetimes read from the database. I may be able to fix this bug at the same time.

[Aymeric Augustin]

​https://github.com/django/django/pull/4601 fixes this.

[Aymeric Augustin <aymeric.augustin@…>]

In d9521f6:

Removed global timezone-aware datetime adapters.

Refs #23820.

Fixed #19738.

Refs #17755. In order not to introduce a regression for raw queries,
parameters are passed through the connection.ops.value_to_db_* methods,
depending on their type.

[Tim Graham <timograham@…>]

In e707e4c:

Refs #19738 -- Removed timezone conversion in SQL queries executed outside of the ORM.

Per deprecation timeline.