Cloning test database fails with mariadb-server 8.x

[Stephen Finucane]

I'm attempting to run tests in parallel using MariaDB. When running tests, all access to the additional databases fail. Scrolling back to the top, I see the following error messages:

❯ tox -e py38-django41 -- patchwork.tests.api.test_user patchwork.tests.api.test_event
py38-django41 run-test-pre: PYTHONHASHSEED='631429049'
py38-django41 run-test: commands[0] | python /home/patchwork/patchwork/manage.py test --noinput --parallel -v 2 --timing -- patchwork.tests.api.test_user patchwork.tests.api.test_event
Found 25 test(s).
Creating test database for alias 'default' ('test_patchwork')...
Operations to perform:
  Synchronize unmigrated apps: django_filters, humanize, messages, rest_framework, staticfiles
  Apply all migrations: admin, auth, authtoken, contenttypes, patchwork, sessions, sites
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying contenttypes.0001_initial... OK
  ... {skipped} ...
  Applying sites.0002_alter_domain_unique... OK
Cloning test database for alias 'default' ('test_patchwork')...
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
Cloning test database for alias 'default' ('test_patchwork')...
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
System check identified no issues (0 silenced).
...

The same issue happens on multiple Django versions. With a bit of hacking on the code in the tox venv, I was able to inspect the command that Django is actually executing:

Dumping! cmd = mysqldump --user=patchwork --host=localhost --routines --events test_patchwork

Executing this locally, I see the same issue:

❯ MYSQL_PWD=password mysqldump --user=patchwork --host=localhost --routines --events patchwork
-- MySQL dump 10.13  Distrib 8.0.30, for Linux (x86_64)
--
-- Host: localhost    Database: patchwork
-- ------------------------------------------------------
-- Server version       5.5.5-10.5.16-MariaDB

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!50503 SET NAMES utf8mb4 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `auth_group`
--

DROP TABLE IF EXISTS `auth_group`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!50503 SET character_set_client = utf8mb4 */;
CREATE TABLE `auth_group` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(150) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `auth_group`
--

LOCK TABLES `auth_group` WRITE;
/*!40000 ALTER TABLE `auth_group` DISABLE KEYS */;
/*!40000 ALTER TABLE `auth_group` ENABLE KEYS */;
UNLOCK TABLES;
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)

❯ echo $?
2

I'm using Fedora 36 with the default packages:

❯ cat /etc/system-release
Fedora release 36 (Thirty Six)

❯ sudo dnf list installed | grep mariadb
mariadb-backup.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-common.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-connector-c.x86_64                             3.2.7-1.fc36                        @updates
mariadb-connector-c-config.noarch                      3.2.7-1.fc36                        @updates
mariadb-connector-c-devel.x86_64                       3.2.7-1.fc36                        @updates
mariadb-connector-c-doc.noarch                         3.2.7-1.fc36                        @updates
mariadb-cracklib-password-check.x86_64                 3:10.5.16-1.fc36                    @updates
mariadb-devel.x86_64                                   3:10.5.16-1.fc36                    @updates
mariadb-errmsg.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-gssapi-server.x86_64                           3:10.5.16-1.fc36                    @updates
mariadb-server.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-server-utils.x86_64                            3:10.5.16-1.fc36                    @updates

❯ sudo dnf list installed | grep mysql
community-mysql.x86_64                                 8.0.30-2.fc36                       @updates
community-mysql-common.x86_64                          8.0.30-2.fc36                       @updates
mysql-selinux.noarch                                   1.0.5-1.fc36                        @updates

I had previously reported a similar issue using a combination of mysql-client with an older version of mysql-server or mariadb-server. This was reported as #33537. However, I'm now seeing the issue with the latest versions of both the server and client. As noted in the issue, I can work around this on the command line by adding the --column-statistics=0 argument to mysqldump, however, there doesn't appear to be any mechanism to do this in Django. Alternatively, I can add the following to my.cnf (or equivalent):

/etc/my.cnf.d/mysqldump.cnf
[mysqldump]
column-statistics=0

There's a chance that someone has packaged something wrong in Fedora land and that I'm using an incompatible client and server combo (MySQL-derived client, MariaDB server). However, it's surprising that the tests proceeded to run despite the DB clone failing. As you can see above, mysqldump returned a non-zero exit code so there doesn't appear to be any reason Django couldn't have stopped immediately. I think it would make sense to fail fast at a minimum. If we wanted, we could also add the --column-statistics=0 argument to our call to mysqldump (or provide a mechanism for users to do so), however, this isn't compatible with the 5.7.x version of mysqldump or the version provided by e.g. the mariadb-client package on Ubuntu) so this would have to be a version-specific check.

[Mariusz Felisiak]

There is no reason to open a duplicate. Please add a comment to the original ticket. Also, I don't think you've explained the issue in enough detail to confirm a bug in Django. It looks like a misconfiguration. Please reopen the ticket (#33537) if you can debug your issue and provide details about why and where Django is at fault.

We can also consider your suggestion to check if returncode is non-zero. Personally, I think it makes sense. Please suggest this in the original ticket.

Duplicate of #33537.