14 August 2024

Load a mysql table into sqlite quickly

I recently had a need to copy about 18M rows from Mysql to a sqlite database. The mysql dump measured about 1.2GB.

First, dump the data from mysql using the following:

mysqldump -uroot -p --compatible=ansi --skip-extended-insert --compact --single-transaction --no-create-info schema table > table_dump.sql

This creates a file of individual insert statements. The key to loading these quickly in sqlite, is to load them all as a single transaction mentioned in the faq. It also makes sense to allocate additional cache memory for sqlite. Additional pragma options can be used to disable the journal and avoid waiting on disk, but I did not need to use these to get the speed I needed.

PRAGMA cache_size = 400000;

BEGIN;
.read table_dump.sql
END;
blog comments powered by Disqus