14 August 2024
I recently had a need to copy about 18M rows from Mysql to a sqlite database. The mysql dump measured about 1.2GB.
First, dump the data from mysql using the following:
mysqldump -uroot -p --compatible=ansi --skip-extended-insert --compact --single-transaction --no-create-info schema table > table_dump.sql
This creates a file of individual insert statements. The key to loading these quickly in sqlite, is to load them all as a single transaction mentioned in the faq. It also makes sense to allocate additional cache memory for sqlite. Additional pragma options can be used to disable the journal and avoid waiting on disk, but I did not need to use these to get the speed I needed.
PRAGMA cache_size = 400000;
BEGIN;
.read table_dump.sql
END;