Even Faster Django Unit Tests

Last year, I wrote about speeding up django unit tests. With this method, I’ve been able to significantly reduce the time it takes to run my unit tests.

But I still have a real short attention span. If anything, it’s shorter than it was last year. I also have an additional problem: I tend to create databases that have lots of preloaded data. This means that when I run tests or when I’m doing model development, I have to wait for all that data to load. Waiting sucks. Plus, in order to be able to test with the SQLite database, I have to keep my SQL compatible with both the MySQL and sqlite engines (usually not a problem, but sometimes it can be).

Solution? I put my database on a RAM disk. I now have full speed unit tests. If I make a model change, I can toss the whole database and reload everything instantly. When I’m running unit tests, I can run them against the same database engine that I use in production.

Current Method (still a bit rough)

Load up your favorite RAM disk software. I don’t have a favorite yet, but so far I’m having good luck with the free one from http://www.romexsoftware.com.

Create yourself a RAM disk, put it on your favorite drive letter (I use R:\). The disk doesn’t have to be large, since there probably won’t be tons of data on it. I only have 2GB of memory on my laptop, so I made a dainty 64MB RAM disk. (Note: if you are running 32-bit windows and have more than 3GB of RAM, you might be able to make your unusable memory into a RAM disk .. there are some details on the site above).

Make yourself a “Create Ramdisk MySQL.cmd” batch file (see below). Edit the necessary variables so they’ll work on your system, then run the file. Since the contents of ram disks are frequently lost (like when you reboot), the setup here needs to be automated. The batch file does 3 things. It creates the needed folders on the ram disk. It copies over your “mysql” database (so your passwords will still work). And it installs a Windows service for the ramdisk mysql (called MySQLR).

Make a copy of your my.ini file called my_ramdisk.ini (or whatever you like). Edit this new file and change the appropriate data directories to R:\mysql_volatile\data and R:\mysql_volatile\innodb.

If you need to have your normal mysql server running, then alter the my_ramdisk.ini so so the server runs on a different port (you will also have to alter the port number in your settings.py). I only use the mysql server on my computer for development, so can just turn it off before starting the ramdisk mysql.

Start up the ram disk mysql. After running the batch file, you’ll have a new service called MySQLR. Use the service control manager gui, or, since you’re going to be on the command line to run your unit tests anyway, just type sc start mysqlr.

And finally, connect to your new mysql server and create a database for your project.

Listing: “Create Ramdisk MySQL.cmd”

SETLOCAL

REM path to the ramdisk mysql config file
set config=c:\Servers\mysql\my_ramdisk.ini
REM ramdisk drive letter
set ramdisk=R:
REM datadir is your normal mysql data folder
set datadir=C:\data\mysql
REM path to your the mysqld-nt binary
set mysqld=c:\servers\mysql\bin\mysqld-nt.exe

mkdir %ramdisk%\mysql_volatile
mkdir %ramdisk%\mysql_volatile\data

rem copy mysql database to retain permissions
xcopy /ei %datadir%\data\mysql %ramdisk%\mysql_volatile\data\mysql

rem install a service for the ramdisk mysql
%mysqld% --install MySQLR --defaults-file=%config%

Notes:
Remove the MySQLR service by running mysqld-nt.exe --remove MySQLR.

You might want to alter the batch file to copy the django application’s database to the ramdisk as well (saves you having to run that step manually).

Unit tests with the RAM disk MySQL is around 30 times faster than normal MySQL, and only about a second and a half slower than using the SQLite backend.

Advertisements

  1. #1 by Paddy Mullen on January 17, 2010 - 9:26 pm

    do you think you could cook up a solution with the datadir mysql table creation attribute? I have other tables in my local mysql database that I can’t dump to ram.

  2. #2 by Candy Harris on March 6, 2010 - 12:07 pm

    Thanks for your post. I am new at python and this is a big help.

  3. #3 by Cássio Nandi on August 18, 2011 - 4:47 am

    Very Well, nice cheat =)

    You reach the same problem as me (using the sqlite3 for the full suite not work properly). I gonna try this ram-way-of-life in the Postgres. Thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: