Accelerating mysql PHPUnit and Behat Tests with RAMDISK: Episode II – Please support me!

Lately I was trying to test whether using ramdisk was actually faster with phpunit and behat tests. The trick is to use ramdisk. In ubuntu there is already a mounted folder for this which is `/run` we can take a look at what is mounted running `mount` command from the cli:

tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/ramdisk type tmpfs (rw,size=4096M)

We notice the 3 first lines tells us about this default ramdisk folders. We add the fourth one by editing `/etc/fastab` and adding a new line:

none           /run/ramdisk tmpfs      defaults,size=4096M 0 0

My RAM is 8GB so I pass half. This line will mount that folder /run/ramdisk into ramdisk for fast access. But before we go into more we have to stop the mysql service, backup our /var/lib/mysql folder and move it all to ramdisk so it can work off ramdisk.

~ sudo service mysql stop
~ sudo mv /var/lib/mysql /var/lib/mysql_backup
~ sudo cp -rp /var/lib/mysql_backup /run/ramdisk/mysql
~ sudo ln -s /run/ramdisk/mysql /var/lib/mysql
~ chown -R mysql:mysql /run/ramdisk

For the above to work we have to tell ubuntu it is ok for mysql to look into the other dirs under run:

~ sudo pico /etc/apparmor.d/usr.sbin.mysqld
// and add these two lines
  /run/ramdisk/mysql/ r,
  /run/ramdisk/mysql/** rwk,
~ sudo service apparmor restart
~ sudo service mysql restart

Once all is working is time for the automation script:

~ cat /etc/init.d/ramdisk 
#! /bin/sh
# /etc/init.d/ramdisk.sh
#

case "$1" in
 start)
   echo "Copying files to ramdisk"
   cd /run
   tar --lzop -xvf /var/ramdisk-mysql/ramdisk-backup.tar.lzop
   echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched from HD >> /var/log/ramdisk_sync.log
   ;;
 sync)
   echo "Synching files from ramdisk to Harddisk"
   echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
   cd /run
   mv -f /var/ramdisk-mysql/ramdisk-backup.tar.lzop /var/ramdisk-mysql/ramdisk-backup-old.tar.lzop
   tar --lzop -cvf /var/ramdisk-mysql/ramdisk-backup.tar.lzop ramdisk
   ;;
 stop)
   echo "Synching logfiles from ramdisk to Harddisk"
   echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
   tar --lzop -cvf /var/ramdisk-mysql/ramdisk-backup.tar.lzop ramdisk
   ;;
 *)
   echo "Usage: /etc/init.d/ramdisk {start|stop|sync}"
   exit 1
   ;;
esac
 
exit 0

And run to get at it on boot time (currently with some difficulties but following more instructions on this site from where I adapted the script can help initializing it manually):

~ update-rc.d ramdisk defaults 00 99

And add the job to the crontab:

crontab -e
2 * * * * root        /etc/init.d/ramdisk sync >> /dev/null 2>&1

And that is it now you are ready to run it by yourself. Some quick results just drop by a minute or so but in extensive tests it goes much faster. And even faster with optimizations and SSDs :D.

Time: 04:29, Memory: 404.75Mb
Time: 05:25, Memory: 434.75Mb

I have a request, please donate so i can afford training in SymfonyLive in San Francisco. Currently the price for training is 290EUR. And I already spent a lot on my air ticket. If you would like me to share more tips and continue to blog please support me here is the link http://bit.ly/9xbyzM . Thanks!

Behat Parallelization Series: Episode I – Setting Out For San Francisco

Sometimes is time to parallelize work. This is needed because you want to speed up development by having two folders or more. The same project, same files almost, but different branches in each folder, therefore, different development stages and testing at the same time. How can this be possible since there could be many conflicts? Keep reading.

~ cp -rf ~/sites-2/project1 ~/sites-2/project1-running-tests
~ /sites-2/project1 (feature/branch1)
~ /sites-2/project1-running-tests (develop)

This implies you have copy the same folder twice and they are now totally independent. You of course are pushing to the same git repository but they are coming from different development stages. There could be total decoupling when you have set db names for each instance different, and of course also vhost settings have to be different as well in order not to interfere between runs. This last can be achieved with some scripting and randomization (more on that later).

You also want your behat suite to run on the background so no popups or anything shows up to distract you from your work. That is achieved with xvfb. How to set it up with all the misinformation? Keep reading.

Copy this script into `/etc/init.d/xvfb` :

#!/bin/bash
### BEGIN INIT INFO
# Provides:          Xvfb
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start Xvfb.
# Description:       Start the X virtual framebuffer.
### END INIT INFO

XVFB=/usr/bin/Xvfb
XVFBARGS=":5 -nolisten tcp -fbdir /var/run"
PIDFILE=/var/run/xvfb.pid
case "$1" in
    start)
        echo -n "Starting virtual X frame buffer: Xvfb"
        start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
        echo "."
    ;;
    stop)
        echo -n "Stopping virtual X frame buffer: Xvfb"
        start-stop-daemon --stop --quiet --pidfile $PIDFILE
        echo "."
    ;;
    restart)
        $0 stop
        $0 start
    ;;
    *)
        echo "Usage: service xvfb {start|stop|restart}"
        exit 1
esac
 
exit 0

Run then:

sudo chmod u+x /etc/init.d/xvfb
sudo update-rc.d xvfb defaults
sudo service xvfb restart

Now tell sahi you want it to run into :5 display with a new setup called chromeBg (Bg for background):

~ pico ~/sahi/userdata/config/browser_types.xml 
// and add a browser type entry:
        <browserType>
                <name>chromeBg</name>
                <displayName>ChromeBg</displayName>
                <icon>chrome.png</icon>
                <path>env DISPLAY=:5 /usr/bin/google-chrome</path>
                <options>--user-data-dir=$userDir/browser/chrome/profiles/sahi$threadNo --proxy-server=localhost:9999 --disable-popup-blocking</options>
                <processName>chrome</processName>
                <capacity>5</capacity>
        </browserType>

Then make sure you tell your behat configuration behat.yml or other in symfony2 to have a sahi browser:

mink:
    // ...
    goutte:     true
    sahi:       ~
    browser_name: chromeBg

Now that your behat tests are running in one folder then you can calmly work on your other folder without interruptions and interference by fixing tests, doing more develop, etc. On our next series we will consider this in the greater scheme of things.

I have a request, please donate so i can afford training in SymfonyLive in San Francisco. Currently the price for training is 290EUR. And I already spent a lot on my air ticket. If you would like me to share more tips and continue to blog please support me here is the link http://bit.ly/9xbyzM . Thanks!

Looking forward to more!

Set Symfony2 Environment Straight For Your Branch

After switching from work on a branch to a different branch to run tests on it you need quickly a fresh environment ready to go. Your environment is dirty and will mess and confuse things if you don’t rewind and wipe correctly. If you are like me and don’t want to type then good news!

I wrote a script you can try if you are yet not well convinced:

~ curl -s https://raw.github.com/gist/fbbcb4bf61f498fa8ec6 | sh -s dev

If you like it then you can download it and copy it to a file like:

~ wget https://raw.github.com/gist/fbbcb4bf61f498fa8ec6
~ mv fbbcb4bf61f498fa8ec6 set
~ chmod +x ./set

And you can run it easily in your projects like:

~ ./set dev // to reset dev db according to config_dev.yml
~ ./set test // to reset test db according to config_test.yml
...

soon more good news!

PHPStorm Symfony2 View Feature

PHPStorm on its latest EPA 5.0 reveals an interesting feature. Feature works even for 2.0.x projects provided you add an `app/SystemRequirements.php` file from a composer 2.1 symfony2 project directory. In order to enable it you would have to go under `File >> Settings >> Framework Integration` and select `Symfony2` as `name` and `Framework root directory` as your top project directory. After this you would see a tab on the left to have the view `Symfony 2 View` which is totally independent from Project type views.

Here are some screenshots: