My way to verify MySQL bug reports

I promised to write this blog post long time ago at one of conferences in Russia. Don't know why I delayed this, but finally I did.

We, members of MySQL bugs verification group, have to verify bugs in all currently supported versions. We use not only version reported, but test in development source tree for each of supported major versions and identify recent regressions.

You can imagine that even if I would do so for simple bug report about wrong results with perfect test case, which requires me simply run few queries I would have to start 4 or more MySQL servers: one for each of currently supported versions 5.0, 5.1, 5.5 plus one for current development. And unknown number of servers if I could not repeat or if I want to check if this is regression.

Even if I have all these basic 4 servers running I still should type all these queries at least 4 times. How much time it would take to verify single bug report if I did so?

I know some members of my group preferred this way, because typing queries manually is same action which our customers do. Again, some bugs are repeatable only if you type queries manually.

But I prefer to test manually erroneous exceptions only and don't make it my routine job.


So how do I test bug reports?
Every version of MySQL server comes with regression test suite: a program, called mtr (mysql-test-run.pl), its libraries, mysqltest program (you should not call it directly, though) and set of tests. Good thing with MySQL test suite is that you can create your own test cases. So do I.

I write my tests in MTR format, then run MTR with record option and examine result. Actually this is kind of hack, because users expected to create result file first, then compare output of running test with that result file. But my purpose is to repeat bug report, not to create proper test case for it, so I can be lazy.

But simply running MTR manually still takes time. And I found a way to automate this process as well.

I created a BASH script, called do_test.sh, which run through all my MySQL trees and runs tests for me automatically, then prints result.

Let me explain it a little bit.


$ cat ~/scripts/do_test.sh


#!/bin/bash


# runs MySQL tests in all source directories


# prints usage information


usage ()


{


    echo "$VERSION"


    echo "


do_test copies MySQL test files from any place


to each of source directory, then runs them


Usage: `basename $0` [option]... [testfile ...]


    or `basename $0` [option]... -d dirname [test ...]


or `basename $0` [option]... [-b build [build option]... ]...


Options:


    -d --testdir    directory, contains test files


I have a directory, there I store test files. It has subdirectory t where tests to be run are stored, subdirectory r, where results, sorted by MySQL server version number, are stored, and directory named archive, there test are stored for archiving purpose.


-s --srcdir     directory, contains sources directories


This is path to the directory where MySQL package is located. I called it srcdir, but this is actually not so strict: program will work with binary packages as well.


    -b --build      mysql source directory


Name of MySQL source directory. You can specify any package name. For example, to run tests in 5.6.9 package in my current dir I call the program as do_test -s . -b mysql-5.6.9-rc-linux-glibc2.5-x86_64


-c --clean      remove tests from src directory after execution


    -t --suite      suite where to put test


MTR can have test suites with their own rules of how to run test case. If you want to run your tests in specific suite, specify this option. You can also have directory for your own suite, but in this case you need to create directories your_suite, your_suite/t and your_suite/r in mysql-test/suite directory of your MySQL installation prior doing this.

As I told I am lazy, so I run tests in main test suite mostly. This can be not good idea if you use MySQL installation not only for tests of its own bugs, but for some other tests.

Rest of the code speaks for itself, so I would not explain it. What you need to do to run this program is simply call it: do_test.sh and pass paths to your test, src dir and MySQL installation.


    -v --version    print version number, then exit


    -h --help       print this help, then exit


You can also pass any option to mysqltest program.


    "


}


# error exit


error()


{


    printf "$@" >&2


    exit $E_CDERROR


}


# creates defaults values


initialize()


{


This probably not very obvious. These are my default paths and, most importantly, default set of servers I test


    TESTDIR=/home/sveta/src/tests


    SRCDIR=/home/sveta/src


    BUILDS="mysql-5.0 mysql-5.1 mysql-5.5 mysql-trunk"


    CLEAN=0 #false


    MYSQLTEST_OPTIONS="--record --force"


    TESTS_TO_PASS=""


    TESTS=""


    SUITE=""


    SUITEDIR=""


    OLD_PWD=`pwd`


    VERSION="do_test v0.2 (May 28 2010)"


}


# parses arguments/sets values to defaults


parse()


{


    TEMP_BUILDS=""


    


    while getopts "cvhd:s:b:t:" Option


    do


        case $Option in


            c) CLEAN=1;;


            v) echo "$VERSION";;


            h) usage; exit 0;;


            d) TESTDIR="$OPTARG";;


            s) SRCDIR="$OPTARG";;


            b) TEMP_BUILDS="$TEMP_BUILDS $OPTARG";;


t) SUITE="$OPTARG"; SUITEDIR="/suite/$SUITE"; MYSQLTEST_OPTIONS="$MYSQLTEST_OPTIONS --suite=$SUITE";;


            *) usage; exit 0; ;;


        esac


    done


    if [[ $TEMP_BUILDS ]]


    then


        BUILDS="$TEMP_BUILDS"


    fi


}


# copies test to source directories


copy()


{


    cd "$TESTDIR/t"


TESTS_TO_PASS=`ls *.test 2>/dev/null | sed s/.test$//`


    cd $OLD_PWD


    for build in $BUILDS


    do


        #cp -i for reject silent overload


cp "$TESTDIR"/t/*.{test,opt,init,sql,cnf} "$SRCDIR/$build/mysql-test$SUITEDIR/t" 2>/dev/null


    done


}


# runs tests


run()


{


    for build in $BUILDS


    do


        cd "$SRCDIR/$build/mysql-test"


./mysql-test-run.pl $MYSQLTEST_OPTIONS $TESTS_TO_PASS


    done


    cd $OLD_PWD


}


# copies result and log files to the main directory


get_result()


{


    for build in $BUILDS


    do


        ls "$TESTDIR/r/$build" 2>/dev/null


        if [[ 0 -ne $? ]]


        then


            mkdir "$TESTDIR/r/$build"


        fi


        for test in $TESTS_TO_PASS


        do


cp "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} "$TESTDIR/r/$build" 2>/dev/null


        done


    done


}


# removes tests and results from MySQL sources directories


cleanup()


{


    if [[ 1 -eq $CLEAN ]]


    then


        for build in $BUILDS


        do


            for test in $TESTS_TO_PASS


            do


rm "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} 2>/dev/null


rm "$SRCDIR/$build/mysql-test$SUITEDIR/t/$test.test"


            done


        done


    fi


}


# shows results


show()


{


    for build in $BUILDS


    do


        echo "=====$build====="


        for test in $TESTS_TO_PASS


        do


            echo "=====$test====="


cat "$TESTDIR/r/$build/$test".{log,result} 2>/dev/null


            echo


        done


        echo


    done


}


E_CDERROR=65


#usage


initialize


parse $@


copy


run


get_result


cleanup


show


exit 0


After I finished with test I copy it to archive directory, again, with a script, named ar_test.sh:


$ cat ~/scripts/ar_test.sh


#!/bin/bash


# moves MySQL tests from t to archive directory and clean ups r directories


# prints usage information


usage ()


{


    echo "$VERSION"


    echo "


ar_test copies MySQL test files from t to archive folder


Usage: `basename $0` [-v] [-d dirname] [test ...]


Options:


    -d    directory, contains test files


    -v    print version


    -h    print this help


    "


}


# error exit


error()


{


    printf "$@" >&2


    exit $E_CDERROR


}


# creates defaults values


initialize()


{


    TESTDIR=/home/sveta/src/tests


    TESTS_TO_MOVE=""


    OLD_PWD=`pwd`


    VERSION="ar_test v0.2 (Dec 01 2011)"


}


# parses arguments/sets values to defaults


parse()


{    


    while getopts "vhd:" Option


    do


        case $Option in


            v) echo "$VERSION"; shift;;


            h) usage; exit 0;;


            d) TESTDIR="$OPTARG"; shift;;


            *) usage; exit 0;;


        esac


    done


    


    TESTS_TO_MOVE="$@"


}


# copies test to source directories


copy()


{


    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]


    then


cp "$TESTDIR"/t/* "$TESTDIR"/archive 2>/dev/null


    else


        for test in $TESTS_TO_MOVE


        do


cp "$TESTDIR/t/$test".{test,opt,init,sql} "$TESTDIR"/archive 2>/dev/null


        done


    fi


}


# removes tests and results from r directories


cleanup()


{


    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]


    then


        rm "$TESTDIR"/t/* 2>/dev/null


        rm "$TESTDIR"/r/*/* 2>/dev/null


    else


        for test in $TESTS_TO_MOVE


        do


rm "$TESTDIR/t/$test".{test,opt,init,sql} 2>/dev/null


rm "$TESTDIR/r/"*"/$test".{test,opt,init,sql} 2>/dev/null


        done


    fi


}


E_CDERROR=65


initialize


parse $@


copy


cleanup


exit 0


But most important part: what to do if I want to test on some specific machine which is not available at home? Fortunately, we have shared machines to run tests on, so I can simply move them to my network homedir, then choose appropriate machine and run. Since this is BASH script and test cases in MTR format this would work on any operating system.


$ cat ~/scripts/scp_test.sh


#!/bin/bash


# copies MySQL tests to remote box


# prints usage information


usage ()


{


    echo "$VERSION"


    echo "


scp_test copies MySQL test files from t directory on local box to MySQL's XXX


    


Usage: `basename $0` [-v] [-d dirname] [-r user@host:path] [test ...]


Options:


    -d    directory, contains test files


-r    path to test directory on remote server, default: USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t


    -v    print version


    -h    print this help


    "


}


# error exit


error()


{


    printf "$@" >&2


    exit $E_CDERROR


}


# creates defaults values


initialize()


{


    TESTDIR=/home/sveta/src/tests


    MOVETO='USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t'


    TESTS_TO_MOVE=""


    OLD_PWD=`pwd`


    VERSION="scp_test v0.2 (Dec 1 2011)"


}


# parses arguments/sets values to defaults


parse()


{    


    while getopts "vhd:" Option


    do


        case $Option in


            v) echo "$VERSION"; shift;;


            h) usage; exit 0;;


            d) TESTDIR="$OPTARG"; shift;;


            r) MOVETO="$OPTARG"; shift;;


            *) usage; exit 0;;


        esac


    done


    


    TESTS_TO_MOVE="$@"


}


# copies test to source directories


copy()


{


    if [[ "xx" = x"$TESTS_TO_MOVE"x ]]


    then


        scp "$TESTDIR"/t/* "$MOVETO"


    else


        for test in $TESTS_TO_MOVE


        do


scp "$TESTDIR/t/$test".{test,opt,init,sql} "$MOVETO"


        done


    fi


}


E_CDERROR=65


initialize


parse $@


copy


exit 0


Wanted to put them to Launchpad, but stack with name for this package. Does anybody have ideas?