I promised to write this blog post long time ago at one of
conferences in Russia. Don't know why I delayed this, but finally
I did.
We, members of MySQL bugs verification group, have to verify bugs
in all currently supported versions. We use not only version
reported, but test in development source tree for each of
supported major versions and identify recent regressions.
You can imagine that even if I would do so for simple bug report
about wrong results with perfect test case, which requires me
simply run few queries I would have to start 4 or more MySQL
servers: one for each of currently supported versions 5.0, 5.1,
5.5 plus one for current development. And unknown number of
servers if I could not repeat or if I want to check if this is
regression.
Even if I have all these basic 4 servers running I still should
type all these queries at least 4 times. How much time it would
take to verify single bug report if I did so?
I know some members of my group preferred this way, because
typing queries manually is same action which our customers do.
Again, some bugs are repeatable only if you type queries
manually.
But I prefer to test manually erroneous exceptions only and don't
make it my routine job.
So how do I test bug reports?
Every version of MySQL server comes with regression test suite: a
program, called mtr (mysql-test-run.pl), its
libraries, mysqltest program (you should not call it
directly, though) and set of tests. Good thing with MySQL test
suite is that you can create your own test cases. So do I.
I write my tests in MTR format, then run MTR with record option
and examine result. Actually this is kind of hack, because users
expected to create result file first, then compare output of
running test with that result file. But my purpose is to repeat
bug report, not to create proper test case for it, so I can be
lazy.
But simply running MTR manually still takes time. And I found a
way to automate this process as well.
I created a BASH script, called do_test.sh, which run
through all my MySQL trees and runs tests for me automatically,
then prints result.
Let me explain it a little bit.
$ cat ~/scripts/do_test.sh
#!/bin/bash
# runs MySQL tests in all source directories
# prints usage information
usage ()
{
echo "$VERSION"
echo "
do_test copies MySQL test files from any place
to each of source directory, then runs them
Usage: `basename $0` [option]... [testfile ...]
or `basename $0` [option]... -d dirname [test ...]
or `basename $0` [option]... [-b build [build option]... ]...
Options:
-d --testdir directory, contains test files
I have a directory, there I store test files. It has subdirectory
t where tests to be run are stored, subdirectory r,
where results, sorted by MySQL server version number, are stored,
and directory named archive, there test are stored for
archiving purpose.
-s --srcdir directory, contains sources directories
This is path to the directory where MySQL package is located. I
called it srcdir, but this is actually not so strict:
program will work with binary packages as well.
-b --build mysql source directory
Name of MySQL source directory. You can specify any package name.
For example, to run tests in 5.6.9 package in my current dir I
call the program as do_test -s . -b
mysql-5.6.9-rc-linux-glibc2.5-x86_64
-c --clean remove tests from src directory after execution
-t --suite suite where to put test
MTR can have test suites with their own rules of how to run test
case. If you want to run your tests in specific suite, specify
this option. You can also have directory for your own suite, but
in this case you need to create directories your_suite,
your_suite/t and your_suite/r in mysql-test/suite
directory of your MySQL installation prior doing this.
As I told I am lazy, so I run tests in main test suite mostly.
This can be not good idea if you use MySQL installation not only
for tests of its own bugs, but for some other tests.
Rest of the code speaks for itself, so I would not explain it.
What you need to do to run this program is simply call it:
do_test.sh and pass paths to your test, src dir and MySQL
installation.
-v --version print version number, then exit
-h --help print this help, then exit
You can also pass any option to mysqltest program.
"
}
# error exit
error()
{
printf "$@" >&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{
This probably not very obvious. These are my default paths and,
most importantly, default set of servers I test
TESTDIR=/home/sveta/src/tests
SRCDIR=/home/sveta/src
BUILDS="mysql-5.0 mysql-5.1 mysql-5.5 mysql-trunk"
CLEAN=0 #false
MYSQLTEST_OPTIONS="--record --force"
TESTS_TO_PASS=""
TESTS=""
SUITE=""
SUITEDIR=""
OLD_PWD=`pwd`
VERSION="do_test v0.2 (May 28 2010)"
}
# parses arguments/sets values to defaults
parse()
{
TEMP_BUILDS=""
while getopts "cvhd:s:b:t:" Option
do
case $Option in
c) CLEAN=1;;
v) echo "$VERSION";;
h) usage; exit 0;;
d) TESTDIR="$OPTARG";;
s) SRCDIR="$OPTARG";;
b) TEMP_BUILDS="$TEMP_BUILDS $OPTARG";;
t) SUITE="$OPTARG"; SUITEDIR="/suite/$SUITE"; MYSQLTEST_OPTIONS="$MYSQLTEST_OPTIONS --suite=$SUITE";;
*) usage; exit 0; ;;
esac
done
if [[ $TEMP_BUILDS ]]
then
BUILDS="$TEMP_BUILDS"
fi
}
# copies test to source directories
copy()
{
cd "$TESTDIR/t"
TESTS_TO_PASS=`ls *.test 2>/dev/null | sed s/.test$//`
cd $OLD_PWD
for build in $BUILDS
do
#cp -i for reject silent overload
cp "$TESTDIR"/t/*.{test,opt,init,sql,cnf} "$SRCDIR/$build/mysql-test$SUITEDIR/t" 2>/dev/null
done
}
# runs tests
run()
{
for build in $BUILDS
do
cd "$SRCDIR/$build/mysql-test"
./mysql-test-run.pl $MYSQLTEST_OPTIONS $TESTS_TO_PASS
done
cd $OLD_PWD
}
# copies result and log files to the main directory
get_result()
{
for build in $BUILDS
do
ls "$TESTDIR/r/$build" 2>/dev/null
if [[ 0 -ne $? ]]
then
mkdir "$TESTDIR/r/$build"
fi
for test in $TESTS_TO_PASS
do
cp "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} "$TESTDIR/r/$build" 2>/dev/null
done
done
}
# removes tests and results from MySQL sources directories
cleanup()
{
if [[ 1 -eq $CLEAN ]]
then
for build in $BUILDS
do
for test in $TESTS_TO_PASS
do
rm "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} 2>/dev/null
rm "$SRCDIR/$build/mysql-test$SUITEDIR/t/$test.test"
done
done
fi
}
# shows results
show()
{
for build in $BUILDS
do
echo "=====$build====="
for test in $TESTS_TO_PASS
do
echo "=====$test====="
cat "$TESTDIR/r/$build/$test".{log,result} 2>/dev/null
echo
done
echo
done
}
E_CDERROR=65
#usage
initialize
parse $@
copy
run
get_result
cleanup
show
exit 0
After I finished with test I copy it to archive directory, again,
with a script, named ar_test.sh:
$ cat ~/scripts/ar_test.sh
#!/bin/bash
# moves MySQL tests from t to archive directory and clean ups r directories
# prints usage information
usage ()
{
echo "$VERSION"
echo "
ar_test copies MySQL test files from t to archive folder
Usage: `basename $0` [-v] [-d dirname] [test ...]
Options:
-d directory, contains test files
-v print version
-h print this help
"
}
# error exit
error()
{
printf "$@" >&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{
TESTDIR=/home/sveta/src/tests
TESTS_TO_MOVE=""
OLD_PWD=`pwd`
VERSION="ar_test v0.2 (Dec 01 2011)"
}
# parses arguments/sets values to defaults
parse()
{
while getopts "vhd:" Option
do
case $Option in
v) echo "$VERSION"; shift;;
h) usage; exit 0;;
d) TESTDIR="$OPTARG"; shift;;
*) usage; exit 0;;
esac
done
TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{
if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
cp "$TESTDIR"/t/* "$TESTDIR"/archive 2>/dev/null
else
for test in $TESTS_TO_MOVE
do
cp "$TESTDIR/t/$test".{test,opt,init,sql} "$TESTDIR"/archive 2>/dev/null
done
fi
}
# removes tests and results from r directories
cleanup()
{
if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
rm "$TESTDIR"/t/* 2>/dev/null
rm "$TESTDIR"/r/*/* 2>/dev/null
else
for test in $TESTS_TO_MOVE
do
rm "$TESTDIR/t/$test".{test,opt,init,sql} 2>/dev/null
rm "$TESTDIR/r/"*"/$test".{test,opt,init,sql} 2>/dev/null
done
fi
}
E_CDERROR=65
initialize
parse $@
copy
cleanup
exit 0
But most important part: what to do if I want to test on some
specific machine which is not available at home? Fortunately, we
have shared machines to run tests on, so I can simply move them
to my network homedir, then choose appropriate machine and run.
Since this is BASH script and test cases in MTR format this would
work on any operating system.
$ cat ~/scripts/scp_test.sh
#!/bin/bash
# copies MySQL tests to remote box
# prints usage information
usage ()
{
echo "$VERSION"
echo "
scp_test copies MySQL test files from t directory on local box to MySQL's XXX
Usage: `basename $0` [-v] [-d dirname] [-r user@host:path] [test ...]
Options:
-d directory, contains test files
-r path to test directory on remote server, default: USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t
-v print version
-h print this help
"
}
# error exit
error()
{
printf "$@" >&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{
TESTDIR=/home/sveta/src/tests
MOVETO='USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t'
TESTS_TO_MOVE=""
OLD_PWD=`pwd`
VERSION="scp_test v0.2 (Dec 1 2011)"
}
# parses arguments/sets values to defaults
parse()
{
while getopts "vhd:" Option
do
case $Option in
v) echo "$VERSION"; shift;;
h) usage; exit 0;;
d) TESTDIR="$OPTARG"; shift;;
r) MOVETO="$OPTARG"; shift;;
*) usage; exit 0;;
esac
done
TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{
if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
scp "$TESTDIR"/t/* "$MOVETO"
else
for test in $TESTS_TO_MOVE
do
scp "$TESTDIR/t/$test".{test,opt,init,sql} "$MOVETO"
done
fi
}
E_CDERROR=65
initialize
parse $@
copy
exit 0
Wanted to put them to Launchpad, but stack with name for this
package. Does anybody have ideas?