You may have noticed that I stopped posting the “weekly news”
from the Mozilla DB Team. After going through the Operations Report
Card and applying it to DBAs in OurSQL Podcast numbers
started thinking that the updates were really more like metrics,
and it would better serve my own purposes better to do the
The purposes of doing this type of blog post are:
0) Answering “So what does a DBA do, anyway?”
1) Answering “DBA? At Mozilla? Does Firefox have a database? Why does Mozilla have databases, and what for?”
2) Showing what the DB team does for Mozilla, so that folks will understand that “just keeping things working” is actually a lot of work. It also helps compile yearly reviews of accomplishments.
We are also starting to get some metrics information. This month we started easy – number of MySQL and Postgres machines, number of unique databases (mysql, information_schema, performance_schema and test are ignored, and duplicates, like the same database on a master and 2 slaves, are ignored), and version information.
As of today, we have 9 unique databases across 8 Postgres servers in 4 clusters, with 6 being Postgres 9.0 and 2 being Postgres 9.2 – we are currently upgrading all our Postgres databases to 9.2 and expect that by the end of December all servers will be using 9.2.
We have 427 unique databases across 98 MySQL DB machines in 20 clusters, with 3 being MySQL 5.0, 71 being MySQL 5.1 (mostly Percona’s patched 5.1), and 24 being MariaDB 5.5.
And in the last week of October and the month of November, we have:
- Documented 4 more of our Nagios checks.
- Started to upgrade Postgres databases to Postgres 9.2
- Decommissioned a legacy database cluster for Firefox
- Built a new database cluster (complete with monitoring and
backups) for a new Sentry implementation.
- Upgraded X machines for general operating system updating
purposes and to ensure that be2net drivers are up-to-date;
out-of-date drivers can (and have!) caused servers to crash.
(b1-db1, b1-db2, addons1, addons2, 3,4,5,
- Upgraded MySQL 5.0 to MySQL 5.1 across X clusters and Y
machines (a01, b02, b2, 7)
- Did a quarterly purge of Crash Stats
- Had to re-sync 6 slaves when a transaction rolled back on the
master, but some of the tables modified were MyISAM. So the
master had data in some tables that was out of sync with the
- Assisted in converting to use UTC timestamps in the
database behind the Mozilla localization portal and the
Bugzilla Anthropology Project, prompting
a blog post on converting timezone-specific
times in MySQL.
- Decommissioned a legacy “production generic” database cluster
that had over 60 databases on it.
- Built a 5th database backup instance due to growing backup
- Changed binary log format to MIXED on our JIRA installation due to JIRA requirements
and an upgrade to MySQL 5.1 issuing warnings that MySQL 5.0 had
- Added checksums to the database cluster that runs Input and
Firefox about:home snippets.
- Archived and dropped the database behind Rock Your Firefox.
- Exported Bugzilla data for a research project. Did you know if you are doing academic research,
you can get a copy of Mozilla’s public Bugzilla data?
- Gave read-only database access to a developer behind the
- Updated the email list for vouched Mozillians.
- Backfilled missing crash-stats data after some failed cron
- Cleared some junk data from the Datazilla databases.
- Added new custom fields to our implementation of Bugzilla for
upcoming release versions: Firefox 20, Thunderbird 20, Thunderbird ESR 20 and seamonkey 217.
- Added 10 new machines to the Graphs database,
and added new sets of machines for Mozilla ESR 17 and Thunderbird
- Gave read-only database access to the two main leads of the
- Debugged and turned off a 10-second timeout in our load
balancing pool that was causing Postgres monitors and processors
to lose connection to their databases.
- Discovered that the plugins database actually does better with the
query_cache turned on, and tuned its
- Tweaked tokudb_cache_size and innodb_buffer_pool_size on our
Datazilla databases so that less swap would be
- Created 2 read/write accounts for 2 people to access the
development database for Mozillians.
- Gave access to Datazilla databases for staging.
Wednesday, Nov 28th was my 1-year anniversary at Mozilla. Tomorrow is December! 2012 went by very quickly.