You may have noticed that I stopped posting the “weekly news”
from the Mozilla DB Team. After going through the Operations Report
Card and applying it to DBAs in OurSQL Podcast numbers
111,
112,
114,
115 and
116, I
started thinking that the updates were really more like metrics,
and it would better serve my own purposes better to do the
updates monthly.
The purposes of doing this type of blog post are:
0) Answering “So what does a DBA do, anyway?”
1) Answering “DBA? At Mozilla? Does Firefox have a database? Why
does Mozilla have databases, and what for?”
2) Showing what the DB team does for Mozilla, so that folks will
understand that “just keeping things working” is actually a lot
of work. It also helps compile yearly reviews of
accomplishments.
We are also starting to get some metrics information. This month
we started easy – number of MySQL and Postgres machines, number
of unique databases (mysql, information_schema,
performance_schema and test are ignored, and duplicates, like the
same database on a master and 2 slaves, are ignored), and version
information.
As of today, we have 9 unique databases across 8 Postgres servers
in 4 clusters, with 6 being Postgres 9.0 and 2 being Postgres 9.2
– we are currently upgrading all our Postgres databases to 9.2
and expect that by the end of December all servers will be using
9.2.
We have 427 unique databases across 98 MySQL DB machines in 20
clusters, with 3 being MySQL 5.0, 71 being MySQL 5.1 (mostly
Percona’s patched 5.1), and 24 being MariaDB 5.5.
And in the last week of October and the month of November, we
have:
- Documented 4 more of our Nagios checks.
- Started to upgrade Postgres databases to Postgres 9.2
- Decommissioned a legacy database cluster for Firefox
themes.
- Built a new database cluster (complete with monitoring and
backups) for a new Sentry implementation.
- Upgraded X machines for general operating system updating
purposes and to ensure that be2net drivers are up-to-date;
out-of-date drivers can (and have!) caused servers to crash.
(b1-db1, b1-db2, addons1, addons2, 3,4,5,
- Upgraded MySQL 5.0 to MySQL 5.1 across X clusters and Y
machines (a01, b02, b2, 7)
- Did a quarterly purge of Crash Stats
data.
- Had to re-sync 6 slaves when a transaction rolled back on the
master, but some of the tables modified were MyISAM. So the
master had data in some tables that was out of sync with the
slaves.
- Assisted in converting to use UTC timestamps in the
Elmo
database behind the Mozilla localization portal and the
Bugzilla Anthropology Project, prompting
a blog post on converting timezone-specific
times in MySQL.
- Decommissioned a legacy “production generic” database cluster
that had over 60 databases on it.
- Built a 5th database backup instance due to growing backup
needs.
- Changed binary log format to MIXED on our JIRA installation due to JIRA requirements
and an upgrade to MySQL 5.1 issuing warnings that MySQL 5.0 had
not.
- Added checksums to the database cluster that runs Input and
Firefox about:home snippets.
- Archived and dropped the database behind Rock Your Firefox.
- Exported Bugzilla data for a research project. Did you know if you are doing academic research,
you can get a copy of Mozilla’s public Bugzilla data?
- Gave read-only database access to a developer behind the
Datazilla project.
- Updated the email list for vouched Mozillians.
- Backfilled missing crash-stats data after some failed cron
scripts.
- Cleared some junk data from the Datazilla databases.
- Added new custom fields to our implementation of Bugzilla for
upcoming release versions: Firefox 20, Thunderbird 20, Thunderbird ESR 20 and seamonkey 217.
- Added 10 new machines to the Graphs database,
and added new sets of machines for Mozilla ESR 17 and Thunderbird
ESR 17.
- Gave read-only database access to the two main leads of the
Air Mozilla
project.
- Debugged and turned off a 10-second timeout in our load
balancing pool that was causing Postgres monitors and processors
to lose connection to their databases.
- Discovered that the plugins database actually does better with the
query_cache turned on, and tuned its
size.
- Tweaked tokudb_cache_size and innodb_buffer_pool_size on our
Datazilla databases so that less swap would be
used.
- Created 2 read/write accounts for 2 people to access the
development database for Mozillians.
- Gave access to Datazilla databases for staging.
Wednesday, Nov 28th was my 1-year anniversary at Mozilla.
Tomorrow is December! 2012 went by very quickly.