The theme this week was “data integrity and security”:

• Upgraded the kernel in our Puppet Dashboard DB master/slave cluster after one of the machines crashed.
  
• Verified that none of our databases were running a version of MySQL that was affected by the serious MySQL authentication bypass vulnerability discovered, patched/fixed and reported by MariaDB. (On a personal note, the way they handled finding, fixing and reporting the bug makes me happy that they are such good community stewards)
  
• We removed duplicate entries from our Bugzilla instance that were put in when assumptions were made about nonexistent unique keys.
  
• Continued the database consistency check for bugzilla.
  
• Backed up 2 more database clusters
  
• Dealt with our development instance of developer.mozilla.org having some latin1 MyISAM tables – converted them to utf8 InnoDB.
  
• Added permissions for the metrics team to read new fields in Bugzilla.
  
• Did a database consistency check for addons after a duplicate key error was triggered on several slaves, and re-sync’d the databases that had issues.
  
• Repaired a crashed table that looked like it worked, other than not showing recent inserts.
  
• Gave the Bugzilla team a place to test development releases of their code (as opposed to production releases)
  
• Updated the Firefox Flicks staging database with a fresh copy of the database from development.
  
• Decommissioned 2 database servers put into production for Addons due to 2 other machines that were crashing. The other 2 machines are healed, and the temporary servers were running out of disk space (they were smaller because they were temporary). We are back to having an n+2 configuration (without the 2 servers, we were exactly at n servers, meaning we had exactly as many servers as we needed. We like to run n+2, for just such an occasion)