In the past year, GitHub engineers shipped GitHub Packages, Actions, Sponsors, Mobile, security advisories and updates, notifications, code navigation, and more. Needless to say, the development pace at GitHub is accelerated.
With MySQL serving our backends, updating code requires changes to the underlying database schema. New features may require new tables, columns, changes to existing columns or indexes, dropping unused tables, and so on. On average, we have two schema migrations running daily on our production servers. Some days we have a half dozen migrations to run. We’ll cover how this amounted to a significant toil on the database infrastructure team, and how we searched for a solution to automate the manual parts of the process.
At first …
[Read more]How we wrote our own Service Worker
As we continue the process of reinvigorating Tumblr’s frontend web development, we’re always on the lookout for modern web technologies, especially ones that make our mobile site feel faster and more native. You could have guessed that we are making the mobile dashboard into a progressive app when we open-sourced our webpack plugin to make web app manifests back in August. And you would’ve been right. But to make a high quality progressive web app, you need more than just a web app manifest—you also need a …
[Read more]Last week, GitHub experienced an incident that resulted in degraded service for 24 hours and 11 minutes. While portions of our platform were not affected by this incident, multiple internal systems were affected which resulted in our displaying of information that was out of date and inconsistent. Ultimately, no user data was lost; however manual reconciliation for a few seconds of database writes is still in progress. For the majority of the incident, GitHub was also unable to serve webhook events or build and publish GitHub Pages sites.
All of us at GitHub would like to sincerely apologize for the impact this caused to each and every one of you. We’re aware of the trust you place in GitHub and take pride in building resilient systems that enable our platform to remain highly available. With this incident, we failed you, and we are deeply sorry. …
[Read more]Uber is committed to delivering safer and more reliable transportation across our global markets. To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks…
The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.
Tumblr is a big user of MySQL, and MySQL automation at Tumblr is centered around a tool we built called Jetpants. Jetpants does an incredible job making risky operations safe and reliable, even fairly complex tasks like replacing failed master servers, or splitting a shard.
While Jetpants is an incredibly effective and valuable tool for Tumblr’s day-to-day operation, it has remained very difficult to implement a meaningful testing framework. Integration testing at this level is very challenging. In this article I’ll go through these challenges and how we’ve tackled them at Tumblr.
Requirements
Jetpants operates under the assumption you’re managing MySQL daemons on a fully functional host, and that it can:
- ssh to the target system
- manage processes via service or systemctl commands
- copy data around between systems …
I am always fascinated about the cleanliness of UNIX . One tool only should do one thing, but it has to be the best in that way. The operating system itself will glue all the modules together and give you a complex feel of a system, you don’t have to take care of huge, bloated software, don’t […]
When I joined Pinterest, my first three weeks were spent in Base
Camp, where the newest engineering hires work on real production
issues across the entire software stack. In Base Camp, we learn
how Pinterest is built by building it, and it’s not uncommon to
be pushing code and making meaningful contributions within just a
few days. At Pinterest, newly hired engineers have the
flexibility to choose which team they’ll join, and working on
different parts of the code as part of the Base Camp experience
can help with this decision. Base Campers typically work on a
variety of tasks, but my project was a deep dive into a MySQL
performance optimization project.
Pinterest, MySQL and AWS, oh my!
We work with MySQL running entirely inside Amazon Web Services (AWS). Despite using fairly high-powered instance types with RAID-0 SSDs and a fairly simple workload (many point selects by PK or simple ranges) that peaks around 2,000 …
[Read more]
A good portion of the startups I meet and advise want to use the
newest, hottest technology to build something that’s cool, but
not technologically groundbreaking. I have yet to meet a startup
building a time machine, teleporter or quantum social network
that would actually require some amazing new tech. They have
awesome new ideas with down-to-earth technical requirements, so I
kept wondering why they choose this shiny (and risky) new stuff
when all they need is a good ol’ trustworthy database. I think
it’s because many assume that building the latest and greatest
needs the latest and greatest!
It turns out that’s only one of three bad reasons (traps) why people go for the shiny and new. Reason two is people mistakenly assume older stuff is slow, not feature rich or won’t scale. “MySQL is sluggish,” they say. “Java is slow,” I’ve heard. “Python won’t scale,” they claim. None of it’s true.
…
[Read more]A Pattern for a Newly Hired DBA? I don’t think this experience is unique. It has been shared repeatedly among those starting a job as a DBA (database administrator) at a new company, especially when the organization has never had a dedicated DBA. The conversation usually goes something like this: – “Welcome aboard <insert name here>! Here [...] …
[Read more]