Amazon is releasing new products & services to it’s global cloud compute network at a rate that has all of our heads spinning. Join 32,000 others and follow Sean Hull on twitter @hullsean. Here’s new stuff worth mentioning around databases & data. 1. For ETL – AWS GLUE Moving data from your transactional MySQL or … Continue reading What products & improvements are new on AWS? →
10 Older Entries »
When moving your MySQL instances to AWS, you would probably have to deal with this decision. This article will highlight some of the key factors that need to be considered in order to help you determine which is the most cost-effective solution for serving your data.
RDS is based on EC2, so in this post I will focus on the benefits and disadvantages of using the former, versus migrating MySQL into user-managed EC2 instances.
The value of RDS resides in simplified provisioning and administration. Several of the most common maintenance tasks such as minor version upgrades, backups, and slave creation are automated and can be managed from the AWS console. On the other hand, RDS imposes some limitations to MySQL functionality so it could be offered “as a Service” while delivering a safe and consistent user experience. Within those limitations we can mention:
- Storage …
When working with Amazon AWS Aurora, there are some steps to consider when trying to get data out of an active Aurora master into a slave, potentially into a EC2 instance or offsite in another data centre. Creating an external mysql to Aurora gives the option to move out of Aurora, or to have the flexibility to move data around as desired. With AWS RDS instances this task is pretty simple because you can do the following :
- Create a read replica
- Stop the slave process
- Capture the positioning
- Dump the database
With Aurora it’s a little trickier, because a read replica in Aurora has no slave process. All of the replication is handled on the back end and cannot be controlled. However, setting up an external slave can be done.
Amazon AWS Documentation
In …[Read more]
In this blog, we’ll use some test results to look at the rationale for using provisioned IOPS volumes for AWS databases.
One piece of advice you often hear running MySQL, MongoDB or other databases in the AWS EC2 environment is that you should use volumes with provisioned IOPs. This kind of makes sense on the “marketing” level, where provisioned IOPS (io1) volumes are designed for IO-intensive database workloads, while General Purpose (gp2) volumes are not. But if you go to the AWS volume type description, you will find that gp2s are shown to have pretty good IO performance. So where do all these supposed database performance problems for Amazon Elastic Block Store (EBS), with no provisioned IOs, come from?
Here is what I found out running experiments with a beta of …[Read more]
Some time ago, I published the article on AWS Aurora Benchmarking (AWS Aurora Benchmarking – Blast or Splash?), in which I analyzed the behavior of different solutions using synchronous replication in AWS environment. This blog follows up with some of the comments and suggestions I received regarding that post from the community and Amazon engineers.
I decided to perform another round of tests, keeping in mind comments and suggestions received.
I presented some of the results during the Percona conference in Santa Clara last April 2016. The following is the transposition that presentation, with more details.
Not interested in the preliminary descriptions? Go to the results section
Why new tests?…[Read more]
Testing functionalities and options for a database can be challenging at times, as a live production environment might be required. As I was looking for different options, I was directed by Derek Downey to this post in the Percona blog.
The blog discussed an interesting and fun tool from Percona, tpcc-mysql. I was interested in testing the tool so I decided to play around with it in an AWS EC2 server.
In this post I will expand on the Percona blog post, since the tool lacks documentation, as well as explain how I used it to create a MySQL Benchmark in AWS.
There are various reasons why tpcc-mysql could be a good …[Read more]
One of the most common engagements I do is working with firms in and around the NYC startup sector. I evaluate AWS infrastructures & applications built in the Amazon cloud. Join 32,000 others and follow Sean Hull on twitter @hullsean. I’ve seen some patterns in customers usage of Amazon. Below is a laundry list of … Continue reading 5 core pieces of the Amazon Cloud puzzle to get your project off the ground → …[Read more]
There’s a strong trend to automation across the cloud. That’s a great thing for startups because it reduces operational headaches & lets them focus on building products. Join 31,000 others and follow Sean Hull on twitter @hullsean. But as that trend begins to touch the database tier, all sorts of complications emerge. Let’s take a … Continue reading When hosting data on Amazon turns bloodsport →
When I joined Pinterest, my first three weeks were spent in Base
Camp, where the newest engineering hires work on real production
issues across the entire software stack. In Base Camp, we learn
how Pinterest is built by building it, and it’s not uncommon to
be pushing code and making meaningful contributions within just a
few days. At Pinterest, newly hired engineers have the
flexibility to choose which team they’ll join, and working on
different parts of the code as part of the Base Camp experience
can help with this decision. Base Campers typically work on a
variety of tasks, but my project was a deep dive into a MySQL
performance optimization project.
Pinterest, MySQL and AWS, oh my!
We work with MySQL running entirely inside Amazon Web Services (AWS). Despite using fairly high-powered instance types with RAID-0 SSDs and a fairly simple workload (many point selects by PK or simple ranges) that peaks around 2,000 …[Read more]
Greetings everyone! Let’s discuss a 3rd Party auditing solution to MySQL and how we can leverage MongoDB® to make sense out of all of that data.
The McAfee MySQL Audit plugin does a great job of capturing, at low level, activities within a MySQL server. It does this through some non-standard APIs which is why installing and configuring the plugin can be a bit difficult. The audit information is stored in JSON format, in a text file, by default.
There is 1 JSON object for each action that takes place within MySQL. If a user logs in, there’s an object. If that user queries a table, there’s an object. Imagine 1000 active connections from an application, each doing 2 queries per second. That’s 2000 JSON objects per second being written to the audit log. After 24 hours, that would be almost 173,000,000 audit entries!
How does one make sense of that many JSON objects? One option would be to write your own parser in …[Read more]
10 Older Entries »