Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 58 Next 28 Older Entries

Displaying posts with tag: sharding (reset)

Shard-Query loader gets a facelift and now Amazon S3 support too
+0 Vote Up -0Vote Down

Shard-Query (source) now supports the MySQL “LOAD DATA INFILE” command.

When you use LOAD DATA LOCAL INFILE a single threaded load from the current process will be performed.  You can specify a path to a file anywhere readable by the PHP script.  This allows loading without using the Gearman workers and without using a shared filesystem.

If you do not specify LOCAL, then the Gearman based loader is used.  You must not specify a path to the file when you omit the LOCAL keyword.  This is because the shared path will the pre-pended to the filename automatically.  The shared path must be a shared or network filesystem (NFS,CIFS,etc) and the files to be loaded must be placed on the shared filesystem for the Gearman

  [Read more...]
MySQL Fabric Release Candidate – adding High Availability and Scaling to MySQL
Employee +4 Vote Up -0Vote Down


MySQL Fabric is a new framework that adds High Availability (HA) and/or scaling-out for MySQL. This is the third in a series of posts on the new MySQL Fabric framework; the first article (MySQL Fabric – adding High Availability to MySQL) explains how MySQL Fabric can deliver HA and then stepped through all of the steps to configure and use it. The second (MySQL Fabric – adding Scaling to MySQL)

  [Read more...]
MySQL Fabric – adding Scaling to MySQL
Employee +3 Vote Up -0Vote Down

MySQL Fabric is a new framework that adds High Availability (HA) and/or scaling-out for MySQL. This is the second in a series of posts on the new MySQL Fabric framework; the first article (MySQL Fabric – adding High Availability to MySQL) explained how MySQL Fabric can deliver HA and then stepped through all of the steps to configure and use it.

This post focuses on using MySQL Fabric to scale out both reads and writes across multiple MySQL Servers. It starts with an introduction to scaling out (by partitioning/sharding data)

  [Read more...]
Oracle’s Mats Kindahl to weave MySQL Fabric into Percona Live session
+2 Vote Up -0Vote Down

Mats Kindahl of Oracle is lead developer of MySQL Fabric

MySQL Fabric is an integrated framework for managing farms of MySQL servers with support for both high-availability and sharding. Its development has been spearheaded by Mats Kindahl, senior principal software developer in MySQL at Oracle.

Mats is leading the MySQL Scaling and High-Availability effort covering the newly released MySQL Fabric and the MySQL Applier for Hadoop. He is also the architect and implementer of several features (mostly replication features), including the row-based replication available in 5.1 and the binary log group commit available in MySQL 5.6. Before starting MySQL he earned a doctoral degree in the area of automated

  [Read more...]
MaxScale has now its own public irc channel
+1 Vote Up -0Vote Down

MaxScale is a Proxy for the MySQL protocol built with a modular architecture. The underlying concept of modules allows to extend the MaxScale proxy services. The current version implements Read Write splitting and Connection Load Balancing. Internally MySQL queries go through a SQL parsing phase. This gives MaxScale great capabilities regarding queries routing.

So if [...]

MaxScale, ProxySQL and MySQL Proxy
+2 Vote Up -0Vote Down

At FOSDEM 2014 ProxySQL and MaxScale were both presented. Both are proxy that can help build sophisticated MariaDB/MySQL architectures. But currently what is the most used proxy with MySQL? It is HAproxy. HAproxy is a level 4 proxy that has no knowledge of the MySQL protocol. Being low level makes it very fast but it [...]

Webinar – Automated Sharding and High Availability with MySQL Fabric
Employee +5 Vote Up -0Vote Down


On Tuesday 17th December, we’ll be presenting a webinar on the latest developments for MySQL Fabric (a framework for managing pools of MySQL server – together with 2 applications: automated sharding and High Availablity). As always, the webinar is free and you should register here (http://www.mysql.com/news-and-events/web-seminars/automated-sharding-and-high-availability-with-mysql-fabric/" target="_blank).

This is your opportunity to hear the details directly from the engineering team and put your questions to them.

This session will present


  [Read more...]
MariaDB CONNECT Storage Engine replay & slides available
+1 Vote Up -1Vote Down

The slides and replay of yesterday’s webinar on the MariaDB CONNECT storage engine have just been posted. First I want to thank the numerous attendees. You have shown great interest on the parallel execution of query on distributed MySQL Servers. I agree this is cool. The ODBC capabilities seems also to generate interest. This make [...]

MariaDB CONNECT Storage Engine and parallelism
+1 Vote Up -1Vote Down

The CONNECT Storage engine implement the concept of a table made of multiple tables. These underlying tables can be distributed remotely. For example the underlying remote tables can be of ODBC or MySQL table type. this allows to execute distributed queries. What is nice is that we can execute this distributed query with parallelism.

How [...]

MySQL Fabric with MariaDB Galera Cluster ?
+2 Vote Up -0Vote Down

MySQL Fabric is a very promising sharding framework. If I take Ulf Wendel definition of MySQL Fabric :

MySQL Fabric is an administration tool to build large “farms” of MySQL servers. In its most basic form, a farm is a collection of MySQL Replication clusters. In its most advanced form, a farm is a collection of [...]

MySQL Connect presentations on MySQL Fabric
Employee +1 Vote Up -0Vote Down
MySQL Connect Conference was a great success and I am really happy for being
able to attend it this year. Oracle showed interesting improvements and
exciting features in the upcoming MySQL 5.7 and released a very early alpha
version of MySQL Fabric which is a framework for managing farms of MySQL
servers.

You can find the presentations about MySQL Fabric on SlideShare:

  . MySQL Sharding: Tools and Best Practices for Horizontal Scaling
  . MySQL High Availability: Managing Farms of Distributed Servers

If you haven't watched yet Edward Screven and Tomas Ulin keynote on “The State
of the Dolphin”,











  [Read more...]
A new kid in the MySQL sharding world
+3 Vote Up -0Vote Down

MySQL Connect 2013 has been a great edition. There was of course a lot of nice announcements of improvements in the the core MySQL server technology. One of the major announcement that received a lot of buzz was MySQL Fabric. MySQL Fabric is an infrastructure component aimed at simplifying construction of a highly available, sharded, [...]

MySQL Connect presentations on MySQL Fabric available on SlideShare
Employee +1 Vote Up -0Vote Down
Going to MySQL Connect was truly a blast. We got a lot of good questions and feedback in the sessions and there were a lot of interest in both MySQL Fabric and the MySQL Applier for Hadoop.

A big thank you to all that attended the talks, I got a lot of good questions and comments that will help us build good solutions.

The talks are available on SlideShare:





  [Read more...]
Installing MySQL Fabric on Windows
Employee +3 Vote Up -0Vote Down

One of the major announcements made at the MySQL Connect conference just over a week ago was the labs release of MySQL Fabric, which supports management of MySQL Server farms in a sharded deployment.  It’s available on labs.mysql.com, which means it is just an early release with some rough edges.  One of those rough edges that I’d like to see resolved soon is that it’s difficult to install on Windows, as there is no installer package provided.  There is some documentation on how to install Fabric (you’ll find it in section 15.8.2 of the PDF that makes up the downloadable

  [Read more...]
Going to MySQL Connect 2013
Employee +2 Vote Up -0Vote Down


MySQL Connect 2013 is coming up with several interesting new sessions. Some sessions that I am participating in got accepted for the conference, so if you are going there, you might find the following sessions interesting. For your convenience, the sessions have hCalendar markup, so it should be easier to add them to your calendar.

MySQL Sharding, Replication, and HA (September 21, 5:30-6:30pm in Imperial Ballroom B)

This session is an opportunity for you to meet the MySQL engineering team and discuss the latest tools and best practices for sharding MySQL across distributed





  [Read more...]
New Shard-Query features checked into SVN
+0 Vote Up -0Vote Down
I checked some updates to Shard-Query into SVN.

Partitioning support has been extended for MySQL 5.6+ to ALL partitioning types.

This includes all previously unsupported types including RANGE LIST/COLUMNS partitioned tables that are partitioned over more than one column, and HASH/KEY/LINEAR variants as well. Shard-Query now exclusively uses the PARTITION hint for partition elimination instead of WHERE clauses in MySQL 5.6. For 5.5 and previous versions, support remains limited to LIST,RANGE, and LIST/RANGE COLUMNS over a single column.

The old mysql interface DAL has been replaced completely by the PDO DAL.

There is no major difference for end users except that you have to check that the return of the query() method is an object with the is_object() function instead of checking that it is a resource with the is_resource() function.

  [Read more...]
Tips for working with append-only databases using sharding and log structured tables
+0 Vote Up -0Vote Down

This post is structured like a series of questions and answers in a conversation.  I recently had a number of conversations that all pretty much went this same way.  If you, like others, have many basic questions about how to proceed when faced with an append-only store for the first time, well then hopefully this post will help provide some answers for you.  The post focuses on column stores, the most common append-only store, but there are others.

Why do I want to use a column store?

Column stores are optimal for OLAP analysis

Column stores offer substantial performance increases for OLAP  compared to row stores.  Row stores are optimized for OLTP workloads.  While a row store can be used for OLAP, it may not perform well because a row store has to retrieve every column for a row (unless there is a covering

  [Read more...]
MySQL thread pool and scalability examples
+2 Vote Up -0Vote Down
Nice article about SimCity outage and ways to defend databases: http://www.mysqlperformanceblog.com/2013/03/16/simcity-outages-traffic-control-and-thread-pool-for-mysql/

The graphs showing throughput with and without the thread pool are taken from the benchmark performed by Oracle and taken from here:
http://www.mysql.com/products/enterprise/scalability.html (http://www.mysql.com/products/enterprise/scalability.html)

The main take away is this graph (all rights reserved to Oracle, picture original URL (http://www.mysql.com/common/images/enterprise/MySQL_Threadpool_Benchmark_RW.png" target="_blank)):

Scalability is






  [Read more...]
They say: "Relational Databases Aren't Dead"
+2 Vote Up -0Vote Down
This is a good read, claiming: "Relational Databases Aren't Dead. Heck, They're Not Even Sleeping", http://readwrite.com/2013/03/26/relational-databases-far-from-dead. A key quote:
"While not comprehensive, the uses for NoSQL databases center around the acquisition of fast-growing data or data that does not easily fit within uniform structures."

There were 2 parts in the statement about NoSQL's uses. I'll start with the latter:


"data that does not easily fit within uniform structures" - NoSQL is probably the right choice, hmm although I always encourage thinking and architecting in advance. And also online structure changes do exist in the RDBMS world and recently in MySQL:




  [Read more...]
Searching document stores in 2013: from 1983 to SQL:2003 in a blink?
Employee +2 Vote Up -0Vote Down

I love the new NoSQL systems: more choices! After years of RDBMS dominance there are hundrets of NoSQL systems offering a wide range of data models, data distribution strategies and interfaces. Polyglot persistence describes the market change. I am most fascinated by document stores: nested data and data distribution go hand-in-hand. Nested data, finally. And, for those who like it: schemaless or even schemafree. Maybe something to learn for MySQL? But their search capabilities… A word or two on SQL (SELECT … FROM … WHERE – SFW) and nested data.

Learn from NoSQL document stores

The classical relational data model requests all data to be in

  [Read more...]
The Data Day, Two days: January 9/10 2013
+0 Vote Up -0Vote Down

SAP on HANA. Funding for Guavus and ScaleArc. And more

It’s alive! @451research‘s 2013 Database survey is available now at bit.ly/451db13 #mysql #nosql #newsql #postgresql etc etc

— Matt Aslett (@maslett) January 9, 2013

#SAPonHANA is official. Read the press release for the SAP Business Suite powered by #SAP

  [Read more...]
Some sharding support and cache locality optimization support for PHP MySQL driver
Employee +5 Vote Up -0Vote Down

It is time for christmas presents: some sharding support and cache locality optimizations are coming with PECL/mysqlnd_ms 1.5. PECL/mysqlnd_ms is a plugin for the mysqlnd library. The plugin adds replication and load balancing support to any PHP MySQL API (mysql, mysqli, PDO_MySQL) if compiled to use the mysqlnd library.

As a MySQL user you can choose between a wide variety of clustering solutions to scale-out. Your options range from eventual consistent solutions to strong consistent ones, from built-in (MySQL Replication, MySQL Cluster) to third party or home-grown. PECL/mysqlnd_ms is a client side load

  [Read more...]
Being successful like Pinterest without its DB adventures...
+1 Vote Up -0Vote Down
I just came across this: "Scaling Pinterest and adventures in database sharding"  (http://gigaom.com/data/scaling-pinterest-and-adventures-in-database-sharding/)
"Pinterest has learned about scaling the way most popular sites do — the architecture works until one day it doesn’t"
Pinterest found out that "the architecture" is not scalable and they turned to development of a Scale Out mechanism also called Sharding.

I find it amazing that sharding, or in other words, the idea of "scale out by splitting and parallelizing data across shared-nothing commodity-hardware" is not supplied "out of the box" by "the architecture" (such as database, load-balancer, any other IT stuff). I'm wondering who was the one that


  [Read more...]
Facebook makes big data look... big!
+1 Vote Up -0Vote Down
Oh I love these things: http://techcrunch.com/2012/08/22/how-big-is-facebooks-data-2-5-billion-pieces-of-content-and-500-terabytes-ingested-every-day/

Every day there are 2.5B content items shares, and 2.7B "Like"s. I care less about GiGo content itself, but metadata, connections, relations are kept transactionally in a relational database. The above 2 use-cases generate 5.2B transactions on the database, and since there are only 86400 seconds a day, we get over 60000 write transactions per second on the database, from these 2 use-cases alone, not to mention all other use-cases, such as new profiles, emails, queries...

And what's the



  [Read more...]
Scale Up, Partitioning, Scale Out
+1 Vote Up -0Vote Down
On the 8/16 I conducted a webinar titled: "Scale Up vs. Scale Out" (http://www.slideshare.net/ScaleBase/scalebase-webinar-816-scaleup-vs-scaleout):


ScaleBase Webinar 8.16: ScaleUp vs. ScaleOut from ScaleBase
The webinar was successful, we had many attendees and great participation in questions and answers throughout the session and in the end. Only after the webinar it only occurred to me that one specific graphic was missing from the webinar deck. It was occurred to me after answering



  [Read more...]
ARM based data center. Inspiring.
+1 Vote Up -1Vote Down
In a previous post I wrote ARM based servers. Since then, and thanks to all the comments and responses I got, I looked more into this ARM thing and it's absolutely fascinating...

Look at this beauty (taken from the site of Calxeda, the manufacturer):

What is it? A chip? A server? No, it's a cluster of 4 servers...

And this:







  [Read more...]
Why shared-storage DB clusters don't scale
+4 Vote Up -2Vote Down
Yesterday I was asked by a customer for the reason why he had failed to achieve scale with a state-of-the-art "shared-storage" cluster. "It's a scale-out to 4 servers, but with a shared disk. And I got, after tons of work and efforts, 130% throughput, not even close to the expected 400%" he said.

Well, scale-out cannot be achieved with a shared storage and the word "shared" is the key. Scale-out is done with absolutely nothing shared or a "shared-nothing" architecture. This what makes it linear and unlimited. Any shared resource, creates a tremendous burden on each and every database server in the cluster.

In a previous post, I identified database engine activities such as buffer management, locking, thread locks/semaphores,



  [Read more...]
Impressions from Amazon's AWS Summit in NYC
+0 Vote Up -0Vote Down
Yesterday (4/19) I attended the AWS Summit in NYC (http://aws.amazon.com/aws-summit-2012/nyc).

I'm a big fan and also a heavy user of AWS especially S3, EC2, and naturally, RDS. In every point in time I have several dozens of AWS machines running for me out there in the East region, and in some cases when we do some special benchmarks and tests, number of EC2 and RDS machines can easily reach 3-digit. As I said, I'm a fan...

A few quotes I was able to catch and document on my laptop, on my laps...:
"When you develop an app for facebook, you must be prepared (and be afraid) that to your party, not noone will show up, but everybody will show up!"
So true! Simple and true. We all want to succeed, to have success with our app. We have to think about scaling




  [Read more...]
So how can we scale databases?
+0 Vote Up -0Vote Down
There are ways to scale databases, unfortunately some are limited, some introduce complexities, some are do not fit the cloud...

By scaling solution I mean a solutions that help me scale my existing environment, my existing RDBMS. Some magic or technology that will take my existing Oracle or MySQL for example, to the next level, without porting to a new DB engine/vendor and without completely recoding my app.

Let's try to organize things a bit in this very summarized table, just to get the hunch of it. I can't imagine to cover it all in 1 table or even 100 pages, but that should be a start of a meaningful discussion to continue in next posts:

Solution Scales reads? Scales writes? Scales data? Scales sessions? Cloud? Bottom line Scale-Up: faster HW, CPU, memory,





  [Read more...]
Applications come and go. Databases are here to scale.
+0 Vote Up -0Vote Down
In my heart, I'm a DBA, always was and always will be. People say I'm a database guy by the way I think, keep my car, and file my music and also bank statements... However I did great deal of development, design, architecture on the apps side. I (hope to) have some perspective.

Applications come and go. The second programming language I've ever learned and worked on was COBOL, some still say most of the world's lines of code are written in this language, maybe so, but anyway I since then have known and written in dozens of programming languages, from Assembly to Force.com, from Pascal to Delphi, from functional C to Object Oriented SmallTalk, C++, Java and , from compiled C/CGI to interpreted Perl, ASP and Ruby back to compiled node.js... My first applications ran on Main-Frame with green screen, later I created beautiful graphic

  [Read more...]
Showing entries 1 to 30 of 58 Next 28 Older Entries

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.