Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 89 Next 30 Older Entries

Displaying posts with tag: Main (reset)

SlackDB Updates
+2 Vote Up -1Vote Down

Since I announced SlackDB a few weeks ago, I’ve had a number of questions and interesting conversations in response. I thought I would summarize the initial feedback and answer some questions to help clarify things. One of the biggest questions was “Isn’t this what Drizzle is doing?”, and the answer is no. They are both being designed for “the cloud” and speak the MySQL protocol, but they provide very different guarantees around consistency and high-availability. The simple answer is that SlackDB will provide true multi-master configurations through a deterministic and idempotent replication model (conflicts will be resolved via timestamps), where Drizzle still maintains transactions and ACID properties, which imply single master. Drizzle could add support for clustered configurations and distributed transactions (like the NDB storage engine),

  [Read more...]
Let’s Build a Relational Database for the Cloud
+2 Vote Up -0Vote Down

As you might have guessed from my last couple blog posts, I’ve been experimenting with a few languages and libraries for a new project. I’ve finally gotten things far enough along to the point where I’d like to start getting other developers and potential users involved. I’m introducing SlackDB, an open source project that combines the functionality of relational databases with the ideas behind eventually consistent, shared-nothing data stores to provide a new database to support new and existing web applications in the cloud (enough buzzwords in there?). This is an idea I wrote about a while ago and recently I started putting a lot of night and weekend time into it.

It is still very early on in the development process, but the ideas behind it are starting to solidify and a fair amount of code is already written.

  [Read more...]
Yet Another Language Comparison
+1 Vote Up -0Vote Down

Over the past year or so I’ve found myself evaluating my overall programming experience with the languages I’m working with. I might just be getting impatient in my old age (turning the big three-oh in a couple months), but I like to think I’m trying to find the most efficient way to solve the problem at hand. This has led me to learn and experiment with a number of languages, taking a look at each one’s strengths and weaknesses. I realize programming language selection is very subjective and folks can get quite passionate in the debate, but I’m still going to present my personal opinions on the matter. Flame away.

The main question that I’m trying to try to answer is: What language will enable me to solve the problem at hand correctly and in the

  [Read more...]
OSCON and OpenStack
+0 Vote Up -0Vote Down

The past two weeks have been both exciting and extremely busy, first traveling to Austin, TX for the first OpenStack Design Summit, and then back home to Portland, OR for The O’Reilly Open Source Conference (OSCON) and Community Leadership Summit. The events were great in different ways, and there was some overlap with OpenStack since we announced it on the first day of OSCON and created quite a bit of buzz around the conference. I want to comment on a few things that came up during these two weeks.

New Role

I’m now focusing on OpenStack related projects at Rackspace. I’m no

  [Read more...]
MySQL Server Protocol Bug
+7 Vote Up -0Vote Down

A few months ago I wrote a tool that verified MySQL and Drizzle protocol compatibility, along with testing for all sorts of edge cases. In analyzing protocol command interactions in mysqld, I found that the MySQL server will happily read an infinite amount of data if you exceed the maximum packet size while using a special sequence of protocol packets. The reasoning behind this behavior is so that the server can be polite and flush your data before sending a “max packet exceeded” error message, but perhaps there should be a limit to one’s politeness. What’s more interesting is that you can do this during the client handshake packet without authorization, so anyone could do this to any open MySQL server. The appropriate thing to do here would be to set some maximum limit of data to read and force a connection close when it is reached, otherwise your bandwidth and CPU could be consumed (essentially a

  [Read more...]
Open Source Bridge Database Sessions
+0 Vote Up -1Vote Down

Open Source Bridge, the “conference for open source citizens,” is right around the corner! The sessions were just announced and it’s going to be packed with quite a variety of really interesting talks. From open cloud computing topics to hardware hacking to language hacks (like HipHop from Facebook), I’m really looking forward to being there (I’m helping organize the event, but hopefully I’ll have time to attend sessions as well).

I wanted to point out a few of the great database talks:

  [Read more...]
Threads with Events
+0 Vote Up -0Vote Down

Last week I was surprised to see this paper bubble back up on Planet MySQL. It describes the pros and cons of thread and event based programming for high concurrency applications (like a web server), arguing that thread-based programming is superior if you use an appropriate lightweight threading implementation. I don’t entirely disagree with this, but the problem is such a library does not exist that is standard, portable, and useful for all types of applications. We have POSIX threads in the portable Linux/Unix/BSD world, so we need to work with this. Other experimental libraries based on lightweight threads or “fibers” are really interesting as they can maintain your stack without all the normal overhead, but it is hard to get the scheduling correct for

  [Read more...]
Drizzle Developer Day Recap
+0 Vote Up -0Vote Down

Last Friday we held the Drizzle Developer Day at the Santa Clara convention center, taking advantage of the fact that many developers and interested contributors were already there for the MySQL Conference & Expo. Minus a few small glitches like wifi and pizza consumption location, I would say it was an overall success. There were a lot of new folks interested in learning about Drizzle and getting the server up and running. The day was organized by splitting folks up into small groups with matching interests, and then switching up groups every hour or so. We had groups focused on replication, documentation, writing plugins, the optimizer, Boots (the new client tool), and a “getting started” group.

The first group I participated in was about Boots, the new command line tool developed by a

  [Read more...]
Boots: A Modular CLI for Databases
+1 Vote Up -0Vote Down

Back in October I wrote about a student group I was sponsoring to create a new command line tool for Drizzle. The group wrapped up their part of the project (the term ended), and we now have a new tool called Boots! A few of the developers are still active in the project, and I’m planning to get involved more as well. We also have a couple students interested in hacking on it for Drizzle’s Google Summer of Code.

Boots is written in Python and aims to replace the the previous ‘drizzle’ tool (which was modified from the ‘mysql’ command line tool). It doesn’t support everything that the old tool has yet (like tab completion), but it adds some new features. For example, there are multiple

  [Read more...]
Scale Stack and Database Proxy Prototype
+1 Vote Up -0Vote Down

Back in January when I was between jobs I had a free weekend to do some fun hacking. I decided to start a new open source project that had been brewing in the back of my head and since then have been poking at it on the weekends and an occasional late night. I decided to call it Scale Stack because it aims to provide a scalable network service stack. This may sound a bit generic and boring, but let me show a graph of a database proxy module I slapped together in the past couple days:

I setup MySQL 5.5.2-m2 and ran the sysbench read-only tests against it with 1-8192 threads. I then started up the database proxy module built on Scale Stack so sysbench would route through that, and you can see the concurrency improved quite a bit at higher thread counts. The

  [Read more...]
Gearman Releases and Talks at the MySQL Conference
+0 Vote Up -0Vote Down

I spent some time this weekend fixing up the Gearman MySQL UDFs (user defined functions) and fixed a few bugs in the Gearman Server. You can find links to the new releases on the Gearman website. The UDFs now use Monty Taylor’s pandora-build autoconf files instead of the old fragile autoconf setup that relied on pkgconfig.

If you are attending the MySQL Conference & Expo next week and want to learn more about Gearman, be sure to check out one of the three sessions Giuseppe Maxia and I are giving:

  [Read more...]
Writing Authentication Plugins for Drizzle
+2 Vote Up -0Vote Down

In this post I’m going to describe how to write an authentication plugin for Drizzle. The plugin I’ll be demonstrating is a simple file-based plugin that takes a file containing a list of ‘username:password’ entries (one per line like a .htpasswd file for Apache). The first step is to setup a proper build environment and create a branch, see the Drizzle wiki page to get going. From here I’ll assume you have Drizzle checked out from bzr and are able to compile it.

Setup a development branch and plugin directory

Change to your shared-repository directory for Drizzle and run (assuming you branched ‘lp:drizzle’ to ‘drizzle’):

shell$ bzr branch drizzle auth-file
Branched 1432 revision(s).
shell$ cd auth-file

Next, we’ll want to create the

  [Read more...]
Thoughts on “NoSQL”
+2 Vote Up -0Vote Down

I’ve decided to jump on the bandwagon and spill my thoughts on “NoSQL” since it’s been such a hot topic lately ([1], [2], [3], [4]). Since I work on the Drizzle project some folks would probably think I take the SQL side of the “debate,” but actually I’m pretty objective about the topic and find value in projects on both sides. Let me explain.

Last November at OpenSQL Camp I assembled a panel to debate “SQL vs NoSQL.” We had folks representing a variety of

  [Read more...]
Drizzle Protocol Changes
+0 Vote Up -0Vote Down

On an entirely unrelated note to the MySQL protocol discussions happening yesterday, the MySQL protocol is now the default protocol in Drizzle as of Monday’s tarball (3/15). Drizzle supports a limited version of the MySQL protocol, only supporting the subset of commands Drizzle cares about (no server-side prepared statements, replication, or deprecated commands due to SQL query equivalents). Not all MySQL clients have been fully tested with it, but our entire test suite is using it now with the libdrizzle MySQL implementation. The latest release of libdrizzle also includes defaulting to the MySQL protocol and port for Drizzle connections.

There has been

  [Read more...]
Open Source Bridge 2010
+1 Vote Up -0Vote Down

A couple months ago Selena Deckelmann asked if I wanted to co-chair the Open Source Bridge Conference this year, and I was thrilled to say yes! This conference is all volunteer run by some of the most dedicated volunteers I have ever seen, I’m excited to be working with such a fantastic group of people. The conference is also backed by the 501(c)3 non-profit Technocation which is primarily run by Sheeri Cabral who is well known in the MySQL community.

The conference is June 1-4 in Portland, OR, and will be held at the Portland Art Museum. The

  [Read more...]
Drizzling from the Rackspace Cloud
+5 Vote Up -0Vote Down

Since I left Sun back in January, folks have been asking what was next. I’m happy to say that I’m going to continue hacking on open source projects like Drizzle and Gearman, but now at the Rackspace Cloud. Not only will I be there, but I get to continue working closely with a few of the amazing Drizzle hackers who have also joined, including Monty Taylor, Jay Pipes, Stewart Smith, and Lee Bieber.

Why Rackspace Cloud? Late last year I was considering what I wanted to do next with the Oracle acquisition looming near, and this was one of the options that presented itself. Rackspace had been a supporter of Drizzle from early on by

  [Read more...]
C++, or Something Like It
+1 Vote Up -0Vote Down

I’ve developed primarily in C most of my career, and recently decided to give C++ a shot as my “primary language” due to hacking on Drizzle and MySQL. The past few months I’ve read and experimented with most features C++ provides over C, including reading Scott Meyer’s excellent “Effective” series books (highly recommended). Along the way I’ve been developing a project I’ve wanted to write for a while, and I’m finding some features to be problematic. I thought I’d share these issues so others can be aware of them and perhaps I can learn better workarounds.

The project I’ve been working on uses dynamic shared object loading at runtime (using dlopen() and friends), is threaded, and has about every strict compiler warning on you can find and being treated as errors

  [Read more...]
Moving On
+3 Vote Up -1Vote Down

Friday was my last day at Sun Microsystems, and today is the first day at my new job (location coming soon). I’ve had a great time at Sun, and thank them for all the opportunities given to me there. I’ll be doing mostly the same work at the new gig, working on projects like Drizzle, but with a slightly different focus. For the most part my day-to-day won’t change much.

Right now I’m focusing on libdrizzle again and am implementing the prepared statement API, cleaning up the MySQL protocol support a little, and also implementing the new Drizzle client/server protocol. I’ll continue to work on Gearman as well, especially where it is relevant to Drizzle. I also need to start blogging again with specific topics in the projects I’m working on, I’ve been fairly quiet lately.

I’ll be in New Zealand next week at Linux Conf AU

  [Read more...]
Pluggable Database Client Tool
+0 Vote Up -0Vote Down

A few weeks ago I wrote about a student group who will be working with the Drizzle community to build a new database client tool. While the tool will be the primary replacement for the Drizzle client tool, we hope it will be generic (using the Python DB API) so it will work with others like MySQL and PostgreSQL. We’ve had a number of great discussions, including a session at OpenSQL camp last weekend. I wanted to toss out a few ideas of how such a tool could be structured to allow for maximum extensibility.

One possibility is to borrow from typical Unix shells and DSP processing systems where you have a number of modules with I/O interfaces and data exchange formats between each module. Each module provides a specific signature so you know what other modules it can plug into. Here is a simple example:

  [Read more...]
New Database Command Line Client
+1 Vote Up -0Vote Down

A few weeks ago I proposed a project to students at Portland State University for their senior capstone class, and this weekend I found out it was chosen by a group! The project will be a rewrite of the command line tool (the Drizzle tool is currently based on the ‘mysql’ tool), plus a lot of new features. We’re really excited to be working with them, and they seem equally excited about the project too. I hope DBAs, developers, and other folks in the Drizzle/MySQL/MariaDB communities will work with them to help define what features should be part of this new command line client. Some new features we have in mind are background queries, piping and redirection of queries (like a normal shell), and plugin support. It will also support at least the MySQL/MariaDB protocol too since it will be built on libdrizzle, but possibly more if we end up using a common DB API

  [Read more...]
OpenSQL Camp, SQL vs NoSQL
+1 Vote Up -0Vote Down

The upcoming OpenSQL Camp is almost full! We have space for 130 people to register, and as of this writing only 10 spots are free. If you want to attend, sign up before it’s too late!

We’re still looking for a few sponsors if anyone is interested in helping cover food and t-shirt costs.

I’m organizing the closing keynote panel, “SQL vs NoSQL”, which will include core community members and committers from a number of open source databases. Selena has offered to take the PostgreSQL position if we don’t find another worthy contender. So far, it will include:

  • Brian Aker – Drizzle
  • Eric Evans – Cassandra
  • Joydeep Sen Sarma – Hive/Hadoop
  • Mike Dirolf –
  [Read more...]
Eventually Consistent Relational Database?
+0 Vote Up -0Vote Down

This weekend I attended Drupal Camp PDX and listened to a session titled “Drupal in the Cloud”. The presenter, Josh Koenig from Chapter Three, gave a great introduction of what moving to “the cloud” really means, especially in the context of a typical web application like Drupal. The problem, which is of course no fault of Josh’s, is that the best high availability database practices are harder to deploy because you’re working within a different set of constraints in the cloud. Sure, you can setup MySQL replication, but without the ability to insert a hardware load balancer or better control over floating IPs, reliable single-master solutions are difficult at best.

I spoke with Josh for a bit after and discussed how

  [Read more...]
Non-blocking State Machines
+0 Vote Up -0Vote Down

If you’ve ever done any non-blocking programming (usually for socket I/O), you’ve probably had to come up with a non-trivial state machine to handle all the places where everything can pause. Say you’re reading an application level packet from a socket, and half way through the read() system call it screams EAGAIN. You need to stop, save any state, and exit out of whatever chain of functions got you there so the calling application can regain control. I’m going to explain a few techniques I’ve come up with over the years, each with their strengths and weaknesses, and I hope this will spur some conversation of what other folks have done. While I’m fairly happy with how I handle these state machines now, but I’m always looking for a more succinct way of handling things. Please share your thoughts!

Switch Statements

The obvious way to handle non-blocking I/O is

  [Read more...]
Gearman News and Releases
+2 Vote Up -1Vote Down

The past week has brought a surge of Gearman related releases. They include:

C Server and Library
Perl Server and Modules
Gearman::XS Perl Module (based on C library)
PHP Extension (based on C library)
PHP Extension Documentation
Drizzle UDFs
PostgreSQL UDFs
MySQL UDFs

Some of these releases were driven by the C API changes I made to clean a few things up, but a fair amount of functionality was added to








  [Read more...]
Debug Console in drizzled, Part 2
+0 Vote Up -0Vote Down

About a month ago I blogged about the debug console I was adding to drizzled. I finished this work up and it’s now in the trunk and latest release. This is implemented using the Client and Listen plugin points (which are heavily modified versions of MySQL’s Protocol class), and can be enabled using the ‘–console-enable’. For example:

hades> drizzled --datadir=/Users/eday/drizzle.data --console-enable
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
090928 15:22:07  InnoDB: highest supported file format is Barracuda.
090928 15:22:07 InnoDB Plugin 1.0.3 started; log sequence number 46409
Listening on :::4427
Listening on 0.0.0.0:4427
./drizzled/drizzled: Forcing close of thread 0  user: '(null)'
./drizzled/drizzled: ready for connections.
Version: '2009.09.1144' Source distribution (trunk)
  [Read more...]
Gearman Slides from San Francisco Meetup
+0 Vote Up -0Vote Down

Thanks to everyone who came out to the San Francisco PHP and MySQL meetup! Also, thanks to Michael for organizing such a great event, and Percona for sponsoring the food. I put the slides from the talk up on my wiki for reference or in case you missed it. I believe that there will be a video up at some point as well. While down there I also had a chance to stop by Digg and talked to them about Gearman (they’ve been using it for a while). It was interesting to see how they were using it in a large scale deployment. I was able to get some valuable feedback to future development, and a cool t-shirt. :) Thanks Digg!

Gearman at San Francisco PHP and MYSQL Meetup
+1 Vote Up -0Vote Down

I’ll be talking about Gearman this Thursday (October 1st, 2009) at the San Francisco PHP and MySQL Meetup groups (these are two separate groups, but sometimes share the topic). A few other folks involved in the Gearman community should also be there to help out, including James Luedke (the PHP extension main author), Eric Lambert (the Java API author), Dormando, and Hachi (Perl version maintainers at SixApart). You can sign up at either the MySQL or PHP meetup groups. We’ll be discussing the basics for those of you who don’t even know what Gearman is, common use cases, new features, advanced topics for folks already using Gearman, and of course Q&A throughout. Hope to see you there!

OpenSQL Camp 2009 in Portland, OR
+8 Vote Up -0Vote Down

Sign up today for OpenSQL Camp 2009! The space is confirmed so go ahead and make your travel arrangements. The event is free and will be taking place in Portland, OR on November 14-15th, 2009. If you are interested in leading a session or presenting a talk, be sure to add it to the session ideas page. Also, we are still looking for sponsors! Please visit the sponsors page if you or your company/organization might be interested. All donations are tax deductible.

We should have representatives from many open source database (and database-related) projects. PostgreSQL, MySQL, Drizzle, memcached, and Gearman, just to name a few. If you work closely with one of

  [Read more...]
Debug Console in drizzled
+4 Vote Up -0Vote Down

Last week on the train to Seattle I was hacking on the new libdrizzle protocol plugin when realized I could really use a debugging console that did not require any socket communication. Since I abstracted a Client plugin in the server (Protocol class for the MySQL folks), this can easily be done by just writing a new plugin. I needed to do some refactoring with the listener plugin to not require a TCP socket/accept and instead poll any arbitrary file descriptor. This allows the the Listen plugin to create the client instance any way it wants. After a bit of hacking and cleanup, we now have this:

shell$ ./drizzled/drizzled --datadir=/home/eday/other/drizzle.data
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
090827  0:29:34  InnoDB: highest supported file format is Barracuda.
090827  0:29:34 InnoDB Plugin 1.0.3 started; log sequence number 8157346
Listening on
  [Read more...]
Drizzle, Gearman, and memcached Meetings
+2 Vote Up -0Vote Down

As Brian mentioned, a number of us traveled up to Seattle last week to discuss the road map for Drizzle, Gearman, and memcached. Thanks to everyone who was able to make it! It was great to see folks again (Northscale guys, Robert Hodges, Padraig), and meet a couple new people like Nathan, one of the Google Summer of Code students for Drizzle. I thought I’d take a moment to mention some of the discussions related to the tasks I’m working on.

For Drizzle, we talked about the new configuration and plugin system I’ve been digging into lately. Monty Taylor has been doing a great job refactoring the plugin loading, but there are still some steps to be taken to get things where we want. One of the big goals with all this is to have the plugin and config system not specific to Drizzle at all so we can use this in other projects as well (one being

  [Read more...]
Showing entries 1 to 30 of 89 Next 30 Older Entries

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.