Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Previous 30 Newer Entries Showing entries 31 to 60 of 89 Next 29 Older Entries

Displaying posts with tag: Main (reset)

C vs C++
+3 Vote Up -0Vote Down

Linux vs FreeBSD, vi vs emacs, MySQL vs PostgreSQL, your habit or favorite technology vs another’s. At the end of the day there is no winner, just a matter of preference for the task at hand. I learned C++ 13 years ago, I forgot most of my C++ knowledge 10 years ago, I discouraged the use of C++ in this period in between, and in the past year I’ve been re-learning C++ (mostly due to Drizzle). So what did I use after unlearning C++ 10 years ago? I wrote everything in C (and by everything I mean this was my performance programming language of choice). This worked quite well, but it’s an interesting evolution that I think is now coming full circle.

When I first started programming C, it was a bit clumsy, and I look back at my old code and cringe. I began to develop a certain programming style that can best be described as object-oriented C programming due to the conventions used. The structs,

  [Read more...]
Open Source Cloud
+0 Vote Up -0Vote Down

There has been a lot of talk lately about the “Open Source Cloud.” What will it look like, who will be behind it, and can I use it now? These were hot topics at OSCON, and Stephen O’Grady had two excellent posts on them recently as well (one, two). As many other folks do, I see a few major drivers of this: the need for private clouds, the prevention of vendor lock-in (proprietary services), and open source hackers just wanting to be able to extend and fix the source code. Some layers of this open source cloud stack are further along than others, and I’m going to attempt to outline what I’ve found so far along with what I think is missing.

Virtual Machine

  [Read more...]
Drizzle and Gearman PHP talk in Portland
+0 Vote Up -0Vote Down

I’m going to be giving a talk at the PHP user group here in Portland, OR on August 11th. Details can be found here. If you’re in the Portland area please join the group and come check it out! This will be similar to the Boston MySQL Meetup group talk I gave earlier this month with Patrick, but with more focus on PHP.

Gearman Updates
+1 Vote Up -0Vote Down

There is a lot happening with Gearman right now. Last week at OSCON we received quite a bit of good feedback with the project which will certainly help direct our priorities moving forward. In the past couple weeks we’ve had the following Gearman releases:

  [Read more...]
OpenSQL Camp comes to Portland, OR! November 14-15, 2009
+3 Vote Up -0Vote Down

Stealing the announcement news from Selena’s blog post:

I’m happy to announce that Portland, OR will be host to the next OpenSQL Camp, Portland. We’ll be holding it November 14-15, 2009.

Patterned after the first Open SQL camp held in Charlottesville, Virginia, USA, in November 2008, we’re inviting any and all database practitioners (even the non-SQL, non-relational ones!) to join us for two days of deep database geeking.

To quote:

“Attendees of this conference are mostly open source developers and end users/open source enthusiasts. The goal of this event is to spread the word about the vibrant communities and large ecosystems around Open Source Databases and to educate the attendees about what alternatives exist to commercial

  [Read more...]
pgGearman 0.1 Released
+3 Vote Up -1Vote Down

Similar to the MySQL and Drizzle user defined functions for Gearman, we now have PostrgeSQL functions too! These allow you to submit jobs to the job server from within your SQL query, trigger, or stored procedure. Here’s a snippet of how this looks in PostgreSQL:

shell$ psql test
test=# SELECT gman_do('reverse', 'Hello World!');
   gman_do
--------------
 !dlroW olleH
(1 row)

test=#

Special thanks to Selena Deckelmann for helping get these working!

MySQL Webinar on Gearman
+0 Vote Up -0Vote Down

I’ll be talking about the latest features in Gearman along with covering a few Gearman-powered applications tomorrow, July 14th, at 10AM Pacific time. Follow this link for details on how to sign up: Gearman: New Features and Applications for Distributed Job Queuing (http://mysql.com/news-and-events/web-seminars/display-387.html)

Brian Aker and I will also be talking about Gearman at the PostgreSQL pgDay San Jose on Sunday, July 19th, and at OSCON the week after. See the Gearman presentations page for more information.

libdrizzle and PHP Extension Released
+2 Vote Up -0Vote Down

Version 0.4 of libdrizzle has been released. This was mostly a maintenance release with build system changes and small bug fixes. This is the client and protocol library for Drizzle and MySQL that provides both client and server interfaces.

Version 0.4.1 of the Drizzle PHP Extension has also been released. James Luedke has moved development and releases of the extension into PECL, and has also fixed a number of bugs, extended the interface, and worked with the PHP/PECL developers to get the extension up to the proper PHP coding standards. Thanks James!

Gearman Releases
+1 Vote Up -0Vote Down

Version 0.8 of the Gearman C Server and Library has been released. This includes basic HTTP protocol support, build system improvements, and bug fixes.

Version 0.4.0 of the Gearman PHP Extension has also been released.

If you want to learn more about Gearman, be sure to check out the upcoming Boston MySQL Meetup, MySQL Webinar (http://mysql.com/news-and-events/web-seminars/display-387.html), or the one of the events at OSCON (tutorial, session, and BoF).

Drizzle and Gearman in Boston Next Week
+0 Vote Up -0Vote Down

I’ll be heading back to my home state (Maine) this week for a visit, and while I’m back there Patrick Galbraith and I will be talking at the Boston MySQL Meetup Group on Monday night about Drizzle, Gearman, and how to combine the two with projects like Narada. If you are in the Boston area, be sure to check it out!

Why You Won’t See a Drizzle Proxy
+3 Vote Up -0Vote Down

I’ve been following the excellent work that Jan, Kay, and others have been doing with MySQL Proxy, it has really matured into a great piece of software. I talked to Jan at the MySQL UC and toyed with the idea of integrating libdrizzle into MySQL Proxy. I’ve also been asked by a number of folks when a Drizzle Proxy project will be started and if it will be as feature rich as MySQL Proxy. For a while I just said “Someday, I just don’t have the time.” Lately though I am hoping we never have a Drizzle Proxy project.

Let me explain.

One of the fundamental ideas in software engineering is code reuse through libraries or modules. Rather than create a Drizzle Proxy

  [Read more...]
Gearman Pluggable Protocol
+2 Vote Up -0Vote Down

I just finished adding pluggable protocol support to the Gearman job server, this will enable even more methods of submitting jobs into Gearman. If all the various Gearman APIs, MySQL UDFs, and Drizzle UDFs are not enough, it’s now fairly easy to write a module that takes over the socket I/O and parsing hooks to map any protocol into the job server. As an example module, I added basic HTTP protocol support:

> gearmand -r http &
[1] 29911
> ./examples/reverse_worker > /dev/null &
[2] 29928
> nc localhost 8080
POST /reverse HTTP/1.1
Content-Length: 12

Hello World!

HTTP/1.0 200 OK
X-Gearman-Job-Handle: H:lap:1
Content-Length: 12
Server: Gearman/0.8

!dlroW olleH

I’ve added a few headers for setting things like background, priority, and unique key. For example, if you want to run the above job in the background:

POST /reverse HTTP/1.1
Content-Length: 12
  [Read more...]
Drizzle Regression Hunting
+3 Vote Up -0Vote Down

We’ve been looking for a Drizzle regression for some time now, and today I decided I would take a step back and make another attempt to find it. The first step in doing this was to reproduce this consistently and find a baseline. We’ve noticed it most dramatically with a 16 concurrent connection test from sysbench in read-only mode. I used two 16-core Intel machines running Linux we have for development. We’ve noticed the regression on certain machines but not all, and these two machines provided one of each. I also setup a MySQL 5.1.35 server to use as a baseline to give some comparisons outside of Drizzle. So first, a few more details on the machines:

Machine 1: 16 core, 16GB RAM, cache sizes from dmesg:
[    0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.010000] CPU: L2 cache: 4096K
From /proc/cpuinfo:
cache_alignment : 64

Machine 2: 16 core,
  [Read more...]
libdrizzle 0.3 Released
+2 Vote Up -0Vote Down

I’m pleased to announce a new version of libdrizzle! This is mostly a bug fixing and maintenance release before I start in on more significant development. One of the new features I added was a hook to be able to use your own I/O event mechanism rather than the default poll(). This will allow you to use libraries like libevent, which can be useful when dealing with a large number of file descriptors, or to mix with other file descriptors in your application (for example, you could listen on other fd’s alongside the non-blocking Drizzle/MySQL socket connections). There is not much for examples or documentation yet with this feature, but for now you can email or find me in #drizzle on irc.freenode.net if you would like to know more.

One of the next steps with libdrizzle is a better protocol abstraction, since the Drizzle protocol is diverging quite a bit

  [Read more...]
Narada - A Scalable Open Source Search Engine
+2 Vote Up -0Vote Down

I’ve been working with Patrick Galbraith for the past couple weeks on a new project that started as an example in his upcoming book. It is a search engine built using Gearman, Sphinx, Drizzle or MySQL, and memcached. Patrick wrote the first implementation in Perl to tie all these pieces together, but there is also a Java version underway bring written by Trond Norbye and Eric Lambert that will be shown at the CommunityOne and JavaOne conferences next week. I’ve been helping get the system setup on a new cluster and with the port to Drizzle.

Narada provides interfaces that allow you to submit URLs to be

  [Read more...]
Cache Line Sizes and Concurrency
+1 Vote Up -0Vote Down

We’ve been looking at high concurrency level issues with Drizzle and MySQL. Jay pointed me to this article on the concurrency issues due to shared cache lines and decided to run some of my own tests. The results were dramatic, and anyone who is writing multi-threaded code needs to be aware of current CPU cache line sizes and how to optimize around them.

I ran my tests on two 16-core Intel machines, one with a 64 byte cache line, and one with 128 byte cache line. First off, how did I find these values?

one:~$ cat /proc/cpuinfo | grep cache_alignment
cache_alignment : 64
...

two:~$ cat /proc/cpuinfo | grep cache_alignment
cache_alignment : 128
...

You will see one line for each CPU. If you are not familiar with /proc/cpuinfo, take a closer look at the full output. It’s a nice quick reference

  [Read more...]
Gearman and Drizzle at OSCON
+2 Vote Up -0Vote Down

If you missed Gearman or Drizzle at the MySQL Conference, have no fear, a number of folks will be at OSCON too! There will be a many opportunities to learn more or get involved with the two projects:

  • 3-Hour Gearman Tutorial - Learn about the latest developments and participate in hands-on demos to help build your own Gearman-powered applications.
  • 45-Minute Gearman Session - Get a more concise glimpse about what Gearman is and how to use it.
  • 45-Minute Drizzle Panel - Get an update and ask questions about the project and community.
  • Gearman and Drizzle Booths - We’ll have a booth for each
  [Read more...]
Drizzle patch for Sphinx 0.9.9-rc2
+1 Vote Up -0Vote Down

Last week I blogged about a patch for Sphinx 0.9.8.1 that allowed Drizzle to be used as a data source (this works much like the MySQL and PostgreSQL options). This past weekend I made a new patch that would work with the latest version, 0.9.9-rc2. It can be found here:

sphinx-0.9.9-rc2-drizzle.patch

The instructions are pretty much the same:

Download sphinx-0.9.9-rc2.tar.gz and sphinx-0.9.9-rc2-drizzle.patch
tar xzf sphinx-0.9.9-rc2.tar.gz
cd sphinx-0.9.9-rc2
patch -p1 

See my previous blog post for configuration changes to make Sphinx index from Drizzle.

Gearman, Now With Persistent Queues
+0 Vote Up -0Vote Down

I’m pleased to announce version 0.6 of the Gearman C server and library. The major new feature of this release is a pluggable persistent queue for the job server. It comes bundled with a libdrizzle module (so your queue can live in Drizzle or MySQL), but Brian has already written a libmemcached module and there is a flat-file module in the works as well. The persistent queue allows background jobs to be stored via the pluggable module, so if the job server crashes or is shutdown, the queue module can repopulate the job server with any jobs that were not yet complete. This is just the first version of the queue support, so expect more modules and features in the future!

On a related note, James Luedke has also released version 0.3 of the

  [Read more...]
Gearman UDF now in Drizzle
+0 Vote Up -0Vote Down

If you’ve pulled the latest Drizzle code from lp:drizzle, you may have noticed a new plugin/gearman_udf directory in there. This is a new UDF that acts as a Gearman client. This is mostly a port of the Gearman MySQL UDF I wrote, but I did it the proper C++ way to fit in better with Drizzle. It also uses the new plugin system Monty Taylor has been working in, which makes it much easier. :)

To use it, just make sure you have the Gearman C library installed and Drizzle will pick it up and build it for you. No extra configuration required!

The following example assumes you have a Gearman job server and a reverse worker running (see examples/reverse_worker in the C library package).

drizzle> SELECT
  [Read more...]
Sphinx Now Indexing from Drizzle
+2 Vote Up -0Vote Down

In preparation for an upcoming demo, we decided to use Sphinx for the full text search. We want to do this with Drizzle but hit an obvious problem - Sphinx does not know how to speak to Drizzle to grab documents for indexing. At least that was the case until last night when I created a patch to do so using the new libdrizzle library.

You can download the patch here: sphinx-0.9.8.1-drizzle.patch

I’m releasing this patch BSD, so do with it what you want. I’m hoping Andrew (the Sphinx author) will include this in the main package in an upcoming release. :)

So for now, to get things working, we first need to patch the sphinx-0.9.8.1 source and build/install:

Download sphinx-0.9.8.1.tar.gz
  [Read more...]
The Joy of Open Source
+1 Vote Up -1Vote Down

I read Jeremy’s post last week on jobs that matter and was also reminded of Tim O’Reilly’s post about the same topic from earlier this year (which is a great read by the way). This got me thinking, and I realized it has been about one year since I made the commitment to myself to get more involved with open source. I had been a passive member for over ten years, but I decided it was time for a change. After looking at various GNU projects, the Linux kernel, MySQL, and a few others, I decided to focus on the MySQL community. After attending OSCON I was even more inspired with the announcement of Drizzle, along with learning about other projects such as

  [Read more...]
Gearman C Server 0.5 Released - Now Threaded!
+0 Vote Up -0Vote Down

Last week at the MySQL Conference & Expo I made a release right before my first Gearman talk. I didn’t get a chance to blog about it, so here it is now. This is a fairly big release since it includes a major refactoring of the job server to now be threaded. Doing some simple tests on it with a 16-core Intel machine:

The Y axis is total jobs/second, and the X axis is number of clients/workers (8×8 means 8 clients and 8 workers). The clients and workers are the ‘blobslap’ utilities included in the source, and they are shoving as many jobs/second through the server as they can. Job size was random between 0 and 1024 bytes. The job server peaked out at about 43,000 jobs/second, most likely due to the machine being so busy (running

  [Read more...]
Drizzle Modularity
+0 Vote Up -0Vote Down

Last week at the Drizzle developer day a few people were surprised at the number of plugin interfaces we’ve created already. I created this image to give a sampling of interfaces and plugins:

As you can see, things are quite different from MySQL.

Gearman and libdrizzle Slides
+1 Vote Up -0Vote Down

Thanks to everyone who came out for the Gearman and libdrizzle talks at the MySQL Confrence & Expo and the Percona Performance Conference! Links to the slides are provided below:

If you want to learn more about Gearman, be sure to check out gearman.org or the upcoming 3-hour tutorial or session at OSCON in July!

More on Drizzle Protocol Plugin
+1 Vote Up -0Vote Down

The other day I wrote about what I did inside of Drizzle to create the new Protocol plugin type. A few people have asked me to expand on this a bit more, so here is a diagram of how this will work:

As you can see, you will be able to communicate with the server not just through the traditional client library, but through anything that can implement the Protocol class. The first new class will be the new libdrizzle (which also supports MySQL protocol so Drizzle can talk to those clients), but soon we’ll also be able to handle direct HTTP, Gearman, and possibly memcached requests. This will eliminate the need for the client driver altogether in some applications (for example, an AJAX request could query Drizzle directly and ask for a JSON object to be returned by the module).

You will

  [Read more...]
Coming soon to a gearmand near you…
+0 Vote Up -0Vote Down

I’m wrapping up some Gearman work and packing for the users conference, it should be fun and exciting! I thought I’d send out a reminder that Brian and I will be talking about Gearman at both the MySQL UC and Percona Performance conference, including the possibility of a new release of Gearman. Here’s a hint:

lap> ./gearmand/gearmand -vv -t 4
Starting up
Initializing libevent for main thread
Method for libevent: epoll
Trying to listen on 0.0.0.0:4730
Listening on 0.0.0.0:4730
Creating wakeup pipe
Creating 4 threads
Initializing libevent for IO thread
Creating IO thread wakeup pipe
Thread 1 created
[   1] Entering thread event loop
Initializing libevent
  [Read more...]
Drizzle Now Has Protocol Plugin Support
+0 Vote Up -0Vote Down

Over the past couple days I’ve been tearing through the network I/O code inside of Drizzle. I moved anything that touched NET into the Protocol class (reading commands, preferences, error checking) and wrapped them with new Protocol methods. I then moved the NET structure out of Session (or THD for the MySQL folks) and into Protocol. At this point all network I/O was going through Protocol, which allowed me to do a proper abstraction layer with the old libdrizzle as a derived implementation class. The next step was to add in a new plugin hook for Protocol, and move the old libdrizzle implementation into a plugin. Since the server is the only thing using the old libdrizzle, all of that code also went into the plugin (note: this means the final death of old libdrizzle!). So now each new connection grabs a Protocol object from the default registered Protocol plugin and Session will

  [Read more...]
Upcoming Gearman Events
+0 Vote Up -0Vote Down

Good news! If you want to learn more about Gearman, you’ll have plenty of opportunities in the coming months:

Gearman Releases
+0 Vote Up -0Vote Down

I’m pleased to announce three Gearman releases today! They are:

  • Gearman C Server and Library - This includes changes to support new part of the protocol, a new Gearman command line tool, and a few other bug fixes and cleanup. Note: This now installs ‘gearmand’ into your preferred ’sbin’ directory, so if you are upgrading be sure to remove the old one from ‘bin’
  • Gearman PHP Extension - This is a brand new PHP extension that wraps the C library to provide PHP with a faster client and worker interface. See the README file and the exampes directory for instructions and how to use it.
  • Gearman MySQL UDF - New version of the UDF to include updates
  [Read more...]
Previous 30 Newer Entries Showing entries 31 to 60 of 89 Next 29 Older Entries

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.