Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 83 Next 30 Older Entries

Displaying posts with tag: gearman (reset)

Shard-Query 2.0 performance on the SSB with InnoDB on Tokutek’s MariaDB distribution
+2 Vote Up -0Vote Down
Scaling up a workload to many cores on a single host

Here are results for Shard-Query 2.0 Beta 1* on the Star Schema Benchmark at scale factor 10.  In the comparison below the “single threaded” response times for InnoDB are the response times reported in my previous test which did not use Shard-Query.

Shard-Query configuration

Shard-Query has been configured to use a single host.  The Shard-Query configuration repository is stored on the host.  Gearman is also running on the host, as are the Gearman workers.  In short, only one host is involved in the testing.

The

  [Read more...]
Talking about Gearman at Etsy Labs
+0 Vote Up -0Vote Down
I find myself flying to New York on Monday for some dealnews related business. Anytime I travel I try and find something fun to do at night. (Watching a movie by myself in Provo, Utah was kinda not that fun.) So, this week I asked on Twitter if anything was happening while I would be in town. Anything would do. A meetup of PHP/MySQL users or some design/css/js related stuff for example. Pretty much anything interesting. Well, later that day I received an IM from the brilliant John Allspaw, Senior VP of Technical Operations at Etsy. He wanted me to swing by the Etsy offices and say hi. Turns out it is only a block away from where I would be. Awesome! He also mentioned that he would like to have me come and speak at their offices some time. That would be neat too. I will have to plan better next time  [Read more...]
Gearman Priorities And Persistent Storage
+0 Vote Up -0Vote Down

I have been writing a bit about Gearman lately, including installing it for PHP and Ubuntu, actually using it from PHP and also how I use persistent storage with Gearman. I'm moving on to look at adding jobs of different priorities.

I use Gearman entirely as a point to introduce asynchronous-ness in my application. There is a complicated and image-heavy PDF to generate and this happens on an automated schedule. To do this, I use the

  [Read more...]
Using Persistent Storage with Gearman
+0 Vote Up -0Vote Down

I'm using gearman for the first time in a new project, and two things in particular were bothering me. Firstly, there doesn't seem to be a built-in way to see what's in the queue. Secondly, if the gearman server dies (which seemed quite likely when I was first getting to grips with this stuff and writing really buggy code!) you lose your queue. Therefore I decided that I would switch gearman over to running with persistent storage. My config file (this is Ubuntu 10.10) is in /etc/default/gearman-job-server and it contains the following snippet:

# Use mysql as persistent queue store                                                                                                  
# PARAMS="-q libdrizzle --libdrizzle-host=10.0.0.1 --libdrizzle-user=gearman \
  [Read more...]
Dealing with MySQL Gone Away in Zend Framework
+0 Vote Up -0Vote Down

I wrote recently about having gearman in my application, however I have been seeing problems with the long-running PHP worker scripts. My logs had entries like this:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

The worker is a Zend Framework application, run from the CLI, and it seemed like the Zend_Db_Adapter had no way of knowing when MySQL had let go of its end of the connection. I tried a few different things, including Zend_Db_Adapter::getConnection(), but without success - until I dug through the source code (with some help from a friend) and realised that ZF was not reconnecting at all if it thought it already had a connection. So instead, I

  [Read more...]
CB1 Ubuntu 10.10 Linux Development Setup
+0 Vote Up -0Vote Down

I use a MacBook Pro for my day-to-day operations here at CB1, INC. I’m a huge believer that a development environment should mimic the production environment, so I find myself running a couple virtual machines in VMware Fusion.

The following guide is a reference for myself as well as possibly a helpful resource for setting up your own Linux development environment. Here’s an checklist of the tasks to perform and software to install:

  • Operating System
    • Ubuntu 10.10 64-bit: I use Ubuntu Desktop in dev and Ubuntu Server in production
    • Package updates and upgrades
    • Network configuration (at least 2 static IP addresses)
  • Development Tools
    • C/C++ development environment
    • Autotools
    • Sun Java JDK
  [Read more...]
OSCON and OpenStack
+0 Vote Up -0Vote Down

The past two weeks have been both exciting and extremely busy, first traveling to Austin, TX for the first OpenStack Design Summit, and then back home to Portland, OR for The O’Reilly Open Source Conference (OSCON) and Community Leadership Summit. The events were great in different ways, and there was some overlap with OpenStack since we announced it on the first day of OSCON and created quite a bit of buzz around the conference. I want to comment on a few things that came up during these two weeks.

New Role

I’m now focusing on OpenStack related projects at Rackspace. I’m no

  [Read more...]
OpenSQLCamp Boston Pages are online
+5 Vote Up -1Vote Down

OpenSQLCamp is less than 4 months away, and I have finally gotten around to updating the site. Special thanks go to Bradley Kuzsmaul and the folks at Tokutek for getting the ball rolling and making the reservation at MIT. Using MIT means that we will have *free* reliable wireless guest access and projects.

OpenSQL Camp is a free unconference for people interested in open source databases (MySQL, SQLite, Postgres, Drizzle), including non-relational databases, database alternatives like NoSQL stores, and database tools such as Gearman. We are not focusing on any one project, and hope to see representatives from a variety of open source database projects attend. As usual I am one of the main organizers of Open SQL Camp (in previous years, Baron Schwartz, Selena Deckelmann and Eric Day have been main organizers too; this year Bradley

  [Read more...]
Threads with Events
+0 Vote Up -0Vote Down

Last week I was surprised to see this paper bubble back up on Planet MySQL. It describes the pros and cons of thread and event based programming for high concurrency applications (like a web server), arguing that thread-based programming is superior if you use an appropriate lightweight threading implementation. I don’t entirely disagree with this, but the problem is such a library does not exist that is standard, portable, and useful for all types of applications. We have POSIX threads in the portable Linux/Unix/BSD world, so we need to work with this. Other experimental libraries based on lightweight threads or “fibers” are really interesting as they can maintain your stack without all the normal overhead, but it is hard to get the scheduling correct for

  [Read more...]
MySQL Conference Review
+1 Vote Up -0Vote Down
I am back home from a good week at the 2010 O'Reilly MySQL Conference & Expo. I had a great time and got to see some old friends I had not seen in a while.

Oracle gave the opening keynote and it went pretty much like I thought it would. Oracle said they will keep MySQL alive. They talked about the new 5.5 release. It was pretty much the same keynote Sun gave last year. Time will tell what Oracle does with MySQL.

The expo hall was sparse. Really sparse. There were a fraction of the booths compared to the past. I don't know why the vendors did not come. Maybe because they don't want to compete with Oracle/Sun? In the past you would see HP or Intel have a booth at the conference. But, with Oracle/Sun owning MySQL, why even try. Or maybe they are not allowed? I don't know. It was just sad.

I did stop by the Maatkit booth and was embarrassed to





  [Read more...]
Expert PHP and MySQL published!
+4 Vote Up -0Vote Down
I'm very pleased to announce the publication of my book Expert PHP and MySQL, published by Wrox. This book was written by myself, Andrew Curioso and Ronald Bradford. The short of it is, upon finishing my previous book, Developing Web Applications with Apache, MySQL, memcached, and Perl, Wiley asked me if I knew of anyone who would like to write a MySQL/PHP book. I had worked with Andrew at Lycos and found him to be a brilliant PHP developer, having been the primary developer of Lycos's Webon product-- which has some excellent usage of PHP, Javascript and MySQL. When I friend of mine Bob Wilkins, who started MyVBO, was looking for a developer I suggested Andrew (he now works at MyVBO), and for this book  [Read more...]
Gearman Releases and Talks at the MySQL Conference
+0 Vote Up -0Vote Down

I spent some time this weekend fixing up the Gearman MySQL UDFs (user defined functions) and fixed a few bugs in the Gearman Server. You can find links to the new releases on the Gearman website. The UDFs now use Monty Taylor’s pandora-build autoconf files instead of the old fragile autoconf setup that relied on pkgconfig.

If you are attending the MySQL Conference & Expo next week and want to learn more about Gearman, be sure to check out one of the three sessions Giuseppe Maxia and I are giving:

  [Read more...]
Logging with MySQL
+3 Vote Up -0Vote Down
I was reading a post by Dathan Vance Pattishall titled "Cassandra is my NoSQL solution but..". In the post, Dathan explains that he uses Cassandra to store clicks because it can write a lot faster than MySQL. However, he runs into problems with the read speed when he needs to get a range of data back from Cassandra. This is the number one problem I have with NoSQL solutions.

SQL is really good at retrieving a set of data based on a key or range of keys. Whereas NoSQL products are really good at writing things and retrieving one item from storage. When looking at redoing our architecture a few years ago to be more scalable, I had to consider these two issues. For what it is worth, the NoSQL market was not nearly as mature as it is now. So, my choices were much more limited. In the end, we decided to stick with

  [Read more...]
Drizzling from the Rackspace Cloud
+5 Vote Up -0Vote Down

Since I left Sun back in January, folks have been asking what was next. I’m happy to say that I’m going to continue hacking on open source projects like Drizzle and Gearman, but now at the Rackspace Cloud. Not only will I be there, but I get to continue working closely with a few of the amazing Drizzle hackers who have also joined, including Monty Taylor, Jay Pipes, Stewart Smith, and Lee Bieber.

Why Rackspace Cloud? Late last year I was considering what I wanted to do next with the Oracle acquisition looming near, and this was one of the options that presented itself. Rackspace had been a supporter of Drizzle from early on by

  [Read more...]
MySQL Conf & Drizzle Dev Day
+1 Vote Up -1Vote Down

I’m glad to announce that we’ll be having a Drizzle developer day again this year on the Friday after the MySQL Conference! Be sure to sign up and add any topic ideas you may have so we know what folks are interested in. Space is limited!

While at the MySQL Conference, I’ll be speaking with Monty Taylor on “Using Drizzle.” This will take a non-developer approach to the project, so everyday DBAs and web developers should find this interesting. I’ll also be teaming up with Giuseppe Maxia to talk about

  [Read more...]
Gearman meets MySQL Cluster (NDBAPI)
+7 Vote Up -0Vote Down
After a discussion with my colleague Stephane Varoqui we decided to see how Gearman and the NDBAPI could be used together. The result of the POC was a Gearman worker and a couple of clients (clients and workers use Google Protocol Buffers as the protocol). The worker can:
  • set/get/delete records on a single table in MySQL Cluster using the primary key
  • set/get/delete "any" type. It is not possible to dynamically add types but this is done at compile time.
  • supports the following SQL data types: (UNSIGNED) INTEGER, (UNSIGNED) BIGINT, CHAR, VARCHAR/VARBINARY
  • supports the following Google Protocol Buffer


  [Read more...]
Multi dimensional cubes in MySQL through Gearman
+4 Vote Up -0Vote Down



I gave two presentations about Gearman at the Linux.conf.au. As part of the preparation for these talks, I created several sample applications. One of them, about remote replication administration, I will cover in a separate post. The most amazing one, which I cover here, is a quick and painless solution for multiple level crosstabs in MySQL.

Some background is needed. Crosstabs (also called data cubes or pivot tables, have been one of my favorite hacks for long time. In 2001 I wrote an article about a simple




  [Read more...]
Using ini files for PHP application settings
+0 Vote Up -0Vote Down
At dealnews we have three tiers of servers. First is our development servers, then staging and finally production. The complexity of the environment increases at each level. On a development server, everything runs on the localhost: mysql, memcached, etc. At the staging level, there is a dedicated MySQL server. In production, it gets quite wild with redundant services and two data centers.

One of the challenges of this is where and how to store the connection information for all these services. We have done several things in the past. The most common thing is to store this information in a PHP file. It may be per server or there could be one big file like:

<?php

if(DEV){
    $server = "localhost";
} else {
    $server = "10.1.1.25";
}

?>











  [Read more...]
Gearman meets MySQL Cluster (NDBAPI)
+0 Vote Up -0Vote Down
After a discussion with my colleague Stephane Varoqui we decided to see how Gearman and the NDBAPI could be used together. The result of the POC was a Gearman worker and a couple of clients (clients and workers use Google Protocol Buffers as the protocol). The worker can:
  • set/get/delete records on a single table in MySQL Cluster using the primary key
  • set/get/delete "any" type. It is not possible to dynamically add types but this is done at compile time.
  • supports the following SQL data types: (UNSIGNED) INTEGER, (UNSIGNED) BIGINT, CHAR, VARCHAR/VARBINARY
  • support the following Google Protocol Buffer


  [Read more...]
Moving On
+3 Vote Up -1Vote Down

Friday was my last day at Sun Microsystems, and today is the first day at my new job (location coming soon). I’ve had a great time at Sun, and thank them for all the opportunities given to me there. I’ll be doing mostly the same work at the new gig, working on projects like Drizzle, but with a slightly different focus. For the most part my day-to-day won’t change much.

Right now I’m focusing on libdrizzle again and am implementing the prepared statement API, cleaning up the MySQL protocol support a little, and also implementing the new Drizzle client/server protocol. I’ll continue to work on Gearman as well, especially where it is relevant to Drizzle. I also need to start blogging again with specific topics in the projects I’m working on, I’ve been fairly quiet lately.

I’ll be in New Zealand next week at Linux Conf AU

  [Read more...]
Speaking, speaking, speaking: Dubai-Sydney-Wellington
+5 Vote Up -0Vote Down


(*)
From January 12th to 27th I will be traveling to the Southern Hemisphere and speaking at two user groups and two conferences.
The schedule (see below) is almost scary. I will be talking about Partitioning (Dubai and Wellington), MySQL Sandbox (Sydney and Wellington), Gearman (Wellington), and some general topics now and then.

The complete schedule and location follows:







  [Read more...]
Gearman: distributed computing and Codebits pictures
+3 Vote Up -0Vote Down



The first Codebits> day lasted until long past midnight. So the attendees were a bit sleepy today, but they were brave and got up early enough for my session.
The presentation covered the basics of Gearman, some advanced magic to install remote MySQL servers, and more magic to enable MySQL users to shoot themselves in the foot repeatedly by combining a gearman/MySQL UDF and some clever scripts.
As usual, the slides are available on slideshare.
Some pictures from codebits 2009 are on Flickr.









  [Read more...]
Codebits 2009, coders conference and competition in Lisbon
+3 Vote Up -0Vote Down

Codebits is approaching. Form December 3rd to 5th, this gathering of 600 developers for a conference, which is also and foremost a competition, will occupy the mind of the best coders in Europe.
I will be a speaker, with two sessions:

Also Lenz will be there, and quite busy. He will also have two sessions:





  [Read more...]
Catching up with Mark Atwood on memcached
Employee +0 Vote Up -0Vote Down
I had an opportunity to catch up with Mark Atwood last week to discuss his new role at Gear6 and some of the interesting developments currently going on around memcached, including Gearman integration and its suitability for cloud computing environments.




Gearman for MySQL
+2 Vote Up -0Vote Down


If you haven't yet heard about Gearman, it's time to have a look at it. Its distributed client server architecture are a perfect match for today's cloud oriented applications.
When talking about Gearman, much stress is often given on its scalability features, such as map/reduce and distributed loads. But Gearman has also a distinctive advantage in the feature department, because of its design that I like to describe as cooperation of the fittest.
While a traditional application is written entirely in a given language, or it must find ways of integrating multiple languages parts, Gearman encourages easy cooperation between parts written in different



  [Read more...]
MySQL University: Gearman for MySQL
Employee +0 Vote Up -0Vote Down

This Thursday (November 12th, 14:00 UTC), Giuseppe Maxia of the MySQL Community Team will present Gearman for MySQL. Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture. Giuseppe's talk will show examples of how to use Gearman for remote installation and how to call a complicate data warehousing function written in Perl from any other language, with no knowledge of Perl at

  [Read more...]
MySQL University: Gearman for MySQL
Employee +0 Vote Up -0Vote Down

This Thursday (November 12th, 14:00 UTC), Giuseppe Maxia of the MySQL Community Team will present Gearman for MySQL. Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture. Giuseppe's talk will show examples of how to use Gearman for remote installation and how to call a complicate data warehousing function written in Perl from any other language, with no knowledge of Perl at

  [Read more...]
MySQL University: Gearman for MySQL
Employee +0 Vote Up -0Vote Down

This Thursday (November 12th, 14:00 UTC), Giuseppe Maxia of the MySQL Community Team will present Gearman for MySQL. Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture. Giuseppe's talk will show examples of how to use Gearman for remote installation and how to call a complicate data warehousing function written in Perl from any other language, with no knowledge of Perl

  [Read more...]
Non-blocking State Machines
+0 Vote Up -0Vote Down

If you’ve ever done any non-blocking programming (usually for socket I/O), you’ve probably had to come up with a non-trivial state machine to handle all the places where everything can pause. Say you’re reading an application level packet from a socket, and half way through the read() system call it screams EAGAIN. You need to stop, save any state, and exit out of whatever chain of functions got you there so the calling application can regain control. I’m going to explain a few techniques I’ve come up with over the years, each with their strengths and weaknesses, and I hope this will spur some conversation of what other folks have done. While I’m fairly happy with how I handle these state machines now, but I’m always looking for a more succinct way of handling things. Please share your thoughts!

Switch Statements

The obvious way to handle non-blocking I/O is

  [Read more...]
Gearman News and Releases
+2 Vote Up -1Vote Down

The past week has brought a surge of Gearman related releases. They include:

C Server and Library
Perl Server and Modules
Gearman::XS Perl Module (based on C library)
PHP Extension (based on C library)
PHP Extension Documentation
Drizzle UDFs
PostgreSQL UDFs
MySQL UDFs

Some of these releases were driven by the C API changes I made to clean a few things up, but a fair amount of functionality was added to








  [Read more...]
Showing entries 1 to 30 of 83 Next 30 Older Entries

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.