Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Previous 30 Newer Entries Showing entries 61 to 90 of 245 Next 30 Older Entries

Displaying posts with tag: Programming (reset)

Win a free book at the February Python Book Contest
+2 Vote Up -0Vote Down

This month is a special month. It’s not because of Valentines day or even the exciting day where we see groundhogs. No, this month is special because I’m have a book contest where you, the reader, get to win something free for doing absolutely nothing more than posting a comment saying that you want one of the several books I have available in the contest.

So without getting into boring details I’ll keep this short. I’ve been reviewing a lot of books lately and I think it’s time to get some books into people’s hands to enjoy themselves. This month the giveaways are all Python oriented.

So, all you have to do is take a look at the following titles and post a comment here saying that you want one of them. At the end of the month two readers will be chosen via a random list sorting python script I’ve whipped up for just this purpose. You

  [Read more...]
Python for Automation: using pdsh for a menu-driven command execution environment
+1 Vote Up -0Vote Down

I’ve been playing around with some quick system automation scripts that are handy to use when you don’t want / need to setup a chef or puppet action. I like to keep all of my hostnames and login details in a MySQL database (a cmdb actually) but for this example we’ll just use a couple of nested lists. This script executes commands in parallel across the hosts you choose in the menu system via the “pdsh” command, so make sure you have that installed before running. Alternately you can change the command call to use ssh instead of pdsh for a serialized execution, but that’s not as fun or fast. With some customizations here and there you can expand this to operate parallelized jobs for simplifying daily work in database administration, usage reporting, log file parsing, or other system automation as you see fit. Here’s the code. Comments welcome as

  [Read more...]
Tech Messages | 2011-02-02
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2011-01-27 through 2011-02-02:




  [Read more...]
Review: MySQL for Python by Albert Lukaszewski
+1 Vote Up -0Vote Down

Packt Publishing recently sent me a copy of MySQL for Python to review and after reading through the book I must say that I’m rather impressed at the variety of topics that the book covers.

It starts off with the basics of setting up MySQL for your testing/development needs by going over several of the common installation and configuration methods. After that it’s a quick intro for connection methods and simple error reporting for connections. The author gives a quick intro to CRUD and how it relates to databases and python before heading into the common tasks of simple queries. I was surprised to see some database profiling discussion; which is rather handy for a new coder or a

  [Read more...]
Simple Python: a job queue with threading
+0 Vote Up -0Vote Down

Every so often you need to use a queue to manage operations in an application. Python makes this very simple. Python also, as I’ve written about before, makes threading very easy to work with. So in this quick program I’ll describe via comments, how to make a simple queue where each job is processed by a thread. Integrating this code to read jobs from a mysql database would be trivial as well; simply replace the “jobs = [..." code with a database call to a row select query.

#!/usr/bin/env python
## DATE: 2011-01-20
## FILE: queue.py
## AUTHOR: Matt Reid
## WEBSITE: http://themattreid.com
from Queue import *
from threading import Thread, Lock

'''this function will process the items in the queue, in serial'''
def processor():
    if queue.empty() == True:
        print "the Queue is empty!"
        sys.exit(1)
    try:
        job = queue.get()
        print "I'm
  [Read more...]
Tech Messages | 2011-01-15
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2011-01-12 through 2011-01-15:

  [Read more...]
Tech Messages | 2011-01-08
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2011-01-07 through 2011-01-08:

  [Read more...]
Tech Messages | 2010-12-21
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2010-12-14 through 2010-12-21:

  [Read more...]
MySQL 5.5: CMake replaces autoconf/automake on all platforms, support for autotools has now been removed
Employee +6 Vote Up -0Vote Down

There has been a lot of buzz about the MySQL 5.5 GA release and its new features and other user-visible improvements. In this blog post, I'd like to touch on a less noticeable, but still important change.

CMake has already been used to build the MySQL Server on Windows for a long time, while the GNU autotools were used on all other platforms. Since MySQL 5.5, all builds on all platforms are now performed using the same tool chain. With the latest release of MySQL 5.5, we've made

  [Read more...]
Adopting RAD in the Enterprise: The 14 Biggest Misconceptions
+0 Vote Up -0Vote Down

Rapid Application Development (RAD) is a way of developing computer software applications with less effort than the traditional means.

RAD tools focus on providing code generation and automated testing capabilities with the use of convention over configuration to provide a streamlined workflow to create applications.

Even with the most advanced and easiest to use RAD tools, there are times which the traditional enterprise and the business software development vendors which are having their own implementations and in-house built frameworks are continuously refusing to adopt them.

Most of the misconceptions on the RAD are based on FUD (Fear, Uncertainty

  [Read more...]
Developer Week in Review
+0 Vote Up -2Vote Down

If you live in the U.S., this is the week to gorge on turkey. I wondered out loud last night to my wife if Thanksgiving is the day of the year when the most people eat the same meal. Can any of our overseas readers add to the conversation? Is there a holiday in your country where everyone eats pretty much the same thing? Anyway, before American brains shut down from an overdose of stuffing, here's some developer news you can use.

Oracle announces Plan B for Java

The Java language has continued to evolve over the years, adding features such as Generics. There's an ambitious wishlist of things that developers would like to see in Java 7, but apparently not enough time to do it all and still get a timely release out. As a result, the JCP has decided to forego

  [Read more...]
Developer Week in Review
+0 Vote Up -1Vote Down

Here's what's new for the trendy developer this week:

Java's future on Apple: Slightly less in doubt

Last week, it looked like Apple was all "You're not welcome here, Java." In the changeable world that is Jobsland, this week Apple was offering to marry the language, reiterating their support for Java in OS X, and indicating that they would be supplying code and resources to the OpenJDK project.

As I've noted before, this makes sense for Apple, because it gets them out of the JVM business, and makes Oracle the one-stop shopping solution for all your JDK and JRE needs. It also means that the Mac can be added as a regression-tested target for a new version of Java, hopefully

  [Read more...]
Developer Week in Review
+0 Vote Up -1Vote Down

Here's your weekly helping of developer info:

The sudden but inevitable Apple news

Several pieces of news on the Apple front this week. First up: the Gold Master seed (which either sounds like something you plant to get nice apples, or something out of a bad SF eugenics novel) for iOS 4.2 dropped, signaling the green light for iPad/iPhone/iPod developers to submit 4.2-ready applications to the App Store. Traditionally, the pre-release to developers is followed about a week later by the general release, and is identical.

Meanwhile, continuing to muddy the waters about what is and isn't allowed on the iPhone, Adobe gave a

  [Read more...]
The future of replication revealed in Istanbul
+7 Vote Up -0Vote Down

A very good meeting in Istanbul is drawing to an end. People from Monty Program, Facebook, Galera, Percona, SkySQL, and other parts of the community are meeting with one foot on the European continent and another in Asia to discuss all things MariaDB and MySQL and experience the mystery of the Orient.

At the meeting I had the opportunity to present my plans and visions for the future development of replication in MariaDB. My talk was very well received, and I had a lot of good discussions afterwards with many of the bright people here. Working from home in a virtual company, it means a lot to get this kind of inspiration and encouragement from others on occasion, and I am looking forward to continuing the work after an early flight to Copenhagen tomorrow.

The new interface for transaction coordinator plugins is what

  [Read more...]
Always Test with Real Data
+0 Vote Up -0Vote Down

As I previously noted, I’m in the midst of converting some data (roughly 2 billion records) into documents that will live in a MongoDB cluster. And any time you move data into a new data store, you have to be mindful of any limitations or bottlenecks you might encounter (since all systems have had to make compromises of some sort or another).

In MySQL one of the biggest compromises we make is deciding what indexes really need to be created. It’s great to have data all indexed when you’re searching it, but not so great when you’re adding and deleting many rows.

In MongoDB, the thing that gets me is the document size limit. Currently an object stored in MongoDB cannot be larger than 4MB (though that’s likely to be raised soon). Now, you can build your

  [Read more...]
Dynamic linking costs two cycles
+9 Vote Up -0Vote Down

It turns out that the overhead of dynamic linking on Linux amd64 is 2 CPU cycles per cross-module call. I usually take forever to get to the point in my writing, so I thought I would change this for once :-)

In MySQL, there has been a historical tendency to favour static linking, in part because to avoid the overhead (in execution efficiency) associated with dynamic linking. However, on modern systems there are also very serious drawbacks when using static linking.

The particular issue that inspired this article is that I was working on MWL#74, building a proper shared libmysqld.so library for the MariaDB embedded server. The lack of a proper libmysqld.so in MySQL and MariaDB has caused no end of grief for packaging Amarok for the various Linux distributions.

  [Read more...]
dbbenchmark.com – MySQL (basic) connection pool support added
+1 Vote Up -0Vote Down

In this latest release I’ve added a basic MySQL connection pool to the benchmarking script which improves the method in which connections to MySQL are handled and reused. In addition, there have been some optimizations made to the thread handler functions for better debug reporting. Download the latest release now and see how your MySQL server performs against the rest of the community! Download here: download page.

Micro-benchmarking pthread_cond_broadcast()
+2 Vote Up -0Vote Down

In my work on group commit for MariaDB, I have the following situation:

A group of threads are going to participate in group commit. This means that one of the threads, called the group leader, will run an fsync() for all of them, while the other threads wait. Once the group leader is done, it needs to wake up all of the other threads.

The obvious way to do this is to have the group leader call pthread_cond_broadcast() on a condition that the other threads are waiting for with pthread_cond_wait():

  bool wakeup= false;
  pthread_cond_t wakeup_cond;
  pthread_mutex_t wakeup_mutex

Waiter:

  pthread_mutex_lock(&wakeup_mutex);
  while (!wakeup)
    pthread_cond_wait(&wakeup_cond, &wakeup_mutex);
  pthread_mutex_unlock(&wakeup_mutex);

  [Read more...]
dbbenchmark.com – configuring OpenBSD for MySQL benchmarking
+1 Vote Up -0Vote Down

Here are some quick commands for installing the proper packages and requirements for the MySQL dbbenchmark program.

export PKG_PATH="ftp://openbsd.mirrors.tds.net/pub/OpenBSD/4.7/packages/amd64/"
pkg_add -i -v wget
wget http://dbbenchmark.googlecode.com/files/dbbenchmark-version-0.1.beta_rev26.tar.gz
pkg_add -i -v python
Ambiguous: choose package for python
 a       0: 
         1: python-2.4.6p2
         2: python-2.5.4p3
         3: python-2.6.3p1
Your choice: 2

pkg_add -i -v py-mysql
pkg_add -i -v mysql
pkg_add -i -v mysql-server
ln -s /usr/local/bin/python2.5 /usr/bin/python
gzip -d dbbenchmark-version-0.1.beta_rev26.tar.gz
tar -xvf dbbenchmark-version-0.1.beta_rev26.tar
cd dbbenchmark-version-0.1.beta_rev26
./dbbenchmark.py --print-sql
 - login to mysql and execute sql commands
./dbbenchmark.py
dbbenchmark.com – Debian Lenny, MEMORY_ACTIVE bug fix
+1 Vote Up -3Vote Down

Quick solution to an issue that the affected Debian Lenny release: the process used to collect the MEMORY_ACTIVE_BYTES variable has been modified to correct a situation where some systems report an array of memory information instead of the expected single integer value. The bug has been fixed in revision 21 and the current download (revision 22) is available for download or svn update. As usual, you can download the MySQL dbbenchmark script here: [downloads].

Thanks goes to Brian Vowell at Evernote.com for bringing this bug to my attention. The original bug report can be found here: [link]

Easy Python: multi-threading MySQL queries
+2 Vote Up -3Vote Down

There are many times when writing an application that single threaded database operations are simply too slow. In these cases it’s a matter of course that you’ll use multi-threading or forking to spawn secondary processes to handle the database actions. In this simple example for Python multi-threading you’ll see the how simple it is to improve the performance of your python app.

#!/usr/bin/python
## DATE: 2010-08-30
## AUTHOR: Matt Reid
## WEBSITE: http://themattreid.com
## LICENSE: BSD http://www.opensource.org/licenses/bsd-license.php
## Copyright 2010-present Matt Reid

from __future__ import division
from socket import gethostname;
import threading
import sys
import os
import MySQLdb

class threader(threading.Thread):
    def __init__(self,method):
        threading.Thread.__init__(self)
        self.tx =
        self.method = method
    def run(self):
  [Read more...]
Easy Python: display LVM details in XML
+1 Vote Up -1Vote Down

If you need to work with LVM in your scripts but haven’t found a good method to access details about Logical Volume Groups, here’s a simple Python script that will print the details about any volumes on your system. This could be useful for writing a partition check script for your MySQL data directory (if you’re not using a standard monitoring system like Nagios).

import sys
import os
import commands
import subprocess
import select

def lvm():
    print ""
    LVM_PATH = "/sbin"
    LVM_BIN = os.path.join(LVM_PATH, 'lvm')
    argv = list()
    argv.append(LVM_BIN)
    argv.append("lvs")
    argv.append("--nosuffix")
    argv.append("--noheadings")
    argv.append("--units")
    argv.append("b")
    argv.append("--separator")
    argv.append(";")
    argv.append("-o")
    argv.append("lv_name,vg_name,lv_size")

    process = subprocess.Popen(argv,
  [Read more...]
Easy Python: MySQL connection and iteration
+3 Vote Up -0Vote Down

If you’ve been looking for a simple python script to use with MySQL that you can use to expand upon for your next project, check this one out. It has error handling for the connection, error handling for the sql call, and loop iteration for the rows returned.

#!/usr/bin/python
import sys
import MySQLdb

my_host = "localhost"
my_user = "user"
my_pass = "password"
my_db = "test"

try:
    db = MySQLdb.connect(host=my_host, user=my_user, passwd=my_pass, db=my_db)
except MySQLdb.Error, e:
     print "Error %d: %s" % (e.args[0], e.args[1])
     sys.exit (1)

cursor = db.cursor()
sql = "select column1, column2 from table";
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
    column1 = row[0]
    column2 = row[1]
    print "column1: %s, column2: %s"%(column1,column2)

db.close()
N900 – control all of your accounts with this script
+1 Vote Up -1Vote Down

If you own a Nokia N900 cellular device you might be interested in the ability to control all of your IM accounts from the command line. For those that do not know, the N900 runs Maemo Linux and is capable of running MySQL embedded if you so choose. Here’s a quick script I wrote to provide that functionality for IM accounts. It’s at the bottom of the page, called “im-connections”.

wiki: http://wiki.maemo.org/N900_Mission_Control#Set_all_SIP_accounts_to_online_or_offline
pastebin: http://pastebin.com/qAC57E1N

MySQL/MariaDB replication: applying events on the slave side
+4 Vote Up -2Vote Down

Working on a new set of replication APIs in MariaDB, I have given some thought to the generation of replication events on the master server.

But there is another side of the equation: to apply the generated events on a slave server. This is something that most replication setups will need (unless they replicate to non-MySQL/MariaDB slaves). So it will be good to provide a generic interface for this, otherwise every binlog-like plugin implementation will have to re-invent this themselves.

A central idea in the current design for generating events is that we do not enforce a specific content of events. Instead, the API provides accessors for a lot of different information related to each event, allowing the plugin flexibility in choosing what to

  [Read more...]
Dissecting the MySQL replication binlog events
+1 Vote Up -0Vote Down

For the replication project that I am currently working on in MariaDB, I wanted to understand exactly what information is needed to do full replication of all MySQL/MariaDB statements on the level of completeness that existing replication does. So I went through the code, and this is what I found.

What I am after here is a complete list of what the execution engine needs to provide to have everything that a replication system needs to be able to completely replicate all changes made on a master server. But not anything specific to the particular implementation of replication used, like binlog positions or replication event disk formats, etc.

The basic information needed is of course the query (for statement-based replication), or the column values (for row-based replication). But there are lots of extra details

  [Read more...]
Four short links: 21 June 2010
+0 Vote Up -0Vote Down

  • Law of Success 2.0 -- a blog of interviews with famous and/or interesting people, from Brad Feld to Uri Geller.
  • Pioneer One -- crowdsourced funding for TV show, perhaps a hint of the future. Pilot shot for $6,000 which was raised through KickStarter. Distributed via BitTorrent.
  • DrasticTools -- PHP/MySQL visualisation tools, including TreeMap, tag cloud, hierarchical bar chart, and animated list. (via TomC on Delicious)
  • GoogleCL -- command-line interface to Google services. At the moment the services are Picasa, Blogger, YouTube, Contacts, Docs, and
  •   [Read more...]
    Fixing MySQL group commit (part 4 of 3)
    +3 Vote Up -0Vote Down

    (No three-part series is complete without a part 4, right?)

    Here is an analogy that describes well what group commit does. We have a bus driving back and forth transporting people from A to B (corresponding to fsync() "transporting" commits to durable storage on disk). The group commit optimisation is to have the bus pick up everyone that is waiting at A before driving to B, not drive people one by one. Makes sense, huh? :-)

    It is pretty obvious that this optimisation of having more than one person in the bus can dramatically improve throughput, and it is the same for the group commit optimisation. Here is a graph from a benchmark comparing stock MariaDB 5.1 vs. MariaDB patched

      [Read more...]
    Down the rabbit hole
    +0 Vote Up -0Vote Down

    Generally I avoid going down rabbit holes but today I decided to see how deep a particular testing rabbit hole went. This post is a third in what seems be a continuing series of programming anecdotes. It’s not particularly MySQL-related so you can stop reading here unless you grok code stuff.

    Before beginning work on issue 720 I ran the mk-table-checksum test suite to make sure it was in working order. No sense writing new tests and code when the old tests and code aren’t reliable. I actually made one seemingly innocuous change to the test suite in preparation for the issue: I changed the –replicate checksum table from MyISAM to InnoDB.

    Surprisingly, the test suite proved unstable. Random tests would fail at random times. Some

      [Read more...]
    MongoDB Early Impressions
    +0 Vote Up -0Vote Down

    I’ve been doing some prototyping work to see how suitable MongoDB is for replacing a small (in number, not size) cluster of MySQL servers. The motivation for looking at MongoDB in this role is that we need a flexible and reliable document store that can handle sharding, a small but predictable write volume (1.5 – 2.0 million new documents daily), light indexing, and map/reduce operations for heavier batch queries. Queries to fetch individual documents aren’t that common–let’s say 100/sec in aggregate at peak times.

    What I’ve done so far is to create a set of Perl libraries that abstract away the data I need to store and provide a “backend” interface to which I can plug in a number of modules for talking to different data stores (including some

      [Read more...]
    Previous 30 Newer Entries Showing entries 61 to 90 of 245 Next 30 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.