Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Previous 30 Newer Entries Showing entries 61 to 90 of 249 Next 30 Older Entries

Displaying posts with tag: Python (reset)

Invalid dates returning None, or raise error using Connector/Python?
Employee +0 Vote Up -0Vote Down

In this blog we discuss invalid dates in MySQL (http://www.mysql.com" target="_blank), how to retrieve them using Connector/Python and we raise the question: Should Connector/Python raise an error or just keep returning None on invalid dates?

If you run MySQL without proper SQL Modes, you will be able to update and
read invalid dates such as ’2012-06-00′. If you’ve payed attention the past decade, you’ll know that you can prevent this configuring your MySQL server setting SQL Mode to ‘TRADITIONAL’.

Now, the problem if this is allowed, how do we get invalid dates using MySQL Connector/Python?

Lets look at an example inserting an invalid date and trying to read


  [Read more...]
Tech Messages | 2012-05-11
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2011-09-21 through 2012-05-11:

MySQL Utilities and Global Transaction Identifiers
Employee +7 Vote Up -0Vote Down
The new MySQL 5.6 Development Milestone Release (DMR) includes many new enhancements. One of the most impressive is the use of Global Transaction Identifiers (GTIDs) for replication. With GTIDs enabled, administrators no longer need to keep track of binary log files and positions. In a nutshell, GTIDs simplify the setup and maintenance of replication.

MySQL Utilities has taken this a step further by providing two new utilities that automate two of the most complex replication administration tasks - switchover and failover. Switchover is changing the role of an active, healthy master to one of its slaves whereas failover is the act of promoting a candidate slave to become the new master. Clearly, switchover is an elective operation and failover is performed when there are issues with the master.

The GTID utilities are included in release-1.0.5 of



  [Read more...]
MySQL Connector/Python bugs reports on bugs.mysql.com
Employee +0 Vote Up -0Vote Down

We have moved bugs for MySQL Connector/Python from Launchpad to the MySQL Bugs website http://bugs.mysql.com. Reports which are (probably) fixed in newer code were not taken with. If there is a bug which you really want to get tracked: please report it again.

Please use the MySQL Bugs website to report problems using MySQL Connector/Python. To see a list of active reports, click here.

  [Read more...]
Splitmytab ready for the public!
+1 Vote Up -2Vote Down

Splitmytab.net is finally for the public to check out. Splitmytab is a bill splitting and IOU system for friends. It uses facebook’s login, so you won’t need to put in anyone’s emails, names, or get people to sign up for an account.

It’ll automatically keep balances of who owes who, so you can keep a running tab with friends and always know who’s buying the next case of beer.

Please note: I’m not a designer, so there’s a few rough corners, but what’s there is simple and it works.

Tech Notes:

  • Backend is MySQL 5.5
  • Written in Python
  • Nginx with tornado
  • Redis used on occasion
  • Originally was writing pure JS then switched to Coffeescript

Enjoy, and please leave feedback!

Pythonic Database API: Now with Launchpad
Employee +2 Vote Up -1Vote Down
In a previous post, I demonstrated a simple Python database API with a syntax similar to jQuery. The goal was to provide a simple API that would allow Python programmers to use a database without having to resort to SQL, nor having to use any of the good, but quite heavy, ORM implementations that exist. The code was just an experimental implementation, and I was considering putting it up on Launchpad.
I did some basic cleaning of the code, turned it into a Python package, and pushed it to Launchpad. I also added some minor changes, such as introducing a define function to define new tables instead of automatically creating one when an insert was executed. Automatically constructing a table from values seems neat, but
  [Read more...]
Open APIs are the new open source
+1 Vote Up -0Vote Down

We’ve seen the rise of open source software in the enterprise and also beyond the IT industry, but the real keys to openness and its advantages in today’s technology world — where efficient use of cloud computing and supporting services are paramount — exist in open application programming interfaces, or APIs.

Open source software continues to be a critical part of software development, systems administration, IT operations and more, but much of the action in leveraging modern cloud computing and services-based infrastructures centers on APIs. Open APIs are the new open source.

Read the full story at LinuxInsider.

MySQL: Python, Meta-Programming, and Interceptors
Employee +2 Vote Up -0Vote Down
I recently found Todd's posts on interceptors which allow callbacks (called interceptors) to be registered with the connector so that you can intercept a statement execution, commit, or any of the many extension points supported by Connector/Java. This is a language feature that allow you to implement a number of new features without having to change the application code such as load-balancing policies, profiling queries or transactions, or debugging an application.

Since Python is a dynamic language, it is easy to add

  [Read more...]
Automatic reconnect in MySQL Connector/Python?
Employee +0 Vote Up -0Vote Down

There have been some request to have some reconnect possibilities in Connector/Python. I’m wondering now whether there should be some automatic reconnect on certain errors within the database driver.

My personal feeling is to have no automatic reconnect within Connector/Python and the programmer has to come up with retrying transactions herself.

For example:

	cnx.disconnect() # For testing..
	tries = 2
	while tries > 0:
		tries -= 1
		try:
			cursor.execute("INSERT INTO t1 (c1) VALUES ('ham')")
			cnx.commit()
		except mysql.connector.InterfaceError:
			if tries == 0:
				print "Failed inserting data after retrying"
				break
			else:
				print "Reconnecting.."
				cnx.reconnect()
		else:
			break

The above mimics how you would handle

  [Read more...]
Installing MySQLdb on MacOS Lion
+0 Vote Up -0Vote Down

I ran into an issue installing the MySQLdb module.

>>> import MySQLdb
/Library/Python/2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.7-intel.egg/_mysql.py:3: UserWarning: Module _mysql was already imported from /Library/Python/2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.7-intel.egg/_mysql.pyc, but /Users/jhaddad/Downloads/MySQL-python-1.2.3 is being added to sys.path
Traceback (most recent call last):
File ““, line 1, in
File “MySQLdb/__init__.py”, line 19, in
import _mysql
File “build/bdist.macosx-10.7-intel/egg/_mysql.py”, line 7, in
File “build/bdist.macosx-10.7-intel/egg/_mysql.py”, line 6, in __bootstrap__
ImportError:







  [Read more...]
MySQL Connector/Python available through the Python Package Index
Employee +2 Vote Up -0Vote Down

Today we registered MySQL Connector/Python with the Python Package Index (PyPI). It makes installing your favorite connector even easier (provided you first install setuptools or pip):

shell> easy_install mysql-connector
shell> pip install mysql-connector

Please report problems either using Launchpad or MySQL Bugs website.

MySQL Connector/Python bug category on bugs.mysql.com
Employee +0 Vote Up -0Vote Down

In addition to reporting MySQL Connector/Python bugs on Launchpad, it is now also possible to enter them using http://bugs.mysql.com.

My New Job at Oracle: Working on MySQL Connector/Python
Employee +2 Vote Up -0Vote Down

After more than 6 years doing MySQL Support for MySQL (http://www.mysql.com) AB, Sun Microsystems, and Oracle, it’s time for a change. Time to get back to development!

As of November 2011 I’ll be working full-time on MySQL Connector/Python and other goodies within the MySQL development team at Oracle. Before, this was more or less a pet project done after working hours. However, with the birth of our son Tomas more than a year ago, I’ve been slacking and family got priority.

The idea is to make MySQL Connector/Python the best choice for connecting to MySQL from within your Python code. We still got a long road

  [Read more...]
MySQL Utilities Release 1.0.3
Employee +3 Vote Up -0Vote Down
The MySQL Utilities project continues to evolve with key new features for
replication and export. The latest release, 1.0.3, is no exception.

MySQL Utilities is included in the MySQL Workbench product which can be
downloaded from HTTP://dev.mysql.com/downloads/workbench/5.2.html

If you want the latest developments for MySQL Utilities, you can create a
bazaar branch using the following command:

bzr branch lp:~mysql/mysql-utilities/trunk

New Utility - mysqlrplshow

You can now view a list of the slaves attached to your master with
mysqlrplshow. The utility displays a graph of the master and its slaves y
default but you can also get a list of the slaves in GRID, CSV, TAB, or
VERTICAL format as follows.

  • GRID - Displays output formatted like that of the mysql monitor


















  [Read more...]
Python Interface to MySQL
Employee +1 Vote Up -0Vote Down
There has been a lot of discussions lately about various non-SQL languages that provide access to databases without having to resort to using SQL. I wondered how difficult it would be to implement such an interface, so as an experiment, I implemented a simple interface in Python that similar to the document-oriented interfaces available elsewhere. The interface generate SQL queries to query the database, but does not require any knowlegdge of SQL to use. The syntax is inspired by JQuery, but since JQuery works with documents, the semantics is slightly different.

A simple example would look like this:

from native_db import *
server = Server(host='127.0.0.1')
server.test.t1.insert({'more': 3, 'magic': 'just a test', 'count': 0})
server.test.t1.insert({'more': 3, 'magic': 'just another test', 'count': 0})
server.test.t1.insert({'more': 4,

  [Read more...]
On Password Strength
+6 Vote Up -0Vote Down
XKCD (as usual) makes a very good point – this time about password strength, and I reckon it’s something app developers need to consider urgently. Geeks can debate the exact amount of entropy, but that’s not really the issue: insisting on mixed upper/lower and/or non-alpha and/or numerical components to a user password does not really improve security, and definitely makes life more difficult for users. So basically, the functions that do a “is this a strong password” should seriously reconsider their approach, particularly if they’re used to have the app decide whether to accept the password as “good enough” at all. Update: Jeff Preshing has written an   [Read more...]
Generating dimension data for dates
+0 Vote Up -0Vote Down

Most analytical and BI databases have date dimension table(s). One frequently needs to generate and populate such data. I present a solution below for such data generation, written in Python. Please use different database drivers/modules to connect to your specific database server (MySQL, SQL Server, Oracle, etc.) for data population.

Notes:

1. It takes 2 parameters, start date and end date, in YYYYMMDD format, inclusive. Extensive error checking is built in, but let me know if you have comments/suggestions;

2. The script produce a Python dictionary (associated array) and print out its content;

3. The output includes dayNumber: a day’s position in a year. For example, 2011-02-01 is the 32ed day in 2011, therefore its dayNumber is 32;

4. The output includes weekNumber: a week’s position in a year. The week number in year is based

  [Read more...]
Connecting to MySQL with Perl
+0 Vote Up -0Vote Down

When I was designing web sites, for a long time I wrote my HTML code the “hard” way – by opening a text editor and manually typing in the code (this was before I purchased Adobe DreamWeaver).

During that time of manual HTML writing, I had a project that required forms on a web page, and I needed a place to store the information. After talking with a few tech friends, I decided to use MySQL as my web site database, and Perl as my scripting language.

I had written complex Bourne shell scripts before, but Perl was something entirely new. With a little help from a buddy of mine, after a few hours I was off and running. I was amazed at how easy it was to connect to a MySQL database with Perl.

This example will show you how to

  [Read more...]
A comparison of HandlerSocket and mysql client libraries with Python
+0 Vote Up -0Vote Down

I’ve done some benchmark testing of 2 Python modules for MySQL data retrieval: MySQLdb and pyhs. MySQLdb uses MySQL’s client libraries, whereas pyhs uses HandlerSocket that bypasses MySQL’s client layer and interfaces Innodb storage engine’s files directly. In my testing, HandlerSocket results in 82% improvement over mysql client libraries based on number of rows retrieved. The tests were conducted under different conditions: right after a start when cache is cold, a warmed up cache after running SELECT * FROM customer, and alternating the execution order of those 2 Python files.

  [Read more...]
Finding long running INNODB transactions
+0 Vote Up -0Vote Down

Notes:
1. The script prints out elapsed time since transaction started, MySQL thread id, and the kill statement for transactions running longer than a defined threshold value, in seconds. Just copy, paster, and then execute the kill statement if you want to terminate the long transaction(s);
2. Adjust shellCmd variable;
3. Adjust longRunningThreshold value as needed. It is measured in seconds;
4. No special libraries/modules needed, as long as there is a working mysql client;
5. re module is used for regex processing. Good place to find examples of regular expression search and grouping. A status variable is used to assist locating MySQL thread id once a transaction running longer than the defined threshold is found.

import re, shlex, subprocess 

def runCmd(cmd):
    proc = subprocess.Popen(shlex.split(cmd),





  [Read more...]
Quadrant Framework – rev7 update adds DyGraphs support
+0 Vote Up -0Vote Down

Quick update to the framework that was released yesterday; I’ve added automatic graph generation. I chose DyGraphs due to the quick ability to enable support – the HTML is very quick and simply loads the CSV data. It has the same zooming features of Highcharts without the JS overhead.

Now when you run a load test you will get (in the output directory) a mixture of files: the main cumulative CSV and HTML file for the hostname that was tested, and then one CSV and HTML per report variable that was tested. This means you don’t have to drag the main CSV file into an alternate program or spend time parsing out certain variables one at a time to generate specific graphs.  I’ve also added support for limiting output of SNMP variables (LOAD,CPU,MEM). Head over here and download the update: 

  [Read more...]
MySQL Load Testing Framework – initial release
+0 Vote Up -0Vote Down

It seems that everyone loves load testing these days. Problem is that everyone is using their own quick scripts, simple or complex, to drive their tests without the ability for other DBAs to duplicate those tests. Let’s say I write a great test and share my results and graphs on the blog – you want to run the same tests to see how your new DB servers compare in performance: this framework allows you to do that without duplicating any work or writing code. This is a basic release that will get the ball rolling. I’ve included some sample tests in the README file, so give them a try.

This codebase offers a user friendly framework for creating and visualizing MySQL database load test jobs. It is based around Sysbench, which is generally considered the industry standard load test application. The framework allows you to do the following:

    standardize your tests
  [Read more...]
MySQL Community – what do you want in a load testing framework?
+0 Vote Up -0Vote Down

So I’ve been doing a fair number of automated load tests these past six months. Primarily with Sysbench, which is a fine, fine tool. First I started using some simple bash based loop controls to automate my overnight testing, but as usually happens with shell scripts they grew unwieldy and I rewrote them in python. Now I have some flexible and easily configurable code for sysbench based MySQL benchmarking to offer the community. I’ve always been a fan of giving back to such a helpful group of people – you’ll never hear me complain about “my time isn’t free”. So, let me know what you want in an ideal testing environment (from a load testing framework automation standpoint) and I’ll integrate it into my existing framework and then release it via the BSD license. The main goal here is to have a standardized modular framework, based on sysbench,

  [Read more...]
Another Attempt At Python
+0 Vote Up -1Vote Down

I tried Python out a while ago, but stopped trying it to learn it after some major frustrations. Maybe I didn’t dig deep enough into it. I found the documentation hard to read, and the module layout seemed a little random at times. For some reason I found executing an external process and getting the results to be a little convoluted. (Since then I’ve learned to use popen(..).communicate())

I ended up messing with other languages to try to find one that suits my tastes, like Erlang and

  [Read more...]
Refactored: Poor man’s MySQL replication monitoring
Employee +2 Vote Up -0Vote Down

This is a reply to the blog post Poor man’s MySQL replication monitoring. Haidong Ji had a few problems using MySQLdb (could use the ‘dict’ cursor) and apparently he doesn’t want to much dependencies. I agree that using the mysql client tool is a nice alternative if you don’t want to use any 3rd party Python modules. And the MySQL client tools are usually and should be installed with the server.

However, since MySQL Connector/Python only needs itself and Python, dependencies are reduced to a minimum.

  [Read more...]
Poor man’s MySQL replication monitoring
+1 Vote Up -0Vote Down

Using MySQL replication slave(s) for reporting (with potentially different storage engines) is a very popular way of scaling database read activities. As usual, you want to be on top of things when replication breaks so end users can be notified and issues addressed. When Nagios, Zabbix, or whatever monitoring tools are not available or otherwise not accessible, there got to be another way. I wrote the following Python script for that purpose. The script can then be scheduled via crontab to check replication status in an interval you define. When things break, you get a notification email.

Notes:
1. I toyed with MySQLdb Python module for this task, but I don’t like the fact that I cannot easily retrieve values via column names in a MySQLdb cursor. If there is an easier way that I am not aware of due to my ignorance, I’d appreciate it if you could

  [Read more...]
Install MySQLdb module for Python
+0 Vote Up -0Vote Down

Update:

Commenter MarkR made a great point: if possible, use some packaging tools, to try to maintain proper dependencies, to the extent that is possible. Install from the source should be Plan B. So, try yum install MySQL-python first.

This is mostly for my own future reference. It’ll be icing on the cake if it helps you!

This is geared for CentOS or Red Hat. Use apt-get or other packaging tools for different flavours of Linux.

1. Get Python module setuptools called easy_install. I love easy_install, by the way, sort of like CPAN for Perl modules;
2. To install MySQLdb package, you would think easy_install MySQLdb would do. But that is not the case. I hope the developer would fix that. Instead, you need:

easy_install MySQL-python

3. If you have build errors, you may


  [Read more...]
Tech Messages | 2011-03-07
+0 Vote Up -0Vote Down

A special extended edition of Tech Messages for 2011-02-10 through 2011-03-07:

  [Read more...]
Win a free book at the February Python Book Contest
+2 Vote Up -0Vote Down

This month is a special month. It’s not because of Valentines day or even the exciting day where we see groundhogs. No, this month is special because I’m have a book contest where you, the reader, get to win something free for doing absolutely nothing more than posting a comment saying that you want one of the several books I have available in the contest.

So without getting into boring details I’ll keep this short. I’ve been reviewing a lot of books lately and I think it’s time to get some books into people’s hands to enjoy themselves. This month the giveaways are all Python oriented.

So, all you have to do is take a look at the following titles and post a comment here saying that you want one of them. At the end of the month two readers will be chosen via a random list sorting python script I’ve whipped up for just this purpose. You

  [Read more...]
Python for Automation: using pdsh for a menu-driven command execution environment
+1 Vote Up -0Vote Down

I’ve been playing around with some quick system automation scripts that are handy to use when you don’t want / need to setup a chef or puppet action. I like to keep all of my hostnames and login details in a MySQL database (a cmdb actually) but for this example we’ll just use a couple of nested lists. This script executes commands in parallel across the hosts you choose in the menu system via the “pdsh” command, so make sure you have that installed before running. Alternately you can change the command call to use ssh instead of pdsh for a serialized execution, but that’s not as fun or fast. With some customizations here and there you can expand this to operate parallelized jobs for simplifying daily work in database administration, usage reporting, log file parsing, or other system automation as you see fit. Here’s the code. Comments welcome as

  [Read more...]
Previous 30 Newer Entries Showing entries 61 to 90 of 249 Next 30 Older Entries

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.