I have a small EC2 instance running with a 25GB EBS volume attached. It has a database on it that I need to manipulate by doing things like dropping indexes and creating new ones. This is on rather large (multi-GB, millions of rows) tables. After running one DROP INDEX operation that ran all day without finishing, I killed it and tried to see what was going on. Here’s the results of the first 10 minutes of testing:

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=1000
1000+0 records in
1000+0 records out
131072000 bytes (131 MB) copied, 0.818328 seconds, 160 MB/s

This looks great. I’d love to get 160MB/s all the time. But wait! There’s more!

-bash-3.2# dd if=/dev/zero of=/vol/128.txt bs=128k count=100000
dd: writing `/vol/128.txt': No space left on device
86729+0 records in
86728+0 records out
11367641088 bytes (11 GB) copied, 268.191 seconds, 42.4 MB/s

Ok, well… that’s completely miserable. Let’s try …

Short Version: You can find a fantastic video here about bundling customized AMIs and registering them with Amazon so that you can launch as many instances of your new AMI as you want. The video is so good that I don’t bother writing out the steps to do the bundling (it would be pretty darn long). These are some notes about launching an AMI, customizing it, and mounting an EBS volume to it (the video linked above doesn’t cover EBS). Also, check out the ElasticFox tool which is a very good GUI for doing simple EC2 operations. Nice if you’re just getting started or doing some simple tests.

There are two ways you can go about creating a custom machine image (AMI) for use with Amazon EC2: You can create an image locally by dd’ing to a file, mounting it with “-o loop” creating a filesystem on it, …

Hi all,

The schedule for PyWorks has been posted! I’m really excited about three things:

1) there are some really cool talks that I’m looking forward to attending. There are a couple of sysadmin-related talks, AppEngine, TurboGears, Django, and an area I’ve been especially slow to move into: testing (I know, shame on me). There’s lots more so be sure to check it out.

2) the conference scheduling process is over

3) I get to meet a lot of people face-to-face that I’ve worked with in the past on Python Magazine developing articles, or interacted with on IRC, etc. One thing I like about conferences surrounding open source technologies is you get to thank people face-to-face for the sweat they poured into some of the tools I use regularly. Mark Ramm, Kevin Dangoor, Michael Foord, Brandon Rhodes, and a collection of Python Magazine authors …

Holy cow... It really works!! (Just like they promised!)

read more

According to Wikipedia, in April 2008, the number of videos on Youtube was 83.4 million (ref: http://en.wikipedia.org/wiki/YouTube#cite_note-5). However, the link in the cite note now displays “*†video results 1 – 20 of millions, without showing the real count.

Here's one way I found to get an estimated, but relatively accurate, number of videos on the popular video sharing site Youtube. The idea is simple. Get this feed: http://gdata.youtube.com/feeds/api/videos/-/* and parse out the number inside the <opensearch:totalresults> tag.

…

Problem Statement : 386 GB of MySQL 5.0 Compressed MyISAM tables (Merge engine used)  need to be migrated to MySQL 5.1 Compressed/Partitioned tables. Also we had to add a few new columns and change the datatype of few existing columns.

Initial plan : Setup the MySQL 5.1 database with the schema with partitioning to which the data needs to be migrated and then run scripts to copy the .MYD ,.MYI and .frm files to the 5.1 database. Disable the keys using “Alter table <> disable keys“. Then migrate the data using the insert into newtable select * from old table. Then to compress the tables run myisampack and myisamchk.

As per plan we started the migration process. The plan worked perfectly though a bit on the slower side for most of the tables. For few of the tables, the import process kept dragging on and on. It was well past a week since the import was started and the system …

I have a lot of interaction with publishing types. I write a lot, and I edit some, and I do tech reviews and stuff for some publishers, and I co-authored a book, and I’ve worked on two magazines, and a newspaper, and I’m generally fascinated by the technical book market and stuff like that. I’m also someone who is lucky enough that his job is also his hobby. I work in technology, and am always doing something technology related at home in my spare time. Needless to say, I read tons upon tons of technical books.

I almost never post book reviews, in spite of the fact that I read all of these books. Why? Well, to be honest, I couldn’t tell you. It just hasn’t occurred to me to write a book review. Could be because I don’t really value book reviews too much myself I guess. I mean, if there’s a really obvious consensus across a huge number of reviews, I might be swayed. But in general, I find that book reviews are too often the …

I got to actually speak to Brian Aker for maybe a total of 5 minutes after his micro-presentation about Drizzle, which took place at the Sun booth at OSCON 2008. I was a bit nervous to ask what questions I had out loud, because the things I had wondered about were things I really didn’t see too much discussion about out in the intarweb. I’m happy to report that, if Brian Aker is to be considered any kind of authority (hint: he is), my ideas are not completely ridiculous, so maybe I’ll start talking a bit more about them.

UPDATE: lest anyone get the wrong idea, Brian Aker did, in fact, state that views are not on the short list of priority items for Drizzle, but he did say that views are one of the features he finds most useful, and that they’d probably be higher on any future priority list than, say, …

Launching a Startup in 3 Hours was a great talk given by Andrew Hyde (of techstars.org) and Gavin Doughtie (of Google). Both of the speakers are heavily involved in the recent trend of doing “Startup Weekends”, and techstars.org is an organization that hosts startup weekends all around the US (and I think internationally as well - Andrew mentioned one in Germany if I heard correctly).

The first half of the talk was about the general concept of a startup weekend, the problems it avoids (”we’ve been working for 9 months and haven’t launched anything”), the problems it brings up (”If you’re not using Java, you’re an idiot, so count me out!!”), and lots of details about how to organize, how to assign roles, and some common tools they use (like Basecamp and whatever your IM of choice is). There was also talk of legal issues, how (basically) to think about forming the company with the people involved, and decisions that need to …

Since I didn’t have any MySQL public courses planned this summer, I’ve been using my work time from the last week in the new eBox website.

I’m still far away from what I’d like, but I’m proud my design skills have improved considerably.

For those of you who still don’t know what it is, eBox is a server for the easy administration of corporate networks. eBox was included with the last release of Ubuntu. See eBox in Ubuntu