I caught Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos on Flowgram. First up, I’m not a big fan of Flowgrams - the format is sensible, slide and voice, is excellent, but the delivery in a web browser isn’t optimal… make downloadable videos!

The talk however, was excellent. Do watch it, and learn a bit more about Facebook’s infrastructure. Anyway, some notes I took from the talk:

• “We’re one of the largest MySQL installations in the world”
• Use memcache - “We have memcache because databases aren’t fast” (later on in the questions)
• Separate team focusing on APE (Apache, PHP and Extensions that they work on)
• 6.5 billion total images, 4-5 sizes stored for each, so 30 billion files, of about 540TB total… During peak? 475,000 images served per second, and growing by …

In yet-another sign of the consolidation of the infrastructure business, Progress Software announced the acquisition of IONA. IONA, with headquarters in Dublin and Boston, was a high flyer in the 90s with the development of the first implementation of CORBA, a key middleware technology for integration. While the company stumbled in later years, they have been retooling around SOA (Service Oriented Architecture) including their open source Artix technology. So raise a pint of Guiness for IONA as it enters into the next chapter of its history. Go mbeire muid beo ar an am seo arís.... READ MORE

Red Hat releases Spacewalk. It is described as: “the upstream community project from which the Red Hat Network Satellite product is derived“. Congratulations to all whom have worked on it, especially my friends who tired endlessly over it in the past.

Red Hat, is sticking true to its promise, of open sourcing everything they make. Best of all, they recognise Fedora (they always did, since say, Fedora Core 2 or 3), CentOS (a direct “competitor”/rebuild of RHEL), and Scientific Linux (I know of a certain university’s sysadmin who will be blessing Spacewalk, as her life will now be a lot easier).

There have been a few blogs about it… Matt Asay asks about a community (Red Hat traditionally wasn’t good at this, but with Fedora, I believe they’ve learned, and I’m happy …

Joe Stump, Lead Architect, Digg. Slides should make its way at Joe’s website soon enough.

Mainly works on the backend, makes sure its scalable, can all the Digg buttons be served, et al.

Application layer is loosely coupled from your data. Whole point of SOA? You can put a service in front of the DB, and move between DB’s if required.

They do use MySQL, but its pretty vanilla.

Old habits die hard
- Data requests are sequential (I need foo, bar, bleh, ecky)
- Data requests are blocking (When you need foo, nothing else is happening)
- Tightly coupled (mysql_query, and if you’re using DB abstraction layer even, you’re still using SQL… you then can’t use CouchDB for instance)
- Scaling is not abstracted (a lot of caching are in the front end code. Its a problem when you start scaling your teams out). They …

Ahead in the Cloud - The power of Infrastructure as a Service
CTO Amazon.com, Dr. Werner Vogels

Pretty much everyone in the audience uses Amazon!

Announced: Persistent Storage for Amazon EC2.

Hitting one page, might actually go to 250 different services, before the page is generated for you. Shows the use of a tool (Amazon internal), that graphs it.

SaaS: Develop -> Test -> Operate

Hardware costs? Software costs? Maintenance? Load balancing? Scaling? Utilisation? Idle machines? Bandwidth management? Server hosting? Storage management? High availability? All this is the differentiated heavy lifting that Amazon bases their services on.

SaaS comes at a very big cost that you have to address.

70/30 switch: 30% of time, energy and dollars on differentiated value creation; 70% of time, energy and dollars on differentiated heavy lifting.

At Amazon, we …

Courtesy of our friends at Oracle cost containment company Miro Consulting, I am giving a webinar tomorrow at 1pm EST (click this link for the time in other timezones please. The subject I’ve chosen is how to apply the best practices around advanced supply management that are extremely successful and mature in the product supply [...]

John Quinn writes about Scaling Drupal he is taking a one step at a time approach and is still writing his 4th and 5 stages.

His first step obviously is separating the drupal from a separate database server, and he chooses mysql for this purpose, moving your DB to a different machine is a good thing to do.

However then he gets this crazy idea of using NFS to share his his drupal shared files :(
(he even dares to mention that the setup ease is good) Folks, we abandonned NFS in the late nineties. NFS is still a recipe for disaster, it has performance issues , it as stability issues (stale locks), and no security admin in his right mind will tolerate portmap to be running in his DMZ.
(Also think about the IO path that …

I spoke with Zmanda CEO Chander Kant last week about the company's growth and some new things they have been working on. I first met Chander at the MySQL conference about a year ago. Since then the company has expanded its support for MySQL and is now starting to target other applications like Oracle and SAP.

Today Zmanda announced a program that I think is an interesting marketing move.

Beginning March 1st and through April 15th of 2007, the leader in open source backup Zmanda will offer existing Veritas NetBackup customers the ability to switch to its brand new Amanda Enterprise Edition 2.6 with Zmanda Management Console for free!

Simply show your Zmanda Account Manager the current renewal quote for Veritas NetBackup and Zmanda will set you up with a year long subscription …

I recently had a chance to chat with Nora Denzel, former Senior Vice President and General Manager of the Software Global Business Unit of Hewlett Packard, to talk shop about IT operations management and open source's role / opportunity in the space. Here's a snippet of the QA.

Open Sources: How big is the IT monitoring/management space today? Who are the major players that occupy it?

Denzel: In 2006, Gartner released a study that pegged the IT operations management software market at about $7 billion a year with more than 50% of the number going towards IT infrastructure monitoring and management. The dominant legacy players that offer proprietary monitoring solutions are the usual suspects - HP OpenView, IBM Tivoli, CA Unicenter, and to a lesser extent BMC Patrol. Their software is filled to the gills with features, which often goes unutilized I'd like to point out, and is very expensive.

Open …

I spent some time with Ingres CTO Dave Dargo today talking about Ingres and the market in general. Dave was at Oracle for many moons and is definitely one of the important database thinkers.

My initial conversation question to Dave was simple: Why does Ingres matter?

Dargo responded that he joined because Ingres had some good technology and a strong customer base. He also said that market-wise database customers were not terribly pleased with the existing products from Oracle and Microsoft.

"The success of Linux and Apache emboldened people to experiment with open source. Customers felt like they could go with free and not get features, or go with closed and be stuck. Ingres was already through the maturity process and had better technology than the other open source databases. I wanted to take Ingres beyond the database. If you look at MySQL and EnterpriseDB, …