Showing entries 1 to 5
Displaying posts with tag: software engineering (reset)
A Quick Look at Parallel Rsync and How it Can Save a System Hours

In this post, we'll take a quick look at rsync ("remote sync") and parallel rysnc—a way to increase the efficiency and speed of traditional rsync. At VividCortex, we've found each to be effective and handy at various times.

Image Credit

Rsync is a tool for copying files between volumes in the same or separate servers. The advantage of rsync is that instead of copying data blindly, it compares the source and destination directories, so that only the difference between the two is sent through the network (or between volumes). 

Rsync can still be slow in certain situations, however—especially when there's a high volume of data that needs to be copied. In such a case, the process can take hours. Additionally, if the volume io has high latency—such as when cold Amazon EBS volumes are involved—the throughput can …

[Read more]
How I became a Data Engineer

A reminiscence of a personal timeline of events; please, excuse minor errors and/or let me know if my memory has shuffled things around a bit.

My first job was at a company that had their own CRM and they wanted a web version of some parts of the system. At that time, I only had “experience” with ASP and Microsoft Access (I know, tough). They wanted the web version in PHP, their reasoning, I think, being that they wanted the integration to run directly into the database. The web app would write directly into the DB. The CRM system was written using Delphi and Firebird. So I learned PHP and my first database, which wasn’t MySQL (I don’t count MS Access as a DB). After that I got a job in which MySQL was used. I was really fresh on MySQL, and I didn’t know about the engines and such, so it was a bit weird learning about MyISAM (which didn’t have foreign keys for instance).

After that I got a job in a huge …

[Read more]
Why Stored Programs?

Why should you use stored programs? Great question, here’s my little insight into a situation that I heard about in a large organization.

A very large organization is having a technology argument. In someway, like politics, half-truth drives this type of discussion. This company has hundreds of databases and they’re about half SQL Server and Oracle. The argument (half-truth) states that using T-SQL or PL/SQL yields “spaghetti” code!

It seems like an old argument from my perspective. After all, I’ve been working with T-SQL and PL/SQL for a long time. Spaghetti code exists in every language when unskilled programmers solve problems but the point here is one of software architecture, and an attempt to malign stored programming in general. Let’s examine the merit of the argument against stored programs.

First of all, the argument against stored programs is simply not true. SQL DML statements, like the …

[Read more]
Recommended Reading (Business, Engineering)

As part of an internal programme at Sun, I am a “SEED mentor” for another Sun employee (not a former employee of MySQL, but what we Sun Dolphins call Sun Classics). He is called Alok and lives in Bangalore, and sadly, our schedules crossed so that I couldn’t meet him when I was at our Bangalore offices in July. So I am mentoring someone I’ve met only over phone — but we’re getting along just fine.

Two of the topics we’ve discussed recently are blogging and books. So after hanging up after our 9 CET 12:30 Indian time mentoring session, I got the idea to combine the two: write a blog entry about the books I recommended Alok.

One thing Alok is contemplating at the moment is the degree to which he should spend time on developing his business skills vs his engineering skills. That’s a familiar topic for many of us in …

[Read more]
The Architecture Layer

Contemporary software engineering models include many loosely-defined layers. Database developers might help with other layers, but for the most part a database administrator’s domain is the persistence layer.

  • Presentation
  • Application
  • Business Logic
  • Persistence (also called Storage)

The Daily WTF has an article on The Mythical Business Layer makes the case for not separating the business layer and the application layer:

A good system (as in, one that’s maintainable by other people) has no choice but to duplicate, triplicate, or even-more-licate business logic. If Account_Number is a seven-digit required field, it should be declared as CHAR(7) NOT NULL in the database and …

[Read more]
Showing entries 1 to 5