Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 27

Displaying posts with tag: Admin-tips (reset)

Interesting Resources for Technical Operations Engineers
+0 Vote Up -0Vote Down

As you may have already heard, I am looking for good techops engineers to join my team at Swiftype. This process involves a lot of interviews with candidates and during those interviews along with many challenging practical questions I really love to ask questions like “What are the most important resources you think an Operations Engineer should follow?” or “What books in your opinion are must-read for a techops?” or “Who are your personal heroes in IT community?”. Those questions often give me a lot of information about candidates, their experience, who they are looking up to in the community, what they are interested in, and if they are actively working on improving their professional level.

Recently, one of the

  [Read more...]
MySQL Monitoring With Cacti Using Percona Monitoring Plugins (1-minute resolution)
+0 Vote Up -0Vote Down

Today, just like many times before, I needed to configure a monitoring server for MySQL using Cacti and awesome Percona Monitoring Templates. The only difference was that this time I wanted to get it to run with 1 min resolution (using ganglia and graphite, both with 10 sec resolution, for all the rest of our monitoring in Swiftype really spoiled me!). And that’s where the usual pain in the ass Cacti configuration gets really amplified by the million things you need to change to make it work. So, this is a short checklist post for those who need to configure a Cacti server with 1 minute resolution and setup Percona Monitoring Plugins on it.

Configuring Cacti for 60-seconds polling

First of all, we

  [Read more...]
Momentum MTA Performance Tuning Tips
+0 Vote Up -1Vote Down

This post is being constantly updated as we find out more useful information on Momentum tuning. Last update: 2012-05-05.

About 2 months ago I’ve joined LivingSocial technical operations team and one of my first tasks there was to figure out a way to make our MTAs perform better and deliver faster. We use a really great product called Momentum MTA (former Ecelerity) and it is really fast, but it is always good to be able to squeeze as much performance as possible so I’ve started looking for a ways to make our system faster.

While working on it I’ve created a set of scripts to integrate Momentum with Graphite for all kinds of crazy stats graphing, those scripts will be opensourced soon, but for now I’ve decided to share a few

  [Read more...]
Quick (and dirty) Patch for Ruby Enterprise Edition 2011.03 to Prevent Hash Collision Attacks
+1 Vote Up -2Vote Down

As you may have heard, this week on December 28, 2011, a group of security experts released information about a nasty problem in almost all languages and platforms related to hash function collisions and possibility of using those for DoS attack on web applications.

Ruby core team released new 1.8.7-p357 version with the problem fixed. JRuby development team came out with the new 1.6.5.1 release. Unfortunately 2 days after the release there is still no information from Ruby Enterprise Edition team on what to do with all the ree deployments.

So, since there is no

  [Read more...]
Softlayer Cloud: a Scary Story of One Bad Service
+3 Vote Up -0Vote Down

Disclaimer: the information in this post is the author’s personal opinion and is not the opinion or policy of his employer.


It was spring 2010 when we decided that even though Softlayer‘s server provisioning system is really great and it takes only a few hours to get a new server when we need it, it is still too long sometimes. We wanted to be able to scale up when needed and do it faster. It was especially critical because we were working hard on bringing up Facebook integration to our site and that project could have dramatically changed our application servers cloud capacity requirements.

What buzzword comes to your mind when we talk about scaling up really fast, sometimes within minutes, not hours or days? Exactly – cloud computing! So, after some initial testing and playing around

  [Read more...]
Nginx-Fu: X-Accel-Redirect From Remote Servers
+0 Vote Up -0Vote Down

We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.

First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your

  [Read more...]
Advanced Squid Caching in Scribd: Cache Invalidation Techniques
+0 Vote Up -0Vote Down

Having a reverse-proxy web cache as one of the major infrastructure elements brings many benefits for large web applications: it reduces your application servers load, reduces average response times on your site, etc. But there is one problem every developer experiences when works with such a cache – cached content invalidation.

It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at

  [Read more...]
Installing Midnight Commander 4.7 on Mac OS X
+1 Vote Up -2Vote Down

Another short post just to remember the procedure for the next time I’ll be setting up a new mac. For those of my readers who do not know what Midnight Commander (aka mc) is, GNU Midnight Commander is a visual file manager, created under a heavy influence of Norton Commander file manager from dark DOS ages For more information, you can visit their web site. Now, get to the installation topic itself.

To install mc on a Mac OS X machine, you need macports installed and then first thing you’ll need to do is to install some prerequisite libraries:

1
$ sudo port install libiconv slang2

Next thing, download the sources


  [Read more...]
Enabling IPv6 Support in nginx
+0 Vote Up -0Vote Down

This is going to be a really short post, but for someone it could save an hour of life.

So, you’ve nothing to do and you’ve decided to play around with IPv6 or maybe you’re happened to be an administrator of a web service that needs to support IPv6 connectivity and you need to make your nginx server work nicely with this protocol.

First thing you need to do is to enable IPv6 in nginx by recompiling it with --with-ipv6 configure option and reinstalling it. If you use some pre-built package, check if your nginx already has this key enabled by running nginx -V.

The results should have --with-ipv6 option in configure arguments:

1
2
3
4
5
[root@node ~]# nginx -V
nginx version: nginx/0.7.64





  [Read more...]
Advanced Squid Caching in Scribd: Hardware + Software Used
+0 Vote Up -0Vote Down

After the previous post in this caching related series I’ve received many questions on hardware and software configuration of our servers so in this post I’ll describe our server’s configs and the motivation behind those configs.

Hardware Configuration

Since in our setup Squid server uses one-process model (with an asynchronous requests processing) there was no point in ordering multi-core CPUs for our boxes and since we have a lots of pages on the site and the cache is pretty huge all the servers ended up being highly I/O bound. Considering these facts we’ve decided to use the following hardware specs for the servers:

CPU: One pretty cheap dual-core Intel Xeon 5148 (no need in multiple cores or really high frequencies –

  [Read more...]
Advanced Squid Caching in Scribd: Logged In Users and Complex URLs Handling
+0 Vote Up -0Vote Down

It’s been a while since I’ve posted my first post about the way we do document pages caching in Scribd and this approach has definitely proven to be really effective since then. In the second post of this series I’d like to explain how we handle our complex document URLs and logged in users in the caching architecture.

First of all, let’s take a look at a typical Scribd’s document URL: http://www.scribd.com/doc/1/Improved-Statistical-Test.

As we can see, it consists of a document-specific part (/doc/1) and a non-unique human-readable slug part (/Improved-Statistical-Test). When a user comes to the site with a wrong slug in the document URL, we need

  [Read more...]
ActiveMQ Tips: Flow Control and Stalled Producers Problem
+0 Vote Up -0Vote Down

It’s been a few months since we‘ve started actively using ActiveMQ queue server in our project. For some time we had pretty weird problems with it and even started thinking about switching to something else or even writing our own queue server which would comply with our requirements. The most annoying problem was the following: some time after activemq restart everything worked really well and then activemq started lagging, queue started growing and all producer processes were stalling on push() operations. We rewrote our producers from Ruby to JRuby, then to Java and still – after some time everything was in a bad shape until we restarted the queue server.

So, long story short, after a lots of docs and source code reading we’ve found really interesting thing. There is a

  [Read more...]
Using SSH tunnel connection as a SOCKS5 proxy
+0 Vote Up -0Vote Down

Month ago I was on a vacation and as usual even though our hotel provided us with an internet connection on a pretty decent speeds, I wasn’t able to work there because they’ve banned all tcp ports but some major ones (like 80, 21, etc) and I needed to be able to use ssh, mysql, IMs and other non-web software.

After a short research I’ve found a pretty simple to set up and easy to use approach to such a connection problems I’d like to describe here.

First, you’ll need someone (or you can do it before leaving home) to start an ssh daemon on port 80 on one of your servers. I use one of my Slicehub slices for this to permanently have an ability to use it. You can do it like this (if it is a temporary solution):

1
# `which sshd` -p 80

Notice: this `which sshd` was used because on some OSes sshd does not want to start w/o an


  [Read more...]
Lighttpd Book from Packt – Great Thanksgiving Present
+0 Vote Up -0Vote Down

Many people know me as a nginx web server evangelist. But as (IMHO) any professional I think that it is really rewarding to know as much as possible about all the tools available on the market so every time you need to make a decision on some technical issue, you’d consider all pros and cons based on my own knowledge.

This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.

First, when I received the book, I was in doubt: how such a small book could cover so flexible and multi-purpose piece of software

  [Read more...]
ActiveMQ + Ruby Stomp Client: How to process elements one by one
+0 Vote Up -0Vote Down

Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something

  [Read more...]
Found an Ideal I/O Scheduler for my MySQL boxes
+0 Vote Up -0Vote Down

Today I was doing some work on one of our database servers (each of them has 4 SAS disks in RAID10 on an Adaptec controller) and it required huge multi-thread I/O-bound read load. Basically it was a set of parallel full-scan reads from a 300Gb compressed innodb table (yes, we use innodb plugin). Looking at the iostat I saw pretty expected results: 90-100% disk utilization and lots of read operations per second. Then I decided to play around with linux I/O schedulers and try to increase disk subsystem throughput. Here are the results:

Scheduler Reads per second cfq 20000-25000 noop 35000-60000 deadline 33000-45000 anticipatory 22000-29000

Notice: The box can’t be restarted to check with clean caches and stuff, but I was doing full reads from this huge table on a machine with 16Gb RAM so all caches were washed out

  [Read more...]
Using Sphinx for Non-Fulltext Queries
+0 Vote Up -0Vote Down

How often do you think about the reasons why your favorite RDBMS sucks? Last few months I was doing this quite often and yes, my favorite RDBMS is MySQL. The reason why I was thinking so because one of my recent tasks at Scribd was fixing scalability problems in documents browsing.

The problem with browsing was pretty simple to describe and as hard to fix - we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “

  [Read more...]
32bit VS 64bit - what do you use?
+0 Vote Up -0Vote Down

Hello my dear readers.

Today I have a question for all of you. What platforms (32bit or 64 bit) do you use for your servers with more than 4Gb RAM? I’m asking because recently we‘ve hit few really weird bugs in Linux kernels 2.6.18 to 2.6.22 and all those bugs were PAE-related. Now I’d really love to move all machines to 64-bit, but I’m in doubt because we don’t know too much about Rails stack (ruby, mongrel, haproxy) on 64-bit platforms (all our DB boxes are 64-bit of course).

So, please drop me a line if you have any experience (negative or positive) with Rails platform on 64-bit machines. I’d really appreciate your help.

Puppet - Admin’s Best Friend
+0 Vote Up -0Vote Down

If you’ve ever worked in companies with 5-10+ servers and it was your responsibility to install new boxes, change some configuration files and install new software on many boxes you definitely know how painful this work is. Every time you need to change something on 3-5-100 boxes, you go there and make those changes. Most experienced of us used some weird scripts to perform some task on many boxes or used some stuff like dsh. Even with those tricks I’d never wish this work to anyone.

While I was working in Galt, I’ve asked our junior admin to check out puppet and try to use it on our servers. After a week of screaming he’s managed to install and

  [Read more...]
Small Tip: How to set up two interface Xen machine
+0 Vote Up -0Vote Down

This will be one of those posts I’d like to publish primarily to be able to coma back later and check it out instead of reading docs again

So, we have a server with two (or more) network interfaces are we need to be able to use more than one interface in our VDS machines. How do we set it up?

(more…)

Why I HATE “smart” Software: Cpanel vs Consulter
+0 Vote Up -0Vote Down

Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…

After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) - it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.

Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up

  [Read more...]
Useful Cacti Templates to Monitor Your Servers
+0 Vote Up -0Vote Down

Recently I had one customer for consulting and aside from mysql optimization, etc they asked me for cacti installation/setup to monitor their pretty generic LAMP application. I’ve started setting up all this stuff and I’ve never thought it could be so painful… lots of different templates for the same tasks, all of them are incompatible with recent cacti releases, etc, etc… So, this post is generally a list of used templates with a fixes I’ve made to make them work on recent cacti release.

(more…)

How to run GUI-programs on a server without any monitor
+0 Vote Up -0Vote Down

This weekend I was doing some development for one of our projects and we needed to make screenshots of a web pages (see my next posts about this task). I’ve managed to develop small piece of code which uses GtkMozEmbed component (Mozilla Gecko-based renderer for web pages) to create screenshots of any page, but there was some problem… The problem was a following: GTK+ library can’t work w/o fully-functional X server running on your machine. Obviously I didn’t want to run such software (no monitor/keyboard/mouse, dumb graphics adapter on the server, etc, etc) so I’ve tried to find some solution… And in this tiny article I’ll describe the method I’ve managed to find.

(more…)

Linux User in Solaris 10 Survival Guide
+0 Vote Up -0Vote Down

This week aside from tons of different tasks I was working on one of MMM users complaint regarding some issues with MMM on Solaris 10. I knew that this OS has not so user (admin) friendly environment (especially for people with strong GNU-related background), but had no other options and decided to install Solaris 10 in VMWare Fusion on my desktop.

Installation was a bit strange comparing to Debian/RHEL/Ubuntu and FreeBSD where I have a strong experience, but I’ve managed to install it successfully. The major problem after my first boot was a lack of knowledge about how things could be done in Solaris… Below I’ll describe what generic Linux admin could do with Solaris to make it easier to use and more friendly for

  [Read more...]
How to unmount NFS share mounted with hard option
+0 Vote Up -0Vote Down

Few days ago I worked on some customer’s server and there was a problem - their nfs server went down and we were forced to change some settings on their FC4 clients to prevent shares from dieing because of kernel bug. But when we’ve changed settings in /etc/fstab there was one more step before task was completed - we need to remount this share (I mean unmount/mount). But how to perform this operation if there are some processes in D (non-interruptible sleep) waiting for dead share and prevent it from unmounting? They wait because of hard option on the share and lack of intr option and any unmount request would produce a following results:

streaming01:~# umount /storages/2
umount: /storages/2: device is busy
umount: /storages/2: device is busy

So, here is a list of steps you need to do to be able to remount your share.



  [Read more...]
Using Nginx, SSI and Memcache to Make Your Web Applications Faster
+0 Vote Up -0Vote Down

If you’d take a look at any web site, you will notice, that almost all of the pages on this given site are pretty static in their nature. Or course, this site could have some dynamic elements like login field or link in the header, some customized menu elements and some other things… But entire page could be considered static in many cases.

When I started thinking about my sites from this point of view, I understood, how great it would be to be able to cache entire page somewhere (in memcache for example) and be able to send it to the user without any requests to my applications, which are pretty slow (comparing to memcache ) in content generation. Then I came up with a pretty simple and really powerful idea I’ll describe in this article. An idea

  [Read more...]
Small Tip: How to fix “There are no public key available for the following key IDs” Error in Debian
+0 Vote Up -0Vote Down

Few days ago I’ve started migration of some of my non-critical servers to Debian Etch (from Sarge). Just after first apt-get update && apt-get dist-upgrade, when apt has been upgraded, I noticed really strange (as for me) error: when I’ve tried to do “apt-get update” it worked fine, but there was annoying message like following:

# apt-get update
......
Fetched 5562B in 13s (421B/s)
Reading package lists... Done
W: There are no public key available for the following key IDs:
A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems
#


UPDATE: Thanks to Kurt Fitzner we know, that:

There is already a mechanism to do this automatically:

$ apt-key update

This will obtain the necesary keys and import them. No need to








  [Read more...]
Showing entries 1 to 27

Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.