Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 29

Displaying posts with tag: My Projects (reset)

Interesting Resources for Technical Operations Engineers
+0 Vote Up -0Vote Down

As you may have already heard, I am looking for good techops engineers to join my team at Swiftype. This process involves a lot of interviews with candidates and during those interviews along with many challenging practical questions I really love to ask questions like “What are the most important resources you think an Operations Engineer should follow?” or “What books in your opinion are must-read for a techops?” or “Who are your personal heroes in IT community?”. Those questions often give me a lot of information about candidates, their experience, who they are looking up to in the community, what they are interested in, and if they are actively working on improving their professional level.

Recently, one of the

  [Read more...]
Fwd: Scalexis Inc is Hiring
+0 Vote Up -0Vote Down

Scalexis Inc, web performance/scalability consulting firm in Toronto, is looking for a full-time consultant. Being a consulting company on the high-demand market of web application scalability consulting and high-performance web applications development, we need an employee that could perform both web application development and web application performance consulting work.

The main duty of the employee would be to work directly with the customers’ operations teams to detect, analyze and fix performance problems and plan application scalability in accordance with customers’ needs/requests.

The employee would work in a small team of highly professional consultants helping the company to handle growing demand for web applications performance consulting services.

Position is in Toronto. Telecommuting is not


  [Read more...]
DbCharmer 1.7.0 Release: Rails 3.0 Support and Forced Slave Reads
+0 Vote Up -0Vote Down

This week, after 3 months in the works, we’ve finally released version 1.7.0 of DbCharmer ruby gem – Rails plugin that significantly extends ActiveRecord’s ability to work with multiple databases and/or database servers by adding features like multiple databases support, master/slave topologies support, sharding, etc.

New features in this release:

  • Rails 3.0 support. We’ve worked really hard to bring all the features we supported in Rails 2.X to the new version of Rails and now I’m proud that we’ve implemented them all and the implementation looks much cleaner and more universal (all kinds of relations in rails 3 work in exactly the same way and we do not need to implement connection switching for all kinds of weird corner-cases in ActiveRecord).
  • Forced
  [Read more...]
Advanced Squid Caching in Scribd: Cache Invalidation Techniques
+0 Vote Up -0Vote Down

Having a reverse-proxy web cache as one of the major infrastructure elements brings many benefits for large web applications: it reduces your application servers load, reduces average response times on your site, etc. But there is one problem every developer experiences when works with such a cache – cached content invalidation.

It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at

  [Read more...]
DbCharmer – Rails Can Scale!
+0 Vote Up -1Vote Down

Back in November 2009 I was working on a project to port Scribd.com code base to Rails 2.2 and noticed that some old plugins we were using in 2.1 were abandoned by their authors. Some of them were just removed from the code base, but one needed a replacement – that was an old plugin called acts_as_readonlyable that helped us to distribute our queries among a cluster of MySQL slaves. There were some alternatives but we didn’t like them for one or another reasons so we’ve decided to go with creating our own ActiveRecord plugin, that would help us scale our databases out. That’s the story behind the first release of DbCharmer.

Today, six months after the first release of

  [Read more...]
DB Charmer – ActiveRecord Connection Magic Plugin
+1 Vote Up -1Vote Down

Today I’m proud to announce the first public release of our ActiveRecord database connection magic plugin: DbCharmer.


DB Charmer – ActiveRecord Connection Magic Plugin

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

  • Allows you to easily manage AR models’ connections (switch_connection_to method)
  • Allows you to switch AR models’ default connections to a separate servers/databases
  • Allows you to easily choose where your query should go (on_* methods family)
  • Allows you to automatically send read queries to your slaves while masters would handle all the updates.
  • Adds multiple databases migrations to ActiveRecord

    Installation

    There are two options when

  •   [Read more...]
    Advanced Squid Caching in Scribd: Logged In Users and Complex URLs Handling
    +0 Vote Up -0Vote Down

    It’s been a while since I’ve posted my first post about the way we do document pages caching in Scribd and this approach has definitely proven to be really effective since then. In the second post of this series I’d like to explain how we handle our complex document URLs and logged in users in the caching architecture.

    First of all, let’s take a look at a typical Scribd’s document URL: http://www.scribd.com/doc/1/Improved-Statistical-Test.

    As we can see, it consists of a document-specific part (/doc/1) and a non-unique human-readable slug part (/Improved-Statistical-Test). When a user comes to the site with a wrong slug in the document URL, we need

      [Read more...]
    The Blog v.2.0
    +0 Vote Up -0Vote Down

    Long time ago, in 2002 I decided to create my own point of presence in the Internet. Back then I’ve got pretty nice domain (scoundrel.kremenchug.net), hacked up a few pages on php, added a guestbook and that was it. Many years it was almost static and I did a few updates on my resume page few times a year. Later I’ve switched the site to wordpress to make it easier to manage my resume and stuff

    And 3 years ago in March 2006 I’ve decided to start my own blog. I took a standard template and started the blog on a separate domain while the domain was on its own domain name… This spring my

      [Read more...]
    Loops plugin for rails and merb released
    +0 Vote Up -0Vote Down

    loops is a small and lightweight framework for Ruby on Rails and Merb created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).

    Originally loops plugin was created to make our (Scribd.com) own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

    After a short time of writing our

      [Read more...]
    ActiveMQ + Ruby Stomp Client: How to process elements one by one
    +0 Vote Up -0Vote Down

    Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

    All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something

      [Read more...]
    Advanced Squid Caching for Rails Applications: Preface
    +0 Vote Up -0Vote Down

    Since the day one when I joined Scribd, I was thinking about the fact that 90+% of our traffic is going to the document view pages, which is a single action in our documents controller. I was wondering how could we improve this action responsiveness and make our users happier.

    Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

    Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems

      [Read more...]
    Bounces-handler Released
    +0 Vote Up -0Vote Down

    Today I’ve managed to finish initial version of our bounces-handler package we use for mailing-related stuff in Scribd.

    Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

    This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

    The package itself consists of two parts:

    • Perl scripts to process incoming email:
      • bounces processor — could be
      [Read more...]
    Using Sphinx for Non-Fulltext Queries
    +0 Vote Up -0Vote Down

    How often do you think about the reasons why your favorite RDBMS sucks? Last few months I was doing this quite often and yes, my favorite RDBMS is MySQL. The reason why I was thinking so because one of my recent tasks at Scribd was fixing scalability problems in documents browsing.

    The problem with browsing was pretty simple to describe and as hard to fix - we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “

      [Read more...]
    InnoDB Recovery toolset Version 0.3 Released
    +0 Vote Up -0Vote Down

    Even though I didn’t go to MySQL conf this year (really sad about this), this week is gonna be most active in the community so I decided to do some community stuff too Today I’ve released version 0.3 of our innodb recovery toolkit. Now it became much faster, stable and accurate. At this moment it is possible to recover almost any table from corrupted/deleted tablespace without so much effort as it was before. Here is a short changes list (since 0.1 announced here):

    • More MySQL data types added: DECIMAL (both old and new), DATE, TIME
    • CHAR data type handling improved in table definitions generator
    • Indexes filtering added to
      [Read more...]
    FastSessions Rails Plugin Released
    +0 Vote Up -0Vote Down

    How often do we think about our http sessions implementation? I mean, do you know, how your currently used sessions-related code will behave when sessions number in your database will grow up to millions (or, even, hundreds of millions) of records? This is one of the things we do not think about. But if you’ll think about it, you’ll notice, that 99% of your session-related operations are read-only and 99% of your sessions writes are not needed. Almost all your sessions table records have the same information: session_id and serialized empty session in the data field.

    Looking at this sessions-related situation we have created really simple (and, at the same time, really useful for large Rails projects) plugin, which replaces ActiveRecord-based session store and makes sessions much more effective. Below you can find some information about implementation details

      [Read more...]
    Data Recovery Toolkit for InnoDB Released
    +0 Vote Up -0Vote Down

    I’m returned from my 1-week vacation today and want to say - I’ve never been so productive as I was there Blue ocean, hot sun and white sand really helped me to finish my work on the first release of one really awesome project.

    Today I’m proud to announce our first public release of the Data Recovery Toolkit for InnoDB - set of tools for checking InnoDB tablespaces and recovering data from damaged tablespaces or from dropped/truncated InnoDB tables.

    This release already has a pretty decent set of features:

    • Supports both REDUNDANT (pre mysql 5.0) and COMPACT (mysql 5.0+) versions of tablespaces
    • Works
      [Read more...]
    MySQL Master-Master Replication Manager 1.0 Released
    +0 Vote Up -0Vote Down

    It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

    Changes from previous release:

    • Perl semaphores implementation caused huge memory leaks (mmmd_mod).
    • Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
    • Removed legacy StartSlave method from agent code which caused problems on some Perl versions
    • Added a few fixes to prevent non-exclusive roles
      [Read more...]
    MyWebER is going to die
    +0 Vote Up -0Vote Down

    My google summer of code project called MyWebER is going to die. The reason - I have no time to proceed developing it. I have a lot of ideas, but not sure that I’ll find any time to implement them.

    As I can see, MySQL AB is not interested in this project. If anyone wants to help me - you’re welcome.
    “Skilled JavaScript developers are wanted.” (C)

    MMM checkers memory leak?
    +0 Vote Up -0Vote Down

    One of MMM users reported that they’re experiencing really weird memory leaks in checker processes used by MMM. After a deep investigation I’ve found out that Perl part of the checker and checker modules does not leak (at least I didn’t found these leaks), so I think it could be caused by some problems in MySQL DBD module (client uses Ubuntu server).

    So, I’d like to ask all users to check if their checker processes use more memory than expected and if yes, what OS, MySQL libraries versions and Perl version used on their servers.

    Thanks in advance for any help.

    Linux User in Solaris 10 Survival Guide
    +0 Vote Up -0Vote Down

    This week aside from tons of different tasks I was working on one of MMM users complaint regarding some issues with MMM on Solaris 10. I knew that this OS has not so user (admin) friendly environment (especially for people with strong GNU-related background), but had no other options and decided to install Solaris 10 in VMWare Fusion on my desktop.

    Installation was a bit strange comparing to Debian/RHEL/Ubuntu and FreeBSD where I have a strong experience, but I’ve managed to install it successfully. The major problem after my first boot was a lack of knowledge about how things could be done in Solaris… Below I’ll describe what generic Linux admin could do with Solaris to make it easier to use and more friendly for

      [Read more...]
    1000 Great Technical Videos in BestTechVideos Library!
    +0 Vote Up -0Vote Down

    Yes! We did it! Just before I went to bed at 3:40AM (heh ), I noticed, that someone submitted new video to BTV. I’ve approved it instantly and then noticed a following information in the top part of the site: “We have 1000 videos / 569+ hours of high-quality content!” So, we broke trough 1k videos barrier! And we’ll try to do 2k videos before the end of this year, we have some new features in closed beta testing which would greatly increase number of submitted videos each day.

      [Read more...]
    MMM Documentation Wiki
    +0 Vote Up -0Vote Down

    Today I’ve started moving (actually copying) all information I have about MMM to the project’s wiki.

    If someone is interested in adding some information there or modifying my descriptions (yes, I know about by awful English and I hope someone will fix my mistakes), ping me (contacts are here) and I’ll add you to the project members list.

    Thanks all for participation and support.

    Best Tech Videos 2.0 has been released!
    +0 Vote Up -0Vote Down

    So, we did it! New Best Tech Videos site version has been released today. Of course, it could have some issues, and we are really looking forward for your feedback on our support forums.

    Let’s review main improvements we’ve made here:

    • First of all, I want to mention our first step to socialization of the BTV - all our users could signup now and get their own lists of favorited, commented and voted videos and much more - they could have their own personal RSS feeds.
    • Next cool thing we’ve prepared for you is user-generated content! You can find some great videos on the Net and share them with fellow BTV readers. At this moment our posting system works in pre-moderated mode to keep really high
      [Read more...]
    MyWebEr - creating documentation on the fly
    +0 Vote Up -0Vote Down

    In my last post I’ve asked for any help on my project. So, tanks to Jay Pipes for his tip.
    One of most developer problems is documentation. And a good solution was initially appeared in Java with javadoc. At now many languages have special javadoc-like tools. I’ve made a little research, and decided to use jsdoc and phpdoc in my project. I am still open for any suggestions

    MySQL Proxy Released
    +0 Vote Up -0Vote Down

    Few month ago I heard about some initial work on MySQL Proxy software by Jan Kneschke and I thought about implementing some type of MySQL Replication Aggregator based on this software. The idea was to create some piece of software which could get many replication streams, merge them and feed to some mysql slave. This software could be used for backup purposes and many other interesting things. But back then mysqlproxy distribution has been suspended (afaik, by MySQL AB because of some legal issues).

    And at last, today MySQL Proxy project has been released to public and it became much more flexible so I think we need to take a look at it and try to implement such replication aggregator patches for it.

    MyWebEr - info
    +0 Vote Up -0Vote Down

    So, now I am working on this project. This project is a part of google SoC. I believe in the end it should be something like mysql workbench, but web-two-zerofied. At current moment there is not much to look into (http://myweber.googlecode.com/svn/trunk/). But I will be very grateful to you for any comments, questions and suggestions.

    MMM Release 1.0-pre4
    +0 Vote Up -0Vote Down

    New alpha release 1.0-pre4 of the MySQL Master-Master Replication Manager. This release has lots of major fixes and I’m glad to announce first sponsored port of mmm to non-linux platform - it has been ported to Solaris 10. So, here are our changes in this version:

    • Real checks timeouts - I’ve found and fixed lots of problems in checks timeout code and now if you specified in your mmm_mon.conf, that some check should timeout in 5 sec, it would timeout correctly on all supported platforms.
    • External third-party tools using - On all supported non-linux platforms mmm will use system binaries for fping and arp_ping so porting to another platforms would be much easier.
    • Agent
      [Read more...]
    Best Tech Videos On The Net
    +0 Vote Up -0Vote Down

    Few days ago I’ve started new site which name is BestTechVideos.com. I’ve created this site because there are lots really interesting videos on the Net, but if you’d like to fine some good tech video, it is not so simple to find them because of tons of crappy “funny videos” like “funny cats” and so on.

    So, If you like to attend technical conferences and watch conference sessions on video, if you like idea of screencasts, etc, then this site is for you! Welcome to Best Tech Videos you’ll be impressed by amount of hi-quality and useful videos on the Net.

    P.S. If you like this site idea, support it on Digg.com, please. Great thanks in advance.

    UltimateGeoCoder - a PHP library inteface to miscellaneous geocoding services
    +0 Vote Up -0Vote Down

    Sometimes we may need to know geocode of some specific postal location. For example, you have some shops catalogue and want to show shop locations on the map. Then, you will need to ask some service about map marker coordinates. If geocode service is not available or it can’t parse provided address, you will not be able to show map to your users. To solve such problem I’ve created small PHP-module, that helps me to get reliable results from set of miscellaneous geocoding services.

    As for now, module can use Google Maps API, Yahoo Maps Services,

      [Read more...]
    Showing entries 1 to 29

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.