Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 27

Displaying posts with tag: Architecture (reset)

Five More Things Deadly to Scalability
+0 Vote Up -0Vote Down

Read the original article at Five More Things Deadly to Scalability

Join 6000 others and follow Sean Hull on twitter @hullsean. 1. Slow Disk I/O – RAID 5 – Multi-tenant EBS Disk is the grounding of all your servers, and the base of their performance. True with larger and larger main memory, much is available in cache, a server still needs to constantly read from disk [...]

For more articles like these go to Sean Hull's Scalable Startups

Related posts:
  • Mobile Scalability – What is it and why is it important?
  • 3 Ways to Boost Cloud
  •   [Read more...]
    MySQL Web Reference Architectures: On Demand
    Employee_Team +2 Vote Up -0Vote Down

    Last week we ran a live webinar presenting the new MySQL Web Reference Architectures (http://www.mysql.com/why-mysql/white-papers/mysql-reference-architectures-for-scalable-web-infrastructure/), a set of documented and repeatable best practices for building highly available, scaleable and secure database infrastructure to power new generations of web and mobile services.

    The webinar replay is now available on-demand (http://www.mysql.com/news-and-events/web-seminars/mysql-reference-architectures-best-practices-for-web-mobile-cloud-applications/) so you can listen in from the comfort of your own desk…or commute.

    As a taster - we discuss sizing and design patterns - you can see a sample below:

      [Read more...]
    Serving Clients Rather than Falling Over
    +1 Vote Up -0Vote Down

    Dawnstar Australis (yes, nickname – but I know him personally – he speaks with knowledge and authority) updates on The Real Victims Of The Click Frenzy Fail: The Australian Consumer after his earlier post from a few months ago.

    Colourful language aside, I believe he rightfully points out the failings of the organising company and the big Australian retailers. From the Open Query perspective we can just review the situation where sites fall over under load. Contrary to what they say, that’s not a cool indication of popularity. Let’s compare with the real world:

  • Brick & Mortar store does something that turns out popular and we see a huge queue outside, people need to wait for
  •   [Read more...]
    Calling all next gen app providers: Who’s got your back?
    +1 Vote Up -0Vote Down
    Next gen app providers (and perhaps more specifically, database architects) are clamoring for database technologies that just work. At least, that’s the message we got from one of our newest customers: Mozilla. Earlier this month, we caught up with Sheeri Cabral, database architect at Mozilla and and overall MySQL rock star, to get the down-and-dirty on why [...] Read More
    Mysql HA solutions
    +1 Vote Up -0Vote Down
    Lets see what HA solutions can be designed in mysql and where are they suited. 1. Single master - single slave. M(RW)|S(R) A simple master slave solution can be used for a small site - where all the inserts go into the master and some (non-critical) requests are served from the slave. In case if the master crashes, the slave can be simply promoted as the master - once it has replicated the "
    Working with ScaleBase and NOSQL
    +0 Vote Up -0Vote Down

    There is a huge amount of buzz around NOSQL, and we at ScaleBase are happy to see companies making the move to NOSQL. Despite what some people might think, we consider it a blessed change. It is time for applications to stop having a single data store – namely a relational database (probably Oracle) – and start using the best tool for the job.

    In the last couple of years, since NOSQL technologies broke into our world, a lot of experience has been gathered on how to use them. Mainly, we see NoSQL technologies used for one of the following scenarios:

    • Queries that require a very short response time
    • Storing data without a well-defined schema, or storing data with a frequently modified schema

    Now, I’m not in any way saying that NOSQL solutions are not used for other scenarios as well; I’m only saying that from our experience here at ScaleBase ,

      [Read more...]
    ScaleBase achieves 180K NO-TPM DBT2 results on Amazon RDS
    +1 Vote Up -0Vote Down

    ScaleBase Releases Database DBT2 Performance Results

    Technology achieves unprecedented transaction speed for a MySQL database at a low cost


    Boston, Mass., December 12, 2011ScaleBase, Inc. today announced the results of its MySQL database benchmark, based on the industry-standard DBT-2 test. ScaleBase has achieved an unmatched 180,000 Transactions per Minute – the highest result for a MySQL database – while running on an Amazon RDS environment. Cost per Transaction was reported to be 50 cents, which demonstrates the cost-effectiveness of the ScaleBase solution on the Amazon EC2 cloud. Full details of the benchmark

      [Read more...]
    Making the case for Database Sharding using a Proxy
    +2 Vote Up -0Vote Down

    There are several ways to implement sharding in your application. The first and by far the most popular, is to implement it inside your application. It can be implemented as part of your own Data Access Layer, database driver, or an ORM extension. However, there are many limitations with such implementation, which drove us, at ScaleBase, to look for an alternative architecture.

    As the above diagram shows, ScaleBase is implemented as a standalone proxy. There are several benefits to using such an architecture.

    First and foremost, since the sharding logic is not embedded inside the application, third party applications can be used, be it MySQL Workbench, MySQL command line interface or any other third party product. This translates to a huge saving in the day-to-day costs of both

      [Read more...]
    How do you know when to shard your database?
    +1 Vote Up -0Vote Down

    We at ScaleBase talk about sharding so much, it’s difficult for us to see why someone wouldn’t want to shard. But just because we’re so enthusiastic about our transparent sharding mechanism, it doesn’t mean we can’t understand the very basic question, “When do I shard?”
    Well, it’s not the most difficult question to answer. I’ll keep it short: if your database exceeds the memory you have on a single machine, you should shard. If you hit I/O, your performance suffers, and sharding will assist.
    Why? That’s easy to explain.
    Databases in general (and MySQL is no exception) try to cache data. Because accessing memory is so much faster than accessing disk (even with SSDs), database providers have developed rather sophisticated caching algorithms. For instance, running a query caches the query and its results. Indexes are stored in memory so that,

      [Read more...]
    Oracle's NoSQL
    +0 Vote Up -1Vote Down

    Oracle's turn-about announcement of a NoSQL product wasn't really surprising. When Oracle spends time and effort putting down a technology, you can bet that its secretly impressed, and trying to re-implement it in its back room. So Oracle's paper "Debunking the NoSQL Hype" should really have been read as a backhanded product announcement. (By the way, don't click that link; the paper appears to have been taken down. Surprise.)

    I have to agree with DataStax and other developers in the NoSQL movement:

      [Read more...]
    How to Implement MySQL Sharding – Part 3
    +1 Vote Up -0Vote Down

    In the previous post of this series (which can be found here) I discussed how to migrate your data once you have decided how to shard your schema.

    Once your data is sharded, it’s time to modify your application code. I will not dive into the many open source platforms that provide partial sharding support (Hibernate Shards, Gizzard, and the like), and will take Java (sorry, old habits are hard to overcome) as an example – however, the same holds true for any programming language.

    Without Using ORM

    If you wrote your code without an Object/Relational Mapping tool, kudos to you. Sharding will be easier, as you control the SQL statements.

    Upgrading Connection Pool

    Your first task is to write a connection pool that is “sharding” aware.  The class should

      [Read more...]
    How to Implement MySQL Sharding – Part 2
    +2 Vote Up -0Vote Down

    In the previous post of this series (which can be found here) I discussed how to identify tables that can serve as good candidates for sharding.

    Once you have decided which tables should be sharded (all the rest should be global tables), the choice of sharding keys is rather straightforward, as most will use the table primary key as the shard key. Of course, if multiple tables are sharded, and there is a foreign key relationship between these tables, then the foreign key will serve as the shard key for some tables.

    Many people attempt to shard based on customer_id or a resource id, but I have seen how this usually fails in production environments. It is very hard to know in advance which customers belong together in the same database, and since customers can suddenly increase their traffic,

      [Read more...]
    Backing Up MySQL With ScaleBase
    +1 Vote Up -0Vote Down

    Backing up data is critical for production databases – and there are a lot of well-known solutions for backing up databases.

    When the database is sharded, backing up data becomes problematic. If the backup is not synchronized across all shards, data inconsistency might occur. In this blog post I’ll try to detail the possible backup scenarios for sharded databases when using ScaleBase.

    Backup Types

    Let’s start by understanding the different backup types that are out there. You can read all about it here.

    A physical backup involves copying all database files to a different location. Copying can take several hours for a decent database if it’s done to a disk or a tape. It might take only seconds if the database files reside on SAN/NAS storage hardware that supports snapshot

      [Read more...]
    Standard Query Language (SQL) for NoSQL databases?
    +0 Vote Up -0Vote Down

    I recently came across an interesting blog post on RedMonk (not surprising, as I read most of their posts). It’s called It’s Beginning to Look a Lot Like SQL and basically it talks about query language for NoSQL databases. It seems that as NoSQL becomes more popular, users want to do more with it – a good level of querying, for example, is needed.

    Now of course, since NoSQL is a family of products that work in radically different ways, it’s not certain that this is possible (or even desirable – read Alex Popescu’s post on the subject).

    But my question is – why do you even need a query language for NoSQL data stores? After all, running queries on distributed data might be complex to

      [Read more...]
    OpenDBCamp: Information Lifecycle Architecture
    +2 Vote Up -2Vote Down
    The Open DB Camp in Sardinia 2011 has had a number of sessions on varying topics. Topics range from MySQL over MongoDB to replication and High Availability.

    I decided to tap into the database expert resources present here at Sardegna Ricerche by discussing a non-database issue, where one can expert database experts to have insights beyond those of end users. And they did.

    The topic was the particular case of information overload many of us suffer from on our hard disks: Too many files, too hard to find.
    • How do we find the bank statement from April 2007 from the

      [Read more...]
    MySQL 5.6: InnoDB scalability fix – Kernel mutex removed
    +3 Vote Up -0Vote Down

    For those interested in InnoDB internals, this post tries to explain why the global kernel mutex was required and the new mutexes and rw-locks that now replace it. Along with the long term benefit from this change.

    InnoDB’s core sub-systems up to v5.5 are protected by a global mutex called the Kernel mutex. This makes it difficult to do even some common sense optimisations. In the past we tried optimising the code but it would invariably upset the delicate balance that was achieved by tuning of the code that used the global Kernel mutex, leading to unexpected performance regression. The kernel mutex is also abused in several places to cover operations unrelated to the core e.g., some counters in the server thread main loop.

    The InnoDB core sub-systems are:

  • The Locking sub-system
  • The Transaction
  •   [Read more...]
    Outliers and coexistence are the new normal for big data
    +0 Vote Up -0Vote Down

    Letting data speak for itself through analysis of entire data sets is eclipsing modeling from subsets. In the past, all too often what were once disregarded as "outliers" on the far edges of a data model turned out to be the telltale signs of a micro-trend that became a major event. To enable this advanced analytics and integrate in real-time with operational processes, companies and public sector organizations are evolving their enterprise architectures to incorporate new tools and approaches.

    Whether you prefer "big," "very large," "extremely large," "extreme," "total," or another adjective for the "X" in the "X Data" umbrella term, what's important is accelerated growth in three dimensions: volume, complexity and speed.

    Big data is not without its limitations. Many organizations need to revisit business processes, solve data silo

      [Read more...]
    Calculating your database size
    +2 Vote Up -0Vote Down

    I generally use the following MySQL INFORMATION_SCHEMA (I_S) query to Calculate Your MySQL Database Size. This query and most others that access the MySQL INFORMATION_SCHEMA can be very slow to execute because they are not real tables and are not governed by physical data, memory buffers and indexes for example but rather internal MySQL data structures.

    Mark Leith indicates in his post on innodb_stats_on_metadata that Innodb performs 8 random(ish) dives in to the index, when anybody accesses any of SHOW TABLE STATUS, SHOW INDEX, INFORMATION_SCHEMA.TABLES,INFORMATION_SCHEMA.STATISTICS for InnoDB tables. This can have an effect on performance, especially with a large number of Innodb tables, and a poor ratio of innodb_buffer_pool_size to disk data+index

      [Read more...]
    My favorite MySQL data type – DECIMAL(31,0)
    +1 Vote Up -0Vote Down

    It may seem hard to believe, but I have seen DECIMAL(31,0) in action on a production server. Not just in one column, but in 15 columns just in the largest 4 tables of one schema. The column was being used to represent a integer primary or foreign key column.

    In a representative production instance (one of a dozen plus distributed production database servers) the overall database footprint was decreased from ~10 GB to ~2 GB, a 78% saving. In total, 15 columns across just 4 tables were changed from DECIMAL(31,0) to INT UNSIGNED.

    One single table > 5GB was reduced to under 1GB (a 81% saving). This being my record for any GB+ tables in my time working with the MySQL database.

    Had this server for example had 4GB of RAM, and say 2.5GB allocated to the innodb_buffer_pool_size, this one change moved the system from requiring more consistent disk access (4x

      [Read more...]
    Kickfire Basics – The KFDB columnar storage engine
    +0 Vote Up -0Vote Down

    This is the first post in a new series of “Kickfire Basics” blog posts by myself and others here at Kickfire.  This series will review the basics of the Kickfire appliance starting from this post describing how data is stored on disk, to future posts on topics such as loading data into the appliance and writing queries which best leverage the capabilities of the SQL chip.

    The Kickfire Equation
    Column store + Compression + SQL Chip = performance

    The Kickfire Analytic Appliance features the new KFDB storage engine which was built from scratch to handle queries over vast amounts of data.  KFDB is a column store in contrast to most MySQL storage engines which are row stores.  What follows is a description of our column oriented storage engine and how it improves performance over typical row stores.

    This post concerns itself with the first part

      [Read more...]
    Scalable Internet Architectures
    +1 Vote Up -0Vote Down

    My old friend and collaborator Theo Schlossnagle at OmniTI posted his slides from his Scalable Internet Architectures talk at VelocityConf 2009.

    The slides are brilliant even without seeing Theo talk and I highly recommend the time it takes to flip through them, for anyone who is interested in systems performance. If anyone took an mp3 of this talk I’m dying to hear it, please let me know.

    For those of you unfamiliar with OmniTI, Theo is the CEO of this rather remarkable company specializing in Internet-scale architecture consulting. They generalize on Internet-scale architecture, not on one specific dimension the way Pythian specializes on the database tier. This allows them to see Internet-scale workloads from a unique systemic, multidisciplinary point of view; from the user experience all

      [Read more...]
    Do I have a 32-bit or 64-bit MySQL?
    +0 Vote Up -0Vote Down
    I was asked the question last week of how to tell whether a MySQL installation is 64-bit or 32-bit. Often, MySQL is pre-installed or installed via a package and it is forgotten when and how it was installed. I scratched my head for a little while and just run

    /usr/sbin/mysqld --verbose --help
    /usr/local/mysql/libexec/mysqld Ver 5.1.25-rc-debug for suse-linux-gnu on i686
    (Source distribution)

    This is somewhat telling and I can tell from having used MySQL that this is 32-bit. However, there's nothing that sticks out like a sore thumb that says one way or the other that is is 64 or 32 bit (ok, maybe "i686" (?))

    I was working outside, which seems to be where I become inspired and had a "DUH" moment. The command "file" is the key for this:

    32-bit linux:

    patg@ishvara:~> file /usr/local/mysql/libexec/mysqld

      [Read more...]
    Scalability Best Practices: eBay
    +0 Vote Up -0Vote Down

    Following a link from the High Scalability blog, I found this really great article about scalability practices, as told by Randy Shoup at eBay. Randy is very good at explaining some of the more technical aspects in more or less plain English, and it even helped me find some wording I was looking for to help me explain the notion (and benefits) of functional partitioning. He also covers ideas that apply directly to your application code, your database architecture (including a little insight into their sharding strategy), and more. Even more about eBay’s architecture can be found here.

    addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F05%2F29%2Fscalability-best-practices-ebay%2F'; addthis_title = 'Scalability+Best+Practices%3A+eBay'; addthis_pub = 'jonesy';
    Technical Podcasts I Listen To
    +0 Vote Up -0Vote Down

    There are a few podcasts I tend to listen to as I have time. Since I work with a wide range of technologies, I've tried to group them together into a semblance of order. There are a few others I am evaluating, but since I haven't listened to a large enough body of work, I'll refrain from listing them at this time. If there's one you think is particularly valuable or interesting that I don't have listed, please leave it in the comments.


    .NET Rocks - http://www.dotnetrocks.com/

    This is one of the best done podcasts out there and they cover anything and everything related to Microsoft .NET. That's a broad brush of most anything that interacts with Microsoft technologies. This one runs twice a week and is about an hour each

      [Read more...]
    The Architecture Layer
    +0 Vote Up -0Vote Down

    Contemporary software engineering models include many loosely-defined layers. Database developers might help with other layers, but for the most part a database administrator’s domain is the persistence layer.

    • Presentation

    • Application

    • Business Logic

    • Persistence (also called Storage)

    The Daily WTF has an article on The Mythical Business Layer makes the case for not separating the business layer and the application layer:

    A good system (as in, one that’s maintainable by other people) has no choice but to duplicate, triplicate, or even-more-licate business logic. If Account_Number is a seven-digit required field, it should be declared as CHAR(7) NOT NULL in

      [Read more...]
    Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community - Video
    +0 Vote Up -0Vote Down
    The video of one of my three sessions, "Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community", presented at MySQL Conference & Expo 2008 has been uploaded by Sheeri. I am very thankful to her for doing all the hard work and making it available.

    There are a few slides that were edited out of video because of reasons beyond my control. However, you should still be able to enjoy most of the video.

    There is one point related to this video that I would like to make: Based on my particular experience I was leading to believe that

      [Read more...]
    The Impact of Architecture/Design Choices
    +0 Vote Up -0Vote Down
    I try my best to get things right the first time. So often, correcting a mistake or bad choice is costly: more costly than it would have been to take the time to do it right the first time. However, as SQL Server MVP Andy Leonard points out, sometimes you can't correct an issue.

    He starts with the example of a camera and taking a picture out of focus and then brings it around to development. His example is great: you can't provide up to the second updates to a system than only collects data every five seconds. Well, you can, but the data is only going to change ever five seconds. This raises the question often asked by learning styles expert,

      [Read more...]
    Showing entries 1 to 27

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.