Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Previous 30 Newer Entries Showing entries 31 to 60 of 66 Next 6 Older Entries

Displaying posts with tag: data (reset)

Strata Gems: DIY personal sensing and automation
+1 Vote Up -3Vote Down

We're publishing a new Strata Gem each day all the way through to December 24. Yesterday's Gem: Turn MySQL into blazing fast NoSQL.

Tomorrow's augmented reality is being built today on mobile devices. The Tasker application for Android is a fun platform for prototyping personal automation and sensing applications. Described modestly as an application which "performs tasks based on contexts," it gives non-programmers access to the sensing and system features of the

  [Read more...]
The growing importance of data journalism
+0 Vote Up -0Vote Down

One of the themes from News Foo that continues to resonate with me is the importance of data journalism. That skillset has received renewed attention this winter after Tim Berners-Lee called analyzing data the future of journalism.

When you look at data journalism and the big picture, as USA Today's Anthony DeBarros did at his blog in November, it's clear the recent suite of technologies is part of a continuum of technologically enhanced storytelling that traces back to computer-assisted reporting (CAR).

As DeBarros pointed out,

  [Read more...]
Strata Gems: Turn MySQL into blazing fast NoSQL
+1 Vote Up -1Vote Down

We're publishing a new Strata Gem each day all the way through to December 24. Yesterday's Gem: What your inbox knows.

The trend for NoSQL stores such as memcache for fast key-value storage should give us pause for thought: what have regular database vendors been doing all this time? An important new project, HandlerSocket, seeks to leverage MySQL's raw speed for key-value storage.

NoSQL databases offer fast key-value storage for use in backing web applications, but years of work on regular relational

  [Read more...]
Why clouds and web services will continue to take over computing
+0 Vote Up -0Vote Down

Series

What are the chances for a free software cloud?

  [Read more...]
Strata Gems: Who needs disks anyway?
+0 Vote Up -1Vote Down

We're publishing a new Strata Gem each day all the way through to December 24. Yesterday's Gem: Kinect democratizes augmented reality.

Today's databases are designed for the spinning platter of the hard disk. They take into account that the slowest part of reading data is seeking: physically getting the read head to the part of the disk it needs to be in. But the emergence of cost effective solid state drives (SSD) is changing all those assumptions.

Over the course of 2010, systems designers have been realizing

  [Read more...]
Defining clouds, web services, and other remote computing
+0 Vote Up -0Vote Down

Series

What are the chances for a free software cloud?

  [Read more...]
Energy data in action
+0 Vote Up -0Vote Down

One of the Department of Energy's flagship open government initiatives, Open Energy Information (OpenEI), was on display at the recent National Science Festival. Ryan McKeel, an engineer at the National Renewable Energy Laboratory, talks about how OpenEI provides access to data, models and tools in the following clip:

OpenEI is "open" in any number of important ways, including open source and open linked data, as Debbie Brodt-Giles, the OpenEI project leader, notes in an Amazon case study (OpenEI is hosted on Amazon's Web Services platform). "Key platform software for OpenEI includes

  [Read more...]
Four short links: 21 October 2010
+0 Vote Up -0Vote Down

  • Using MysQL as NoSQL -- 750,000+ qps on a commodity MySQL/InnoDB 5.1 server from remote web clients.
  • Making an SLR Camera from Scratch -- amazing piece of hardware devotion. (via hackaday.com)
  • Mac App Store Guidelines -- Apple announce an app store for the Macintosh, similar to its app store for iPhones and iPads. "Mac App" no longer means generic "program", it has a new and specific meaning, a program that must be installed through the App store and which has limited functionality (only
  •   [Read more...]
    LCA Miniconf Call for Papers: Data Storage: Databases, Filesystems, Cloud Storage, SQL and NoSQL
    +3 Vote Up -0Vote Down

    This miniconf aims to cover many of the current methods of data storage and retrieval and attempt to bring order to the universe. We’re aiming to cover what various systems do, what the latest developments are and what you should use for various applications.

    We aim for talks from developers of and developers using the software in question.

    Aiming for some combination of: PostgreSQL, Drizzle, MySQL, XFS, ext[34], Swift (open source cloud storage, part of OpenStack), memcached, TokyoCabinet, TDB/CTDB, CouchDB, MongoDB, Cassandra, HBase….. and more!

    Call for Papers open NOW (Until 22nd October).

    The SMAQ stack for big data
    +0 Vote Up -0Vote Down

    SMAQ report sections

    → MapReduce

    → Storage

    → Query

    → Conclusion

    "Big data" is data that becomes large enough that it cannot be processed using conventional methods. Creators of web search engines were among the first to confront this problem. Today, social networks, mobile phones, sensors and science contribute to petabytes of data created daily.

    To meet the challenge of processing such large data sets, Google created MapReduce. Google's work and Yahoo's creation of the Hadoop MapReduce implementation has spawned an ecosystem of big data processing tools.

    As MapReduce has grown in  [Read more...]

    MySQL GIS – Part 4
    +1 Vote Up -0Vote Down

    WHAT CAN YOU DO WITH GEO DATA?

    Geo spatial indexes are what make this type of data valuable.  With shape and point data you can find relationships between object in our physical world.  How close is the lightning in the storm front?  What homes where hailed on? (WDT) What schools are in my city?  With a list of homes for sale, how fare are they from their nearest school?  What picture where take in this area. (TwitPic)

    Lets start with a simple grid of coordinates by creating a table for it call geom, adding our data points in and out of our grid and then searching with a small bounding box. The grid looks like this.

    0,0 0,1 0,2 1,0 1,1 1,2 2,0 2,1 2,2
    CREATE TABLE geom
      [Read more...]
    MySQL GIS – Part 3
    +4 Vote Up -0Vote Down

    What data is available?

    GEO data is expensive to create, so has been created by governments.  In the past governments charged for this data.  In 1980 the USGS was charging $300 (usd) per county for Oklahoma GEO data. (I complained to my congressman.) Today, a quick Internet search turns up lots of free GIS data.

    I was hoping to find a neat collection of basic GEO data.   It would be nice if there was one place you could get world political borders (Polygons), postal codes (Polygons) and  points of interest like hospitals and airports.  What you can find is lots of  lists, often collections of odd data created for a virility of complex political purpose.  For example, The Global Change Master Directory is a large list of data sources on earth

      [Read more...]
    MySQL GIS – Part 1
    +3 Vote Up -1Vote Down

    In my business (weather) we use lots map based (Geo) information.  Almost every table has latitude and longitude. Working with this kind of data can be exciting and frustrating.  This should give you a quick start into GIS with MySQL.

    “A geographic information system (GIS), or geographical information system, is any system that captures, stores, analyzes, manages, and presents data that are linked to location. In the simplest terms, GIS is the merging of cartography, statistical analysis, and database technology. GIS systems are used in cartography, remote sensing, land surveying,

      [Read more...]
    Does Size or Type Matter?
    +4 Vote Up -0Vote Down

    MySQL seems to be happy to convert types for you. Developers are rushed to complete their project and if the function works they just move on. But what is the costs of mixing your types? Does it matter if your are running across a million rows or more? Lets find out.

    Here is what the programmers see.

    mysql> select 1+1;
    +-----+
    | 1+1 |
    +-----+
    |   2 |
    +-----+
    1 row in set (0.00 sec)
    
    mysql> select "1"+"1";
    +---------+
    | "1"+"1" |
    +---------+
    |       2 |
    +---------+
    1 row in set (0.00 sec)

    Benchmark

    What if we do a thousand simple loops?  How long does the looping itself take?

    The BENCHMARK() function executes the expression

      [Read more...]
    Is OpenStack Cloud Computing Rocket Science?
    +0 Vote Up -0Vote Down



    There’s a real explosion of cloud platforms and management tools, it seems you can’t swing a dead cat without hitting one these days. In the commercial proprietary solutions space you have – CA’s 3Terra AppLogic, Enomaly, Nimbula, RightScale. In open source there are Eucalyptus



      [Read more...]
    Federated Tables
    +0 Vote Up -0Vote Down

    Your searching for how to create a join across two databases on two different servers and it can’t be done directly.   select  d1.a, d2.b from db1@server1 join db2@server2 where db1.c = db2.c; does not work.

    You learn about federated databases.  The federated storage engine allows accesses data in tables of remote databases.  Now how do you make it work?

    1) Check if the federated storage engine is supported.  Federation is OFF by default!

    mysql> show engines;
    +------------+---------+----------------------------------------------------------------+
    | Engine     | Support | Comment                                                        |
    +------------+---------+----------------------------------------------------------------+
    | InnoDB     | YES    
      [Read more...]
    mg_hot_replace_table.pl
    +0 Vote Up -0Vote Down

    Do you have MyISAM tables you reload with new data?

    Do your queries, using that table, get blocked because the table is locked?

    Do the waiting queries create idle connections slowing down the table load?

    Do you wish you could just replace the table?

    Years ago I was told you can replace CSV tables by simply replacing the CSV file. I figured this would also be true of a MyISAM file and it is. I use this perl script to replace MyISAM tables forcast and current observation weather data. The processing and tables are created on another computer. Weather forecasting is CPU and database expensive. I then copy (rsync) the files to the production system and run this script.

    #!/usr/bin/perl
    ################################################################################
      [Read more...]
    Looking just at the data
    +1 Vote Up -0Vote Down

    There are many areas you need to review when addressing MySQL performance such as current database load, executed SQL statements, connections, configuration parameters, memory usage, disk to memory ratio, hardware performance & bottlenecks just to name a few.

    If you were to just look at the data that is held in the database, what would you consider?
    Here are my tips, when looking just at the data.

  • What is the current database size?
  • What is the growth of data over time, say daily, weekly?
  • Which are the 2 largest tables now?
  • What 2 tables are growing the fastest?
  • What tables have greatest churn, specifically DELETE’s?
  • How often do you optimize your tables?
  • What is your archiving/purging strategy? Do you even have one?
  • Review data types? I average 25% reduction in footprints, just by

  •   [Read more...]
    My favorite MySQL data type – DECIMAL(31,0)
    +1 Vote Up -0Vote Down

    It may seem hard to believe, but I have seen DECIMAL(31,0) in action on a production server. Not just in one column, but in 15 columns just in the largest 4 tables of one schema. The column was being used to represent a integer primary or foreign key column.

    In a representative production instance (one of a dozen plus distributed production database servers) the overall database footprint was decreased from ~10 GB to ~2 GB, a 78% saving. In total, 15 columns across just 4 tables were changed from DECIMAL(31,0) to INT UNSIGNED.

    One single table > 5GB was reduced to under 1GB (a 81% saving). This being my record for any GB+ tables in my time working with the MySQL database.

    Had this server for example had 4GB of RAM, and say 2.5GB allocated to the innodb_buffer_pool_size, this one change moved the system from requiring more consistent disk access (4x

      [Read more...]
    Seeking public data for benchmarks
    +3 Vote Up -0Vote Down

    I have several side projects when time permits and one is that of benchmarking various MySQL technologies (e.g. MySQL 5.0,5.1,5.4), variants (e.g. MariaDB, Drizzle) and storage engines (e.g. Tokutek, Innodb plugin) and even other products like Tokyo Cabinet which is gaining large implementations.

    You have two options with benchmarks, the brute force approach such as Sysbench, TPC, sysbench, Juice Benchmark, iibench, mysqlslap, skyload. I prefer the realistic approach however these are always on client’s private data. What is first needed is better access to public data for benchmarks. I have compiled this list to date and I am seeking additional sources for reference.

    • Feebase – Data is in clean loadable format
    • IMDB – Not
      [Read more...]
    Four short links: 4 August 2009
    +0 Vote Up -0Vote Down

  • NASA Nebula Services/Platform Stack -- The NEBULA platform offers a turnkey Software-as-a-Service experience that can rapidly address the requirements of a large number of projects. However, each component of the NEBULA platform is also available individually; thus, NEBULA can also serve in Platform-as-a-Service or Infrastructure-as-a-Service capacities. Bundles RabbitMQ, Eucalyptus, LUSTRE storage, Fabric deployment, Varnish front-end, MySQL and more. (via Jim Stogdill)
  • A Short History of btrfs -- Now for some
  •   [Read more...]
    MySQL University: Starring Sakila - A Data Warehouse Mini-Tutorial
    Employee +0 Vote Up -0Vote Down

    This Thursday (July 2nd, 13:00 UTC), Roland Bouman will will take the Sakila sample database and explain how to design and load a data warehouse for it, and how to use that to create those reports and charts that make managers smile. Along the way, he'll explain often uttered terms and concepts such as dimensional model, denormalization, star schema, OLAP, data staging, business intelligence, ETL and data integration. Roland's session is titled Starring Sakila - A Data Warehouse Mini-Tutorial.

    For MySQL University sessions, point your browser

      [Read more...]
    MySQL University: Starring Sakila - A Data Warehouse Mini-Tutorial
    Employee +0 Vote Up -0Vote Down

    This Thursday (July 2nd, 13:00 UTC), Roland Bouman will will take the Sakila sample database and explain how to design and load a data warehouse for it, and how to use that to create those reports and charts that make managers smile. Along the way, he'll explain often uttered terms and concepts such as dimensional model, denormalization, star schema, OLAP, data staging, business intelligence, ETL and data integration. Roland's session is titled Starring Sakila - A Data Warehouse Mini-Tutorial.

    For MySQL University sessions, point your browser

      [Read more...]
    Network Management Data Reduction and Smoothing -- A MySQL Webinar
    Employee +1 Vote Up -0Vote Down

    ScienceLogic embeds MySQL (http://www.mysql.com/" target="_blank) in its EM7 network management appliances. An installation of EM7 can perform over half a billion database queries daily, storing massive amounts of data for both real-time and trended performance reporting.

    Michael McFadden, senior software architect with ScienceLogic, will discuss all this in an upcoming MySQL webinar (http://mysql.com/news-and-events/web-seminars/display-361.html).

    Network Management Data Reduction and Smoothing -- A MySQL Webinar
    Employee +0 Vote Up -0Vote Down

    ScienceLogic embeds MySQL (http://www.mysql.com/" target="_blank) in its EM7 network management appliances. An installation of EM7 can perform over half a billion database queries daily, storing massive amounts of data for both real-time and trended performance reporting.

    Michael McFadden, senior software architect with ScienceLogic, will discuss all this in an upcoming MySQL webinar (http://mysql.com/news-and-events/web-seminars/display-361.html).

    Network Management Data Reduction and Smoothing -- A MySQL Webinar
    Employee +0 Vote Up -0Vote Down

    ScienceLogic embeds MySQL (http://www.mysql.com/" target="_blank) in its EM7 network management appliances. An installation of EM7 can perform over half a billion database queries daily, storing massive amounts of data for both real-time and trended performance reporting.

    Michael McFadden, senior software architect with ScienceLogic, will discuss all this in an upcoming MySQL webinar (http://mysql.com/news-and-events/web-seminars/display-361.html).

    Data Backup and Recovery for MySQL - a MySQL Time Machine is Born.
    Employee +0 Vote Up -0Vote Down

    Sure, you've heard it before: [some company's logo] has a new MySQL backup tool that promises to solve all of your data recovery needs. The good news is most of these tools work pretty well. However, they tend to suffer from a similar set of limitations. Most require sophisticated infrastructures or complex setup and maintenance and can become a resource drain for some organizations. You're probably wondering, "Why can't someone build a fully automated MySQL backup solution that you can just turn on and forget?"

    I am happy to say that the MySQL Developers at Sun are doing just that. In fact, a prototype will be demonstrated at the 2009 MySQL Users' Conference that will show the feasibility of a fully automated MySQL backup and recovery tool. It's being called the MySQL Time Machine and (with all due respect to all vendors with products of similar names) it allows you to

      [Read more...]
    Data Backup and Recovery for MySQL - a MySQL Time Machine is Born.
    Employee +0 Vote Up -0Vote Down

    Sure, you've heard it before: [some company's logo] has a new MySQL backup tool that promises to solve all of your data recovery needs. The good news is most of these tools work pretty well. However, they tend to suffer from a similar set of limitations. Most require sophisticated infrastructures or complex setup and maintenance and can become a resource drain for some organizations. You're probably wondering, "Why can't someone build a fully automated MySQL backup solution that you can just turn on and forget?"

    I am happy to say that the MySQL Developers at Sun are doing just that. In fact, a prototype will be demonstrated at the 2009 MySQL Users' Conference that will show the feasibility of a fully automated MySQL backup and recovery tool. It's being called the MySQL Time Machine and (with all due respect to all vendors with products of similar names) it allows you to

      [Read more...]
    Data Backup and Recovery for MySQL - a MySQL Time Machine is Born.
    Employee +0 Vote Up -0Vote Down

    Sure, you've heard it before: [some company's logo] has a new MySQL backup tool that promises to solve all of your data recovery needs. The good news is most of these tools work pretty well. However, they tend to suffer from a similar set of limitations. Most require sophisticated infrastructures or complex setup and maintenance and can become a resource drain for some organizations. You're probably wondering, "Why can't someone build a fully automated MySQL backup solution that you can just turn on and forget?"

    I am happy to say that the MySQL Developers at Sun are doing just that. In fact, a prototype will be demonstrated at the 2009 MySQL Users' Conference that will show the feasibility of a fully automated MySQL backup and recovery tool. It's being called the MySQL Time Machine and (with all due respect to all vendors with products of similar names) it allows you to

      [Read more...]
    Sun talks out Cloud: Open Cloud Platform
    Employee +0 Vote Up -0Vote Down

    Sun's Open Cloud Vision unveilled: Open Cloud Platform, an open infrastructure powered by Java, MySQL, OpenSolaris, and Open Storage software technologies.  Open APIs, Open formats and Open source.

    On March 18th, at CommunityONE aka CloudONE, Sun unveiled the open cloud platform for powering public and private clouds. We also  announced that we are building our own Public Cloud. This will include a Storage and Compute Cloud. Our Cloud will be compatible with Amazon S3 and EC2 at the API level. Meaning, we will provide S3 and EC2 compatibility APIs in addition to our own, hence enabling an easy migration from Amazon services to Sun Cloud. All clouds - public, private or hybrid, built on Sun's Open Cloud

      [Read more...]
    Previous 30 Newer Entries Showing entries 31 to 60 of 66 Next 6 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.