Showing entries 1 to 10 of 17
7 Older Entries »
Displaying posts with tag: flexviews (reset)
‘Indexing’ JSON documents for efficient MySQL queries over JSON data

MySQL meets NoSQL with JSON UDF

I recently got back from FOSDEM, in Brussels, Belgium. While I was there I got to see a great talk by Sveta Smirnova, about her MySQL 5.7 Labs release JSON UDF functions. It is important to note that while the UDF come in a 5.7 release it is absolutely possible to compile and use the UDF with earlier versions of MySQL because the UDF interface has not changed for a long time. However, the UDF should still be considered alpha/preview level of quality and should not be used in production yet! For this example I am using Percona Server 5.6 with the UDF.

That being said, the proof-of-concept that I’m about to present here uses only one JSON function (JSON_EXTRACT) and it has worked well enough in my testing to present my idea here. The JSON functions will probably be GA sometime soon anyway, and this is a useful test of the JSON_EXTRACT function. …

[Read more]
Flexviews 1.8.0 beta 1 released

You can get Flexviews at the project page:
Flexviews project page

This release includes the following new features:

Table change logs now include a new column: fv$gsn. This column carries the "global sequence number" which uniquely identifies the order in which to make changes in the database. This was required to fix some problems identified in updating views that did not contain aggregate functions.

Updated refresh algorithm takes GSN into account and applies changes in GSN order.

All reported bugs have been fixed.

FlexCDC now supports all MySQL data types properly.

Flexviews 1.7.0 GA released

I am happy to announce the GA release of Flexviews. I've numbered the release 1.7.0 GA.
There are a small number of bug fixes and enhancements:

  • Fixes for views which have aggregate functions but no GROUP BY expressions could have had problems during incremental update.
  • GA support for all aggregate functions except GROUP_CONCAT and AVG(distinct).
  • The PERCENTILE_XX (ie PERCENTILE_90 for a 90th percentile calculation) is now stable too

FlexCDC got a number of fixes and improvements:

  • TIMESTAMP columns are now properly supported
  • Now mysqlbinlog errors are detected, and the program exits gracefully
  • FlexCDC now logs to a log file instead of writing to STDOUT/STDERR
  • There are new PHP scripts for adding and removing table changelogs …
[Read more]
Using Flexviews – part one, introduction to materialized views

If you know me, then you probably have heard of Flexviews. If not, then it might not be familiar to you. I’m giving a talk on it at the MySQL 2011 CE, and I figured I should blog about it before then. For those unfamiliar, Flexviews enables you to create and maintain incrementally refreshable materialized views.

You might be asking yourself “what is an incrementally refreshable materialized view?”. If so, then keep reading. This is the first in a multi-part series describing Flexviews.

You can find part 2 of the series here:

The output of …

[Read more]
Flexviews 1.6.0-RC2 is released

Available immediately Flexviews 1.6.0-RC2.

This release is a bugfix release.  This is the second Flexviews release candidate.  If no major bugs are discovered the next release will be the GA release.

Flexviews is a stored procedure managed materialized view system for MySQL 5.1 or greater. 

What is fixed in Flexviews 1.6.0-RC2?

  • Numerous performance fixes. 
    • Flexviews uses fewer temporary tables and subqueries
    • A full table scan of the view is no longer required (only changed GB keys are scanned)
    • Dead code has been removed
  • Bug fixes
    • Removing tables and adding them to a view again could result in the WHERE clause being generated in the wrong order
    • Fix a problem with applying deltas to views which use PERCENTILE …
[Read more]
Flexviews 1.6.0-RC1 is released

Whats new in Flexviews 1.6.0RC1

  • This is the first release candidate before the final release.  If no major bugs are uncovered, then the next release will be the first GA release. 
  • Flexviews now has a test suite for all major features.  The creation of these tests uncovered a number of issues which have been resolved in this release. 
  • All MySQL aggregate functions except GROUP_CONCAT are now supported. 
  • A special aggregate function called PERCENTILE is now also supported.  The calculation uses a modified version of the GROUP_CONCAT based solution suggested by Roland Bouman for percentiles.  This function should be considered experimental.  Please report bugs if you find any.
  • You can add indexes to enabled materialized views using SQL_API/add_expr
  • Adding PRIMARY KEY indexes is no longer …
[Read more]
Check out my Flexviews talk in the Open Space today at 3PM.

Flexviews allows you to cache (materialize) SQL statements and then quickly apply only the changes to underlying tables to update the cache. This can phenomenally increase the performance of your application and allow you to more easily maintain summary tables automatically.

I'll be demonstrating the usage of Flexviews and how to create 'fast refresh' materialized views.

I'll be showing off:
FlexCDC - Flexible change data capture for MySQL
Flexviews - Uses FlexCDC and stored procedures to incrementally maintain mateialized views, including those with inner JOINS and aggregation.

A handy regular expression for 'tokenizing' a SQL statement

Often times I find myself having to handle a particular portion of a SQL statement via a script. I've written a lot of specialized regular expressions over time to handle these tasks, but the one that I've had to write the most is a basic 'tokenizer' which understands the quoting semantics of a MySQL statement. That is, it understands `BackTick`, 'Single Quote' and "Double Quoted" strings.

#PHP Heredoc syntax
$regex = <<< END_OF_REGEX
  [^ \"'`(),]*\([^)]*\)    #match functions like concat(x,"y",`a`.`z`) or sum(`xyz`);
  |\([^)]*?\)              #match grouped items
  |"[^"]*?"                #match double quoted items
  |'[^']*?'                #match single quoted items
  |`[A-Za-z0-9_ .'\"()+\\-&^%\$+?%\\/\\\\!`]+`  #match backtick mysql names
  |[^ ,]+                  #match keywords, operators and aliases

EDIT: After some comments and further testing, I've …

[Read more]
How to support COUNT(DISTINCT expression) expressions with Flexviews.

Unlike COUNT(expression), COUNT(DISTINCT expression) is not a distributable function. That is, the expression can not be computed by looking at only the changed rows in the table change logs.

The following view can not be FAST refreshed with Flexviews today:

SELECT a a_alias, 
       b b_alias, 
       c c_alias, 
       COUNT(DISTINCT d) d_alias
 GROUP BY a, b, c;

However, a dependent child materialization could be created to support the value for COUNT(DISTINCT d):

-- child materialization, dependent subview
--There will be one row for each DISTINCT value of d

SELECT a a_alias, 
       b b_alias, 
       c c_alias, 
       count(*) d_alias_cnt
 GROUP BY a, b, c, d;

The original view could then be rewritten as:

SELECT a a_alias, 
       b b_alias, 
       c c_alias, …
[Read more]
Can't wait for table change logs.

At today's keynote by Mark Callaghan one of the new options he talked about are table change logs. He mentioned they might be of use to external applications, like Flexviews, which he mentioned but not directly. He asked if the guy who wrote it was in the audience, so I got to wave my hand and yell 'Flexviews!'.

I caught up with Mark at the Facebook party this evening. I had a chance to talk to him not only about the change logs, but also about Kickfire and the SQL chip. He asked me what I thought about working at Kickfire and I smiled and said I love it. I think I said "I've never been able to join a billion row table to a hundred million row table, sort, group and get results back in less than a minute" and I'm sure the smile never left my face.

As far as the table change logs, he verified:

  • The global transaction id will be stored in the table
  • OLD and NEW …
[Read more]
Showing entries 1 to 10 of 17
7 Older Entries »