From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, …

The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Continuent is very pleased and excited to announce that the new Tungsten Clustering 6.0 and Tungsten Replicator 6.0 are now available for download by our customers. The 6.0 release is the culmination of over a years work within our clustering product in order to improve the functionality and manageability of what we now call our ‘Multimaster Clustering’ solution. This is the replacement for what we called the multi-site, multi-master (MSMM) clustering functionality in earlier releases. The multimaster clustering allows for multiple clusters, in multiple locations, to be linked together into a single composite cluster. Because it’s a composite cluster, you gain all of the functionality that’s already available in a cluster, such as:

• High availability
• Failover
• Automated recovery
• Read-write split
• Maintenance without downtime

But it’s now applied to the …

You may know Continuent Tungsten for our highly advanced MySQL replication tool, Tungsten Replicator, and for our state-of-the-art MySQL clustering solution, Tungsten Clustering. Our solutions are used by leading SaaS vendors, e-commerce, financial services and telco customers.

But there are more, many more, Tungsten deployments out there. Tungsten Replicator can be used for real-time data

Continuent are pleased to announce the release of Tungsten Replicator and Tungsten Clustering 5.2.0

This release is one of our most exciting new releases for a while, as it contains some significant new features and lays the groundwork for some additional new functionality in the upcoming 5.3.0 and 6.0 releases due later this year.

In particular, this release includes the following new features:

• New replicator filtering environment to make filtering quicker and easier to use, and more flexible
  • New filter configuration standard for new filters
  • New filter to make replication out of a cluster easier
  • New filters for filtering events and data
• New applier for sending Apache Kafka messages directly from an incoming data stream
• New applier for adding incoming records directly to Elasticsearch for indexing
• New …

Introduction As previously explained, CDC (Change Data Capture) is one of the best ways to interconnect an OLTP database system with other systems like Data Warehouse, Caches, Spark or Hadoop. Debezium is an open source project developed by Red Hat which aims to simplify this process by allowing you to extract changes from various database … Continue reading How to extract change data events from MySQL to Kafka using Debezium →

Hello again everybody.

Well, I promised it a couple of weeks ago, and I’m sorry it has been so long (I’ve been working on other fun stuff in addition to this). But I’m pleased to say that we now have a fully working applier that takes data from an incoming THL stream, whether that is Oracle or MySQL, and converts that into a JSON document and message for distribution over a Kafka topic.

Currently, the configuration is organised with the following parameters:

• The topic name is set according to the incoming schema and table. You can optionally add a prefix. So, for example, if you have a table ‘invoices’ in the schema ‘sales’, your Kafka topic will be sales_invoices, or if you’ve added a prefix, ‘myprefix_schema_table’.
• Data is marshalled into a JSON document as part of the message, and the structure is to have a bunch of metadata and then an embedded record. You’ll see an …

Hello again everybody.

Well, I promised it a couple of weeks ago, and I’m sorry it has been so long (I’ve been working on other fun stuff in addition to this). But I’m pleased to say that we now have a fully working applier that takes data from an incoming THL stream, whether that is Oracle or MySQL, and converts that into a JSON document and message for distribution over a Kafka topic.

Currently, the configuration is organised with the following parameters:

• The topic name is set according to the incoming schema and table. You can optionally add a prefix. So, for example, if you have a table ‘invoices’ in the schema ‘sales’, your Kafka topic will be sales_invoices, or if you’ve added a prefix, ‘myprefix_schema_table’.
• Data is marshalled into a JSON document as part of the message, and the structure is to have a bunch of metadata and then an embedded record. You’ll see an …

Uber Engineering

Uber’s mission is transportation as reliable as running water, everywhere, for everyone. Last time, we talked about the foundation that powers Uber Engineering. Now, we’ll explore the parts of the stack that face riders and drivers, starting …

The post The Uber Engineering Tech Stack, Part II: The Edge and Beyond appeared first on Uber Engineering Blog.

Ever since MySQL replication has existed, people have dreamed of a good solution to automatically split read from write operations, sending the writes to the MySQL master and load balancing the reads over a set of MySQL slaves. While if at first it seems easy to solve, the reality is far more complex.

First, the tool needs to make sure it parses and analyses correctly all the forms of SQL MySQL supports in order to sort writes from reads, something that is not as easy as it seems. Second, it needs to take into account if a session is in a transaction or not.

While in a transaction, the default transaction isolation level in InnoDB, Repeatable-read, and the MVCC framework insure that you’ll get a consistent view for the duration of the transaction. That means all statements executed inside a transaction must run on the master but, when the transaction commits or rollbacks, the following select statements on the session can be again …

This is an idea which i presented at percona live 2015.

Is MySQL an avatar of Apache Kafka ?

Can it be Kafka ?

Yes, it can.

This talk takes a shot at modeling MySQL as Kafka.

PS: it’s unconventional, hence a WILD supposition

slides @

http://www.slideshare.net/jaihind213/can-mysql-bekafka

 or

http://www.percona.com/live/mysql-conference-2015/sessions/wild-supposition-can-mysql-be-kafka