Hi, I'm Ben Osheroff, an engineer on the infrastructure team at Zendesk. My team began this year with a single goal: to better scale Zendesk's view system.
If you haven't tried Zendesk yet, views are lists of tickets with user-specified constraints, generally consumed by support agents as a workflow tool. As implemented, views are a SQL generation and execution engine, and this poses some interesting scale challenges: giving your customers a fully-featured bridge to SQL means that they will find all sorts of creative ways to generate queries your databases aren't prepared to handle.
To solve this, we've beenbuilding a system based on adaptive caches, where Zendesk keeps "materialized views" of expensive-to-execute queries, created on the fly as needed. At the heart of this caching system is a data firehose, a change capture system like …
[Read more]