Hi, I'm Ben Osheroff, an engineer on the infrastructure team at
Zendesk. My team began this year with a single goal: to better
scale Zendesk's view system.
If you haven't tried Zendesk yet, views are lists of tickets with
user-specified constraints, generally consumed by support agents
as a workflow tool. As implemented, views are a SQL generation
and execution engine, and this poses some interesting scale
challenges: giving your customers a fully-featured bridge to SQL
means that they will find all sorts of creative ways to generate
queries your databases aren't prepared to handle.
To solve this, we've beenbuilding a system based on adaptive
caches, where Zendesk keeps "materialized views" of
expensive-to-execute queries, created on the fly as needed. At
the heart of this caching system is a data firehose, a change
capture system like …
[Read more]