It seems to be impossible to perform client-side paging full
table scans within MySQL.
For example, say you want to take a 1GB file and page through it
10MB at a time.
With a flat file I could just read 10MB off disk, read the next
10MB, etc.
This would be amazingly efficient as you could write the data
sequentially, without updating indexes, and then read the data
back out by byte offset.
You could page through MySQL tables by adding an index
to a table:
SELECT * FROM FOO WHERE PTR > 10000 LIMIT 10000;
for example … but I REALLY want to avoid an index step because it
is not cheap and only required since MySQL doesn’t support
cursors.
This index slows down the import stage and I have to buffer the
index data in memory which is just a waste.
I could use LIMIT with OFFSET but this isn’t going to be
amazingly efficient because it will either require us to use …
[Read more]