If you are using MyDumper as your Logical Backup solution and you store your backups on S3, you need to take a local backup and then upload it to S3. But what if there is not enough space to hold the backup on the server where we are taking the backup? Even if we have enough disk space, we will need to wait until the end to start to upload the files, making the whole process longer.
MyDumper implemented stream backup in v0.11.3 and we have been polishing the code since then. We also implemented two ways of executing external commands:
--exec-per-thread: The worker that is getting the
data from the database will write and redirect to the STDIN of
the external command. It will be similar to execute
FILE | command per every written and closed file.
--exec: In this case, the worker writes in the local
storage and when the file is closed, the filename is enqueued.
The exec threads are …