As you might know, mysqldump is single-threaded and STDOUT is its
default output. As MyDumper is multithreaded, it has to write on
different files. Since version 0.11.3 was released in Nov 2021,
we have the possibility to stream our backup in MyDumper. We
thought for several months until we decided what was the simplest
way to implement it and we also had to add support for
compression. So, after fixing several bugs, and we now consider
it is stable enough, we can explain how it works.
How Can You Stream if MyDumper is Multithreaded?
Receiving a stream is not a problem for myloader, it receives a
file at a time and sends it to a thread to process it. However,
each worker thread in mydumper is connected to the database, and
as soon as it reads data, it should be sent to the stream, which
might cause collisions with other worker threads that are reading
data from the database. In order to avoid this issue, we ended up
with the …
[Read more]