A small thing that provides a huge help.
The other day I was writing some code to process a very large amount of items coming from a social media API. My items were ending in a queue in MySQL and then needed to be processed and eventually moved.
The task was not so strange, but what I have to do is to develop a queue processor. Now when you need to process a queue you have two types of queue: static and dynamic.
The static comes in a batch of N number of items in a given time interval and is normally easier to process given you have a defined number of items that you can split in chunks and process in parallel.
The dynamic is… well… more challenging. One option is to wait to have a predefined number of items, and then process them as if they were a static queue.
But this approach is not very good, given it is possible that it will delay a lot …
[Read more]