Quantcast
Channel: Topliners: Message List
Viewing all articles
Browse latest Browse all 3431

Re: Is it a queue managing the BULK jobs?

$
0
0

Hi Biao,

 

Based on my own observation, jobs are processed one by one, and the order is generally the order in which they were added to the queue. A data transfer service runs first to parse the amount of rows on the file, and then puts it back into the queue again for loading into the db. You might see this in your sync logs as .json files being converted to .csv once you trigger the sync.

 

Depending on the amount of data present after parsing, it can affect what order it is processed in due to some advanced load balancing logic in place. For example, if you push one really big file and a few smaller files, chances are, the system will more frequently have small chunks of time/resources free to process a smaller file. The worker looks at the oldest file in the queue, sees that it is 'too big' to handle in the time it has, skips it and then grabs the next file which may be smaller. In most scenarios however, you don't see the system skip over large files that often or for very long. The net effect of this behaviour is that if you load a massive file into the system that it can't process right away, it's not going to cause a few minutes of unnecesssary blockage for other smaller files it can manage right away.

 

Again, this is based on observation and such backend logic is subject to change/tweaking at any time.

 

Regards,

Bojan


Viewing all articles
Browse latest Browse all 3431

Trending Articles