This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
with pytorch, do you know what periodic, relatively fixed slowdowns in random places in cpu execution could be? It is when I load 24MB tensors from files to cpu and then from cpu to pinned memory a lot (like every 15ms). Each step of the load I see happening fast at times, but at random spots in the execution there is a 60ms delay. I considered if it is swap but I don't have swap on the system it happens.
Could be a few different things. Possibly the garbage collector stuggling.
Maybe you are not freeing the 24MB tensors when you are done with them and they are getting stuck in memory.
If you are using a very weak/small machine I would try your code on a more powerful machine (more cores/more memory/faster disks).
I would keep an eye on htop and iotop to see if there are any clues there. e.g. is memory just growing?
I would also add additional logging to see if you can spot the exact line the pauses are ocuring at.
Aside: Swap can sometimes help performance on machines with small amounts of memory - because rarely used operating system files can be swapped out, leaving more space for currently running program.