Pytorch memory leak when threading

I get significant memory leak when running Pytorch model to evaluate images from dataset. Every new image evaluation is started in a new thread.

It doesn't matter if the code waits for the thread to finish or not. When the threads are not used (just evaluate function is called), there is no any leak. I've tried to delete the thread variable every iteration, but that doesn't help.

Here is the code:

hidden_sizes = [6336, 1000]   class Net(torch.nn.Module):      def __init__ (self):          super(Net, self).__init__()          self.fc1 = torch.nn.Linear(hidden_sizes[0], hidden_sizes[1])        def forward(self, x):          x= x.view(-1,hidden_sizes[0])          x = torch.nn.functional.log_softmax(self.fc1(x), dim=1)          return x  #---------------------------------------------------------------  def newThread(i):     image=cv2.imread(pathx+filenames[i], cv2.IMREAD_GRAYSCALE)       images = tran(image)     images = tran1(images)     images = images.unsqueeze(0)       #Run model     with torch.no_grad():        logps = model(images)     ps = torch.exp(logps)     probab = list(ps.numpy()[0])     pred_label = probab.index(max(probab))  #---------------------------------------------------------------  model = Net ()  model.load_state_dict(torch.load("test_memory_leak.pt"))  #normalize image  tran = transforms.ToTensor()  tran1 = transforms.Normalize((0.5,), (0.5,))       pathx="images\\"  filenames=os.listdir(pathx)    for i in range(len(filenames)):        thread1  = threading.Thread(target = newThread, args = (i, ))       thread1.start()      thread1.join()  

What could be the reason for that?

UPD: Tried to detect memory leaks with guppy, but the reason still isn't clear. Here are some stats: First table is program memory usage on the beginning, the second one is when the memory usage increased x2.4 fold (up to 480mb) after analyzing 1000 images:

Partition of a set of 260529 objects. Total size = 33587422 bytes.   Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)       0  72188  28  9511536  28   9511536  28 str       1  70921  27  5418336  16  14929872  44 tuple       2  32926  13  2526536   8  17456408  52 bytes       3  16843   6  2434008   7  19890416  59 types.CodeType       4   2384   1  2199952   7  22090368  66 type       5  14785   6  2010760   6  24101128  72 function       6   4227   2  1631384   5  25732512  77 dict (no owner)       7    794   0  1399928   4  27132440  81 dict of module       8   2384   1  1213816   4  28346256  84 dict of type       9     38   0   704064   2  29050320  86 dict of torch.tensortype  <575 more rows. Type e.g. '_.more' to view.>  -------------------------------------------    Partition of a set of 265841 objects. Total size = 34345930 bytes.   Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)       0  72203  27  9523346  28   9523346  28 str       1  70924  27  5418584  16  14941930  44 tuple       2  32928  12  2530722   7  17472652  51 bytes       3  16844   6  2434152   7  19906804  58 types.CodeType       4   2384   1  2200488   6  22107292  64 type       5  14786   6  2010896   6  24118188  70 function       6   4232   2  1637736   5  25755924  75 dict (no owner)       7    794   0  1399928   4  27155852  79 dict of module       8   2384   1  1213816   4  28369668  83 dict of type       9    265   0   840672   2  29210340  85 set  <577 more rows. Type e.g. '_.more' to view.>  

https://stackoverflow.com/questions/67380780/pytorch-memory-leak-when-threading May 04, 2021 at 03:53PM