error: (-4:Insufficient memory) Failed to allocate 36578304 bytes in function 'cv::OutOfMemoryError'

Intro

I want to train a convolutional neural net to predict how many upvotes a picture posted to r/art will get on Reddit as a personal project.

I've downloaded 30000 pictures -> 18648 valid pictures and I named them using their score and post id so that posts would the same score wouldn't conflict (0_k0un69.jpg for example, only mildly important for the code snippet)

Problem

I'm trying to create X, the input pictures array and Y, an array of the scores for neural net training. About 5.8% of the way through, I get the title error:

error: OpenCV(4.4.0) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-6sxsq0tp\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 36578304 bytes in function 'cv::OutOfMemoryError'  

The code that creates this error is the following:

path = "pics/"  #Get all file names  files = [f for f in os.listdir(path) if os.path.isfile(os.path.join(path, f))]  numpics = len(files)  #Store all image arrays and image names in a list  X = []  Y = []  counter = 0  for file in files:      #Progress bar      sys.stdout.write('\r')      sys.stdout.write("Reading images into DataSets:  {:.1f}%".format((100/(numpics-1)*counter)))      sys.stdout.flush()      counter+=1        X.append(cv2.imread(f'{path}{file}'))      Y.append(file.split(".")[0].split("_")[0]  

It seems obvious to me that the problem is with how many big pictures I'm trying to process. Nonetheless, I'm wondering if there's anything I can do to avoid turning my pictures into grayscale/downsizing them, or if there's just a memory leak or problem with my code. If it helps, I was looking into using Global Pooling to handle variable image dimensions as this medium post and a stack exchange post suggest. If it turns out that scaling wouldn't change the resolution after global pooling, please let me know because I have no idea.

Attempted solutions

I heard that going from 32bit ->64 bit would help, but I'm not sure what that looks like for what I'm doing. I'm running this in a Jupyter notebook in Pycharm, and I checked that Pycharm is a 64 bit executable.

Some extra questions I have for comprehension:

1. The error message seems to occur on the Java Virtual Machine behind Pycharm or something like that because appveyor is not a user on my PC. That, or it's opencv related. Would running this without Pycharm maybe help?
2. Is this memory issue a RAM issue or a storage issue? I find it strange that the pictures themselves take up only ~22 GB of storage, but 6% of them trip up this error.

https://stackoverflow.com/questions/65546532/error-4insufficient-memory-failed-to-allocate-36578304-bytes-in-function-c January 03, 2021 at 12:05PM