Reading multiple raster/netCDF data in R slows down after a few iterations

I have 9000+ netCDF files I am extracting data from at points. The bottleneck is reading in the data to raster::bricks (or stacks).

The netCDF have only 1 time step (single day), but have 72 layers for elevation and 84 variables. I am extracting data for each day, for each variable, and the 72nd layer.

My approach is to use raster::brick followed by extract(). I make use of lapply (or purrr::map, no big difference here). My problem is that after a few iterations the reading in the data gets much much slower. Initially, one variable to read in takes ~0.25 seconds, but then it slows down to ~6 seconds. The raster bricks themselves are not being saved at each iteration - there is nothing else that SHOULD be growing, so I'm at a loss. I'm on a Mac, R402. TIA!

https://portal.nccs.nasa.gov/datashare/merra2_gmi/

Example code: I simply put the print(x) to keep track, but would eventually remove once the issue is resolved.

z <- lapply(1:length(varnames),       function(x) {            print(x)           raster::brick(MERRA2.GMI.filenames.read[i],varname = varnames[x])[[72]]            %>% crop(US.extent)       }  )   %>% raster::brick()  

https://stackoverflow.com/questions/66891447/reading-multiple-raster-netcdf-data-in-r-slows-down-after-a-few-iterations April 01, 2021 at 12:13AM