hey hey. I'm using tablecloth for some data analysis and given the sheer size of my dataset I see that keeping everything in memory at once is fairly demanding (currently "just" 16G but it's bound to be more memory hungry in the future)

is there any way I can limit or otherwise optimize memory usage?

you can lazily process the datasets if that works. Converting to the right types helps as well

some of the lazy and map/reduce (on one machine) style processing is in tech.ml.dataset