If you convert your data to arrow files, then you can use it memory-mapped. So file stay on disk and needed parts gets swapped in by the OS. Very fast, but of course dependent on operations