I think it is important in this space, to think "polyglot" from the beginning, so accepting that Clojure alone will not do it. This means (for me) as well to accept Docker (or any other container technology) as your "development environment". At least inside your Docker environment, you have full control of what is there, and can move to more powerful machines when needed without larger headaches. If you have that, then it is not too difficult to work in both, Clojure and Python in collaboration, either via libpython-clj or via shell scripting. Depending what precisely you want to do, a tool such as http://dvc.org might then help.
@jjttjj > big pretrained models are just data and could maybe be converted to something usable You probably want https://onnx.ai/. For example https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html. Once you have it in onnx, you could pull it into MXNET, for instance, and leave python