Fork me on GitHub
#data-science
<
2023-01-30
>
Carlo19:01:02

Another question on the clojure ml stack. I'm currently using pytorch for the training part of my models (in a bespoke docker container), but I'd like to try my hand with libpython . I also love clerk a lot. So, question, is there someone that has a setup in which a clerk notebook uses libpython , ideally connecting to a docker container for the python dependencies?

Carsten Behring20:01:04

I do this regularly and it does work. You have there several "moving pieces" and Docker is one of them. "development in Docker has its own challanges", you need to be aware of. (but they are independent from Clojure) You need to get the following things right: 1. mount your "working directory" into the Docker and solve the permission issue 2. setup clojure, python + pytorch in the container 3. make the Docker command start and expose a nrepl (configure host address to be 0.0.0.0 and a fixed port you can forward + add cider-nrepl middleware) 4. Run your editor on the host and "connect" to the exposed nrepl port of the container 5. Clerk will start its web server inside Docker, so you need to expose this port as well This is "more complex", then "clojreu-jack-in" specially in the beginning. Using VsCode with the "devcontainer" feature does part of this automatically

Carsten Behring20:01:05

Alternatively you can install all development tools in the container, as long as they are text based. (X11 forwarding does work as well, but need to be configured specifically) This makes certain things from my list not needed , but is not the "Docker way".

Carlo21:01:19

Thanks for the detailed answer @U7CAHM72M! I'll try the first method!

zane06:02:11

We’ve had some of luck using https://nixos.org/ in lieu of Docker for our libpython-clj projects, but it’s not without its own challenges. I can elaborate if you’re interested.

Carlo10:02:52

Thanks @U050CT4HR, I am very interested in that line of thought, as I already use NixOS in my machines. I decided to go with the Docker approach (which honestly served me well) because I wanted to use https://github.com/untitled-ai/jupyter_ascending to get a clerk-like experience, and I could not find a derivation that worked. Also I would be concerned on how the gpu is recognized. That said, if you have some example derivation I'd be very interested!

Carlo20:02:26

Thanks @U050CT4HR, I'll try this approach out early next week!

2