Clojurians Log v2

My gut instinct would be that Python interop cannot really work with AOT, given how dynamic Python itself is, and how reliant it is on env vars. Do you actually need AOT or is that just an incidental artefact of whatever template you started the project with? Taking a step back, have you activated your Python venv in the shell where you started the REPL? I would assume that'd be the easiest way to proceed: first, activate Python's venv, then load Python from Clojure. The JVM cannot change its own env vars and the Python venv approach is based almost entirely on modifying env vars, so that seems like the order that would work the most seamlessly. If your venv's Python is the only one the JVM can see on the PATH, there should be little room for confusion. And if you were able to create an AOT-compiled JAR, I would still expect users to need to run it in a properly-initialized and activated Python venv.

(Unless I'm completely off base and libpython does compilation magic to embed a Python interpreter and all of the required dependencies within the JAR.)

> with the right arguments And what are they?

(ns krew-agent-backend.krew-agent-backend
       (:require [libpython-clj2.python :refer [py. py.. py.-] :as py]
                 [libpython-clj2.require :refer [require-python]]
                 [ :as io])
       (:gen-class))
    
     (def workdir (System/getProperty "user.dir"))
    
     (py/initialize! {:python-executable (str workdir "/.venv/bin/python")})

If I print my py/initialize! call, it's already initialized. Also, if I use py/run-simple-string to import sys; print(sys.path), it's initialized to my system python

> If I print my py/initialize! call, it's already initialized. What do you mean by "print py/initialize! call"? Do you have logging configured? Just to confirm, the initialize! function logs the Python std lib that it's loading, so you can double check. It also prints out all the libraries that will be checked - you can confirm that your venv is at least there.

If I wrap with (println )

I get the symbol :already-initialized as a return value

And the libraries printed,e ven without my call to initialize, are all system libraries

don't see my venv

here's something interesting

I stripped my ns form down to just this and now it's working:

(ns krew-agent-backend.krew-agent-backend
  (:require [libpython-clj2.python :as py]
            [ :as io])
  (:gen-class))

> I get the symbol :already-initialized as a return value Ah, well, then something else has already called initialize!. Maybe with wrong arguments. If you were testing it in a REPL session, then yeah, that will happen if you re-evaluate (py/initialize! ...), even if the arguments change. Also, beware of side-effects happening when a namespace is being loaded. You have (py/initialize! ...) at the top level of your namespace, and that namespace also has (:gen-class), so you're probably using AOT compilation with it. libpython will be initialized during compilation, and who knows what issues it could result in.

Yeah, not great... libpython-clj2.require has a bunch of side-effects at the top level. Apart from that, it requires libpython-clj2.metadata, which calls (py/initialize!) at the top level. So either something really bad is going on, or you're expected to require both ...require and ...metadata namespaces dynamically - outside of the (ns ...) form, via a (require ...) that's not at the top level and that doesn't happen during compilation.

This is what I as just digging into! Because if I require libpython-clj2.require then I get my environment initialized even though I didn't request it

How can I verify that I am not supposed to do what you say, which is require those namespaces dynamically?

The docstring of the libpython-clj2.metadata namespace confirms it:

"Namespace to create metadata from python objects.   This namespace requires
  python helper fns to work correctly and thus cannot be used until after python
  has been initialized."

So now for the nooby clojure question, can I just have a function that calls require for those dynamic requires, and call that in my namespace?

And that would add those symbols to the namespace symbol table?

Agreed on the throw, I would expect that rather than initializing with my system python and me having no idea why

The most direct way to achieve that would be to use requiring-resolve in each function that needs something from one of those namespaces. However, it seems that at least libpython-clj2.require/require-python doesn't return the loaded things themselves and instead modifies the current namespace. That means that you can do stuff like (require-python 'math) math/pi in a REPL but you cannot do them in a plain file that will get compiled because the reader will have no idea about the math alias/namespace. Not sure what the correct way to proceed here is. But what should work is if you have an entry point to your program that requires the absolute minimum to call (py/initialize!), calls it, and then dynamically loads another namespace from your program that uses stuff like (require-python ...) at the top level. Just make sure that the dynamically loaded namespace and any other namespaces that use require-python or anything like that at the top level don't get AOT'ed.

Would using the PYTHON_HOME environment variable fix your problem? https://clj-python.github.io/libpython-clj/libpython-clj2.python.html#var-initialize.21 > :python-home - Python home directory. The system first uses this variable, then the environment variable PYTHON_HOME, and finally information returned from python system info. Another option is to make sure python gets initialized at startup. For jars, you can initialize at the start of main. For dev, you can use middleware.

I imagine PYTHON_HOME could also be problematic with AOT.

For AOT, it doesn't seem like PYTHON_HOME is a problem, it's the top level forms with side effects.

Personally, I would probably avoid those namespaces with side effects altogether. I don't think you need them and there are other approaches that are more convenient.

Yes, but if there are no top-level forms with side-effects then setting the python executable explicitly will also work just fine.

> I don't think you need them and there are other approaches that are more convenient. What would you use instead of require-python?

Looking at my projects, I haven't written any code that was AOT'd so I did just use require-python. If I did have that use case, I would probably write my own macro.

Or maybe there's already something there. I haven't looked since I haven't had the need.

hmmm, looking at various libpython projects isn't encouraging. Many of them assume an environment and couldn't be AOT'd.

hmmm, it seems like this is supposed to be addressed by python.edn. • https://clj-python.github.io/libpython-clj/libpython-clj2.python.html#var-initialize.21 • https://github.com/clj-python/libpython-clj/pull/226 Maybe there's some special handling for AOT that I'm not aware of, but I don't see how that would work.

Why do you think python.edn has anything to do with AOT?

I don't.

I think it's supposed to address the problem of using require-python with a python environment that's potentially not the system environment. I would guess that it doesn't work under AOT (maybe it does, but I'm doubtful).

Ah, it might simply help with avoiding dynamic requires. That's something, right. @windlejacob12 Using python.edn is at least easier. But won't help with AOT, yes. But maybe you don't need it at all?..