Fork me on GitHub
#babashka
<
2021-12-22
>
sheluchin14:12:28

I would like to use babashka for some management tasks in Fulcro. Things like effecting mounted states, reading config state, reading application state, etc.. All things that area easily accomplished through my REPL connected editor once the application is running. Is there some way I can tap into same nrepl with babashka tasks? https://clojurians.slack.com/archives/C68M60S4F/p1640028952193400

borkdude14:12:33

@alex.sheluchin You can interact with nREPL via bencode, which is the format that the nrepl protocol uses: https://book.babashka.org/#_interacting_with_an_nrepl_server

sheluchin15:12:37

Sweet! I saw that part but didn't think that would be all it would take.. for some reason. Got a little PoC task working. Thank you @U04V15CAJ.

👍 1
sheluchin16:12:32

@U04V15CAJ I find I'm getting this error intermittently on some such tasks, and constantly on other ones. For the intermittent ones, simply re-trying the task without any modifications usually makes it run successfully:

$ bb --debug mount:restart-app

 ;; extra-paths
 ;; extra-deps

(ns user-eed26d10-2b4e-4062-88c1-e7ca096fae50 (:require [tasks] [taoensso.timbre :as log]))
(require '[babashka.tasks])
(when-not (resolve 'clojure)
  ;; we don't use refer so users can override this
  (intern *ns* 'clojure babashka.tasks/clojure))

(when-not (resolve 'shell)
  (intern *ns* 'shell babashka.tasks/shell))

(when-not (resolve 'current-task)
  (intern *ns* 'current-task babashka.tasks/current-task))

(when-not (resolve 'run)
  (intern *ns* 'run babashka.tasks/run))

nil
(def mount:restart-app (binding [
  babashka.tasks/*task* '{:name mount:restart-app, :task (println (tasks/restart-app))}]
  nil
(println (tasks/restart-app)))) mount:restart-app


----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: 21:24

----- Exception ----------------------------------------------------------------
clojure.lang.ExceptionInfo: null
{:type :sci/error, :line 21, :column 24, :message nil, :sci.impl/callstack #object[clojure.lang.Volatile 0x41d7a562 {:status :ready, :val ()}], :
file nil, :locals {}}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:85)
    sci.impl.evaluator$eval.invokeStatic (evaluator.cljc:338)
    sci.impl.evaluator$eval.invoke (evaluator.cljc:335)
    sci.impl.evaluator$eval_def.invokeStatic (evaluator.cljc:102)
    sci.impl.analyzer$expand_def$fn__8625.invoke (analyzer.cljc:430)
    sci.impl.evaluator$eval.invokeStatic (evaluator.cljc:341)
    sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:44)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:59)
    sci.core$eval_string_STAR_.invokeStatic (core.cljc:236)
    babashka.main$exec$fn__34865$fn__34866.invoke (main.clj:847)
    babashka.main$exec$fn__34865.invoke (main.clj:847)
    babashka.main$exec.invokeStatic (main.clj:837)
    babashka.main$main.invokeStatic (main.clj:917)
    babashka.main$main.doInvoke (main.clj:904)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    babashka.main$_main.invokeStatic (main.clj:951)
    babashka.main$_main.doInvoke (main.clj:943)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    babashka.main.main (:-1)
Do you happen to know what might be the issue here?

borkdude16:12:16

Let me check

borkdude16:12:55

can you also show me the task?

borkdude16:12:37

if you can make a repro, I can check locally

borkdude16:12:52

when I paste that code into a .clj file and prepend it with:

(ns tasks)

(defn restart-app []
  (prn :hello))
it works fine for me

borkdude16:12:30

I suspect the error is happening in the actual restart-app code

borkdude16:12:33

which I don't see here

sheluchin18:12:53

@U04V15CAJ so here is another example, roughly the same, but a bit simpler than restart:

$ bb --debug mount:stop-app

 ;; extra-paths
 ;; extra-deps

(ns user-6cf46754-4d17-4adf-bc46-f92b1d8b43d7 (:require [tasks] [taoensso.timbre :as log]))
(require '[babashka.tasks])
(when-not (resolve 'clojure)
  ;; we don't use refer so users can override this
  (intern *ns* 'clojure babashka.tasks/clojure))

(when-not (resolve 'shell)
  (intern *ns* 'shell babashka.tasks/shell))

(when-not (resolve 'current-task)
  (intern *ns* 'current-task babashka.tasks/current-task))

(when-not (resolve 'run)
  (intern *ns* 'run babashka.tasks/run))

nil
(def mount:stop-app (binding [
  babashka.tasks/*task* '{:name mount:stop-app, :task (do (log/info "Stopping application states...") (println (tasks/stop-app)))}]
  nil
(do (log/info "Stopping application states...") (println (tasks/stop-app))))) mount:stop-app


2021-12-23T18:44:25.423Z xps INFO [user-6cf46754-4d17-4adf-bc46-f92b1d8b43d7:24] - Stopping application states...
----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: 21:21

----- Exception ----------------------------------------------------------------
clojure.lang.ExceptionInfo: null
{:type :sci/error, :line 21, :column 21, :message nil, :sci.impl/callstack #object[clojure.lang.Volatile 0x556e6925 {:status :ready, :val ()}], :
file nil, :locals {}}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:85)
    sci.impl.evaluator$eval.invokeStatic (evaluator.cljc:338)
    sci.impl.evaluator$eval.invoke (evaluator.cljc:335)
    sci.impl.evaluator$eval_def.invokeStatic (evaluator.cljc:102)
    sci.impl.analyzer$expand_def$fn__8625.invoke (analyzer.cljc:430)
    sci.impl.evaluator$eval.invokeStatic (evaluator.cljc:341)
    sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:44)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:59)
    sci.core$eval_string_STAR_.invokeStatic (core.cljc:236)
    babashka.main$exec$fn__34865$fn__34866.invoke (main.clj:847)
    babashka.main$exec$fn__34865.invoke (main.clj:847)
    babashka.main$exec.invokeStatic (main.clj:837)
    babashka.main$main.invokeStatic (main.clj:917)
    babashka.main$main.doInvoke (main.clj:904)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    babashka.main$_main.invokeStatic (main.clj:951)
    babashka.main$_main.doInvoke (main.clj:943)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    babashka.main.main (:-1)
The nrepl eval is implemented as such:
(defn stop-app []
  (nrepl-eval 9000 "(development/stop)"))
And the actual function being called is:
(defn stop  []
  (mount/stop))
And I've compared states before/after with (mount/runnning-states) and confirmed that it does indeed produce the desired effect of stopping the states.

borkdude19:12:28

Can you make an actual repository for me so I can test this? if I don't have the whole context, I cannot reproduce the issue

borkdude19:12:03

perhaps something in nrepl-eval is going wrong?

borkdude19:12:14

Perhaps you can wrap your task in a try/catch and see if that helps to locate the error

sheluchin19:12:00

I could create a repo for it. My project is a fork of https://github.com/fulcrologic/fulcro-rad-demo/blob/develop/src/xtdb/development.clj. I would just first check that I can reproduce the issue on a clean fork before showing it to you. I added some prints and narrowed it down to the (String. bytes)/`bytes` not being set:

(defn nrepl-eval 
  "Execute some expression string in the nREPL

  Used for calling eval in the same context as the Fulcro application"
  [port expr]
  (println "here")
  (let [s (java.net.Socket. "localhost" port)
        _ (println "here1")
        out (.getOutputStream s)
        _ (println "here2")
        in (java.io.PushbackInputStream. (.getInputStream s))
        _ (println "here3")
        _ (b/write-bencode out {"op" "eval" "code" expr})
        _ (println "here4")
        bytes (get (b/read-bencode in) "value")
        _ (println "here5")]
    (try
      (String. bytes)
      (catch Exception e
        (println "type" (type bytes))
        (println "bytes" bytes)
        (println "in" in)
        ; (println (ex-data e))))))
        (.printStackTrace e)))))
here
here1
here2
here3
here4
here5
type nil
bytes nil
in #object[java.io.PushbackInputStream 0x6b180f95 [email protected]]
java.lang.NullPointerException
        at java.lang.String.<init>(String.java:614)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at sci.impl.Reflector.invokeConstructor(Reflector.java:310)
        at sci.impl.interop$invoke_constructor.invokeStatic(interop.cljc:65)
        at sci.impl.analyzer$analyze_new$fn__8755.invoke(analyzer.cljc:791)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.evaluator$eval.invoke(evaluator.cljc:335)
        at sci.impl.evaluator$eval_try.invokeStatic(evaluator.cljc:150)
        at sci.impl.analyzer$analyze_try$fn__8693.invoke(analyzer.cljc:616)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.evaluator$eval.invoke(evaluator.cljc:335)
        at sci.impl.evaluator$eval_let.invokeStatic(evaluator.cljc:76)
        at sci.impl.analyzer$expand_let_STAR_$fn__8614.invoke(analyzer.cljc:394)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.analyzer$return_do$fn__8007.invoke(analyzer.cljc:115)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.fns$fun$arity_2__7310.invoke(fns.cljc:149)
        at sci.impl.vars.SciVar.invoke(vars.cljc:325)
        at sci.impl.analyzer$return_call$fn__8951.invoke(analyzer.cljc:1040)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.fns$fun$arity_0__7298.invoke(fns.cljc:140)
        at sci.impl.vars.SciVar.invoke(vars.cljc:321)
        at sci.impl.analyzer$return_call$fn__8943.invoke(analyzer.cljc:1040)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.analyzer$return_call$fn__8947.invoke(analyzer.cljc:1040)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.analyzer$return_do$fn__8007.invoke(analyzer.cljc:115)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.analyzer$return_do$fn__8007.invoke(analyzer.cljc:115)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.evaluator$eval.invoke(evaluator.cljc:335)
        at sci.impl.evaluator$eval_try.invokeStatic(evaluator.cljc:150)
        at sci.impl.analyzer$analyze_try$fn__8693.invoke(analyzer.cljc:616)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.evaluator$eval.invoke(evaluator.cljc:335)
        at sci.impl.evaluator$eval_let.invokeStatic(evaluator.cljc:76)
        at sci.impl.analyzer$expand_let_STAR_$fn__8614.invoke(analyzer.cljc:394)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.evaluator$eval.invoke(evaluator.cljc:335)
        at sci.impl.evaluator$eval_def.invokeStatic(evaluator.cljc:102)
        at sci.impl.analyzer$expand_def$fn__8625.invoke(analyzer.cljc:430)
        at sci.impl.evaluator$eval.invokeStatic(evaluator.cljc:341)
        at sci.impl.interpreter$eval_form.invokeStatic(interpreter.cljc:44)
        at sci.impl.interpreter$eval_string_STAR_.invokeStatic(interpreter.cljc:59)
        at sci.core$eval_string_STAR_.invokeStatic(core.cljc:236)
        at babashka.main$exec$fn__34865$fn__34866.invoke(main.clj:847)
        at babashka.main$exec$fn__34865.invoke(main.clj:847)
        at babashka.main$exec.invokeStatic(main.clj:837)
        at babashka.main$main.invokeStatic(main.clj:917)
        at babashka.main$main.doInvoke(main.clj:904)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at babashka.main$_main.invokeStatic(main.clj:951)
        at babashka.main$_main.doInvoke(main.clj:943)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at babashka.main.main(Unknown Source)

sheluchin19:12:19

(b/read-bencode in) has no value when this fails:

{"out" #object["[B" 0x7c0eef04 "[[email protected]"], "session" #object["[B" 0x7a099599 "[[email protected]"]}
But in a working execution it does:
{"ns" #object["[B" 0x4231a6fd "[[email protected]"], "session" #object["[B" 0x62972195 "[[email protected]"], "value" #object["[B" 0x23291ff5 "[[email protected]"]}
:thinking_face: Then if I (println (String. (get (b/read-bencode in) "out"))), I get the first line of the usual stdout I see when evaling that form in my editor:
I 2021-12-23T19:38:40.348Z       com.zaxxer.hikari.HikariDataSource:-350 - HikariPool-76 - Shutdown initiated...
But I can't quite make out why this happens from the bencode docs/tests.

borkdude19:12:53

yeah, "out" can happen when something is printed

borkdude19:12:07

you should read messages until you receive a "done" for the id

borkdude19:12:13

you can take a look at the nrepl docs and the tests for babashka.nrepl to create a better version of what's in the book https://github.com/babashka/babashka.nrepl/blob/master/test/babashka/nrepl/server_test.clj

sheluchin19:12:10

I see. Looks like this should do the trick: https://github.com/babashka/babashka.nrepl/blob/master/test/babashka/nrepl/server_test.clj#L266-L270 Thanks for the pointer. I'll see about getting this working and will post a PR to the book.

sheluchin20:12:05

Got it working. Not terribly pretty, but it works. Good enough for me to return the entire output, but I guess it can be problematic for very large outputs, or just not necessary in some cases.

(defn bytes->str [x]
  (if (bytes? x) (String. (bytes x))
      (str x)))

(defn read-msg [msg]
  (let [res (zipmap (map keyword (keys msg))
                    (map #(if (bytes? %)
                            (String. (bytes %))
                            %)
                         (vals msg)))
        res (if-let [status (:status res)]
              (assoc res :status (mapv bytes->str status))
              res)
        res (if-let [status (:sessions res)]
              (assoc res :sessions (mapv bytes->str status))
              res)]
    res))

(defn read-reply [in session id]
  (loop []
    (let [msg (read-msg (b/read-bencode in))]
      (if (and (= (:session msg) session)
               (= (:id msg) id))
        msg
        (recur)))))

(defn nrepl-eval 
  "Execute some expression string in the nREPL

  Used for calling eval in the same context as the Fulcro application"
  [port expr]
  (let [s (java.net.Socket. "localhost" port)
        out (.getOutputStream s)
        in (java.io.PushbackInputStream. (.getInputStream s))
        ;; totally arbitrary but I think it's sufficient
        id (rand-int 1000)
        _ (b/write-bencode out {"op" "eval" "code" expr "id" id})
        {:keys [session value]} (read-msg (b/read-bencode in))]
    (if value
      value
      (loop [output ""]
        (let [{:keys [status out]} (read-reply in session id)]
          (if (= status ["done"])
            output
            (recur (str output out))))))))

borkdude21:12:06

for id you could use (str (java.util.UUID/randomUUID))

👍 1
borkdude21:12:31

I think you should always start with a loop, read and print "out" and keep the value as a variable in the loop and and the loop on done, then return the value

borkdude21:12:39

as you can get out before or after the value

borkdude21:12:46

but this is great

sheluchin21:12:20

You think it should always print out? It might not be desirable. The way it is now, the invoking task can decide whether to print it or not. Maybe output printing could be parameterized while the value should always be returned. Then again, being able to capture out and do something with it instead of just printing might be handy too :man-shrugging:

borkdude21:12:19

I think you can just print to *out* and the user can re-bind that when needed

borkdude21:12:35

Perhaps see how nrepl.core does this, I expect they do the same

sheluchin00:12:33

I didn't find the relevant snippets in nrepl.core but I did try re-binding it in a task and it works fine. So far, looks something like this:

(defn nrepl-eval 
  "Execute some expression string in the nREPL

  Used for calling eval in the same context as the Fulcro application"
  [port expr]
  (let [s (java.net.Socket. "localhost" port)
        out (.getOutputStream s)
        in (java.io.PushbackInputStream. (.getInputStream s))
        id (str (java.util.UUID/randomUUID))
        _ (b/write-bencode out {"op" "eval" "code" expr "id" id})
        {session :session
         ret :value} (read-msg (b/read-bencode in))]
    (loop [output ""
           return ret]
      ;; TODO: :status [eval-error]} when an exception is thrown
      (let [{:keys [status out value]} (read-reply in session id)]
        (when out
          (print out))
        (if (= status ["done"])
          return
          (recur (str output out) (or value return)))))))
It's good enough for my needs for the moment. I'm not sure the best way to catch possible thrown exceptions. I guess it's more complete than the example currently in the book. If you would like I can add it, and if not that's okay too 🙂 I can see how an nREPL client might be something useful to work towards. Thanks again for the help.

borkdude08:12:02

ok, I'll take your code, modify it somewhat and will paste it back into the book

👍 1
borkdude14:12:27

I'm considering an nREPL client library for babashka at one point, that would be fun, but just evaluating an expression can be done using the above