Fork me on GitHub
#babashka
<
2022-07-08
>
borkdude10:07:02

Run a babashka script from a remote location:

bb -e '(load-string (slurp ""))'

2
borkdude10:07:14

A better solution is coming soon: bb install --name tree ..

👍 2
lilactown15:07:04

is there a way to do that for a script with dependencies?

lilactown15:07:13

(maybe a stupid question)

borkdude15:07:24

@lilactown not a stupid question at all - so currently this is supported when a script uses (babashka.deps/add-deps '{:deps ..})

1
borkdude15:07:28

but for script "authors" that do not want their users to use the internet post-installation, it's better to distribute an uberjar probably and then bb install --name tree

1
borkdude15:07:48

or perhaps we can piggieback on the clojure -T stuff a bit, still need to look at this

Jeffrey Bay17:07:38

<stack trace redacted>

Jeffrey Bay17:07:54

note that even the invocation of clojure.core/requiring-resolve has been lost, and ideally, it would also include the stack trace from the code that was required, considering that's where the error is.

borkdude17:07:07

Could you maybe link to a gist instead of posting a long stacktrace in this channel?

Jeffrey Bay17:07:33

i'm working on creating a small test case but i was hoping there was a pat answer already 😆

Jeffrey Bay17:07:53

is this what you meant by "gist"?

borkdude17:07:04

also fine :)

borkdude17:07:48

so what you are seeing here is the stack trace from the interpreter. what do you expect to see?

borkdude17:07:52

and in which situation?

Jeffrey Bay17:07:30

i'm working on creating an isolated demonstration case

Jeffrey Bay17:07:10

i'm invoking clojure-core/requiring-resolve, and the code that is thus required throws an exception. I need to be able to get the info about what is happening inside the required code, but instead, the interpreter is eating the stacktrace and only giving the underlying exception message, and nothing about the actual trace.

Jeffrey Bay18:07:04

It's possible that in the process of creating the isolated test case, i may also show a problem with our exception handling, but my visual inspections haven't shown that. isolating the test case from the rest of our code will take a bit.

borkdude18:07:35

So, bb vs clojure:

$ clojure -M -e '(try (requiring-resolve (quote foo/bar)) (catch Exception e [(ex-message e) (type e) (ex-data e)]))'
["Could not locate foo__init.class, foo.clj or foo.cljc on classpath." java.io.FileNotFoundException nil]

$ bb -e '(try (requiring-resolve (quote foo/bar)) (catch Exception e [(ex-message e) (type e) (ex-data e)]))'
["Could not find namespace: foo." java.lang.Exception nil]

Jeffrey Bay18:07:56

i'm getting the exception message but not the trace - the trace data shows the interpreter invocation up to, i assume, the point in the interpreter where it caught the exception

borkdude18:07:54

that's correct

Jeffrey Bay18:07:20

that's not helpful though

borkdude18:07:39

what would you expect?

Jeffrey Bay18:07:16

the exception trace to have the path the code took to the exception in the required code

Jeffrey Bay18:07:57

not just the trace to this point:

at java.lang.reflect.Constructor.newInstance (Constructor.java:490)
    sci.impl.Reflector.invokeConstructor (Reflector.java:310)
    sci.impl.interop$invoke_constructor.invokeStatic (interop.cljc:58)
which is in bb internals and has no reference to code in my system

Jeffrey Bay18:07:14

there's no reference to code in on my machine anywhere in the trace

borkdude18:07:18

this is available, but only when you catch the exception that the interpreter threw, but I'm not sure how to do this within the interpreter, since you cannot change Java stack traces and add random information to that

borkdude18:07:40

if you have any ideas how to improve this without sacrificing performance, please share them

borkdude18:07:28

e.g. when you invoke bb your_script.clj then you will see the stack trace printed with references to your code

borkdude18:07:55

but since you can catch any exception like FileNotFoundException you can't add arbitrary clojure data to those

Jeffrey Bay18:07:22

i don't know anything about bb internals, so I can't help with suggestions like that; i just know that I'm losing trace information that is critical to debugging

borkdude18:07:08

I agree it would be helpful to have this information, it's just not there at the moment in the trace on the exception

borkdude18:07:37

But for debugging you can just execute your script and then see the stack trace being spit out to stdout

Jeffrey Bay18:07:16

i'm not sure what you mean by "execute your script" - executing my script includes a call to requiring-resolve, which is the thing that is eating the stack trace

Jeffrey Bay18:07:53

the script involves dynamic lookup of code, in fact, intentionally trying to avoid having to interpret the entire (growing) codebase to execute very small portions of it

borkdude18:07:16

What I mean is this:

$ bb foo.clj
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Could not find namespace: dude.
Location: /private/tmp/foo.clj:4:1

----- Context ------------------------------------------------------------------
1: (defn foo []
2:   (requiring-resolve 'dude/bar))
3:
4: (foo)
   ^--- Could not find namespace: dude.

----- Stack trace --------------------------------------------------------------
user/foo - /private/tmp/foo.clj:1:1
user     - /private/tmp/foo.clj:4:1

borkdude18:07:03

Yeah, that's a good approach.

Jeffrey Bay18:07:22

the error you just posted is not coming from the required code; the require itself is failing

borkdude18:07:52

ah, so the error is thrown when the required code is loaded - not because it's unavailable?

Jeffrey Bay18:07:00

there's no hidden trace information, the error is explicit - it can't find the namespace

Jeffrey Bay18:07:21

the error is thrown when the loaded symbol is run with arguments taken from the command line

Jeffrey Bay18:07:41

(the last sentence is for my situation, the prior for the example you posted)

borkdude18:07:04

I'm not sure I understand your problem. Are you trying to debug a problem, or are you just asserting that debugging is hard due to the missing stack trace?

Jeffrey Bay18:07:49

I'm trying to figure out why debugging a problem is hard; in parallel, i'm trying to construct a test case that i can show you because trying to talk about code without code to look at is hard

Jeffrey Bay18:07:14

currently the test case is still behaving differently than the real use case

👍 1
borkdude18:07:22

In the first case, I can help you debug it if you make a repro (which you are doing). For the second, there might be a solution, but not this week, more longer term, perhaps.

Jeffrey Bay18:07:06

it's possible there's an error in our framework code that handles the dynamic code loading, exceptions and logging; it's possible there's not a solution; you might be able to suggest a better framework pattern as well.

👍 1
Jeffrey Bay18:07:19

okay, i have a tarball - where do you want it?

Jeffrey Bay18:07:32

it's only 2k, i think here should be ok

Jeffrey Bay19:07:01

it's always harder than i expect to isolate a test case. all code is tangled with its web of dependencies 🙂

Jeffrey Bay19:07:21

there's a bit of extra code in here, but hopefully not too much for you to figure out what you're looking at.

borkdude19:07:12

ok. and what should I do once I'm in that project?

Jeffrey Bay19:07:22

there's a readme.txt

Jeffrey Bay19:07:42

bb -f main.clj open wrong

borkdude19:07:57

ok, so, as already stated, there is no useful info for you in the stacktrace, so printing that won't be helpful to the user (right now). Removing that try/catch will result into more helpful info:

$ bb -f main.clj open wrong
running (open wrong)
----- Error --------------------------------------------------------------------
Type:     java.lang.IllegalArgumentException
Message:  No matching clause: wrong
Location: /Users/borkdude/Downloads/trace-fail/main.clj:11:5

----- Context ------------------------------------------------------------------
 7:
 8: (let [[command & args] *command-line-args*]
 9:   (println "running" *command-line-args*)
10:   (try
11:     (resolve-and-run command args)
        ^--- No matching clause: wrong
12:     #_(catch Exception e
13:       (clojure.stacktrace/print-stack-trace e)
14:       (println "\n\nfailed (see trace before help):\n" (str (.getName (.getClass e)) ":") (ex-message e)))))

----- Stack trace --------------------------------------------------------------
main/resolve-and-run - /Users/borkdude/Downloads/trace-fail/main.clj:4:1
main                 - /Users/borkdude/Downloads/trace-fail/main.clj:11:5 
My advice would be to make the scope of your try/catch smaller, e.g. add a fallback to your case statement which prints that it's an unsupported option for example.

borkdude19:07:44

It may not be the answer you want to hear, but that's all I can do at the moment

borkdude19:07:04

What I could maybe support short term is an uncaught exception handler in which you get to see the babashka stacktrace, but I'm not sure why that would be more useful than what bb prints by default

borkdude19:07:27

Your command line pattern looks like it could potentially benefit from babashka cli: https://github.com/babashka/cli It has support for subcommands, but it's a bit new and there may be changes along the road

Jeffrey Bay19:07:15

yeah, that's something that didn't exist when we first started this (like 2 months ago, lol, things are evolving fast for both bb and for us 😆 )

Jeffrey Bay19:07:40

it definitely could and if i were doing it over again i'd totally support it

Jeffrey Bay19:07:07

as it is, i want to try cli out on something smaller before trying to wade into back propagating it to our existing structure

borkdude19:07:33

sure, makes sense, just thought I'd mention it in case you didn't know about it. in the future it's going to be a built-in, but not before it's "ready"

Jeffrey Bay19:07:01

i'm curious - why doesn't the interpreter have a stack trace in this situation, just like it does when it is running code that it required via the namespace require?

borkdude19:07:57

The interpreter does have a stacktrace, it's just not part of the host exception

borkdude19:07:19

and only available on calls to the interpreter, not from the inside

borkdude19:07:46

this is something I want to figure out, but without sacrificing performance

borkdude19:07:06

One problem e.g. is when you catch a thrown java.lang.FileNotExistsException , there is no place to put the interpreter stacktrace (which is different from the host stack trace)

borkdude19:07:20

one possible API could be:

(catch java.lang.WhateverException e (babashka.core/stacktrace))
but then it'd be already too late to get the stacktrace

borkdude19:07:51

so it's a long term thing I'm thinking about how to solve, not so easy

Jeffrey Bay19:07:31

it seems like the ideal would be for the exception to have a cause on it, one way or the other

borkdude19:07:30

so if you caught a FileNotFoundException, and the cause would have an interpreter stacktrace thing, wouldn't that be a bit reversed?

borkdude19:07:01

but surely it could be a loophole

Jeffrey Bay20:07:01

i'd take it reversed if i had to 😆 although i would think you could put them in whichever configuration made the most sense, as long as the exception thrown to the host system worked for dropping a good exception into the main output

Jeffrey Bay20:07:03

you can either find the bottom of the exception you have and call setCause on it (where cause is nil, in other words), or you can create a new exception to throw and include the one you have as the cause on it...

borkdude20:07:02

that's a good point, but if the user says (catch FileNotFoundException ...) they want of course not some wrapped thing right?

Jeffrey Bay20:07:29

hmm - that's a tough one. you'd wind up, i guess, with different exceptions in the requiring-resolve case vs the straight execution path... which is gross

Jeffrey Bay20:07:53

i wonder if you could reify a new exception of the same type but with a new function like "get-other-trace" or something

borkdude20:07:07

that's only possible if the exception type is an interface or abstract class

borkdude20:07:51

but if the users says "catch Exception" then we could return some wrapped thing with a cause, maybe

Jeffrey Bay20:07:14

could you make it an interpreter argument whether to wrap it or not?

Jeffrey Bay20:07:17

or a binding?

borkdude21:07:33

I'm going to track the issue here: https://github.com/babashka/sci/issues/774 I think there are some possibilities. You can share your interest there with a "thumbs up" or a comment. @U038RGYDGUR also asked about something similar a while ago. It'd be nice if we could support this in a good way.

borkdude09:07:44

So I've got this working now:

$ clojure -M:babashka/dev -e "(require '[sci.core :as sci]) (defn foo [] (/ 1 0)) (defn bar [] (try (foo) (catch Exception e (run! prn (sci/format-stacktrace (sci/stacktrace e)))))) (bar)"
"clojure.core// - <built-in>"
"user/foo       - <expr>:1:44"
"user/foo       - <expr>:1:31"
"user           - <expr>:1:71"
Here, I expose a part of sci.core itself in bb and also let the exception hang on to the stack trace info. I just haven't figured out yet the part where the user catches a certain type of Exception. Maybe the contract could be, when the user catches the most general type of exception, then it will contain this info, otherwise it will be the vanilla type or so.

borkdude09:07:55

I may have found a good way: when you just say (catch Exception ...) you get the "wrapped exception. And then your code would become:

(catch Exception e
      (->> e stacktrace format-stacktrace (run! prn))
      (println "\n\nfailed (see trace before help):\n" (str (.getName (.getClass e)) ":") (ex-message e)))))
which then gives (with a locally compiled bb):
$ bb -f main.clj
running nil
"main/resolve-and-run - /Users/borkdude/Downloads/trace-fail/main.clj:4:1"
"main                 - /Users/borkdude/Downloads/trace-fail/main.clj:11:5"


failed (see trace before help):
 clojure.lang.ExceptionInfo: no conversion to symbol

borkdude09:07:32

I get one failing case with babashka's CI tests, though, for example with clj-http-lite:

FAIL in (exception-test) (/Users/borkdude/dev/babashka/test-resources/lib_tests/clj_http/lite/client_test.clj:)
expected: (:headers (ex-data e))

borkdude09:07:13

So it might be better to catch a very specific type of exception when you want the interpreter trace. E.g. (catch sci.lang.Exception) to not break libraries (and also for perf reasons probably)

borkdude09:07:19

@U038RGYDGUR Would that seem good to you too?

borkdude09:07:44

(grump: this would require me to compile a specific exception type, but that's a small price to pay)

borkdude10:07:13

Perhaps going with (catch ^::sci/preserve-stacktrace Exception) would be a better choice for not breaking compatibility

borkdude11:07:07

So now your code is going to look like:

(ns main
  (:require [sci.core :as sci]))

(defn resolve-and-run [command args]
  (let [command-fn (requiring-resolve (symbol (str "foo." command) command))]
    (apply command-fn args)))

(let [[command & args] *command-line-args*]
  (println "running" *command-line-args*)
  (try
    (resolve-and-run command args)
    (catch ^:sci/error Exception e
      (->> e sci/stacktrace sci/format-stacktrace (run! prn))
      (println "\n\nfailed (see trace before help):\n" (->  e ex-cause class .getName (str ":")) (ex-message e)))))
and the output:
$ bb main.clj
running nil
"main/resolve-and-run - /Users/borkdude/Downloads/trace-fail/main.clj:4:1"
"main                 - /Users/borkdude/Downloads/trace-fail/main.clj:11:5"


failed (see trace before help):
 java.lang.IllegalArgumentException: no conversion to symbol

borkdude11:07:09

If you let me know your OS @U03A0EGF82E, I can link you to a binary to try out locally

Jeffrey Bay13:07:21

thanks so much for working on this - the path you are on seems great to me.

borkdude13:07:31

@U03A0EGF82E I merged it to master already. You can install a dev version from : https://github.com/babashka/babashka-dev-builds