Fork me on GitHub
#babashka
<
2021-08-31
>
mike_ananev10:08:03

@borkdude hi! We use babashka in CI Jenkins in a closed environment without Internet. When we run babashka task, babashka tries to download tools.deps 1.10.3.822 and fails. Is there any way to run babashka without internet ?

borkdude10:08:30

@mike1452 I think it only does this when you have :deps in your bb.edn and when you have deps then bb needs internet to download these

mike_ananev10:08:41

ah, thank you. I will check.

mike_ananev10:08:19

Yes, we have `

:deps           {cprop/cprop {:mvn/version "0.1.18"}}
Can I put cprop to classpath or .m2 to make babashka not using internet?

borkdude10:08:53

@mike1452 Perhaps you can use bb uberjar for this, but currently that doesn't really integrate with bb.edn tasks I think

borkdude10:08:09

bb uberjar will package the deps into an uberjar which you can invoke with bb foo.jar

mike_ananev10:08:42

This is CI task, uberjar is not suitable here.

mike_ananev10:08:03

cprop is used to work with env vars for tasks.

mike_ananev10:08:54

Ok, I probably should rewrite my tasks to use (System/getProperties) directly, not from cprop

borkdude10:08:23

or you could install your deps as part of your CI build

borkdude10:08:30

I assume your CI is based on some docker image right

borkdude10:08:47

then just resolve the dep using bb during the image build

borkdude10:08:56

and then it should work without internet thereafter

mike_ananev10:08:40

Yeah, I'm just experimenting right now. I will respond here when I find the solution.

borkdude10:08:26

you could just invoke bb print-deps to force downloading of deps

mike_ananev16:08:46

@borkdude we made several experiments. We completely deleted :deps section from bb.edn. It doesn't help. Babashka tries to download jar file Could not find /home/jenkins/.deps.clj/1.10.3.822/ClojureTools/clojure-tools-1.10.3.822.jar Where should we place clojure-tools-1.10.3.822.jar? What folder? Can't find group-id/artifact-id inside jar? Btw, we run bb not in a docker.

borkdude16:08:20

What version of bb are you using

borkdude16:08:53

You should be placing it in /home/jenkins/.deps.clj/1.10.3.822/ClojureTools like the error message suggests

borkdude16:08:32

it surprises me that bb tries to download if you don't use deps. are you using babashka.deps anywhere else in your scripts

borkdude16:08:26

borkdude@MBP2019 /tmp $ rm -rf ~/.deps.clj
borkdude@MBP2019 /tmp $ bb dude.clj
6
borkdude@MBP2019 /tmp $ cat dude.clj
(+ 1 2 3)

borkdude16:08:52

borkdude@MBP2019 /tmp $ echo '{}' > bb.edn
borkdude@MBP2019 /tmp $ bb dude.clj
6

borkdude16:08:12

borkdude@MBP2019 /tmp $ echo '{:deps {}}' > bb.edn
borkdude@MBP2019 /tmp $ bb dude.clj
Could not find /Users/borkdude/.deps.clj/1.10.3.933/ClojureTools/clojure-tools-1.10.3.933.jar
Attempting download from 
6

mike_ananev17:08:43

Thank you. I will continue to experiment and respond here

borkdude10:08:11

unless you're using :extra-deps during tasks ;)

mike_ananev10:08:17

We have closed Nexus repo with all deps. Can I put :mvn/repos to bb.edn to tell babashka download deps from here ?

borkdude10:08:00

but it will still try to download the tools jar the first time you're downloading deps

mike_ananev10:08:04

In babashka book I can't find how to do this.

borkdude10:08:14

but you can put it there manually as well

borkdude10:08:28

Just look at the deps.edn docs from Clojure

borkdude10:08:31

it uses the same logic

borkdude10:08:51

except it doesn't support aliases yet

borkdude10:08:57

but everything else yes

alexdavis11:08:47

Is there a way to call nbb from a js file? I’m not entirely sure how to import it because I don’t really understand what shadow has generated in nbb_core.js. I want to do something like this

import nbb from 'nbb'
const script = fs.readFileSync('./test.cljs', 'utf8')
nbb(script, 'args');

borkdude11:08:53

@alex395 That is something which I still haven't figured out yet. I think nbb should expose an API but it's not yet been implemented. If you want to help with this, I would appreciate that

borkdude11:08:20

at least make an issue about it

borkdude11:08:29

There is also an #nbb channel

alexdavis11:08:36

Ah ok thanks, I will have a look into it

borkdude11:08:43

@alex395 I don't know how normal nodejs projects do this but perhaps we can make a index.js in the repo which exports things

borkdude11:08:23

or is this the job of shadow...?

borkdude11:08:41

let's continue the chat in #nbb

celebrimbor37912:08:14

Is there something like a :bb or :sci platform tag for reader conditional expressions that would target bb/sci? I've got a script that's using data.xml, and the alias-uri function https://github.com/clojure/data.xml/blob/master/src/main/clojure/clojure/data/xml/name.cljc#L99:

#?(:clj
   (defn alias-uri
     "Define a Clojure namespace aliases for xmlns uris.
  This sets up the current namespace for reading qnames denoted with
So bb fails with:
7: (xml/alias-uri 'Atom "")
     ^--- Could not resolve symbol: xml/alias-uri
I'm not sure if there's something in that function that legitimately won't work on bb, or if it was just not around/not considered when that code was written.

celebrimbor37912:08:23

Reader conditionals are new to me, but the docs only list four possibilities: :clj :cljs :cljr :default: https://clojure.org/guides/reader_conditionals

borkdude12:08:55

I think we could expose that function/macro though

borkdude12:08:30

@horton.wh.dev The precedence for reader conditionals is, first :bb, then :clj, if there is no :bb then bb will take the :clj branch

borkdude12:08:48

Issue welcome about the alias uri part

celebrimbor37912:08:35

Thanks, so in that case since alias-uri only has a single :clj branch, wouldn't we expect bb pick it up as-is?

borkdude12:08:11

yes, but not if it's not available in bb yet.

borkdude12:08:47

$ bb -e "(clojure.data.xml/alias-uri 'foo 'bar)"
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: clojure.data.xml/alias-uri
Location: <expr>:1:2
Phase:    analysis

----- Context ------------------------------------------------------------------
1: (clojure.data.xml/alias-uri 'foo 'bar)
    ^--- Could not resolve symbol: clojure.data.xml/alias-uri
It's a function or macro that needs to be added

borkdude12:08:43

ah it's a normal function

borkdude12:08:03

but we need to override it in bb since it's creating a clojure ns and not a sci ns

borkdude12:08:14

anyway, issue welcome, I'll look at it soon (or someone else)

celebrimbor37912:08:54

Ah, should have clarified that I was trying to pull in data.xml using a bb.edn file. Did not realize it was packaged with bb itself. And this is where my understanding gets really, really fuzzy -- can any arbitrary dependency that's not already available in bb be pulled in using a bb.edn and be expected to work? Or do other libraries need to be "prepared" somehow to run on sci vs. normal JVM Clojure? I'm relatively new to Clojure in general so sorry if that's an obvious question.

borkdude12:08:51

@horton.wh.dev That's an understandable question. bb supports a (large) subset of Clojure. Some libraries work, but ones that go outside of the subset don't.

borkdude12:08:00

For convenience and performance some libraries are built-in.

borkdude12:08:35

The performance of libraries run from source aren't as good as the built-in ones

borkdude12:08:43

because the source is going through an interpreter

celebrimbor37913:08:28

That makes sense, thanks. Is the subset of non-built-in ones that do work known, or is it more of a "try and see what happens" thing? 🙂

celebrimbor37913:08:37

Excellent, thank you. And thanks for all the work on babashka itself too, it is really cool.

celebrimbor37913:08:41

I'll open an issue for the alias-uri thing in a bit.

👍 1
mkvlr14:08:31

there’s currently no way to customize the (excellent!) exception printing in babashka, correct? We’d be interested to show the report but hide the locals because they can leak sensitive information (secrets).

borkdude14:08:34

any proposal?

borkdude14:08:43

a flag, config in bb.edn?

borkdude14:08:27

there is currently no customization for it

borkdude14:08:58

but one could wonder: if the locals contain sensitive info, then anyone having access to that script can already get at that right?

borkdude14:08:21

or is it more like when you're sharing your screen, it can get awkward if someone else sees the output?

mkvlr15:08:01

yeah, in our case the user does has access to the secret (they put it there in the first place) but we don’t want to write it to the logfile where the babashka exception ends up being written to. Our use case is scripting for github and s3 bucket components on nextjournal, so it’s great to see more info when things don’t work but we don’t want the locals there. A cli flag seems like overkill though since our use case is quite unusual.

borkdude15:08:38

CircleCI has some feature to print ****** instead of secrets... perhaps an idea?

borkdude15:08:54

Not sure how easy that is, but this problem might occur more frequently with other tools as well

mkvlr15:08:17

hmm, so wrap it in a thing with a custom .toString impl? Yeah, I guess that could work, will give it a try and let you know, thanks!

borkdude15:08:04

@mkvlr You can also set the uncaught exception handler and then remove :locals from the exception, perhaps (I'll try locally)

mkvlr15:08:15

@borkdude oh, so just remove it from ex-data? We already catch the exception so that should be easy, one sec

☝️ 1
mkvlr15:08:02

ah, it has to be the uncaught exception handler, right?

borkdude15:08:40

if you're already catching the exception that that's the place where you want to do this?

borkdude15:08:52

are you re-throwing the exception yourself?

borkdude15:08:00

the printing is done for whatever exception isn't caught and handled by the script

borkdude15:08:31

try is handled by the interpreter, only when the exception escapes the interpreter, then this stacktrace + locals stuff gets added

borkdude15:08:58

and its turned into an ex-info

borkdude15:08:22

so I think it has to be an option in bb itself probably. or perhaps something that you can set in bb. exposing the stacktrace stuff has been on my list for a while. perhaps we can make this more programmable in scripts

borkdude15:08:45

but that's not going to be a "quick fix" thing that I can do in an hour

mkvlr15:08:49

not so urgent for us for sure

borkdude15:08:51

Perhaps you can just grep the locals portion out:

----- Locals -------------------------------------------------------------------
x: 1

----- Stack trace
At least it's structured so you can do it

mkvlr15:08:06

yeah, was thinking to capture and swallow it as well

mkvlr15:08:44

also the printing ****** is something we’ll try, might work well with protocol extension via metadata

borkdude15:08:42

@mkvlr perhaps solving this at a lower (or higher, depends on how you look at this) is better, so it works independent of language/tech?

1
borkdude15:08:13

I mean, one could also print secrets from Python, R, whatever right

borkdude15:08:26

this is what CircleCI solves at their UI level

mkvlr15:08:54

yeah, not sure how it can be prevented in general, will have to take a look at how circle ci does it

mkvlr18:08:18

> With secret masking, if you inadvertently `echo`, or print, your environment variable or context, we substitute it with `XXXXX`. Before the build logs are printed out by the UI, CircleCI scans the logs’ output to ensure there are no secrets printed that match against the names of project or context environment variables.

mkvlr18:08:15

not sure I fully understand it, do they prevent secret env variable names or contents from being printed out?

mkvlr18:08:24

I guess it must be the contents.

borkdude16:08:23

@mkvlr would it help if bb "masked" the variable if the value came from an environment variable?

borkdude16:08:48

that could be implemented on the bb side but I'm not sure if it would solve the problem generally enough

borkdude16:08:09

we could just look at all the vals of the env vars and then mask it if it's in that set

borkdude16:08:48

this wouldn't really help if the secret came from some edn file or so

borkdude16:08:13

so it's maybe a little bit too specific

borkdude16:08:16

to include in bb

borkdude16:08:01

but that could be a way how nextjournal could do it in its runner

mkvlr16:08:41

in our case it comes in as an edn cli arg

mkvlr16:08:02

but could also pass it as an env var instead

borkdude16:08:32

perhaps bb could always mask the values of locals or dump the locals to a file instead

borkdude16:08:44

or dump the entire error output to a file like the clojure CLI does

borkdude16:08:53

I'm not sure what the right answer is :)

borkdude16:08:03

probably better to wait a bit more for some more feedback on this

👍 1
Dig17:08:05

looking for expect like functionality on babashka, any suggestions?

borkdude17:08:57

What do you mean, "expect like"? There is clojure.test or clojure.core/assert?

Dig18:08:20

sorry had to step out, like tcl expect, or python-expect library

Dig19:08:19

yes exactly, I use python-expect library right now, but of cause would love to use bb

Dig19:08:08

I found some Java libraries too https://en.wikipedia.org/wiki/Expect#Java did not try them yet, also no clojure wrappers to my knowledge

borkdude20:08:01

@i.slack I haven't seen something like this in the bb space yet, but perhaps it can be built in user space?

Dig20:08:15

yes, i would think so, if only there were more then 24 hours in the day for me, maybe it even can be some extension of babashka/process library

Huahai22:08:32

I concur that expect would be a really useful addition to bb. It would greatly help with testing command line programs