Fork me on GitHub
#babashka
<
2021-05-11
>
n2o08:05:09

Hi, I need some rubberducks I think 😄 I want to use echo within a shell command, but want to pass -en as a Parameter. But it seems as if the shell command does not interpret -en as Parameter

(sh "echo" "-en" "foo\n\n")
=> {:exit 0, :out "-en foo\n\n\n", :err ""}
As we can see in :out the -en Parameter seems to be interpreted as the String, which shall be echoed, but not as a Parameter for echo. Can anybody help me with this? In general, I want to pipe this output to another command. I am also trying process, but there’s the same problem with the -en .

n2o08:05:03

process, which should then be used for pipelining, provides similar results:

(-> (process '[sh -c "echo -en $FOO"] {:env {:FOO "BAR"}}) :out slurp)
=> "-en BAR\n"

n2o08:05:50

Expected output: Something like this, without -en:

borkdude09:05:23

@n2o The issue may be that echo is both a built-in shell command and echo being a binary on your system. Shelling out from Java always means it's calling the binary

borkdude09:05:24

What is the purpose of echo -en foo ?

n2o09:05:20

The complete script looks something like this:

(-> (process '[echo -en $SIGNATURE] {:env {:SIGNATURE "something"}})
    (process '[openssl sha1 -hmac $SECRET -binary] {:env {:SECRET "some other stuff"}})
    (process '[base64])
    :out slurp)
To upload a file to my minio server

borkdude09:05:15

Note that this does work for me:

$ bb -e '@(:out (babashka.process/process ["echo" "-n" "foo\n\n"] {:out :string}))'
"foo\n\n"
Just the -e doesn't work, it's also not mentioned in the man echo page on my system

borkdude09:05:04

Why don't you just feed the env var to :in ?

n2o09:05:37

I tried several different things 😄 also using :in. But I think the first problem is the en parameter

borkdude09:05:06

Try:

@(:out (process ["openssl" "sha" "-hmac" <your-secret> "-binary"] {:in "something" :out :string})

borkdude09:05:31

you don't have to pipe something via echo to stdin of another process, you can do that directly

n2o09:05:20

yes, i hoped so, but the resulting base64 encoded string differs from the one when I put it directly on the console. And I was figuring out the root cause of this 😬

n2o09:05:29

thanks, I’ll give it a try.

borkdude09:05:15

btw, there is also https://github.com/babashka/pod-babashka-buddy if you don't want to shell out

n2o09:05:48

Yep, I think I’ll switch to some of your pods, seems easier.

borkdude09:05:51

$ bb -e '@(:out (babashka.process/process ["openssl" "sha1" "-hmac" "dude" "-binary"] {:in "foobar" :err :inherit :out :string}))'
"/��B1SKsi�Ӕ @�\n��"

borkdude09:05:19

$ bb -e '(-> (babashka.process/process ["openssl" "sha1" "-hmac" "dude" "-binary"] {:in "foobar" :err :inherit}) (babashka.process/process ["base64"] {:out :string}) deref :out)'
"L6UQkkIxU0tzaROn05QgQKkKg90=\n"

borkdude09:05:32

(might want to remove the newline)

n2o09:05:28

oh jeah, there it is… Now it is working! Thanks!

borkdude09:05:38

hmm, I see it is different yes

n2o09:05:54

but that’s the base64 encoded string I was looking for (on my machine)

borkdude09:05:19

perhaps it's because of the newline of openssl

borkdude09:05:40

@n2o oooh:

$ bb -e '(-> (babashka.process/process ["openssl" "sha1" "-hmac" "dude" "-binary"] {:in "foobar\n" :err :inherit}) (babashka.process/process ["base64"] {:out :string}) deref :out)'
"89CMnuc40b7mfZpt9domvGCGIm0=\n"

borkdude09:05:51

$ echo foobar | openssl sha1 -hmac dude -binary | base64
89CMnuc40b7mfZpt9domvGCGIm0=

borkdude09:05:55

you see, that's the same...

n2o09:05:07

yes yes, I’ve got it now working on my machine 🙂

n2o09:05:29

I can also produce the same string. Thank you very much! although I currently do not see the difference from my previous approaches :thinking_face: 🤷

borkdude09:05:48

the difference I found was that when you add a newline to the input it's the same as on the command line

borkdude09:05:14

probably echo also adds a newline to the input

n2o09:05:17

but I was working in a babashka script and defed the symbol with the newlines

n2o09:05:25

so they were always present in my script

borkdude09:05:30

This is without the newline:

$ echo -n foobar | openssl sha1 -hmac dude -binary | base64
L6UQkkIxU0tzaROn05QgQKkKg90=

borkdude09:05:47

and that's the same as the first example we had

borkdude09:05:12

$ bb -e '(-> (babashka.process/process ["openssl" "sha1" "-hmac" "dude" "-binary"] {:in "foobar" :err :inherit}) (babashka.process/process ["base64"] {:out :string}) deref :out)'
"L6UQkkIxU0tzaROn05QgQKkKg90=\n"

borkdude09:05:27

so the confusion comes from the newlines :)

n2o09:05:29

Okay, I understand. Great 🎉 again something learned

n2o09:05:53

and now piping works as expected in babashka 🥳 thanks

holmdunc09:05:59

Is there something in the babashka/sci universe that can print a list of the names of the :aliases in the nearest deps.edn file?

borkdude09:05:05

@duncan540 not really built-in, you can just parse the edn

holmdunc09:05:44

I'm doing it from a Python context, so the best I can think of so far is to use the use Python stdlib to find the nearest deps.edn on the filesystem, then use your jet tool to transform that to JSON, then pick out the alias names from that

borkdude09:05:49

oh Python, hmm, I guess you could just shell out to bb from python as well

💡 3
borkdude09:05:59

or use #jet that's also possible

holmdunc09:05:32

yeah, I'll be able to cobble something together, thanks for the pointer 🙂

holmdunc12:05:37

Sublime Text GUI plugin:

pithyless13:05:10

I'm having a problem moving my tasks into a local project root. 🧵

pithyless13:05:44

If I have a bb.edn with:

{:deps
  {some.lib {:local/root "lib/some.lib"}}

  :requires ([some.lib.foo :as foo]) ;; foo is requiring lread.status-line

  :tasks {bar (foo/bar)}}

pithyless13:05:58

Where lib deps.edn has:

{:deps
  {lread/status-line {:git/url ""
                      :sha     "35ed39645038e81b42cb15ed6753b8462e60a06d"}}}

pithyless13:05:14

Then we get an error when running bb bar:

Type:     java.lang.Exception
Message:  Could not find namespace: lread.status-line.

pithyless13:05:33

But if I add that status-line dependency directly to bb.edn's :deps everything works. Am I doing something wrong or is there some bug between the interplay of deps and tasks?

borkdude13:05:52

no, this is by design. bb deps should go into bb.edn

borkdude13:05:15

this used to work by mistake

borkdude13:05:27

but the Clojure deps should not be mixed with bb deps imo

pithyless13:05:06

hmm, yeah, so this did used to work and I'm not going crazy; I depended on the behavior :]

pithyless13:05:49

I don't understand this design decision - if I would like to publish a task as a reusable piece of code that can be used by multiple bb.edns, how should I go about it?

borkdude13:05:51

yeah, it was accidental

borkdude13:05:14

then you could make that reusable piece of code a library I guess?

pithyless13:05:52

and that lib would be imported via :deps in bb.edn, correct?

borkdude13:05:54

oh, you would like to publish a task as a reusable piece of code, I didn't anticipate that use case

borkdude13:05:04

I mean, the task name etc?

borkdude13:05:13

or just some functions?

pithyless13:05:29

no, just some functions that a task can call

borkdude13:05:40

yes, you can use any library in :deps, same as deps.edn, but specific for bb

borkdude13:05:13

so you can e.g. use :local/root or :paths to get at the shared code or if you want to publish it, use :git/url

pithyless13:05:26

yep, that's what I'm trying to do

pithyless13:05:50

it's like bb is not resolving the deps inside the :local/root deps.edn when calling tasks

borkdude13:05:19

bb isn't resolving deps from deps.edn at all

borkdude13:05:54

or do you mean, the deps.edn from the local/root library?

borkdude13:05:59

that should work

pithyless13:05:17

no, when bb has an entry {:deps {foo :local/root "foo"}} -> it's not resolving the "foo/deps.edn"

borkdude13:05:40

it should be {:deps {foo {:local/root "foo}}}

borkdude13:05:17

it might be a caching issue, maybe try rm -rf .cpcache; rm -rf ~/.clojure/.cpcache

pithyless13:05:33

facepalm that did it

borkdude13:05:20

it was the caching issue?

pithyless13:05:37

Yep, state. Mutable state is always the problem.

pithyless13:05:02

maybe bb needs a -Srepro ? XD

borkdude13:05:31

cache is turned off with -Sforce but yeah, we should allow passing args down to the deps tool

pithyless13:05:34

anywho, thanks @U04V15CAJ - I was starting to doubt my sanity

borkdude13:05:30

:) you can debug the classpath with (babashka.classpath/get-classpath)

pithyless13:05:54

I used to always add -Sforce -Srepro to all my Makefiles out of habit just to reduce these vectors of confusion

pithyless14:05:39

(although that comes with its own set of drawbacks)

borkdude17:05:13

if you have some ideas how babashka could pass down args to the deps system...

borkdude17:05:50

bb --deps -Sforce --deps -Srepro ... run foo
or so...

borkdude17:05:24

or directly support the relevant CLI args...

bb -Sforce run foo

borkdude17:05:13

some prefix would be good probably to not get conflicts with prior or future bb args

borkdude17:05:32

like -J-Dfoo=bar but then something else

pithyless06:05:33

> if you have some ideas how babashka could pass down args to the deps system... Nothing directly comes to mind, the -C-Sforce is interesting, because it's consistent with the approach deps.tools have taken; maybe it would be enough to have a :clojure-opts setting in the bb.edn ? The complexity also stems from the fact that you have the clojure that is internal and used to resolve things like bb :deps; but there is also the clojure that can be invoked by a user task. If one were to pass or configure some default clojure args, should they be applied the same to both?

borkdude06:05:12

That’s not really an issue. clojure is a drop-in replacement to which you can pass all args

borkdude06:05:45

The args we are discussing should only affect bb deps

metehan19:05:59

I couldn't install babashka on ubuntu (pop-os)

r@pop-os:~/dev/babashka/babashka$ script/uberjar && script/compile
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
Syntax error (ClassNotFoundException) compiling at (babashka/impl/classes.clj:345:20).
java.lang.ProcessHandle

Full report at:
/tmp/clojure-2871116251297672484.edn
Error encountered performing task 'do' with profile(s): 'base,system,user,provided,dev,xml,yaml,core-async,csv,transit,httpkit-client,httpkit-server,core-match,hiccup,test-check,rewrite-clj,selmer,reflection'
Suppressed exit

borkdude19:05:32

you need graalvm 21.1.0 CE JDK11

❤️ 3
borkdude19:05:40

btw why are you trying to build it yourself rather than downloading the binary?

metehan19:05:09

yes I was trying to build myself somehow I didn't see the binaries 🙂

metehan19:05:42

I installed with asdf now

borkdude19:05:44

ok, just letting you know that bb is available as binaries, so only if you are developing you have to build it yourself

borkdude19:05:52

:thumbsup:

🍺 3
n2o19:05:33

Are there any autocompletion features for the shell to access aliases from the Task Runner? If not, I’d love to have more support in the shell to have it tab-autocompleted 😄

borkdude19:05:10

@n2o there have been some experimental approaches, I'd love to see contributions for this so we can support bash and zsh (and others, but I'm using zsh)

n2o19:05:52

ah, perfect. I’ll take a look at this, Sadly, I am using fish, but maybe we can work on that / I can help you with that

borkdude19:05:42

I'm fine with supporting any shell, but I haven't had the time to look into this myself and I think this would be a good place to contribute

borkdude19:05:59

Perhaps the completion script can be written in bb itself?

borkdude19:05:12

or can bb emit some hints to the shell itself via some subcommand? Not sure how this should work

n2o19:05:14

that’s what I was thinking about 😄

borkdude19:05:53

Shells should be able to ask to a binary: bb --complete <substring> and it will give a list of completions, or something like this right

n2o19:05:06

Jep, sounds great, thanks for the pointers. interesting that these repos are recently created

Jakub Holý (HolyJak)08:05:09

@U01C3LSFZAM Hi! regarding ☝️ and the "replacements to mv, rm, and cp", it would be also note pointing to the more ergonomic https://babashka.org/fs/babashka.fs.html#var-copy included with bb: bb -e '(babashka.fs/copy "server.pem" "server.pem.copy")'

Avi Drucker19:02:19

@U0522TWDA Thank you for this. The link unfortunately appears to be down, but searching for "bb -e" in the book https://book.babashka.org/ seems to give helpful context 👍🤓

borkdude19:02:04

@U01C3LSFZAM Nice, a reply after almost two years :) The new link is here:

😅 1