Fork me on GitHub
#babashka
<
2020-11-21
>
yubrshen01:11:30

How can I access babashka.curl from emacs/cider/JVM environment? I now can access it in babashka repl, but I prefer the repl in emacs/cider. I guess that I need to add babashka/babashka {:mvn/version ???} to my project's deps.edn. What would be the version number, and if adding babashka/babashka {:mvn/version ???} is it the correct way? Do I need to perform a local installation of babashka in the local maven? Thanks!

yubrshen03:11:23

Find the answer: borkdude/babashka {:mvn/version "0.2.3"}

lispyclouds07:11:17

also, you could just add babashka.curl with borkdude/babashka.curl {:git/url "" :sha "59e03c18c7f16f4d2328b52d37b988d6761d2a3d"} without needing to add the whole of babashka 😄

yubrshen03:11:34

What's the equivalent to

(clj-http.client/get "")
by curl? The following
(curl/get "")
seems resulting the zipp'ed:
Class: clojure.lang.PersistentArrayMap
Contents: 
  :status = 200
  :headers = { "content-encoding" "gzip", ... }
  :body = "�\b\0\0\0\0\0\0\n�]ko�8�+��Onb�z9����8�l�]��;���`d�bIIN�-���R�I���qb mM������#�b�M�{����&��\b��O�q|���~~~��b�<O8���+'�&Ӽx����:�M'Q�rp�����$F...
  :err = ""
  :process = Process[pid=15826, exitValue=0]
  :exit = 0
Thank!

lispyclouds06:11:24

you can pass args to the curl command using the :raw-args in an option map. This should work (curl/get "" {:raw-args ["--compressed"]})

borkdude08:11:37

@U7ERLH6JX Do you think we could always add the --compressed flag or would this fail with websites that do not send compressed responses?

lispyclouds08:11:56

We could add it, it should work. My only reason of not adding it would be that there is an implicit behavior and since this is a thin wrapper over curl, its doing more than that and curl users might be surprised. If you ask me i would leave it out. What say?

lispyclouds09:11:59

Ideally this should've been done by curl by inspecting the response header. HTTPie does it

lispyclouds09:11:57

Even web browsers do that too

lispyclouds09:11:34

another reason of not adding would be if later the meaning of --compressed changes based on the curl version, this would start failing for a non obvious reason

borkdude09:11:28

@U7ERLH6JX Yep, I agree. - We could support (get ... {:compress true}) to make this a little nicer. - Alternatively we could look at the content-encoding header ourselves and decompress but that may also fall into the category of doing too much.

borkdude09:11:47

I wonder if stackexchange should have sent the response in gzip encoding even when we didn't ask for that

borkdude09:11:38

> The likelihood of many applications not opting into compression, and being materially worse for it, is unacceptable.

borkdude09:11:21

btw, if I start a server with:

$ bb -e '(org.httpkit.server/run-server (constantly {:body "hello"}) {:port 3000}) @(promise)'
and request that with --compressed it doesn't fail

lispyclouds10:11:31

yeah curl's --compressed works even if the response isnt. maybe a fn to convert a map of opts -> cli args might be a good addition. But that should be about it, nothing more IMO

borkdude10:11:34

hmm, if it "always" works what's the reason not to enable it by default?

borkdude10:11:56

is clj-http doing this by default?

borkdude10:11:10

(i.e. requesting compressed responses)

lispyclouds10:11:22

i would assume its inspecting the response header like HTTPie

lispyclouds10:11:47

its Apache HTTP client under the hood, maybe thats doing it?

borkdude10:11:47

By default, clj-http will add the {"Accept-Encoding" "gzip, deflate"} header to requests, and automatically decompress the resulting gzip or deflate stream if the Content-Encoding header is found on the response. 

borkdude10:11:25

we could also do that ourselves and not let curl do that

borkdude10:11:57

> If this is undesired, the {:decompress-body false} option can be specified:

borkdude10:11:15

but I guess it doesn't hurt to let curl do it

borkdude10:11:47

and it seems unlikely to me that --decompress will change its meaning. If curl will change the meaning of its args, then we're doomed anyway in other cases

lispyclouds10:11:55

thats the thing, where do we draw the line between "a wrapper over curl" and "powered by curl".

borkdude10:11:38

we're already using these sane defaults:

"curl" "--silent" "--show-error" "--location" "--dump-header"

borkdude10:11:55

I would say powered by curl

borkdude10:11:31

I think offering sane defaults with the possibility of opting out of it is a better way

lispyclouds10:11:48

yeah if we are sending some flags already as some opinions, then yes we can do that

borkdude10:11:56

follow locations is also not enabled by default, but this is the sane default

borkdude10:11:19

I think curl is conservative because it was growing over time when compression maybe wasn't the default before

lispyclouds10:11:00

or another thought process could be to implement clj-http's API as close as possible and say this is powered by curl. im of the opinion that if we follow one of the standards its more compilant?

borkdude10:11:02

I think the more important thing is that it's useful for scripting and doesn't cause any beginner questions. Experts can dig into the docs to disable the defaults. I think this is what clj-http has done too.

borkdude10:11:30

clj-http isn't a standard, it's a popular clojure lib that has influenced other libs

lispyclouds10:11:23

right, i guess sticking with some sane defaults which most of us(beginners and others) may use and allow the internal knobs to be turned when needed would be a good choice.

borkdude10:11:14

ok, I'll proceed with implementing this and documenting it

babashka 3
borkdude10:11:45

Thanks for your input. Btw, I'm keeping some notes about http here: https://github.com/borkdude/babashka/wiki/HTTP-client-and-server-considerations

borkdude10:11:06

I am contemplating of building a Java 11 based solution as babashka.http-client similar to what schmee has done.

borkdude10:11:26

To have one canonical way of doing http in bb

lispyclouds10:11:26

that would be super awesome. bb setting the standards 😎

lispyclouds10:11:40

on a side note, had great success with java-http-clj. would love to see that being used. pretty much use it in all of my clj needs now 😄

borkdude10:11:30

at the moment it's a bit messy having to choose between babashka.curl, org.httpkit or clj-http-lite, slurp, etc. once babashka.http-client is there, the other clients can slowly fade to the background and be deprecated over time. I want to give it some incubation time to flatten out bugs etc. of course

borkdude10:11:03

yeah, that one looks great, but isn't yet feature complete.

borkdude10:11:14

but it looks like a good start to build babashka.http-client on

lispyclouds10:11:29

yep my vote would be there too

lispyclouds10:11:20

also this is something we made based on OkHTTP: https://github.com/into-docker/unixsocket-http to be able to do UNIX sockets naturally. but quite skeptical for Graal due to the size bloat. Works on it fine though

borkdude10:11:42

we can still use babashka.curl for that right?

lispyclouds10:11:36

yep, in the off chance you dont have/want to install curl

borkdude10:11:37

Since babashka.curl is so light-weight, I will forever keep it in bb. curl might cover some weird edge cases that Java doesn't out of the box, like this

lispyclouds10:11:47

this lib is useful for embedding and lean docker deployments more i'd say.

borkdude10:11:53

(deftest compressed-test
  (is (-> (curl/get "")
          :body (json/parse-string true) :items))
  (is (thrown?
       Exception
       (-> (curl/get ""
                     {:compressed false})
           :body (json/parse-string true) :items))))

🎉 3
borkdude10:11:55

### Compression

From babashka 0.2.4 onwards, this library will call `curl` with `--compressed`
by default. To opt out, pass `:compressed false` in the options.

lispyclouds10:11:27

> Since babashka.curl is so light-weight, I will forever keep it in bb Just due to the target audience of bb and the use cases, the likely hood of curl being there along with bb is extremely high 😄

borkdude10:11:43

yep. it's also part of the docker image

borkdude10:11:20

it's even on windows nowadays, along with tar

lispyclouds10:11:13

windows is the new Mac. (shots fired) 😛

borkdude10:11:34

it's kinda true ;)

borkdude10:11:06

My next laptop might seriously be Windows, if not Ubuntu

borkdude10:11:26

bozhidar is now running Windows as well

lispyclouds10:11:12

yep, was pleasantly surprised seeing the talk

borkdude10:11:01

haven't watched it yet. any good?

borkdude10:11:09

any new information I should see I mean

borkdude10:11:57

Ugh, another github actions instability thing.

lispyclouds10:11:11

well some of the lesser known features of CIDER, some Emacs hiccups and the fact that Emacs running on Windows like magic

borkdude10:11:47

I kinda found that out myself before bozhidar was on Windows with all the WSL2 stuff

borkdude10:11:58

is he running emacs on Windows natively or through wsl2?

borkdude10:11:16

yeah, I love it

borkdude10:11:56

I'm eyeing this laptop: https://bestware.com/en/schenker-via-15-pro.html AMD + 64GB... if my current laptop stopped working.

🤯 3
borkdude10:11:33

Fix worked!

borkdude10:11:16

## Java 11 client
Java 11 client. Based on  minus the specs.
Adds 2.28MB to the binary. See branch `java-11-client`. 
This is pretty reasonable I'd say

lispyclouds10:11:09

yeah i love the zero dependency aspect of it

lispyclouds10:11:56

quite extensive interceptors

borkdude10:11:08

All of these things are still moving targets, so babashka.http-client will also probably take another year or so

borkdude10:11:16

but the seeds are planted

borkdude10:11:07

anyway @yubrshen, I hope we answered your question :)

borkdude08:11:59

@yubrshen This seems to be the way to do it with curl:

$ bb -e '(-> (curl/get "" {:raw-args ["--compressed"]}) :body json/parse-string keys)'
("items" "has_more" "quota_max" "quota_remaining")

yubrshen02:11:11

@U04V15CAJ Thanks for taking the time to help me. I'll study the documentation of curl/get to understanding the details of the parameters.

borkdude08:11:33

So {:raw-args ["--compressed"]} is only necessary here because stackexchange always sends compressed responses, but curl does not decompress compressed responses automatically. In the next release of babashka I will pass this flag to curl by default, but for now you have to use it like this.

David Pham09:11:51

grasp is really cool by the way. Really helpful.

borkdude09:11:15

Cool! There is also a #grasp channel now

borkdude11:11:49

If someone is looking for some OSS issues to work on, here's a list: https://gist.github.com/borkdude/18af5d96c6465ce64144f03636fda3dc

nivekuil11:11:55

maybe an odd request: has anyone used clara rules with babashka?

nivekuil11:11:39

here's some context -- I'll share more about it if people are interested, but the gist of it is generating docker compose/swarm commands with nice reusable code https://gist.github.com/nivekuil/05f507d9e53cf53da6918728ece54b40 so I never have to write yaml again :)

borkdude11:11:42

@kevin842 currently that doesn't work, it seems:

[email protected] /tmp $ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {com.cerner/clara-rules {:mvn/version "0.21.0"}}}')
[email protected] /tmp $ bb -e "(require '[clara.rules :as cr])"
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: load
Location: clojure/reflect.clj:123:2
Phase:    analysis

----- Context ------------------------------------------------------------------
119:   {:added "1.3"}
120:   [obj & options]
121:   (apply type-reflect (if (class? obj) obj (class obj)) options))
122:
123: (load "reflect/java")
      ^--- Could not resolve symbol: load
We could look into this to make it work, possibly.

borkdude11:11:09

I think we should then add clojure.reflect into bb

borkdude11:11:55

oh hmm, clara also uses its own Java classes: https://github.com/cerner/clara-rules/tree/main/src/main/java/clara/rules so it won't work anyway, unless we add it into bb itself

borkdude11:11:29

so for now I would say: use clojure on the JVM if you want clara

borkdude11:11:09

You could try it out using the same thing as above and let us know what you run into

nivekuil11:11:27

I think spec may be a problem

bb -e "(require '[odoyle.rules :as o])"             ----- Error -------------------------------------------------------------------- Type:     clojure.lang.ExceptionInfo Message:  Could not resolve symbol: clojure.lang.Compiler/demunge Location: clojure/spec/alpha.clj:128:16 Phase:    analysis  ----- Context ------------------------------------------------------------------ 124: (defn- fn-sym [^Object f] 125:   (let [[_ f-ns f-n] (re-matches #"(.*)\$(.*?)(__[0-9]+)?" (.. f getClass getName))] 126:     ;; check for anonymous function 127:     (when (not= "fn" f-n) 128:       (symbol (clojure.lang.Compiler/demunge f-ns) (clojure.lang.Compiler/demunge f-n)))))                     ^--- Could not resolve symbol: clojure.lang.Compiler/demunge 129:  130: (extend-protocol Specize 131:   clojure.lang.Keyword 132:   (specize* ([k] (specize* (reg-resolve! k))) 133:             ([k _] (specize* (reg-resolve! k))))

borkdude11:11:15

ah right, that one again. you could request if doyle moved its specs into a different namespace perhaps so they become optional

borkdude11:11:33

possibly starting out with your own fork

nivekuil11:11:19

that makes sense, thanks

noprompt21:11:15

Is there a ticket for this?

user=> (reify clojure.lang.IFn (invoke [this x]))
clojure.lang.ExceptionInfo: No reify factory for: #{clojure.lang.IFn} [at <repl>:18:1]

borkdude21:11:48

@noprompt why not: (fn [this x])?

noprompt21:11:34

Here’s how I got to this

(defn factory [x]
  (fn [y] ,,,))

(let [a (factory 1)
      b (factory 1)]
  (assoc {} a 1 b 2))
;; => No
{#function[meander.interpreter.epsilon/factory/fn--26726] 1,
 #function[meander.interpreter.epsilon/factory/fn--26726] 2}

(defn factory [x]
  (reify
    clojure.lang.IFn
    (invoke [this y]
      ,,,)))

(let [a (factory 1)
      b (factory 1)]
  (assoc {} a 1 b 2))
;; => NO
{#object[meander.interpreter.epsilon$factory$reify__26734 0x814ee48e "[email protected]"]
 1,
 #object[meander.interpreter.epsilon$factory$reify__26734 0x9a1f044a "[email protected]"]
 2}


(defn factory [x]
  (reify
    Object
    (hashCode [this]
      (hash x))

    (equals [this that]
      (and (identical? (class this) (class that))
           (= (hash this) (hash that))))

    clojure.lang.IHashEq
    (hasheq [this]
      (hash x))

    clojure.lang.IFn
    (invoke [this y]
      ,,,)))

(let [a (factory 1)
      b (factory 1)]
  (assoc {} a 1 b 2))
;; => YES
{#object[meander.interpreter.epsilon$factory$reify__26752 0x3ba9821a "[email protected]"]
 2}

noprompt21:11:48

I could make a record for this but, at the moment, I’d rather not.

noprompt21:11:29

I could also make a protocol for semantic purposes instead of using IFn but I also wanted to avoid that for the moment.

borkdude21:11:41

Due to limitations with GraalVM this is really hard to support.

borkdude21:11:52

Can't you maybe just use maps?

noprompt21:11:00

The heart of it really boils down wanting some leverage as a maintainer. 😛

borkdude21:11:01

or functions with metadata on them

noprompt21:11:25

Ah, good idea! Forgot about that trick. 😉

noprompt21:11:25

(defn factory [x]
  (with-meta (fn [y] ,,,)
    {:id x}))

(let [a (factory 1)
      b (factory 1)]
  (assoc {}
         [(class a) (meta a)] 1
         [(class b) (meta b)] 2))
;; => Yes
{[clojure.lang.AFunction$1 {:id 1}] 2}
😎

noprompt21:11:50

Not ideal but it works.

borkdude21:11:40

There's currently a bug in bb so this doesn't work:

(meta ^:foo (fn [])) ;;=> nil
but with-meta works. I'll make an issue for this.

borkdude21:11:04

@noprompt fyi, in babashka / sci+graalvm I have to create classes ahead of time for all possible combinations of interfaces, because I can't create classes at runtime in GraalVM. So I end up with this macro that generates all combinations of subsets of interfaces that are supported with reify in bb: https://github.com/borkdude/babashka/blob/master/src/babashka/impl/reify.clj Then whenever someone calls reify, the interpreter looks at the reified interfaces and then picks the class that has the matching combination of interfaces.

borkdude21:11:41

So theoretically what you want can be supported (it's not right now because there isn't an AOT-ed class with these combinations of classes), but if fns with metadata work equally well, I would prefer that

borkdude21:11:54

it's also less code to process for the interpreter, so better initial load time

noprompt21:11:04

Thanks for the details. 👍

borkdude21:11:10

That, or you can just use :bb branches to make a little bit different logic for bb only

noprompt21:11:39

I may have to use protocols/records in the end.

noprompt21:11:53

Well, not quite.

noprompt21:11:24

Bit of a hack but it will suffice.