Fork me on GitHub
#babashka
<
2019-12-17
>
sogaiu01:12:38

that works for me when i use java 8 -- but not for java 11

sogaiu01:12:23

$ lein bb examples/process-builder.bb 
Syntax error (ClassNotFoundException) compiling at (babashka/impl/classes.clj:91:16).
java.lang.UNIXProcess$ProcessPipeOutputStream

Full report at:
/tmp/clojure-8153440468259675917.edn

$ java -version
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)

borkdude08:12:32

That is because they renamed UNIXProcess in java 11

borkdude08:12:52

There is an issue for JDK 11 here: https://github.com/borkdude/babashka/issues/120 I'll need to make a custom version of clojure.lang.Reflector for it

borkdude08:12:03

So, it's a known issue

sogaiu13:12:30

oh yeah. forgot about that -- thanks for the reminder :)

borkdude13:12:32

Do you think Windows support would be easier on JDK11? I don't see a pressing need to upgrade to it right now. I also feel the support for it is more experimental in GraalVM, but I'm not sure

sogaiu13:12:35

i'm really not sure about the first question. it's clear that there are various fixes and features in java 11 that aren't in java 8 though (e.g. some process support) i agree support for windows is more experimental in graalvm. it seems like windows support lagging is almost inevitable for some multiplatform efforts because of all of these odd limitations and behaviors in the os. i think if it isn't too much work, exercising some of the jdk11 bits is good because we can: 1) have a better idea of what some upcoming issues might be (like this unixprocess thing) 2) possibly file issues to get stuff fixed sooner (so things will be in place for us later)

oskarkv05:12:45

I tried to use dotimes, like this bb '(dotimes [i 10] (println i))', but I got Could not resolve symbol: i [at line 1, column 11]. Is dotimes not available? How do I know what is and what isn't available?

steveb8n06:12:05

I just started using bb & jet. amazing!!

steveb8n06:12:23

Is there an example somewhere of doing base64 decoding using bb?

borkdude07:12:36

@oskarkv I get this:

$ bb '(dotimes [i 10] (println i))'
0
1
2
3
4
5
6
7
8
9

borkdude07:12:47

Maybe you are using an old version which had a bug

borkdude07:12:27

@steveb8n How would you do base64 in normal Clojure? Maybe I could add support for those classes/functions

steveb8n08:12:37

the simplest is interop

(:import (java.util Base64 UUID))
(.decode (Base64/getDecoder) s)

steveb8n08:12:19

and a few others. if you could add that, it would really help me

steveb8n08:12:58

I was gonna try and figure out the dep/require but in this case I’ll wait till it’s built in

borkdude08:12:45

First step would be adding support for the raw Java class:

$ clj -e '(-> (java.util.Base64/getDecoder) (.decode "YmFiYXNoa2E=") (String.))'
"babashka"
I'll give it a try

borkdude08:12:34

it seems data.codec is from before a certain Java version since they don't use this?

borkdude08:12:19

@steveb8n Note that you can already do this today:

$ bb '(:out (shell/sh "base64" "-D" :in "YmFiYXNoa2E="))'
"babashka"

steveb8n08:12:47

even better. thank you!

steveb8n08:12:56

might be a good one to add to the examples

oskarkv08:12:01

@borkdude yeah, seems like it, with the new version it works

oskarkv08:12:59

Still, some things are not available, like ns-interns . Is there a list of everything available somewhere?

borkdude08:12:03

The README lists some difference with Clojure. Everything related to vars and namespaces is re-implemented.

borkdude08:12:22

Also check out sci which is used to build babashka

oskarkv08:12:52

Can I use these reimplemented namespace things to see what's in the namespaces? πŸ™‚

borkdude08:12:42

What's in the namespaces you can see in the README

borkdude08:12:59

I haven't made most ns-... functions yet

borkdude08:12:39

Most things aren't even real vars, but just direct values

borkdude08:12:46

Except when you define vars yourself

borkdude08:12:23

So the result of ns-interns would be different than in Clojure right now. But it seems most people are just interested in the keys, not the values?

oskarkv08:12:52

I just wanted to see what exactly is missing, and tried to use ns-functions to do it.

oskarkv08:12:09

But maybe only the ns-functions are missing. πŸ˜›

oskarkv08:12:52

Trying to figure out what the differences are between joker and babashka too. They seem pretty similar.

borkdude08:12:53

@oskarkv I guess I could just make ns-interns and fake the values by printing them as vars πŸ˜‰

borkdude08:12:50

I'll make an issue for it

borkdude08:12:01

As for the differences between joker and babashka: I think joker is more mature, since it has been around longer, babashka is still in active development. Also it tries to be more compatible with Clojure by not introducing custom namespaces, so when babashka is not enough, you can just run the JVM without major changes

oskarkv08:12:26

Ok, got it! Thanks!

borkdude08:12:47

And there's the bash integration like:

$ bb -o '(range)' | bb -io '(take 3 *in*)'
0
1
2

oskarkv08:12:24

Yeah, that's nice. Tried to do the same thing with joker just now but couldn't figure it out. But maybe that's just me. πŸ˜›

borkdude08:12:12

babashka also allows interop with a curated set of Java classes, which is another difference

oskarkv08:12:52

OK, which ones?

oskarkv08:12:33

Found them

borkdude08:12:54

I don't know if the docs are up to date, but they are listed here as well: https://github.com/borkdude/babashka/blob/master/reflection.json

borkdude09:12:01

I'm going to add support for more Java classes as we progress (e.g. the Base64 class that Steve requested above)

oskarkv09:12:17

Ok. Nice work! πŸ™‚

borkdude10:12:36

The conch low level namespace will no longer be supported in babashka, now that there will be direct support for ProcessBuilder: https://github.com/borkdude/babashka#spawning-and-killing-a-process

borkdude11:12:24

@sogaiu This now works on master:

#!/usr/bin/env bb

(require '[babashka.wait :as wait])
(def ws (-> (ProcessBuilder. ["python" "-m" "SimpleHTTPServer" "1777"]) (.inheritIO) (.start)))
(wait/wait-for-port "localhost" 1777)
(prn (slurp ""))
(.destroy ws)
Note that I had to make a fix for wait/wait-for-port and close the Socket, since it seems that SimpleHTTPServer is single threaded πŸ™‚

borkdude11:12:52

@sogaiu We might add a PYTHON_CMD environment variable to fix your python2 issues with the tests

sogaiu12:12:53

since python isn't likely to be on windows machines by default, i think switching over to using some jvm-based web server like you mentioned before is a better idea.

borkdude12:12:46

I was thinking to skip the python-required tests when PYTHON_CMD is empty

borkdude12:12:58

and just print a warning

borkdude12:12:18

but a JVM based solution would also be fine

borkdude11:12:11

which defaults to which python if it isn't set

borkdude11:12:27

or type -p python (this is what the clojure shell script uses for some reason)

oskarkv12:12:21

Hm, say I have a library that has reader conditionals for cljs and clj that I also want to use in babashka, and the clj part would work in babashka. Then I could use :default instead of :clj, and when there is only :clj (no :cljs) I could use :default and add a :cljs nil. Or is there an easier way?

borkdude12:12:05

@oskarkv there is also the :bb branch

oskarkv12:12:47

Yeah but if the clj code works in bb too, adding :bb would mean copying the code, right?

borkdude12:12:14

true, then :default would be more appropriate

borkdude12:12:44

if clojure itself supported something like #?(#{:clj :bb} 1 :cljs 2) this would become easier, but it doesn't

borkdude12:12:52

or does it?

oskarkv12:12:18

@borkdude Or maybe you could make it so that if a reader conditional has only :clj and :cljs, then bb uses the :clj one, but if it has :bb, then of course use that. Would that make sense? πŸ™‚

oskarkv12:12:00

Maybe it wouldn't

borkdude12:12:16

I don't think that works, since the reader conditional processing is done by a parser library which you give the features in advance

borkdude12:12:32

it's processed from left to right

borkdude12:12:02

just copy the code I'd say πŸ™‚

oskarkv12:12:15

Nah :default sounds better

borkdude12:12:40

yeah, or that if it works

plexus13:12:48

what would be the current recommendation for converting HTML to Hiccup from the command line?

borkdude13:12:51

I would try that one although I don't know if it does HTML -> Hiccup

borkdude13:12:07

it does do Hiccup -> HTML

plexus13:12:09

can't immediately find anything like that in the README

borkdude13:12:17

@retrogradeorbit?

borkdude13:12:32

I had something like this in jet, but it's in a branch

plexus13:12:38

yeah I saw that

plexus13:12:02

I don't suppose babashka includes an HTML/XML parser?

borkdude13:12:19

I have a branch for clojure.data.xml but I'm not sure if that does HTML

borkdude13:12:41

Maybe I could add Jsoup to the mix

plexus13:12:16

ok, thanks for the input πŸ™‚

borkdude13:12:01

oh maybe joker supports it now btw

borkdude13:12:28

it was recently added

borkdude13:12:36

maybe I can just add the hiccup library to babashka as is

plexus13:12:00

ah but it's again hiccup->html. That part is easy πŸ™‚

borkdude13:12:25

oh yeah... sorry

borkdude13:12:52

so maybe the jet branch is worth another try. It was almost done

retrogradeorbit13:12:24

hey! yes it round trips everything

retrogradeorbit13:12:00

so if you wanted to do html to hiccup you could do:

retrogradeorbit13:12:08

bootleg -d -e '(html "")'

retrogradeorbit13:12:36

another way might be

retrogradeorbit13:12:44

bootleg -o index.clj -d -e '(convert-to (slurp "index.html") :hiccup)'

retrogradeorbit13:12:12

thats basically what html function is doing (can take a filename or a url)

retrogradeorbit13:12:26

-d is output data

retrogradeorbit13:12:42

you could also spit the output out in a file from within the scipt

retrogradeorbit13:12:10

bootleg -e '(spit "index.clj" (html "index.html"))'

retrogradeorbit13:12:18

so a few options

borkdude13:12:27

o wow, so @plexus I think you're set then πŸ™‚

borkdude13:12:59

(and I see another reason why I should maybe just include a link to bootleg instead of baking html support into jet)

retrogradeorbit13:12:17

theres a test suite that tests round trips between all the supported formats

retrogradeorbit13:12:30

might be worth making bootleg a clojars library just for that

retrogradeorbit13:12:28

someone else wanted to bring the conversion functionality into a clojure project. Will look at getting it up to clojars as a library next release

borkdude13:12:21

would it make sense to read from stdin?

./bootleg -d <<< '<html></html>'

retrogradeorbit13:12:17

I will make the default read hiccup from stdin by default.

retrogradeorbit13:12:32

the default is always hiccup

retrogradeorbit13:12:47

but I think I will expose *in* like babashka for this kind of thing

retrogradeorbit13:12:49

so would be something like bootleg -d -e '(convert-to *in* :hiccup)' <<< '<html></html>'

borkdude13:12:43

yeah, makes sense

borkdude13:12:43

why doesn't the html function just work with strings directly ?

retrogradeorbit13:12:46

bootleg -d -e '(html "<html></html>" :data)'
([:html {}])

retrogradeorbit13:12:58

the argument will be interpreted as a filename

retrogradeorbit13:12:09

if :data option is present, it treats it as the data itself

retrogradeorbit13:12:17

all the functions have this

borkdude13:12:52

oh sorry then, I think I was confused by your earlier example with convert-to then

retrogradeorbit13:12:58

so you can for example (yaml filename) or (yaml data :data)

retrogradeorbit13:12:16

same with json and selmer and others

retrogradeorbit13:12:48

yeah. just a bit rusty πŸ˜„

retrogradeorbit13:12:08

convert-to is the underlying function

borkdude14:12:09

ah makes sense

retrogradeorbit14:12:04

you're my marketing department πŸŽ‰

retrogradeorbit13:12:03

yes it would πŸ˜„ add a ticket so I dont forget next time Im working on it!

borkdude13:12:22

@retrogradeorbit How does one get from a literal string to hiccup right now?

./bootleg -d -e '"<html></html>"'

retrogradeorbit13:12:09

bootleg -d -e '(convert-to "<html></html>" :hiccup)'
[:html {}]

borkdude14:12:42

so tl;dr:

$ bootleg -d -e '(html "<div><h1>heading</h1><p>body</p></div>" :data :hiccup)'
[:div {} [:h1 {} "heading"] [:p {} "body"]]

plexus14:12:52

This is great, thanks @retrogradeorbit and @borkdude. And another tool in the clojure+graal space to add my tool belt :)

retrogradeorbit14:12:31

I ❀️ graal.

borkdude14:12:50

@plexus Another blog topic πŸ˜‰

borkdude14:12:03

HTML options for Clojure in the command line πŸ˜‰

plexus14:12:06

reading from *in* would be great, as well as cleaning up the hiccup (removing the empty property maps, also I don't know if you already roll classes/id into the keyword)

retrogradeorbit14:12:08

though id be struggling without @borkdudes amazing work

retrogradeorbit14:12:23

classes and ids are rolled in

plexus14:12:30

yes, I will likely mention this in a blogpost before the week is out

plexus14:12:34

ok that's great!

retrogradeorbit14:12:44

Im not actually generating the markup. so the {} are a relic of the downstream converters

retrogradeorbit14:12:18

there is some magic that I need to do to get those converters to behave. they are a little, um, warty?

plexus14:12:35

I guess one can always prune them with bb

retrogradeorbit14:12:47

thus the full round trip test suite.

retrogradeorbit14:12:13

I could prune them out I suppose. πŸ€”

retrogradeorbit14:12:24

I'll look into it for next release

retrogradeorbit14:12:04

clojure.walk is exposed inside bootleg. let me see if I can hack them out on the command line...

retrogradeorbit14:12:59

bootleg -d -e '(->> (html "") (clojure.walk/postwalk #(if (and (vector? %) (= {} (second %))) (vector (concat [(first %)] (subvec % 2))) %)))'

plexus14:12:42

I don't need it right now so don't do it for me πŸ™‚ just thinking out loud that it would be nice.

borkdude14:12:43

clojure.walk is also now in bb since a couple of releases

plexus14:12:16

since I think converting to Hiccup will mainly be used when people want to copy some outside example into their code

borkdude14:12:25

I have used a webservice for that before, e.g. http://html2hiccup.buttercloud.com

borkdude14:12:57

and I think Cursive does it now as well, when you paste HTML into Clojure (never tried it)

plexus14:12:40

emacs does it too, I actually wrote the package for that. But CLI tools can be used by anyone regardless of their editor.

plexus14:12:19

cat project.clj | bb '(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *in*))] {:paths (into source-paths resource-paths) :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))} ) ' | jet --pretty > deps.edn

borkdude14:12:00

@plexus Awesome idea, if it wasn't for the stupid read-eval in my project.clj πŸ™‚

borkdude14:12:52

I should probably remove that anyway

borkdude14:12:05

In clj-kondo I used selmer to update the version

plexus14:12:13

cat project.clj | sed -e 's/#=//g' -e 's/[email protected]//g' -e 's/~//g' | bb '(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *in*))] {:paths (into source-paths resource-paths) :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))} ) ' | jet --pretty > deps.edn

borkdude14:12:33

wow, that worked πŸ™‚

borkdude14:12:19

totally awesome, feel free to add it to the "gallery"

borkdude14:12:36

@plexus without sed:

$ cat project.clj | bb -io '(str/replace (str/join "\n" *in*) "#=" "")' | bb '(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *in*))] {:paths (into source-paths resource-paths) :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))})' | jet --pretty

plexus14:12:44

I like the sed version better πŸ™‚ no need to snub the classics

borkdude14:12:21

maybe even nicer:

cat project.clj | bb -io --stream '(str/replace *in* "#=" "")'
yeah, that's fine πŸ™‚

borkdude14:12:18

$ cat project.clj |
sed -e 's/#=//g' -e 's/[email protected]//g' -e 's/~//g' |
bb '
(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *in*))]
  {:paths (into source-paths resource-paths) :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))})' |
jet --pretty
a little bit indented etc.

borkdude14:12:58

a total hack, I love it

plexus14:12:23

that's what I'm best at, just random hacks

borkdude22:12:19

I wonder what day it is 10 days from now:

$ ./bb '(-> (java.time.LocalDate/now) (.plusDays 10) (.getDayOfWeek) (str))'
"FRIDAY"

borkdude22:12:15

it seems either plus or minus is redundant in this Java API πŸ˜‰

$ bb '(-> (java.time.LocalDate/now) (.plusDays -1)  (str))'
"2019-12-16"