Fork me on GitHub
#babashka
<
2019-12-10
>
dromar5602:12:56

Most of my scripts need json, http calls and possibly aws stuff. How difficult would it be to add e.g. cheshire, clj-http and aws-api?

sogaiu02:12:00

@dromar56 fwiw, it may be some hours before you get an answer

dromar5602:12:46

ahah yeah I'm used to it, living in Singapore, not exactly the most common timezone in clojurians πŸ˜›

dromar5602:12:19

It seems simple enough to modify reflection.json and :classes in main.clj, but I'm guessing it depends if those libs play well with graalvm

sogaiu02:12:42

yes -- the difficulty varies. on a related note, iiuc, borkdude is working on some method of loading code at runtime. it may be that some things will be doable via whatever he comes up with. i'm confident he'll have a better answer than my attempt here πŸ™‚

borkdude07:12:16

@dromar56 Cheshire works with Graal, I think it would be nice to add

borkdude07:12:43

Http: slurp works, but maybe adding clj-http or something would be nice. Of course you can also shell out to curl or use bash and bb inline

borkdude07:12:49

aws seems a bit too specific for a general purpose tool, maybe that deserves its own cli

dromar5607:12:14

Adding cheshire would be enough to use curl with json payloads in a relatively simple way

borkdude07:12:42

@dromar56 have you tried jet though?

borkdude07:12:59

It’s a complementary tool to babashka

dromar5607:12:27

I haven't, but I'm more interested in using babashka by itself than calling it inline with bash. I also don't want to ask my coworkers to install/learn more than one extra tool

borkdude07:12:35

Yep, understandable. I’m adding Cheshire in the next release then, probably this week

dromar5607:12:04

Thanks! That'd be awesome

borkdude08:12:13

It looks like ClojureD is going to be a devops + shell scripting feast. @rahul080327 with bob, dundalek with closh and me with babashka all in the same room after each other πŸ™‚

borkdude08:12:14

@dromar56 added cheshire: <https://github.com/borkdude/babashka/releases/tag/v0.0.40>

borkdude09:12:35

$ bb '(json/parse-string "{\"foo bar\": 1}" (fn [x] (keyword (str/replace x " " "_"))))'
{:foo_bar 1}

sogaiu20:12:50

looks good here too:

$ ./bb '(json/parse-string "{\"foo bar\": 1}" (fn [x] (keyword (str/replace x " " "_"))))'
{:foo_bar 1}

nate17:12:48

going to try adding clj.java-time. first step is to compile master with no changes.

nate17:12:30

$ java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

nate17:12:15

$ ~/graalvm-ce-java11-19.3.0/bin/native-image --version
GraalVM Version 19.3.0 CE

borkdude18:12:16

that seems alright

nate18:12:11

Error: Error parsing reflection configuration in /home/nate/projects/babashka/reflection.json:
Could not resolve java.lang.UNIXProcess for reflection configuration. To allow unresolvable reflection configuration, use option -H:+AllowIncompleteClasspath
Verify that the configuration matches the schema described in the -H:PrintFlags=+ output for option ReflectionConfigurationFiles.
com.oracle.svm.core.util.UserError$UserException: Error parsing reflection configuration in /home/nate/projects/babashka/reflection.json:
Could not resolve java.lang.UNIXProcess for reflection configuration. To allow unresolvable reflection configuration, use option -H:+AllowIncompleteClasspath
Verify that the configuration matches the schema described in the -H:PrintFlags=+ output for option ReflectionConfigurationFiles.

nate18:12:23

(just a snippet)

borkdude18:12:55

oh, you need: ~/graalvm-ce-java8-19.3.0/bin/native-image --version

borkdude18:12:58

the java 8 one

borkdude18:12:25

there's an issue for JDK 11, but it's not yet finished

borkdude18:12:43

they changed a bunch of classes (the pipe signal stuff)

nate18:12:02

that's cooking much better...

nate18:12:22

awesome, that worked, now to try adding clj.java-time and see what blows up

nate19:12:21

well, it does compile in graalvm

nate19:12:28

but then I get an odd error

nate19:12:37

here's running it via the uberjar:

$ java -jar target/babashka-0.0.41-SNAPSHOT-standalone.jar -e '(jt/local-time "HH:mm" "08:14")'
#object[java.time.LocalTime 0x70d63e05 "08:14"]

nate19:12:08

and via the binary:

$ ./bb -e '(jt/local-time "HH:mm" "08:14")'
java_time.graph.Types cannot be cast to java.lang.Comparable [at line 1, column 1]

nate19:12:43

ah, there's a verbose flag

nate19:12:47

$ ./bb --verbose -e '(jt/local-time "HH:mm" "08:14")'
clojure.lang.ExceptionInfo: java_time.graph.Types cannot be cast to java.lang.Comparable [at line 1, column 1]
{:type :sci/error, :row 1, :col 1, :message "java_time.graph.Types cannot be cast to java.lang.Comparable [at line 1, column 1]"}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:67)
    sci.impl.interpreter$eval_call.invokeStatic (interpreter.cljc:357)
    sci.impl.interpreter$interpret.invokeStatic (interpreter.cljc:391)
    sci.impl.interpreter$eval_form.invokeStatic (interpreter.cljc:423)
    sci.impl.interpreter$eval_string_STAR_.invokeStatic (interpreter.cljc:443)
    sci.impl.interpreter$eval_string.invokeStatic (interpreter.cljc:450)
    sci.core$eval_string.invokeStatic (core.cljc:131)
    babashka.impl.utils$eval_string.invokeStatic (utils.clj:5)
    babashka.main$main$fn__12674.invoke (main.clj:303)
    babashka.main$main.invokeStatic (main.clj:296)
    babashka.main$main.doInvoke (main.clj:187)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:665)
    babashka.main$_main.invokeStatic (main.clj:336)
    babashka.main$_main.doInvoke (main.clj:336)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    babashka.main.main (:-1)

nate19:12:44

is that error recognizable or is it uncharted territory?

borkdude21:12:27

@ It could be that java-time uses reflection of some sort. It could be worth vendoring its source code and plug in some (set! *warn-on-reflection* true)s if they are not there yet

nate21:12:13

ok, thanks, I'll try that

borkdude21:12:52

don't know yet how I feel about this library. using clojure in the name is something that's culturally only reserved for official clojure libs. the single segment namespace name... and the code seems quite complex for something that should be relatively straightforward?

nate21:12:27

yeah, I agree

nate21:12:04

I thought it was going to be a thinner wrapper, but then I dove into the source this morning and saw that there's a decent amount of cleverness

nate21:12:24

I've used the library many times after switching away from clj-time

borkdude21:12:57

writing pure Java is maybe not so horrible?

user=> (java.time.LocalTime/of 10 0)
#object[java.time.LocalTime 0x487db668 "10:00"]
user=> (java.time.LocalTime/parse "10:00")
#object[java.time.LocalTime 0x487db668 "10:00"]

borkdude21:12:09

we can add those classes to bb

borkdude21:12:03

ok, it's slightly more horrible than writing in clojure πŸ™‚

borkdude21:12:01

I needed to parse a RFC_1123 date time, which I could only do in java.time, and then I had to convert it to a java.util.Date and then to a joda time date because the rest of our code expects that πŸ™‚

borkdude21:12:11

(->
     date-text
     (java.time.ZonedDateTime/parse java.time.format.DateTimeFormatter/RFC_1123_DATE_TIME)
     (.toInstant)
     (java.util.Date/from)
     (tc/from-date))

borkdude21:12:11

vs:

(-> (jt/zoned-date-time (jt/formatter :rfc-1123-date-time) date-text)
        (jt/to-java-date)
        (tc/from-date))

borkdude21:12:09

java_time.graph.Types cannot be cast to java.lang.Comparable
It seems that thing doesn't implement Comparable: https://github.com/dm3/clojure.java-time/blob/707f16daf19e04cf182cc532a2cce8db170ea178/src/java_time/graph.clj#L26 But not sure why this manifests only in native image

nate21:12:37

didn't know if this was a new error, sounds like it is

borkdude21:12:40

could also be the result of some reflection thing, so I would first try to check that

nate22:12:16

cool, I'll open an issue to track progress, and when I get to trying out the reflection stuff, I'll ping it

nate22:12:19

thanks for your help!

borkdude22:12:35

I forked the repo and put :global-vars {*warn-on-reflection* true} in project.clj. There doesn't seem to be any reflection going on

borkdude22:12:54

ok, worth some research.

borkdude22:12:27

Small quick POC / repro:

(ns time.core
  (:gen-class)
  (:require [java-time :as jt]))

(defn -main [& args]
  (println (jt/local-time))
  (println (jt/local-time "HH:mm" "08:14")))
$ native-image --no-server --no-fallback --initialize-at-build-time -jar target/time-0.1.0-SNAPSHOT-standalone.jar
$ ./time-0.1.0-SNAPSHOT-standalone
#object[java.time.LocalTime 0x4ce83ffe 23:13:22.667300]
Exception in thread "main" java.lang.ClassCastException: java_time.graph.Types cannot be cast to java.lang.Comparable

borkdude22:12:06

Btw, this library also seems popular and based on Java time: https://github.com/juxt/tick

borkdude22:12:14

It also seems to have quite good docs: https://juxt.pro/tick/docs/index.html

borkdude22:12:30

Might be worth researching also so we can make an informed choice

borkdude22:12:04

@ Have you tried tick with graal?

nate23:12:44

That looks really nice, I'll check it out

dominicm08:12:55

Not at all. I don't think it uses reflection though, I'm pretty sure it uses code generation.

dominicm08:12:12

There might be some accidental reflection though

borkdude08:12:11

Spec is required in the core ns which might cause the locking trouble

dominicm08:12:32

I might be able to influence that away. There's not any real reason for that I think.

borkdude10:12:18

Yeah, first worth a try without any changes. Might just work. Else we could also try to vendor it

borkdude10:12:32

Adding support for some of the java.time.* classes in bb is a no-brainer in either case