Fork me on GitHub
#babashka
<
2020-09-30
>
danielgrosse06:09:33

When I run a script using the bootleg pod, I now get a NullpointerException with the following error:

----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: ***/static_website.clj:1:1

----- Context ------------------------------------------------------------------
1: (ns static-website.core
   ^--- 
2:  (:require [babashka.pods :as pods]))
3: 
4: (pods/load-pod "bootleg")
5: (require '[pod.retrogradeorbit.bootleg.utils :as utils])
6: 

----- Stack trace --------------------------------------------------------------
pod.retrogradeorbit.net.cgrand.xml - ***/static_website.clj:1:1
static-website.core                - ***/static_website.clj:4:1

danielgrosse06:09:51

An verbose call with the classpath and main flag brought this additional info:

clojure.lang.ExceptionInfo: null
{:type :sci/error, :line 1, :column 1, :message nil, :callstack #object[clojure.lang.Delay 0x4d51040 {:status :ready, :val ((#object[clojure.lang.AFunction$1 0x39a2d783 "[email protected]"] "bootleg") (#object[clojure.lang.AFunction$1 0x1c52cd9 "[email protected]"] document? "Document nodes are a parsing impelentation details and should never leak\n   outside of it." [x] (= :document (:type x))))}], :file "***/src/static_website/core.clj", :locals {}}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:104)

danielgrosse06:09:35

As bootleg hadn't released a new version I suppose, bb had made a change here.

borkdude06:09:53

Does it work with bootleg when used not as a pod though? I’m not aware of changes to the pod protocol that would break existing pods

Crispin07:09:20

bootleg does push over a bunch of macro code that could be affecting things

borkdude06:09:47

Maybe a question for @retrogradeorbit

Crispin06:09:09

Hey what version of bb and bootleg is this with. I will try and reproduce.

Crispin07:09:48

@danielgrosse is it possible to get a boiled down reproduction example that triggers the NPE?

danielgrosse07:09:56

I run it with latest released version of bb and bootleg. It occurs with the example from the bootleg repo. But I try to setup something simple

Crispin07:09:05

bootleg example should be sufficient. Which example?

Crispin07:09:27

some how x is going in nil here by the looks

Crispin07:09:16

I did have issues when podifying with passing metadata over the pod boundary. Turns out clojure's zipper implementations rely on embedded metadata in the data structure for functioning. Thought I had dealt with those issues (by keeping the zipper calls on the babashka side by sending that code over to be evaluated bb side). Maybe not. Will need to test to see.

borkdude09:09:15

@danielgrosse Found the bug. It's an issue with babashka.pods and bb 0.2.1. For now use 0.2.0 with the bootleg pod

hugod09:09:04

Babashka seems to use nio.FileVisitor and FileVisitResult to build uberjars, so I thought it would be possible to use FileVisitor with bb, but apparently not. Would it be cheap to add these, since they are already used internally?

borkdude09:09:45

@danielgrosse Now added bb 0.2.0 to glam:

$ glam install org.babashka/[email protected]
$ bb --version
babashka v0.2.0
so you can easily switch per shell: https://github.com/borkdude/glam

borkdude09:09:46

@hugod I guess so. How would the interop look? I'm not sure if the interpreter already can handle that (if reify etc is involved)

hugod09:09:38

the uberjar code is representative of what I was trying to do. Without reify though I guess this is a non-starter.

borkdude09:09:49

The thing is, reify does work, but only for protocols at the moment

borkdude09:09:09

and some Clojure interfaces which create objects that only work with Clojure, not when being passed back to Java

borkdude09:09:39

if you want to have compatibility with both JVM and bb, use reader conditionals in that case and rewrite using map, filter, etc

hugod09:09:07

ok, thanks for explaining

borkdude09:09:14

at interpretation time we can't create new classes which is a fundamental limitation of graalvm, that's basically the problem. although we could maybe have support for a limited set of interfaces / superclasses. feel free to post an issue at bb and it might be addressed in the future

hugod09:09:02

It’s a testament to bb that you don’t have to think about the graalvm limitations most of the time.

borkdude09:09:47

I think we could add support for it like this:

user=> (defn sci-reify [clazz impls] (when (= clazz Object) (reify Object (toString [this] ((get impls :toString) this)))))
#'user/sci-reify
user=> (reify Object (toString [this] "foo"))
#object[user$eval176$reify__177 0x6a0ac48e "foo"]
user=> (sci-reify Object {:toString (fn [_] "foo")})
#object[user$sci_reify$reify__173 0x38704ff0 "foo"]

borkdude09:09:13

so then we will make some pre-defined supported interfaces in sci that you can use with reify

hugod09:09:24

are protocol extensions a similar problem?

borkdude09:09:09

yes, but because they are not passed back to Java I have enough hooks to work around that

hugod09:09:21

so protocol extensions should work? I tried extending http://clojue.java.io/Coercions onto nio.Path but get a nil pointer exception.

borkdude09:09:56

they should work if babashka knows about that protocol. so far it only knows about Datafiable, Navigable, and interfaces IAtom, IDeref

borkdude09:09:17

could add support for that protocol. issue welcome

borkdude09:09:30

Protocols are implemented as multimethods in sci (under the hood), since protocols also do compile time things, but multimethods don't

borkdude10:09:44

so all functions that deal with these protocolized objects should be patched as well, to check for a sci-protocolized objects and if not, then defer to the old function

borkdude10:09:01

major workarounds, but they do work :)

hugod10:09:45

the flexibility the workarounds give seems worth it

borkdude10:09:50

usually the fallback is implemented using a :default multimethod. I was lucky that this all kinda worked

borkdude13:09:03

@hugod Are you on mac or linux or do you build bb from master?

hugod13:09:17

on mac. using 0.2.1 at the moment, though I have built bb myself as well.

borkdude13:09:48

This script now works:

(java.nio.file.Files/walkFileTree
 (.toPath (java.io.File. "."))
 (reify java.nio.file.FileVisitor
   (preVisitDirectory [this p attrs]
     (prn :pre-dir (str p))
     java.nio.file.FileVisitResult/CONTINUE)
   (postVisitDirectory [this p attrs]
     (prn :post-dir (str p))
     java.nio.file.FileVisitResult/CONTINUE)
   (visitFile [this p attrs]
     (prn :file (str p))
     java.nio.file.FileVisitResult/CONTINUE)))

borkdude13:09:16

I've yet to write a unit test for it

hugod13:09:52

I don’t know how you respond so quickly to everything! That’s going to be very useful.

borkdude13:09:53

well I've already had the idea for a long time and I've got a day off today ;)

borkdude13:09:50

Pushed to master. I'll probably release 0.2.2 somewhere today

borkdude09:09:43

can you post a Clojure example?

borkdude10:09:19

@danielgrosse Fix pushed to master. New bb binaries will be available in #babashka_circleci_builds in a few minutes

borkdude10:09:24

Seems to work! @retrogradeorbit @danielgrosse

$ glam install org.babashka/[email protected] retrogradeorbit/bootleg --force
$ cat /tmp/test_pod.clj
(ns mybbscript
  (:require [babashka.pods :as pods]))

(pods/load-pod "bootleg")
(require '[pod.retrogradeorbit.bootleg.utils :as utils])

(-> [:div
     [:h1 "Using Bootleg From Babashka"]
     [:p "This is a demo"]]
    (utils/convert-to :html))
$ bb /tmp/test_pod.clj
"<div><h1>Using Bootleg From Babashka</h1><p>This is a demo</p></div>"

borkdude10:09:21

Hmm I guess bootleg could also be useful in the context of the new web server

borkdude10:09:53

To serve html

Crispin11:09:53

somewhere in there is a web development framework waiting to get out. Just needing someone to tie it together...

mmz12:09:00

Interested to try the bootleg pod soon 🤓

Daniel Tan10:09:25

i think babashka might just replace python for me lmao

motform12:09:37

Is there a way to do oauth1 auth in bb? is it preferred to shell out? (the use case is the twitter api)

borkdude12:09:03

Not sure, but there is an http client and http server now, so it should be possible using those maybe?

borkdude12:09:24

and if you figure it out, an example gist would be highly appreciated

motform12:09:48

that’s what I figured, yeah, I’ve never really used oauth before but I’ll keep at it!

jeroenvandijk13:09:36

I would say bb has all the ingredients with the client and server

borkdude15:09:15

The next bb will support reifying http://java.io.FileFilter and java.nio.file.FileVisitor. What other classes are useful to reify in Clojure?

borkdude15:09:29

@nate_clojurians When you were working on the fs lib for bb, there was something like this you were running into right?

borkdude15:09:29

@nate_clojurians When you were working on the fs lib for bb, there was something like this you were running into right?

borkdude17:09:52

well, as of today yes, in #babashka_circleci_builds ;)

borkdude17:09:26

worth a try again with that lib I'd say

borkdude18:09:56

Hmm, I guess it's FilenameFilter, and there's FileFilter. I'll add both

borkdude18:09:19

pushed to master. fs seems to work now. btw this is the maintained repo: https://github.com/clj-commons/fs

cldwalker18:09:00

Exciting to hear about reify!

borkdude18:09:45

It seems raynes/fs is now also runnable with bb master:

export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {clj-commons/fs {:mvn/version "RELEASE"}}}')

$ clojure -M:bb-local -e "(require '[me.raynes.fs :as fs]) (fs/glob \"*.md\")"
(#object[java.io.File 0x78f2bb1b "/Users/borkdude/Dropbox/dev/clojure/babashka/CHANGELOG.md"] #object[java.io.File 0x651c826c "/Users/borkdude/Dropbox/dev/clojure/babashka/README.md"] #object[java.io.File 0x10323056 "/Users/borkdude/Dropbox/dev/clojure/babashka/CONTRIBUTING.md"])

borkdude18:09:20

Yes. Note that reify will work only with a selection of pre-defined classes, so if you need more, let me know

borkdude18:09:27

Alrighty, pushed 0.2.2 with the fix for bootleg and some nice reify improvements: https://github.com/borkdude/babashka/blob/master/CHANGELOG.md#v022-2020-09-30

borkdude19:09:22

$ which bb
/Users/borkdude/.glam/repository/org.babashka/babashka/0.2.2/bb
$ bb --version
babashka v0.2.2
$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {clj-commons/fs {:mvn/version "1.5.2"}}}')
$ time bb -O -e '(ns foo (:require [me.raynes.fs :as fs])) (map str (fs/glob "*.md"))'
"/Users/borkdude/Dropbox/dev/clojure/babashka/CHANGELOG.md"
"/Users/borkdude/Dropbox/dev/clojure/babashka/README.md"
"/Users/borkdude/Dropbox/dev/clojure/babashka/CONTRIBUTING.md"
bb -O -e   0.03s user 0.02s system 92% cpu 0.057 total