Fork me on GitHub
#babashka
<
2020-05-14
>
mbjarland14:05:29

quick question. Some of my babashka scripts need to deal with complex xml data structures and doing this directly via clojure/data.xml which is included in bb can be somewhat painful. In other words, I would like to use clojure.data.zip.xml namespace and specifically the xml-zip function. I tried adding org.clojure/data.zip to the classpath using add-classpath , this seems to work but then when requiring clojure.data.zip.xmlwhich in turn requires clojure.data.zip which in turn requires clojure.zipI get an exception saying:

Could not require clojure.zip. [at clojure/data/zip.clj, line 11, column 1]
but is not clojure.zip supposed to be part of clojure? i.e. already bundled into babashka? As a side note I find the name space naming and package dependency setup with clojure.zip vs clojure.data.zip et al somewhat unfortunate and confusing. Maybe this is where I fell off the trail?

borkdude14:05:41

I don't think clojure.zip is in babashka yet, but a PR for that is welcome.

borkdude14:05:47

or an issue

mbjarland14:05:28

but in jvm clojure clojure.zip is part of clojure and not a separate dependency right?

mbjarland14:05:19

ok, figured it would be there. I stand corrected.

borkdude14:05:47

well it's not all automatically using clojure core on the JVM and that's it, it takes a little more work sometimes

borkdude14:05:22

and clojure.zip has never been requested before - I'm generally adding stuff by request or stuff I'm using myself already

mbjarland14:05:37

if I add clojure itself using add-classpath then the require works

😮 1
borkdude14:05:51

that's amazing 🙂

mbjarland14:05:01

that is weirdly recursive...

mbjarland14:05:24

now my head hurts

borkdude14:05:04

peak computer science...

borkdude14:05:24

zippers inside an interpreter interpreting the same language it was built in..

borkdude14:05:28

but I think it would be good to add as a bundled namespace

mbjarland14:05:33

I actually ended up writing my own (defn xml-> [d & fns]...)because I got tired of navigating all the :tag and :content stuff returned from xml parse, but yeah, would be nice not to reinvent the weel

mbjarland14:05:19

I can take a look at see if I can understand the required changes to add it

borkdude14:05:26

@mbjarland take a look at babashka.impl.xml and then do the same for clojure.zip under babashka.impl.clojure.zip

borkdude14:05:18

also take a look at https://github.com/borkdude/babashka/blob/master/doc/dev.md how to get started (git pull recursive)

mbjarland14:05:44

not sure if clojure.zip strictly speaking belongs under xml as I think zippers can be used for other things though

borkdude14:05:04

I wasn't saying that

borkdude14:05:27

xml is just an example of how to add a namespace

borkdude14:05:33

actually...

borkdude14:05:40

is clojure.zip also part of clojurescript?

mbjarland14:05:47

but xml is under feature-xml, so would you lika this to be a new feture then ?

borkdude14:05:54

then maybe it should belong in sci... not sure...

mbjarland14:05:30

seems to be yes

borkdude14:05:56

I don't think I've ever used it in CLJS though

borkdude14:05:02

let's just not add it there yet

borkdude14:05:24

clojure.zip is just some pure functions, that doesn't take up a lot of space. xml does because of the dependency on all kinds of Java stuff

mbjarland14:05:41

yes, just a bunch of pure functions

borkdude14:05:51

yeah, no feature flag needed

borkdude14:05:10

we can also measure the difference in binary size as reported in #babashka_circleci_builds

mbjarland14:05:18

including a comment which lets us know that Rich still runs on a macbook:

(comment

(load-file "/Users/rich/dev/clojure/src/zip.clj")
(refer 'zip)

borkdude14:05:33

well, at least 12 years ago?

mbjarland14:05:41

ok so where does that leave us, not in sci, not as a feature but rather directly into bb

borkdude14:05:27

yes. just like tools.cli, http://clojure.java.io, etc.

mbjarland14:05:05

ok will take a look

borkdude14:05:12

I think pods might take the place of feature flags eventually, but only time will tell

borkdude14:05:05

There is an issue about data-readers if anyone wants to chime in with some insights: https://github.com/borkdude/babashka/issues/419

snorremd15:05:17

Hello. Please excuse my ignorance, but is there any way to pretty-print the result of running an expression in sci from JavaScript? I recognise that I could probably achieve this if I had used ClojureScript and then bound the clojure.pprint function to a function inside sci. I'm thinking about embedding sci in my Gatsby blog to enable interactive Clojure-snippets.

snorremd15:05:29

E.g.

sci.evaluateString("(with-out-string (pprint {:a 1 :b 2}))");

borkdude15:05:43

@snorremd clojure.pprint isn't part of the default sci library, but as you say you can bind it yourself and then compile to a JS target

snorremd15:05:53

Right, I suspected so. Thank you for the quick answer and for creating sci. Will have fun playing with it.

borkdude15:05:44

sci's with-out-str picks up on sci/out

snorremd15:05:13

Good to know.

borkdude15:05:32

There is also sci/in and sci/err

borkdude15:05:44

which are all connected to the print-related or read-related functions

snorremd15:05:15

Will definitely take a closer look at the documentation. Now the hard part is only figuring out the cljs compilation and bundling in gatsby pipeline. But this should get me going on the sci part at least. 👍 Thank you again.

borkdude15:05:16

@snorremd Take a look at script/compile-js and script/release-js. It's relatively straightforward.

borkdude15:05:52

And the same code works on nodeJS and in the browser, there is no real difference

borkdude15:05:37

The javascript API may be a bit behind the CLJS api, feel free to comment on that if you need anything

borkdude15:05:32

@snorremd I had to take a look myself as well, the main gist of it is in min.js.edn which is used in a CLJS setting

borkdude15:05:26

this might be easier with the new CLJS version, but I haven't looked at any of that

snorremd15:05:30

Yeah, I read a post by David Nolen that suggests as much: https://clojurescript.org/guides/webpack.

borkdude16:05:58

well maybe we should migrate to that then.

borkdude16:05:02

PR welcome 🙂

snorremd21:05:40

Hi! Saw you retweeted my tweet about interactive clojure support via sci. Just wanted to let you know I solved the pretty-printing thing by making my own clojurescript namespace wrapping sci cwith some minimal stuff like catching exceptions and pretty-printing the results. See the code here: https://github.com/snorremd/snorreio/blob/master/src/cljs/sci_wrapper.cljs

borkdude21:05:01

that seems a fairly minimal wrapper, cool that it worked!

snorremd21:05:44

Essentially I tried bundling the minified version of my wrapper with and without the new bundle option in clojurescript compiler. But it seems that no matter you get compiled output that webpack in Gatsby don't like. So for JavaScript + webpack projects simply serving the compiled and optimized clojurescript as a static file and using ugly side-effecting script loading in react should work. 👍

borkdude22:05:39

Aha. So the wrapper sci for npm is using also doesn't work?

snorremd22:05:22

It works considering it exposes sci as a commonjs module, but webpack (at least with the default bundling settings) consumes so much resources that at least github actions ran out of memory.

snorremd22:05:20

Anyway, I think my new approach is good enough. Again, thanks. 🙂

borkdude22:05:08

@snorremd If you want to capture output from prn etc, you could include this:

(let [sb (goog.string/StringBuffer.)] (sci/binding [sci/out sb] [(sci/eval-string "(do (prn :foo) 1)") (str sb)]))
[1 ":foo\n"]

borkdude22:05:19

you can then also display what people tried to print

snorremd22:05:38

Oh good idea! Adding it to my github issues list on my blog.

jeroenvandijk17:05:27

@snorremd the cljs version should do all of the things the clj version can do if you remain in the clojure domain. That was my experience at least. Only when you go crazy on interop you might find some edge cases (https://github.com/borkdude/sci/pull/286)

jeroenvandijk19:05:18

I’ll not keep it under my name, but it’s to show others this possibility

jeroenvandijk19:05:57

Maybe it makes sense to extend this playground with cool libraries such as Datascript and others. So people have a webtool?

jeroenvandijk19:05:42

Other interesting libraries Meander, core logic, a lot more? If it’s Clojurescript, I think they could also be integrated with Sci, right?

jeroenvandijk19:05:16

Clojure Toolbox, live in your browser? 🙂

jeroenvandijk19:05:45

Can we have pods for Sci clojurescript? 🙌 hahah

mbjarland19:05:30

FYI @borkdude I added a pr for clojure.zip including library support for clojure.data.zip

jeroenvandijk19:05:07

@mbjarland Wouldn’t that be useful at Sci level? Isn’t jvm specific right?

mbjarland19:05:53

we had a chat about that above with @borkdude - he seemed to want to leave it outside of sci for now. I’m agnostic, just want access to the clojure.data.zip.xml goodies

jeroenvandijk20:05:08

ok makes sense, I was just curious

jeroenvandijk20:05:28

I’m a bit (too?) enthusiastic about the Sci repl now

mbjarland20:05:19

16:16 borkdude then maybe it should belong in sci... not sure...
16:17 mbjarland seems to be yes 
16:17 borkdude I don't think I've ever used it in CLJS though
16:18 let's just not add it there yet
but I agree, it’s pure functions all the way so would not be a problem adding/moving it

jeroenvandijk20:05:48

makes sense to wait i guess

mbjarland20:05:07

yeah, I have less than perfect visibility into the sci project and/or the reasoning for the above. I’m happy to change the pr or create a new one if we want it moved

borkdude20:05:42

@jeroenvandijk Feel free to make a PR with datascript and add the gist to the example list at the bottom

borkdude20:05:10

other libs also welcome. it's a demo so adding more libs and examples to it is fine

jeroenvandijk20:05:42

cool, any preference for structure?

jeroenvandijk20:05:57

or shall i just start adding and wait for feedback 🙂

borkdude20:05:05

feel free to make something nice out of it 😉

borkdude20:05:00

@jeroenvandijk pods for CLJS: I think that is possible. the pod lib already supports the JVM. CLJS support can be added there as well. nREPL also works from JS based editors, so I don't see why it wouldn't work for pods

jeroenvandijk20:05:33

it would be BIG 🙂

jeroenvandijk20:05:15

think observablehq for clojurescript

jeroenvandijk20:05:58

Load any clojurescript library directly in your browser

jeroenvandijk20:05:39

ok i’ll think about it how this can work if you don’t beat me to it 🙈

borkdude20:05:18

I think you have more ideas about the JS side of things than I do at the moment 🙂

jeroenvandijk20:05:43

haha np, I’ll show you the power of your pods 😅

borkdude20:05:58

@mbjarland btw if you need a binary without building yourself, they are available pre-release in #babashka_circleci_builds

borkdude20:05:19

One issue I want to solve before a next release is the following one: https://github.com/borkdude/babashka/issues/419#issuecomment-628671008 So the loading of data reader functions. The problem is: it's technically possible, but scanning an entire classpath at the start of each script may waste some startup time (8-15ms).

mbjarland20:05:36

@borkdude no worries I built them to test the pr locally before pushing