Fork me on GitHub
#babashka
<
2023-02-25
>
Benjamin10:02:46

Does virtual thread support mean I can try project loom on the next bb release?

borkdude10:02:31

Yes. But you can also try it now:

bash <(curl ) --dev-build --dir /tmp

Benjamin10:02:11

re-enacted your initial example

borkdude10:02:59

15s is a bit long, maybe a bit old hardware?

borkdude10:02:05

still, 1M threads :)

Benjamin10:02:02

hm 12.5 seconds without ffmpeg running

Benjamin10:02:59

I am trying to override clojure.core.async.impl.exec.threadpool thread-pool-executor can I just call intern? I guess the symbols need to be exposed? From async protols only ReadPort is defined I think

borkdude10:02:45

yeah that won't work I think

borkdude10:02:52

does core.async has a function to set the threadpool executor? clojure itself has this:

user=> (apropos "executor")
(clojure.core/set-agent-send-executor! clojure.core/set-agent-send-off-executor!)

Benjamin11:02:01

I don't think so. There is a a defonce calling thread-pool-executor

borkdude11:02:03

Might be good to request a function for this that explicitly supports this on ask.clojure. It seems it's not well supported currently, barring some hacks changing an impl namespace: https://stackoverflow.com/questions/18779296/clojure-core-async-any-way-to-control-number-of-threads-in-that-go-thread

borkdude11:02:30

Doing the intern in SCI will not have the desired effect since the pre-compiled functions don't see the change

borkdude11:02:36

Hmm, we could hack this by adding a watcher on the SCI var that also changes the core var, but even then it might not work (as you can see in the SO answer, the order of loading things matters)

Benjamin11:02:32

also alter-var-root is the same as intern right

borkdude11:02:52

if the var already exists, yes

Benjamin13:03:33

that's cute

Benjamin12:02:18

https://faster-than-light-memes.xyz/jacking-nbb.html You can now define a cider jack in cmd via .dir-locals.el, or as parameter to a custom function. I was just using this

((nil
  (cider-jack-in-cmd . "/tmp/bb nrepl-server")))
so cider-jack-in starts a dev bb in that dir

ūüĎć 1
Istv√°n Karaszi15:02:12

I am working on a notification tool which is technically a scraper that would fetch HTML pages parse the data with XPath queries and send out notifications. did I get it correctly that currently there is no way to use XPath with Babashka?

borkdude15:02:37

@UJSBQNUG6 Dealing with HTML can be done via the bootleg pod: https://github.com/babashka/pod-registry/blob/master/examples/bootleg.clj It has a lot of HTML library stuff, e.g. it has enlive

Istv√°n Karaszi15:02:07

I saw that, but I could not find any XPath support there

borkdude15:02:51

I see. If you're open to using Node.js, maybe #C029PTWD3HR is also an option here. I'm sure there are lots of Node libraries that support Xpath

Istv√°n Karaszi15:02:17

sure, that is a good alternative

Istv√°n Karaszi15:02:59

I wanted to solve it w/ Babashka since this would be my first project with it

Istv√°n Karaszi15:02:04

and the experience was flawless so far ūüôā

borkdude15:02:27

Bootleg does support parsing the HTML very well though

Istv√°n Karaszi15:02:21

would you go in the hiccup way? or what would give me a similar query like interface?

Istv√°n Karaszi15:02:10

although I guess, I wouldn’t be able to query the HTML like, I want the <h3> with this content and all the next siblings until the next <h3>

borkdude16:02:28

I'm thinking...

borkdude16:02:33

Perhaps @U1QQJJK89 has an idea?

borkdude16:02:40

(the author of bootleg)

ūüĎć 1
borkdude16:02:57

Maybe I would do the hiccup processing manually or use enlive/select perhaps

Istv√°n Karaszi16:02:09

Warning: converting markup from :html to :hiccup lost 1 form                                                                                                                                                        
Cannot read EDN: ...
----- Error --------------------------------------------------------------------
Type:     java.lang.RuntimeException
Message:  Invalid token: /hirdetmeny?id=1931760
Location: /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:15:3

borkdude16:02:09

Hmm, @UJSBQNUG6 perhaps you can make a small reproduction of this.

Istv√°n Karaszi16:02:20

sure, give me a couple of minutes

borkdude16:02:52

another option could be to install a look like xidel which supports xpath: https://formulae.brew.sh/formula/xidel and then shell out from bb with (babashka.process/shell "xidel" ...)

borkdude16:02:14

So the issue is that the HTML gets transformed into EDN (hiccup) and this EDN is sent back to babashka as a string, but I think there may be some escaping issue

Istv√°n Karaszi16:02:09

So I might have found a bug?

borkdude16:02:52

I'm not sure :)

borkdude16:02:32

When I manually paste that string into an .edn file and parse it it's fine...

(clojure.edn/read-string (slurp "/tmp/foo.edn"))

Istv√°n Karaszi16:02:49

I believe the HTML is invalid

Istv√°n Karaszi16:02:53

<a href="/hirdetmeny?id=1931760" title="‚ÄěHejŇĎpapi X.-√°tmeneti t√∂rmel√©kes nyersanyagok" v√©dnevŇĪ b√°nya b√°ny√°szati tev√©kenys√©g√©re vonatkoz√≥ BO/16/7277-22/2016. sz√°m√ļ k√∂rnyezetv√©delmi enged√©ly m√≥dos√≠t√°sa">‚ÄěHejŇĎpapi X.-√°tmeneti t√∂rmel√©kes nyersanyagok" v√©dnevŇĪ b√°nya b√°ny√°szati tev√©kenys√©g√©re vonatkoz√≥ BO/16/7277-22/2016. sz√°m√ļ k√∂rnyezetv√©delmi enged√©ly m√≥dos√≠t√°sa</a>

Istv√°n Karaszi16:02:18

I believe this is the root cause

borkdude16:02:30

$ xidel \?SZO\=szeged --silent --extract //h3
Hirdetm√©ny√©t itt tudja bek√ľldeni!
Keresés a hirdetmények között
Tal√°latok

Istv√°n Karaszi16:02:27

so xidel could be a working solution, thank you!

borkdude16:02:10

It might work with enlive/select too, not sure:

(prn (enlive/select "<html></html>" #_(slurp "/tmp/foo.html") [:html]))

borkdude16:02:21

I can't figure out how enlive/select works, it returns an empty list

borkdude16:02:16

yeah, I'd go for xidel or move to nbb

Istv√°n Karaszi16:02:30

thank you, I’ll check all 3 options

borkdude16:02:14

you can shell out to xidel with:

(shell "xidel" ...)
If you want to get back a string of the output:
(:out (shell {:out :string } "xidel" ...))
etc

borkdude16:02:21

See babashka.process for details

Istv√°n Karaszi16:02:32

yes, but then I need to figure how to properly escape the args for xidel

Istv√°n Karaszi16:02:41

since XPath selectors are quite complex

borkdude17:02:04

escape how so?

Istv√°n Karaszi17:02:07

and I cannot see a good way to do the escaping with babashka.process/shell

Istv√°n Karaszi17:02:30

like what happens if I have both the quotes (`"` and ') in the argument?

Istv√°n Karaszi17:02:03

what would be great to be able to pass a data structure to babashka.process/shell and that would do the escaping automatically and I would not need to build a string

Istv√°n Karaszi17:02:28

["cmd", arg0, arg2, arg3]

Istv√°n Karaszi17:02:33

or something similar

borkdude17:02:36

wouldn't you have the same problem when you would call an xpath function in clojure? I don't see how that would be different than shelling out. the string arrives at xidel exactly how you wrote it, barring Java escaping rules

borkdude17:02:01

yes, this is already the case:

(shell "xidel" arg0 arg1 arg3)

Istv√°n Karaszi17:02:18

then there are no issues, I just could not find an example like that

borkdude17:02:40

then we need to improve the docs :)

ūüôā 1
Istv√°n Karaszi17:02:55

or I need to improve myself

Istv√°n Karaszi17:02:13

this somehow implies it

Istv√°n Karaszi17:02:48

@U04V15CAJ how could you get enlive working? for me it throws:

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: org.xml.sax.ContentHandler
Location: net/cgrand/xml.clj:11:3

----- Context ------------------------------------------------------------------
 7: ;   You must not remove this notice, or any other, from this software.
 8: 
 9: (ns net.cgrand.xml
10:   (:require [clojure.zip :as z])
11:   (:import (org.xml.sax ContentHandler Attributes SAXException XMLReader)
      ^--- Unable to resolve classname: org.xml.sax.ContentHandler
12:            (org.xml.sax.ext DefaultHandler2)
13:            (javax.xml.parsers SAXParser SAXParserFactory)))
14: 
15: (defstruct element :tag :attrs :content)
16: 

----- Stack trace --------------------------------------------------------------
net.cgrand.xml         - net/cgrand/xml.clj:11:3
net.cgrand.tagsoup     - net/cgrand/tagsoup.clj:12:3
net.cgrand.enlive-html - net/cgrand/enlive_html.clj:14:3
pnn.scrape             - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:2:3
pnn.core               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/core.clj:2:3
pnn.main               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/main.clj:2:3
user                   - <expr>:1:10

Istv√°n Karaszi17:02:19

okay, I guess I need to use pods, right?

borkdude17:02:25

(require '[pod.retrogradeorbit.bootleg.enlive :as enlive])

borkdude17:02:36

the same bootleg pod

Istv√°n Karaszi17:02:23

I believe I would simply switch to clojure

Istv√°n Karaszi17:02:39

nbb requires too much rewrites

borkdude17:02:26

yeah, of course that's also an option :)

Istv√°n Karaszi17:02:18

it was a nice experience with Babashka but unfortunately the HTML parsing made it impossible

Istv√°n Karaszi17:02:17

I would need to run tidy, or xidel from shell and that does not sounds too nice ūüėě

borkdude17:02:53

user=> (json/parse-string (:out (babashka.process/shell {:out :string} "xidel" "" "--silent" "--extract" "//h3" "--output-format" "json-wrapped")))
(["Hirdetm√©ny√©t itt tudja bek√ľldeni!" "Keres√©s a hirdetm√©nyek k√∂z√∂tt" "Tal√°latok"])

borkdude17:02:59

and with stdin:

user=> (json/parse-string (:out (babashka.process/shell {:in "<body><h3>\nfoo</h3></body></html>" :out :string} "xidel" "--silent" "--data" "-" "--extract" "//h3" "--output-format" "json-wrapped")))
("foo")

borkdude17:02:36

takes 25ms on my machine to shell out and get the answer back, not too bad, but it depends on your constraints of course

borkdude17:02:27

I added some clarification here to the examples, hope it helps https://github.com/babashka/process/blob/master/API.md#babashka.process/shell

ūüĎć 1
teodorlu17:02:43

Update all your git repos with 49 lines of babashka: https://github.com/teodorlu/bb-scripts/tree/385d310664c676063bdbfcded3ed4f09ff7ddd91/src/teodorlu/bb_scripts/update_repos.clj To try for yourself with #babashka-bbin:

$ bbin install io.github.teodorlu/bb-scripts --latest-sha
$ cd FOLDER_WITH_ALL_MY_GIT_REPOS
$ update-repos
Those 36 lines aren't too special, you could probably have written those yourself. But I think it's quite amazing how much it's possible to get done with a bit of babashka right now. If I were to try to solve that with Bash, I'd get stuck with string manipulation. And now, bbin really solves distribution for babashka scripts. I'd encourage anyone on the fence with Babashka, shell scripts and bbin to give it a shot!

ūüéČ 1
borkdude17:02:06

The syntax for shell is (shell ?opts cmd arg1 arg2), what you're using now is a legacy syntax that was only supported by process, and is now deprecated

teodorlu18:02:09

OK - thanks! Changing to this:

(shell {:out :string
        :continue true
        :dir path}
       '[git symbolic-ref --short HEAD])

borkdude18:02:31

The supported syntax is:

(shell {:out :string
        :continue true
        :dir path}
       "git" "symbolic-ref" "--short" "HEAD")

borkdude18:02:47

or you can pass a single string and will be split automatically

borkdude19:02:17

so:

(shell {:out :string
        :continue true
        :dir path}
       "git symbolic-ref --short HEAD")
would also be valid

ūüĎć 1
adam-james20:02:47

Is there some way I can get the watches for an atom? In Clojure, I can do something like (.getWatches my-atom) but that’s not working in babashka:

Babashka v1.1.173 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (.getWatches (atom {}))
java.lang.NoSuchFieldException: getWatches [at <repl>:1:1]

borkdude20:02:15

Congrats on your Clojure Conj talk! I'll have a look

adam-james20:02:39

ūüėä thanks! I‚Äôm really looking forward to the Conj. No rush

borkdude20:02:44

I guess there is no non-interop way to get the watches from an atom so I'll have to add this to the reflection config.

borkdude20:02:59

What's the use case if I may ask?

adam-james20:02:47

My https://github.com/adam-james-v/solenoid project uses a registry of controls, each of which has a ‚Äėcursor‚Äô into it. Cursors are implemented using watch fns that update any atoms that are dependent on the atom being currently changed. Control Blocks are auto-updating whenever controls that drive them change. But, you can use the dereferenced value of any control block in the body of other blocks. The mechanism I came up with to track dependent blocks uses watches and watch keys to get details‚Ķ

borkdude20:02:50

I see 7 usages of this on http://grep.app of which one is solenoid :)

ūüėČ 1
borkdude20:02:19

that sounds a lot like reagent :)

borkdude20:02:25

I'm adding support for it now

borkdude20:02:45

is your talk also about solenoid?

borkdude20:02:35

Pushed to master. You can test with:

bash <(curl ) --dev-build --dir /tmp
once the CI build has finished

adam-james20:02:54

Whoa, thanks! I’ll give it a spin soon.

adam-james20:02:56

My talk will feature Solenoid, but is a bit more broad than that. I want to tell my motivations for learning Clojure, and how I used various projects to help me learn. I‚Äôll be talking a bit about making tools to help make ‚Äėthings‚Äô (where things are objects, art, machinery, structures, not just code). The idea of interactivity and manipulation is pretty important to making things that actually work (I remember designing things to be welded‚Ķ not every joint is physically possible, even if you can 3D model it). And solenoid is just one example of interactive tools that can help in other design/build processes. I promise the talk itself will be coherent haha

ūüėÉ 1
borkdude20:02:33

It already makes much sense to me :)

ūüôā 1
adam-james20:02:27

Babashka v1.1.174-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (def a (atom {}))
#'user/a
user=> (add-watch a :my-watch (fn [_ _ _ _] nil))
#object[clojure.lang.Atom 0x3a37bb7 {:status :ready, :val {}}]
user=> (.getWatches a)
{:my-watch #object[sci.impl.fns$fun$arity_4__1176 0x34a5f957 "[email protected]"]}

adam-james20:02:58

The snapshot works ūüôā Thanks for working on that so quickly, and on the weekend! Really amazing

borkdude20:02:52

ūüĎć Hopefully a new bb release next week

gratitude 2