Fork me on GitHub
#babashka
<
2023-02-25
>
Benjamin10:02:46

Does virtual thread support mean I can try project loom on the next bb release?

borkdude10:02:31

Yes. But you can also try it now:

bash <(curl ) --dev-build --dir /tmp

❤️ 1
Benjamin10:02:11

re-enacted your initial example

borkdude10:02:59

15s is a bit long, maybe a bit old hardware?

borkdude10:02:05

still, 1M threads :)

clojure-spin 1
Benjamin10:02:02

hm 12.5 seconds without ffmpeg running

Benjamin10:02:59

I am trying to override clojure.core.async.impl.exec.threadpool thread-pool-executor can I just call intern? I guess the symbols need to be exposed? From async protols only ReadPort is defined I think

borkdude10:02:45

yeah that won't work I think

borkdude10:02:52

does core.async has a function to set the threadpool executor? clojure itself has this:

user=> (apropos "executor")
(clojure.core/set-agent-send-executor! clojure.core/set-agent-send-off-executor!)

Benjamin11:02:01

I don't think so. There is a a defonce calling thread-pool-executor

borkdude11:02:03

Might be good to request a function for this that explicitly supports this on ask.clojure. It seems it's not well supported currently, barring some hacks changing an impl namespace: https://stackoverflow.com/questions/18779296/clojure-core-async-any-way-to-control-number-of-threads-in-that-go-thread

borkdude11:02:30

Doing the intern in SCI will not have the desired effect since the pre-compiled functions don't see the change

borkdude11:02:36

Hmm, we could hack this by adding a watcher on the SCI var that also changes the core var, but even then it might not work (as you can see in the SO answer, the order of loading things matters)

Benjamin11:02:32

also alter-var-root is the same as intern right

borkdude11:02:52

if the var already exists, yes

Benjamin13:03:33

that's cute

Benjamin12:02:18

https://faster-than-light-memes.xyz/jacking-nbb.html You can now define a cider jack in cmd via .dir-locals.el, or as parameter to a custom function. I was just using this

((nil
  (cider-jack-in-cmd . "/tmp/bb nrepl-server")))
so cider-jack-in starts a dev bb in that dir

👍 2
István Karaszi15:02:12

I am working on a notification tool which is technically a scraper that would fetch HTML pages parse the data with XPath queries and send out notifications. did I get it correctly that currently there is no way to use XPath with Babashka?

borkdude15:02:37

@UJSBQNUG6 Dealing with HTML can be done via the bootleg pod: https://github.com/babashka/pod-registry/blob/master/examples/bootleg.clj It has a lot of HTML library stuff, e.g. it has enlive

István Karaszi15:02:07

I saw that, but I could not find any XPath support there

borkdude15:02:51

I see. If you're open to using Node.js, maybe #C029PTWD3HR is also an option here. I'm sure there are lots of Node libraries that support Xpath

István Karaszi15:02:17

sure, that is a good alternative

István Karaszi15:02:59

I wanted to solve it w/ Babashka since this would be my first project with it

István Karaszi15:02:04

and the experience was flawless so far 🙂

borkdude15:02:27

Bootleg does support parsing the HTML very well though

István Karaszi15:02:21

would you go in the hiccup way? or what would give me a similar query like interface?

István Karaszi15:02:10

although I guess, I wouldn’t be able to query the HTML like, I want the <h3> with this content and all the next siblings until the next <h3>

borkdude16:02:28

I'm thinking...

borkdude16:02:33

Perhaps @U1QQJJK89 has an idea?

borkdude16:02:40

(the author of bootleg)

👍 2
borkdude16:02:57

Maybe I would do the hiccup processing manually or use enlive/select perhaps

István Karaszi16:02:09

Warning: converting markup from :html to :hiccup lost 1 form                                                                                                                                                        
Cannot read EDN: ...
----- Error --------------------------------------------------------------------
Type:     java.lang.RuntimeException
Message:  Invalid token: /hirdetmeny?id=1931760
Location: /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:15:3

borkdude16:02:09

Hmm, @UJSBQNUG6 perhaps you can make a small reproduction of this.

István Karaszi16:02:20

sure, give me a couple of minutes

borkdude16:02:52

another option could be to install a look like xidel which supports xpath: https://formulae.brew.sh/formula/xidel and then shell out from bb with (babashka.process/shell "xidel" ...)

borkdude16:02:14

So the issue is that the HTML gets transformed into EDN (hiccup) and this EDN is sent back to babashka as a string, but I think there may be some escaping issue

István Karaszi16:02:09

So I might have found a bug?

borkdude16:02:52

I'm not sure :)

borkdude16:02:32

When I manually paste that string into an .edn file and parse it it's fine...

(clojure.edn/read-string (slurp "/tmp/foo.edn"))

István Karaszi16:02:49

I believe the HTML is invalid

István Karaszi16:02:53

<a href="/hirdetmeny?id=1931760" title="„Hejőpapi X.-átmeneti törmelékes nyersanyagok" védnevű bánya bányászati tevékenységére vonatkozó BO/16/7277-22/2016. számú környezetvédelmi engedély módosítása">„Hejőpapi X.-átmeneti törmelékes nyersanyagok" védnevű bánya bányászati tevékenységére vonatkozó BO/16/7277-22/2016. számú környezetvédelmi engedély módosítása</a>

István Karaszi16:02:18

I believe this is the root cause

borkdude16:02:30

$ xidel \?SZO\=szeged --silent --extract //h3
Hirdetményét itt tudja beküldeni!
Keresés a hirdetmények között
Találatok

István Karaszi16:02:27

so xidel could be a working solution, thank you!

borkdude16:02:10

It might work with enlive/select too, not sure:

(prn (enlive/select "<html></html>" #_(slurp "/tmp/foo.html") [:html]))

borkdude16:02:21

I can't figure out how enlive/select works, it returns an empty list

borkdude16:02:16

yeah, I'd go for xidel or move to nbb

István Karaszi16:02:30

thank you, I’ll check all 3 options

borkdude16:02:14

you can shell out to xidel with:

(shell "xidel" ...)
If you want to get back a string of the output:
(:out (shell {:out :string } "xidel" ...))
etc

borkdude16:02:21

See babashka.process for details

István Karaszi16:02:32

yes, but then I need to figure how to properly escape the args for xidel

István Karaszi16:02:41

since XPath selectors are quite complex

borkdude17:02:04

escape how so?

István Karaszi17:02:07

and I cannot see a good way to do the escaping with babashka.process/shell

István Karaszi17:02:30

like what happens if I have both the quotes (`"` and ') in the argument?

István Karaszi17:02:03

what would be great to be able to pass a data structure to babashka.process/shell and that would do the escaping automatically and I would not need to build a string

István Karaszi17:02:28

["cmd", arg0, arg2, arg3]

István Karaszi17:02:33

or something similar

borkdude17:02:36

wouldn't you have the same problem when you would call an xpath function in clojure? I don't see how that would be different than shelling out. the string arrives at xidel exactly how you wrote it, barring Java escaping rules

borkdude17:02:01

yes, this is already the case:

(shell "xidel" arg0 arg1 arg3)

István Karaszi17:02:18

then there are no issues, I just could not find an example like that

borkdude17:02:40

then we need to improve the docs :)

🙂 2
István Karaszi17:02:55

or I need to improve myself

István Karaszi17:02:13

this somehow implies it

István Karaszi17:02:48

@U04V15CAJ how could you get enlive working? for me it throws:

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: org.xml.sax.ContentHandler
Location: net/cgrand/xml.clj:11:3

----- Context ------------------------------------------------------------------
 7: ;   You must not remove this notice, or any other, from this software.
 8: 
 9: (ns net.cgrand.xml
10:   (:require [clojure.zip :as z])
11:   (:import (org.xml.sax ContentHandler Attributes SAXException XMLReader)
      ^--- Unable to resolve classname: org.xml.sax.ContentHandler
12:            (org.xml.sax.ext DefaultHandler2)
13:            (javax.xml.parsers SAXParser SAXParserFactory)))
14: 
15: (defstruct element :tag :attrs :content)
16: 

----- Stack trace --------------------------------------------------------------
net.cgrand.xml         - net/cgrand/xml.clj:11:3
net.cgrand.tagsoup     - net/cgrand/tagsoup.clj:12:3
net.cgrand.enlive-html - net/cgrand/enlive_html.clj:14:3
pnn.scrape             - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:2:3
pnn.core               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/core.clj:2:3
pnn.main               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/main.clj:2:3
user                   - <expr>:1:10

István Karaszi17:02:19

okay, I guess I need to use pods, right?

borkdude17:02:25

(require '[pod.retrogradeorbit.bootleg.enlive :as enlive])

borkdude17:02:36

the same bootleg pod

István Karaszi17:02:23

I believe I would simply switch to clojure

István Karaszi17:02:39

nbb requires too much rewrites

borkdude17:02:26

yeah, of course that's also an option :)

István Karaszi17:02:18

it was a nice experience with Babashka but unfortunately the HTML parsing made it impossible

István Karaszi17:02:17

I would need to run tidy, or xidel from shell and that does not sounds too nice 😞

borkdude17:02:53

user=> (json/parse-string (:out (babashka.process/shell {:out :string} "xidel" "" "--silent" "--extract" "//h3" "--output-format" "json-wrapped")))
(["Hirdetményét itt tudja beküldeni!" "Keresés a hirdetmények között" "Találatok"])

borkdude17:02:59

and with stdin:

user=> (json/parse-string (:out (babashka.process/shell {:in "<body><h3>\nfoo</h3></body></html>" :out :string} "xidel" "--silent" "--data" "-" "--extract" "//h3" "--output-format" "json-wrapped")))
("foo")

borkdude17:02:36

takes 25ms on my machine to shell out and get the answer back, not too bad, but it depends on your constraints of course

borkdude17:02:27

I added some clarification here to the examples, hope it helps https://github.com/babashka/process/blob/master/API.md#babashka.process/shell

👍 2
teodorlu17:02:43

Update all your git repos with 49 lines of babashka: https://github.com/teodorlu/bb-scripts/tree/385d310664c676063bdbfcded3ed4f09ff7ddd91/src/teodorlu/bb_scripts/update_repos.clj To try for yourself with #babashka-bbin:

$ bbin install io.github.teodorlu/bb-scripts --latest-sha
$ cd FOLDER_WITH_ALL_MY_GIT_REPOS
$ update-repos
Those 36 lines aren't too special, you could probably have written those yourself. But I think it's quite amazing how much it's possible to get done with a bit of babashka right now. If I were to try to solve that with Bash, I'd get stuck with string manipulation. And now, bbin really solves distribution for babashka scripts. I'd encourage anyone on the fence with Babashka, shell scripts and bbin to give it a shot!

🎉 2
borkdude17:02:06

The syntax for shell is (shell ?opts cmd arg1 arg2), what you're using now is a legacy syntax that was only supported by process, and is now deprecated

teodorlu18:02:09

OK - thanks! Changing to this:

(shell {:out :string
        :continue true
        :dir path}
       '[git symbolic-ref --short HEAD])

borkdude18:02:31

The supported syntax is:

(shell {:out :string
        :continue true
        :dir path}
       "git" "symbolic-ref" "--short" "HEAD")

borkdude18:02:47

or you can pass a single string and will be split automatically

borkdude19:02:17

so:

(shell {:out :string
        :continue true
        :dir path}
       "git symbolic-ref --short HEAD")
would also be valid

👍 2
adam-james20:02:47

Is there some way I can get the watches for an atom? In Clojure, I can do something like (.getWatches my-atom) but that’s not working in babashka:

Babashka v1.1.173 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (.getWatches (atom {}))
java.lang.NoSuchFieldException: getWatches [at <repl>:1:1]

borkdude20:02:15

Congrats on your Clojure Conj talk! I'll have a look

adam-james20:02:39

😊 thanks! I’m really looking forward to the Conj. No rush

borkdude20:02:44

I guess there is no non-interop way to get the watches from an atom so I'll have to add this to the reflection config.

borkdude20:02:59

What's the use case if I may ask?

adam-james20:02:47

My https://github.com/adam-james-v/solenoid project uses a registry of controls, each of which has a ‘cursor’ into it. Cursors are implemented using watch fns that update any atoms that are dependent on the atom being currently changed. Control Blocks are auto-updating whenever controls that drive them change. But, you can use the dereferenced value of any control block in the body of other blocks. The mechanism I came up with to track dependent blocks uses watches and watch keys to get details…

borkdude20:02:50

I see 7 usages of this on http://grep.app of which one is solenoid :)

😉 2
borkdude20:02:19

that sounds a lot like reagent :)

borkdude20:02:25

I'm adding support for it now

borkdude20:02:45

is your talk also about solenoid?

borkdude20:02:35

Pushed to master. You can test with:

bash <(curl ) --dev-build --dir /tmp
once the CI build has finished

adam-james20:02:54

Whoa, thanks! I’ll give it a spin soon.

adam-james20:02:56

My talk will feature Solenoid, but is a bit more broad than that. I want to tell my motivations for learning Clojure, and how I used various projects to help me learn. I’ll be talking a bit about making tools to help make ‘things’ (where things are objects, art, machinery, structures, not just code). The idea of interactivity and manipulation is pretty important to making things that actually work (I remember designing things to be welded… not every joint is physically possible, even if you can 3D model it). And solenoid is just one example of interactive tools that can help in other design/build processes. I promise the talk itself will be coherent haha

😃 2
borkdude20:02:33

It already makes much sense to me :)

🙂 2
adam-james20:02:27

Babashka v1.1.174-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (def a (atom {}))
#'user/a
user=> (add-watch a :my-watch (fn [_ _ _ _] nil))
#object[clojure.lang.Atom 0x3a37bb7 {:status :ready, :val {}}]
user=> (.getWatches a)
{:my-watch #object[sci.impl.fns$fun$arity_4__1176 0x34a5f957 "sci.impl.fns$fun$arity_4__1176@34a5f957"]}

adam-james20:02:58

The snapshot works 🙂 Thanks for working on that so quickly, and on the weekend! Really amazing

borkdude20:02:52

👍 Hopefully a new bb release next week

gratitude 4