Yes. But you can also try it now:

bash <(curl ) --dev-build --dir /tmp

re-enacted your initial example

15s is a bit long, maybe a bit old hardware?

still, 1M threads :)

hm 12.5 seconds without ffmpeg running

I am trying to override clojure.core.async.impl.exec.threadpool thread-pool-executor can I just call intern? I guess the symbols need to be exposed? From async protols only ReadPort is defined I think

yeah that won't work I think

does core.async has a function to set the threadpool executor? clojure itself has this:

user=> (apropos "executor")
(clojure.core/set-agent-send-executor! clojure.core/set-agent-send-off-executor!)

I don't think so. There is a a defonce calling thread-pool-executor

Might be good to request a function for this that explicitly supports this on ask.clojure. It seems it's not well supported currently, barring some hacks changing an impl namespace: https://stackoverflow.com/questions/18779296/clojure-core-async-any-way-to-control-number-of-threads-in-that-go-thread

Doing the intern in SCI will not have the desired effect since the pre-compiled functions don't see the change

ah

Hmm, we could hack this by adding a watcher on the SCI var that also changes the core var, but even then it might not work (as you can see in the SO answer, the order of loading things matters)

also alter-var-root is the same as intern right

if the var already exists, yes

https://ask.clojure.org/index.php/12692/core-async-add-a-function-to-set-thread-executor

https://twitter.com/borkdude/status/1634257317184040962

that's cute

@UJSBQNUG6 Dealing with HTML can be done via the bootleg pod: https://github.com/babashka/pod-registry/blob/master/examples/bootleg.clj It has a lot of HTML library stuff, e.g. it has enlive

I saw that, but I could not find any XPath support there

I see. If you're open to using Node.js, maybe #C029PTWD3HR is also an option here. I'm sure there are lots of Node libraries that support Xpath

sure, that is a good alternative

I wanted to solve it w/ Babashka since this would be my first project with it

and the experience was flawless so far 🙂

Bootleg does support parsing the HTML very well though

would you go in the hiccup way? or what would give me a similar query like interface?

although I guess, I wouldn’t be able to query the HTML like, I want the <h3> with this content and all the next siblings until the next <h3>

I'm thinking...

Perhaps @U1QQJJK89 has an idea?

(the author of bootleg)

Maybe I would do the hiccup processing manually or use enlive/select perhaps

Warning: converting markup from :html to :hiccup lost 1 form                                                                                                                                                        
Cannot read EDN: ...
----- Error --------------------------------------------------------------------
Type:     java.lang.RuntimeException
Message:  Invalid token: /hirdetmeny?id=1931760
Location: /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:15:3

There is also hickory/select which I've used recently in nbb: https://github.com/babashka/nbb/commit/6feaa7f2e5e0888ccc3a33ea5efbb47041638954

Hmm, @UJSBQNUG6 perhaps you can make a small reproduction of this.

sure, give me a couple of minutes

another option could be to install a look like xidel which supports xpath: https://formulae.brew.sh/formula/xidel and then shell out from bb with (babashka.process/shell "xidel" ...)

So the issue is that the HTML gets transformed into EDN (hiccup) and this EDN is sent back to babashka as a string, but I think there may be some escaping issue

So I might have found a bug?

I'm not sure :)

When I manually paste that string into an .edn file and parse it it's fine...

(clojure.edn/read-string (slurp "/tmp/foo.edn"))

The error is coming from here: https://github.com/babashka/pods/blob/75c2216649bf5caf3ae41e7b8fc202e45b090ea9/src/babashka/pods/impl.clj#L151

I believe the HTML is invalid

<a href="/hirdetmeny?id=1931760" title="„Hejőpapi X.-átmeneti törmelékes nyersanyagok" védnevű bánya bányászati tevékenységére vonatkozó BO/16/7277-22/2016. számú környezetvédelmi engedély módosítása">„Hejőpapi X.-átmeneti törmelékes nyersanyagok" védnevű bánya bányászati tevékenységére vonatkozó BO/16/7277-22/2016. számú környezetvédelmi engedély módosítása</a>

I believe this is the root cause

hmm right

$ xidel \?SZO\=szeged --silent --extract //h3
Hirdetményét itt tudja beküldeni!
Keresés a hirdetmények között
Találatok

so xidel could be a working solution, thank you!

It might work with enlive/select too, not sure:

(prn (enlive/select "<html></html>" #_(slurp "/tmp/foo.html") [:html]))

I can't figure out how enlive/select works, it returns an empty list

yeah, I'd go for xidel or move to nbb

thank you, I’ll check all 3 options

you can shell out to xidel with:

(shell "xidel" ...)

(:out (shell {:out :string } "xidel" ...))

See babashka.process for details

yes, but then I need to figure how to properly escape the args for xidel

since XPath selectors are quite complex

escape how so?

and I cannot see a good way to do the escaping with babashka.process/shell

like what happens if I have both the quotes (`"` and ') in the argument?

what would be great to be able to pass a data structure to babashka.process/shell and that would do the escaping automatically and I would not need to build a string

["cmd", arg0, arg2, arg3]

or something similar

wouldn't you have the same problem when you would call an xpath function in clojure? I don't see how that would be different than shelling out. the string arrives at xidel exactly how you wrote it, barring Java escaping rules

yes, this is already the case:

(shell "xidel" arg0 arg1 arg3)

then there are no issues, I just could not find an example like that

then we need to improve the docs :)

or I need to improve myself

https://github.com/babashka/process/blob/master/API.md#shell-

this somehow implies it

@U04V15CAJ how could you get enlive working? for me it throws:

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: org.xml.sax.ContentHandler
Location: net/cgrand/xml.clj:11:3

----- Context ------------------------------------------------------------------
 7: ;   You must not remove this notice, or any other, from this software.
 8: 
 9: (ns net.cgrand.xml
10:   (:require [clojure.zip :as z])
11:   (:import (org.xml.sax ContentHandler Attributes SAXException XMLReader)
      ^--- Unable to resolve classname: org.xml.sax.ContentHandler
12:            (org.xml.sax.ext DefaultHandler2)
13:            (javax.xml.parsers SAXParser SAXParserFactory)))
14: 
15: (defstruct element :tag :attrs :content)
16: 

----- Stack trace --------------------------------------------------------------
net.cgrand.xml         - net/cgrand/xml.clj:11:3
net.cgrand.tagsoup     - net/cgrand/tagsoup.clj:12:3
net.cgrand.enlive-html - net/cgrand/enlive_html.clj:14:3
pnn.scrape             - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/scrape.clj:2:3
pnn.core               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/core.clj:2:3
pnn.main               - /Users/ikaraszi/_vc/raszi/pnn/src/pnn/main.clj:2:3
user                   - <expr>:1:10

okay, I guess I need to use pods, right?

(require '[pod.retrogradeorbit.bootleg.enlive :as enlive])

the same bootleg pod

I believe I would simply switch to clojure

nbb requires too much rewrites

yeah, of course that's also an option :)

it was a nice experience with Babashka but unfortunately the HTML parsing made it impossible

I would need to run tidy, or xidel from shell and that does not sounds too nice 😞

user=> (json/parse-string (:out (babashka.process/shell {:out :string} "xidel" "" "--silent" "--extract" "//h3" "--output-format" "json-wrapped")))
(["Hirdetményét itt tudja beküldeni!" "Keresés a hirdetmények között" "Találatok"])

and with stdin:

user=> (json/parse-string (:out (babashka.process/shell {:in "<body><h3>\nfoo</h3></body></html>" :out :string} "xidel" "--silent" "--data" "-" "--extract" "//h3" "--output-format" "json-wrapped")))
("foo")

takes 25ms on my machine to shell out and get the answer back, not too bad, but it depends on your constraints of course

hmm

I added some clarification here to the examples, hope it helps https://github.com/babashka/process/blob/master/API.md#babashka.process/shell

thank you!

The syntax for shell is (shell ?opts cmd arg1 arg2), what you're using now is a legacy syntax that was only supported by process, and is now deprecated

OK - thanks! Changing to this:

(shell {:out :string
        :continue true
        :dir path}
       '[git symbolic-ref --short HEAD])

The supported syntax is:

(shell {:out :string
        :continue true
        :dir path}
       "git" "symbolic-ref" "--short" "HEAD")

or you can pass a single string and will be split automatically

so:

(shell {:out :string
        :continue true
        :dir path}
       "git symbolic-ref --short HEAD")

Congrats on your Clojure Conj talk! I'll have a look

😊 thanks! I’m really looking forward to the Conj. No rush

I guess there is no non-interop way to get the watches from an atom so I'll have to add this to the reflection config.

What's the use case if I may ask?

My https://github.com/adam-james-v/solenoid project uses a registry of controls, each of which has a ‘cursor’ into it. Cursors are implemented using watch fns that update any atoms that are dependent on the atom being currently changed. Control Blocks are auto-updating whenever controls that drive them change. But, you can use the dereferenced value of any control block in the body of other blocks. The mechanism I came up with to track dependent blocks uses watches and watch keys to get details…

I see 7 usages of this on http://grep.app of which one is solenoid :)

that sounds a lot like reagent :)

I'm adding support for it now

is your talk also about solenoid?

Pushed to master. You can test with:

bash <(curl ) --dev-build --dir /tmp

Whoa, thanks! I’ll give it a spin soon.

My talk will feature Solenoid, but is a bit more broad than that. I want to tell my motivations for learning Clojure, and how I used various projects to help me learn. I’ll be talking a bit about making tools to help make ‘things’ (where things are objects, art, machinery, structures, not just code). The idea of interactivity and manipulation is pretty important to making things that actually work (I remember designing things to be welded… not every joint is physically possible, even if you can 3D model it). And solenoid is just one example of interactive tools that can help in other design/build processes. I promise the talk itself will be coherent haha

It already makes much sense to me :)

Babashka v1.1.174-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (def a (atom {}))
#'user/a
user=> (add-watch a :my-watch (fn [_ _ _ _] nil))
#object[clojure.lang.Atom 0x3a37bb7 {:status :ready, :val {}}]
user=> (.getWatches a)
{:my-watch #object[sci.impl.fns$fun$arity_4__1176 0x34a5f957 "sci.impl.fns$fun$arity_4__1176@34a5f957"]}

The snapshot works 🙂 Thanks for working on that so quickly, and on the weekend! Really amazing

👍 Hopefully a new bb release next week