Fork me on GitHub
#clojure
<
2022-09-27
>
Martynas Maciulevičius08:09:37

Hey. I was wondering if it's possible to use clojure.walk/walk so that the recursion would be stopped at a specific depth upon finding some type of a node? It is doable by hand but what if I would want to do it using clojure.walk? What I want to do is to do clojure.walk/postwalk but up until I find a specific node type. Is this doable? I wanted to wrap some parent nodes but when I change the structure of the tree it goes deeper and deeper into recursion and wraps too much.

p-himik08:09:57

Not possible. This is a part of the implementation of walk:

(cond
  (list? form) (outer (apply list (map inner form)))
  ...)
And there's no way to stop map in the middle.

Martynas Maciulevičius08:09:53

It has to be possible because I don't expect to stop in the same-level leaves. I wanted only to stop on the depth. I think with the current implementation of walk I would have to do the same if twice :thinking_face: In the outer and in the inner as well. And inner should probably return form when it stops processing.

p-himik08:09:29

Actually, it is possible to prevent wrapping as soon as you stumble upon something specific - your inner and outer functions would have to inspect some flag that you set in inner when a specific node is found. But that won't stop at the depth, that'll stop at a node. And it'll still traverse the whole data structure, so having infinite collections there will not work well.

Martynas Maciulevičius08:09:26

Yes, I meant stopping at the node, not at the level. I.e. find a node and don't propagate down. And no, my collection is finite.

borkdude10:09:21

When I need to do such a thing, I usually just make a copy of the walk source and then tweak it

borkdude10:09:58

This is what I've done e.g. here: https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/hooks_api.clj I stop descending at the level of a certain type in the tweaked pre-walk

teodorlu16:09:55

I think this might be possible with prewalk. It lets you act to composite collections before traversing "down".

(defn walker [form]
  (if (map? form)
    :stop
    (do (prn form) form)))

(def data
  [:hello
   :there
   {:deep '[s t u f f]}])

(clojure.walk/prewalk walker data)

;; prints
[:hello :there {:deep [s t u f f]}]
:hello
:there

;; returns
[:hello :there :stop]
Not sure whether it's a good idea though. Edit: reformatted for clarity

Martynas Maciulevičius16:09:56

clojure.walk/prewalk uses clojure.walk/walk underneath. It's complicated to do what it wasn't intended to do. I already did something else already.

teodorlu21:09:03

Might i ask what "else" you did?

Martynas Maciulevičius04:09:43

I have an internal representation of nested data and my factory function was returning this:

{:my :data
 :inner-atrribute {:other :data}}
So I wanted to transform the :inner-attribute and all other attributes like this (even in the deeper nested data):
{:my {:data-attr :data}                 ; this is easy to do
 :inner/data {:nested {:other {:data-attr :data}}}}  ; this is where I though about using this recursion but I didn't want to transform the leaf because I was applying the :data-attr transformation first
So instead I decided to have this as my input and work with it directly:
{:my {:data-attr :data}
 :inner-atrribute {:other {:data-attr :data}}}
Then I wouldn't need to nest anything using walk This is currently for my own purposes and I'd probably want to change this format into something more similar to malli because I use that underneath.

👍 1
pez14:09:37

I have hiccup and want hickory. Is there a straight path? (I'm using hickory the library)

pez14:09:01

Right now I am going via html, using hickory.render. But it doesn't seem to cope with {:style {:foo 0}} , which gets rendered as style="{:foo 0}" ¯\(ツ)

pavlosmelissinos14:09:25

Seems like hickory https://github.com/clj-commons/hickory/blob/ea248a6387f007dc4c4e8fcbbafbb1b9cbc19c78/src/cljc/hickory/convert.cljc#L16 but maybe still worth checking out in case your solution is different? (I haven't personally tried it)

pez15:09:49

Thanks! So that answers my question if there is a more straight path. render/hiccup-to-html doesn't cut it with :style maps. I should probably write one myself now. Or see if I can use reagent's hiccup to html facility.... Hmmm.

pesterhazy05:09:02

I think style maps is an extension that reagent brought to hiccup, so even hiccup the library doesn't (or didn't used to) support it

🙏 1
pez13:10:36

Follow up: I ended up using reagent for this. Something like so:

(require '[reagent.dom.server :as r-dom-server]) 
  #_{:clj-kondo/ignore [:unused-private-var]} 
  (defn- hiccup-node->hickory
    [hiccup-node]
    (->> hiccup-node
         r-dom-server/render-to-static-markup
         html->ast
         hickorize
         first)) 
  (hiccup-node->hickory [:div [:p {:style {:border-color :red
                                           :margin-top "5px"}}]])
=>
{:attrs nil,
 :content [{:attrs {:style "border-color:red;margin-top:5px"},
            :content nil,
            :tag :p,
            :type :element}],
 :tag :div,
 :type :element}
Where html->ast uses posthtml-parser from npm, because the Hickory parser doesn't want to run in node where my tests run.

emccue14:09:29

Happy "java 19 is on docker hub" day everyone!!!!!!

🎉 8
emccue14:09:31

(ns dev.mccue.vthreads
  (:import (java.lang.management ManagementFactory)
           (java.lang.invoke MethodHandles MethodType)
           (java.util.concurrent Executors ExecutorService)))

(defn started-with-enable-preview
  []
  (let [runtime-mxbean (ManagementFactory/getRuntimeMXBean)]
    (boolean
      (some #{"--enable-preview"}
            (.getInputArguments runtime-mxbean)))))

(defn get-vthread-executor
  []
  (let [lookup (MethodHandles/publicLookup)
        method-type (MethodType/methodType ExecutorService)
        method-handle (.findStatic lookup
                                   Executors
                                   "newVirtualThreadPerTaskExecutor"
                                   method-type)]
    (.invokeWithArguments method-handle [])))

(if (started-with-enable-preview)
  (do
    (println "using vthread executor")
    (set-agent-send-off-executor! (get-vthread-executor))
    (set-agent-send-executor! (get-vthread-executor)))
  (println "not using vthread executor"))

p-himik14:09:56

Should be (if (started-with-enable-preview) ...) ;)

👀 1
jjttjj16:09:02

I'm curious if the new virtual threads will remove the need for http clients and libraries wrapping http clients to provide async options (in cases where older java versions dont need to be supported). Is there ever a need for these types of library developers to have to worry about users' async needs now?

Joshua Suskalo16:09:54

In theory, assuming you are only supporting java versions which have virtual threads enabled by default, you could not worry about the async needs of the consumer, assuming that frameworks also get updated.

👍 1
Joshua Suskalo16:09:48

more or less, if the broader ecosystem is adopting virtual threads, there should be little reason to prefer async APIs to sync APIs with virtual thread support.

Joshua Suskalo16:09:45

That's a key thing to be aware of though, if the IO is happening in a native library, or under a monitor lock, or otherwise using IO that is not provided by http://java.io. or java.nio. then you need to support an async API in order to provide async features, or otherwise wrap it in a manner that's compatible with virtual threads (e.g. a thread pool that you submit IO jobs to that uses monitor locks and .notify calls to wake virtual threads that submitted the jobs).

emccue16:09:01

Need is relative

emccue16:09:26

I think that we really shouldn't cater to old java releases in stuff we do for free for fun

emccue16:09:56

because that helps corporate lethargy more than people

winsome15:09:15

I'm trying to come up with a schema for async channels and promises, is there any prior art I should know of?

Joshua Suskalo15:09:32

I would personally recommend that you don't directly try to create schemas for "promise that resolves to x" or "channel with x on it" as channels and promises are inherently stateful, they aren't data, and schemas are designed for data validation. If you want to ensure that you have valid data on your channels and promises, I would recommend instead making simple macros that validate the types that you receive from them that can be "turned off" in production.

Joshua Suskalo15:09:25

e.g.

(let [p-val (deref-type the-promise ::some-schema)]
  (do-stuff p-val))
where deref-type is a custom macro that derefs the promise, validates the data against the passed schema, and if it isn't then throw an exception with the explain-data if you're using spec or something equivalent if you're not.

emccue15:09:45

...also potentially your design doesn't need as many async channels and promises. see above snippet

JAtkins17:09:30

Has anyone tried out the new AMD 5800x3d chips for clojure performance? I have a pet theory that the v-cache might have an outsized performance improvement for clojure with its' persistant data structures, but I don't have any way to test this theory myself.

kwladyka22:09:34

I need to go throw time series vector with more, than 2 milions of items. [{:open_time …} {:open_time …} {:open_time …} {:open_time …} …] On each item I want to read data from N previous items and M next items. Based on previous and next items, assoc conditional custom calculated values to current item. How can I do it in Clojure with good performance? I was trying to write something for that purpose, but it takes endless to process the collection. PS I am going sleep, so I will read your answers tomorrow. PS2 Thank you in advance.

p-himik22:09:27

> I was trying to write something for that purpose, but it takes endless to process the collection. So what exactly have you tried?

p-himik22:09:11

A naive implementation with partition takes around 18 seconds for 2 million items on my end:

(def data (mapv (fn [i]
                  {:open_time i, :data i})
                (range 2000000)))

(defn process-window [n _m window]
  (let [[front back] (split-at (inc n) window)
        [item & back] back]
    (assoc item :front (apply + (map :data front))
                :back (apply + (map :data back)))))

(defn process [n m data]
  (let [ps (partition (+ n m 1) 1
                      (concat
                        ;; To make sure that first N windows have the right size.
                        (repeat n {:data 0})
                        data))]
    (time (mapv #(process-window n m %) ps))))

(process 10 20 data)
It can be drastically improved if instead of partition + concat you write your own function that uses subvec.

👍 1
Ben Sless03:09:10

There's a version of a stateful transducer of a rolling window which might be useful

👍 1
kwladyka11:09:15

> So what exactly have you tried? some primitive code, not worth to mention

kwladyka11:09:59

thank you, I will take a look

Lone Ranger11:09:50

@U0WL6FA77 the most performant answer is you're going to want to use tech.ml.dataset's group-by-column-agg https://github.com/techascent/tech.ml.dataset/blob/master/src/tech/v3/dataset/reductions.clj#L364. It's extremely complex to use but it doesn't get any faster than this except maybe hand optimized C. Check out this talk from Chris if you're new to this kind of computing: https://www.youtube.com/watch?v=5mUGu4RlwKE I'd also check out the data science channel in the clojurians zulip. Now again I am ONLY suggesting this if speed is an absolute priority -- this is scientific computing, not idiomatic Clojure. tech.ml.dataset is its own paradigm.

👍 1
Lone Ranger11:09:09

The sci-cloj group has done a lot of work on rolling windows using tech.ml.dataset, so they may have a slicker answer. @U066L8B18 do you know how the rolling window project is coming along?

Daniel Slutsky11:09:22

Thanks, @U3BALC2HH. The relevant namespace of dtype-next by Chris is tech.v3.datatype.rolling: https://cnuernber.github.io/dtype-next/tech.v3.datatype.rolling.html If the necessary functionality is not there, then let us chat further -- indeed there have been some other drafts of functions that could be helpful.