This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-09-27
Channels
- # announcements (1)
- # aws (8)
- # babashka (77)
- # babashka-sci-dev (8)
- # beginners (29)
- # biff (2)
- # calva (13)
- # cljs-dev (1)
- # clojure (42)
- # clojure-europe (205)
- # clojure-nl (1)
- # clojure-norway (5)
- # clojure-uk (4)
- # clojurescript (58)
- # conjure (9)
- # data-science (7)
- # datalevin (19)
- # datomic (3)
- # emacs (7)
- # fulcro (15)
- # gratitude (8)
- # lsp (52)
- # meander (3)
- # membrane (92)
- # off-topic (12)
- # re-frame (16)
- # reagent (4)
- # reitit (15)
- # releases (1)
- # sci (30)
- # shadow-cljs (34)
- # tools-deps (5)
- # xtdb (17)
Hey.
I was wondering if it's possible to use clojure.walk/walk
so that the recursion would be stopped at a specific depth upon finding some type of a node?
It is doable by hand but what if I would want to do it using clojure.walk
? What I want to do is to do clojure.walk/postwalk
but up until I find a specific node type. Is this doable?
I wanted to wrap some parent nodes but when I change the structure of the tree it goes deeper and deeper into recursion and wraps too much.
Not possible.
This is a part of the implementation of walk
:
(cond
(list? form) (outer (apply list (map inner form)))
...)
And there's no way to stop map
in the middle.It has to be possible because I don't expect to stop in the same-level leaves. I wanted only to stop on the depth.
I think with the current implementation of walk
I would have to do the same if
twice :thinking_face:
In the outer
and in the inner
as well. And inner
should probably return form
when it stops processing.
Actually, it is possible to prevent wrapping as soon as you stumble upon something specific - your inner
and outer
functions would have to inspect some flag that you set in inner
when a specific node is found.
But that won't stop at the depth, that'll stop at a node. And it'll still traverse the whole data structure, so having infinite collections there will not work well.
Yes, I meant stopping at the node, not at the level. I.e. find a node and don't propagate down. And no, my collection is finite.
When I need to do such a thing, I usually just make a copy of the walk source and then tweak it
This is what I've done e.g. here: https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/hooks_api.clj I stop descending at the level of a certain type in the tweaked pre-walk
I think this might be possible with prewalk
.
It lets you act to composite collections before traversing "down".
(defn walker [form]
(if (map? form)
:stop
(do (prn form) form)))
(def data
[:hello
:there
{:deep '[s t u f f]}])
(clojure.walk/prewalk walker data)
;; prints
[:hello :there {:deep [s t u f f]}]
:hello
:there
;; returns
[:hello :there :stop]
Not sure whether it's a good idea though.
Edit: reformatted for clarityclojure.walk/prewalk
uses clojure.walk/walk
underneath. It's complicated to do what it wasn't intended to do. I already did something else already.
I have an internal representation of nested data and my factory function was returning this:
{:my :data
:inner-atrribute {:other :data}}
So I wanted to transform the :inner-attribute
and all other attributes like this (even in the deeper nested data):
{:my {:data-attr :data} ; this is easy to do
:inner/data {:nested {:other {:data-attr :data}}}} ; this is where I though about using this recursion but I didn't want to transform the leaf because I was applying the :data-attr transformation first
So instead I decided to have this as my input and work with it directly:
{:my {:data-attr :data}
:inner-atrribute {:other {:data-attr :data}}}
Then I wouldn't need to nest anything using walk
This is currently for my own purposes and I'd probably want to change this format into something more similar to malli
because I use that underneath.I have hiccup and want hickory. Is there a straight path? (I'm using hickory the library)
Right now I am going via html, using hickory.render
. But it doesn't seem to cope with {:style {:foo 0}}
, which gets rendered as style="{:foo 0}"
¯\(ツ)/¯
Seems like hickory https://github.com/clj-commons/hickory/blob/ea248a6387f007dc4c4e8fcbbafbb1b9cbc19c78/src/cljc/hickory/convert.cljc#L16 but maybe still worth checking out in case your solution is different? (I haven't personally tried it)
Thanks! So that answers my question if there is a more straight path. render/hiccup-to-html
doesn't cut it with :style
maps. I should probably write one myself now. Or see if I can use reagent's hiccup to html facility.... Hmmm.
I think style maps is an extension that reagent brought to hiccup, so even hiccup the library doesn't (or didn't used to) support it
Follow up: I ended up using reagent for this. Something like so:
(require '[reagent.dom.server :as r-dom-server])
#_{:clj-kondo/ignore [:unused-private-var]}
(defn- hiccup-node->hickory
[hiccup-node]
(->> hiccup-node
r-dom-server/render-to-static-markup
html->ast
hickorize
first))
(hiccup-node->hickory [:div [:p {:style {:border-color :red
:margin-top "5px"}}]])
=>
{:attrs nil,
:content [{:attrs {:style "border-color:red;margin-top:5px"},
:content nil,
:tag :p,
:type :element}],
:tag :div,
:type :element}
Where html->ast
uses posthtml-parser
from npm, because the Hickory parser doesn't want to run in node where my tests run.(ns dev.mccue.vthreads
(:import (java.lang.management ManagementFactory)
(java.lang.invoke MethodHandles MethodType)
(java.util.concurrent Executors ExecutorService)))
(defn started-with-enable-preview
[]
(let [runtime-mxbean (ManagementFactory/getRuntimeMXBean)]
(boolean
(some #{"--enable-preview"}
(.getInputArguments runtime-mxbean)))))
(defn get-vthread-executor
[]
(let [lookup (MethodHandles/publicLookup)
method-type (MethodType/methodType ExecutorService)
method-handle (.findStatic lookup
Executors
"newVirtualThreadPerTaskExecutor"
method-type)]
(.invokeWithArguments method-handle [])))
(if (started-with-enable-preview)
(do
(println "using vthread executor")
(set-agent-send-off-executor! (get-vthread-executor))
(set-agent-send-executor! (get-vthread-executor)))
(println "not using vthread executor"))
I'm curious if the new virtual threads will remove the need for http clients and libraries wrapping http clients to provide async options (in cases where older java versions dont need to be supported). Is there ever a need for these types of library developers to have to worry about users' async needs now?
In theory, assuming you are only supporting java versions which have virtual threads enabled by default, you could not worry about the async needs of the consumer, assuming that frameworks also get updated.
more or less, if the broader ecosystem is adopting virtual threads, there should be little reason to prefer async APIs to sync APIs with virtual thread support.
That's a key thing to be aware of though, if the IO is happening in a native library, or under a monitor lock, or otherwise using IO that is not provided by http://java.io. or java.nio. then you need to support an async API in order to provide async features, or otherwise wrap it in a manner that's compatible with virtual threads (e.g. a thread pool that you submit IO jobs to that uses monitor locks and .notify
calls to wake virtual threads that submitted the jobs).
I think that we really shouldn't cater to old java releases in stuff we do for free for fun
That's fair.
I'm trying to come up with a schema for async channels and promises, is there any prior art I should know of?
I would personally recommend that you don't directly try to create schemas for "promise that resolves to x" or "channel with x on it" as channels and promises are inherently stateful, they aren't data, and schemas are designed for data validation. If you want to ensure that you have valid data on your channels and promises, I would recommend instead making simple macros that validate the types that you receive from them that can be "turned off" in production.
e.g.
(let [p-val (deref-type the-promise ::some-schema)]
(do-stuff p-val))
where deref-type
is a custom macro that derefs the promise, validates the data against the passed schema, and if it isn't then throw an exception with the explain-data
if you're using spec or something equivalent if you're not....also potentially your design doesn't need as many async channels and promises. see above snippet
Has anyone tried out the new AMD 5800x3d chips for clojure performance? I have a pet theory that the v-cache might have an outsized performance improvement for clojure with its' persistant data structures, but I don't have any way to test this theory myself.
I need to go throw time series vector with more, than 2 milions of items.
[{:open_time …} {:open_time …} {:open_time …} {:open_time …} …]
On each item I want to read data from N previous items and M next items. Based on previous and next items, assoc
conditional custom calculated values to current item.
How can I do it in Clojure with good performance? I was trying to write something for that purpose, but it takes endless to process the collection.
PS I am going sleep, so I will read your answers tomorrow.
PS2 Thank you in advance.
> I was trying to write something for that purpose, but it takes endless to process the collection. So what exactly have you tried?
A naive implementation with partition
takes around 18 seconds for 2 million items on my end:
(def data (mapv (fn [i]
{:open_time i, :data i})
(range 2000000)))
(defn process-window [n _m window]
(let [[front back] (split-at (inc n) window)
[item & back] back]
(assoc item :front (apply + (map :data front))
:back (apply + (map :data back)))))
(defn process [n m data]
(let [ps (partition (+ n m 1) 1
(concat
;; To make sure that first N windows have the right size.
(repeat n {:data 0})
data))]
(time (mapv #(process-window n m %) ps))))
(process 10 20 data)
It can be drastically improved if instead of partition
+ concat
you write your own function that uses subvec
.There's a version of a stateful transducer of a rolling window which might be useful
@U0WL6FA77 the most performant answer is you're going to want to use tech.ml.dataset's group-by-column-agg
https://github.com/techascent/tech.ml.dataset/blob/master/src/tech/v3/dataset/reductions.clj#L364. It's extremely complex to use but it doesn't get any faster than this except maybe hand optimized C. Check out this talk from Chris if you're new to this kind of computing: https://www.youtube.com/watch?v=5mUGu4RlwKE
I'd also check out the data science channel in the clojurians zulip.
Now again I am ONLY suggesting this if speed is an absolute priority -- this is scientific computing, not idiomatic Clojure. tech.ml.dataset is its own paradigm.
The sci-cloj group has done a lot of work on rolling windows using tech.ml.dataset, so they may have a slicker answer. @U066L8B18 do you know how the rolling window project is coming along?
Thanks, @U3BALC2HH. The relevant namespace of dtype-next by Chris is tech.v3.datatype.rolling: https://cnuernber.github.io/dtype-next/tech.v3.datatype.rolling.html If the necessary functionality is not there, then let us chat further -- indeed there have been some other drafts of functions that could be helpful.