This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-12-15
Channels
- # adventofcode (110)
- # announcements (30)
- # aws (2)
- # babashka (39)
- # babashka-sci-dev (112)
- # beginners (155)
- # calva (5)
- # cider (12)
- # clj-kondo (11)
- # cljs-dev (1)
- # cljsrn (2)
- # clojure (144)
- # clojure-australia (2)
- # clojure-europe (14)
- # clojure-nl (5)
- # clojure-spec (3)
- # clojure-uk (2)
- # clojurescript (22)
- # core-async (23)
- # cursive (31)
- # data-science (3)
- # emacs (12)
- # events (1)
- # fulcro (8)
- # honeysql (7)
- # jobs-discuss (11)
- # lsp (1)
- # missionary (28)
- # nextjournal (7)
- # off-topic (64)
- # pedestal (3)
- # polylith (19)
- # reagent (14)
- # reitit (12)
- # releases (4)
- # shadow-cljs (33)
- # tools-deps (3)
- # xtdb (3)
Hi! It seems that either
or slurp
cache the content of the file. If I change the file and run (slurp (io/resource "path"))
again, I get the old content. How to fix w/o restarting the repl? 🙏
It's not exactly that it's cached; rather, the resource has been loaded by the classloader already and you need to force a new loader into place. If you're working in your REPL, the easiest way to do that is to reload the namespace where you're slurp
ing the resource. That'll start a new classloader up that will be able to rescan the file when you slurp it.
Ah, thank you very much!
oh wow TIL, I've never experienced this (and personally like/use resources
over files quite a lot and also fiddle a lot with tooling)
I'm looking for a little style assistance ... wondering if this is an idiomatic approach to update a complex data structure (without a new dependency)
(defn set-params
[conformed-data new-value index]
(let [index (or index 0)
index-count (atom 0)]
(walk/postwalk
(fn [node]
(let [return-node (if (and (map? node)
(= index @index-count)
(:params node))
(assoc-in node [:params] new-value)
node)]
(when (and (map? node) (:params node))
(swap! index-count inc))
return-node))
conformed-data)))
I am not sure what the input is. Is it a sequence of maps and you want to update the index
-th one?
Or just the index-th map anywhere in the nested structure that has the expected key?
Walk is quite expensive so it depends on what you need. I would just do the ugly nested update-in
calls.
[:conformed-data
{:defn-type defn,
:defn-args
{:fn-name apply-theme,
:docstring "Apply the theme to a given output environment",
:meta {:api-version "0.1.0"},
:fn-tail
[:arity-n
{:bodies
[{:params {:args [[:local-symbol output]]},
:body
[:prepost+body
{:prepost {:pre [(outputs output)]},
:body [(apply-theme output :terminal)]}]}
{:params {:args [[:local-symbol output] [:local-symbol device]]},
:body
[:body [(str " To be implemented " output " for " device)]]}],
:attr-map {:stub true}}]}}
:new-value
{:args [[:local-symbol outputs]]}
:index
nil]
yeah, spec generates quite complex data structures and I couldn't figure out a way to do update-in
so I feel like postwalk
is probably the right thing to use but open to suggestions obvs ... but I kinda wanted some help on the postwalk
itself
I see. I guess you are right in this case
This works for me (where d
is your data): (update-in d [1 :defn-args :fn-tail 1 :bodies 0] (constantly :REPLACED))
You can of course nest these so the (constantly ...) could actually be #(update-in % ...)
sometimes it generates arity-1
rather than arity-n
and then the :params
are at a different nesting level
just a silly question @U04V5V0V4 ... are you sure that you want to do it to all nodes with a :params
key? Or would it be preferable to be specific about the locations that you want to update. What if there's a :params
key in the metadata (for example)? Will that throw off the index count? Maybe you'd be safer specifying the exact paths that you want? Something like
(defn set-params [conformed-data new-value index]
(let [update-params (fn [p] (when p new-value))]
(-> conformed-data
(cond-> (zero? index) (update-in [1 :defn-args :fn-tail :arity-1 :params] update-params))
(update-in [1 :defn-args :fn-tail 1 :bodies index :params] update-params))))
? (this is entirely untested and probably doesn't work in any way)I like the point about making the path clearer (will adjust for that) though the logic to reach the various bodies is tricky cos the conformed data has a few variations ... must within :fn-tail
though so that could be a good sentinel
after some testing, its not a problem cos I'm doing a postwalk
- any extra :params
always come after the action is finished
after some advice from @U051N6TTC I went with this
(defn set-pre-post
"Insert `new-value` for :prepost into the params at `index` (or 0) of the `conformed-data`"
[conformed-data new-value index]
(let [index (or index 0)
index-count (volatile! 0)]
(walk/postwalk
(fn [node]
(cond
(and (= index @index-count)
(vector? node)) (cond
;; There is an existing prepost property at the index
(and (= (first node) :prepost+body))
[:prepost+body (assoc (last node) :prepost new-value)]
;; There is not an existing prepost property at the index
(and (= (first node) :body)
(= (first (last node)) :body))
[:body [:prepost+body {:prepost new-value
:body (last (last node))}]]
:default node)
(and (map? node)
(:params node)) (do (vswap! index-count inc)
node)
:default node))
conformed-data)))
(defn set-params
[conformed-data new-value index]
(let [index (or index 0)
index-count (volatile! 0)]
(walk/postwalk
(fn [node]
(cond
(and (map? node)
(:params node))
(let [node' (cond-> node
(= index @index-count) (assoc-in [:params] new-value))]
(vswap! index-count inc)
node')
:default node))
conformed-data)))
I'm feeling awkward about the atom
and also the style in which I update it so wondering if some fresh eyes could bring a more elegant approach
Maybe I need to take a further step back, but at first blush I’d probably just tweak it a little:
(defn set-params
[conformed-data new-value index]
(let [index (or index 0)
index-count (volatile! 0)]
(walk/postwalk
(fn [node]
(if (and (map? node) (= index @index-count) (:params node))
(do
(vswap! index-count inc)
(assoc node :params new-value))
node))
conformed-data)))
nice! thanks @U051N6TTC
(defn set-params
[conformed-data new-value index]
(let [index (dec (or index 0))
index-count (volatile! 0)]
(walk/postwalk
(fn [node]
(if (and (map? node) (:params node))
(do
(vswap! index-count inc)
(update node :params #(if (= index @index-count) new-value %)))
node))
conformed-data)))
hah was just going to mention the condition specified by the if
will mean that the inc only occurs once so .... cool update
The update
runs either way, which implies a little work, even when the index doesn’t match, but when the old value is returned then the object doesn’t change
I see the choice between atom and volatile to be based on: • scope • threads Atoms used to be a default, but now that volatiles are here I think they make more sense to only use them when they are in a multithreaded environment, or when the scope is outside of a single function
Can dec
the index
in the let
block. I think I’d rather do that than save a value as I called assoc
on, update the index, and then return the saved value. It works, but it feels clumsy
on reflection I'll adopt the volatile and leave the rest as is cos it's starting to get difficult for my 🧠
At this point, the vswap!
could be embedded in the update
which would avoid the need for the do
. But I don’t know that this is idiomatic. I hate the idea of a side-effect happening during the update
I have other cases with several conditions where I could not do the inc
inside. Given that they will be read together, it makes more sense to me to adopt a more obvious if clunky approach
So, about the log4j / logback vulnerabilities. I don't have a dependency on any of them. We use timbre, is there anything I should update? tools.logging is a dependency
tools.logging does not depend on log4j or logback so you probably don't need to do anything, but would be good to check your full dep tree to ensure you're not including log4j if you don't think you're using it
I did install the plugin com.livingsocial/lein-dependency-check that creates a report that shows all dependencies and there was no log4j or logback, just tools.logging.
It did show some vulnerabilities that we are figuring out how to deal with, but nothing on log4j. I was just wondering if there was some feature on Timbre that could have the same problem.
the double edged sword of using something niche- it probably exists, it isn't being exploited on a mass scale
btw, although I'm biased (as a maintainer), I recommend nvd-clojure over lein-dependency-check. Experience has shown us that this type of job is generally best not offered as a lein plugin, which is why it's a discouraged api in nvd-clojure (soon to be entirely removed)
Is the ordering of the keys in a destruction significant? The compiler says so:
(defonce !db (atom {:config {:bar "baz"}}))
(defn foo
[{:keys [config !state]
:or {!state !db
config (:config @!state)}}]
config)
;; => Syntax error compiling at (destruct.clj:8:17).
;; Unable to resolve symbol: !state in this context
While, if I swap the keys :keys [!state config]
, the compiler is happy.it's undefined so don't rely on it
:or
should not depend on the state of possibly destructured keys like this
I would recommend doing this type of logic in a let binding inside the function
Thanks. Makes sense. And what about the difference in the compiler’s behaviour? Is it the behaviour that is undefined?
there is no defined order that the destructuring occurs or when the :or values are computed
Is there a simple way to find which dependency on project.clj links to a specific java package jar dependency?
When I use that command, it shows some sugestions of :exclusions. Is it just to avoid having two paths to the same library?
Yes, you can setup global exclusions and add a specific version of a given dependency, or setup exclusions on per-direct dependency basis. Leiningen picks a specific version automatically for you if it find multiple artifacts with different versions, most of the time it's not what you want and it's best to ensure you pull in only one version in given project.
For example, I want to upgrade the jetty package used by ring-jetty-adapter. So I could exclude it on the import of ring-jetty-adapter and put a specific version on my project as a top dependency?
Yes, you'd exclude jetty-server from ring-jetty-adapter, and then provide your own version of jetty-server as a direct dependency. here's an example: https://github.com/nomnom-insights/nomnom.duckula/blob/34107b2927f3a5a3db45b0947cc9bd8f66418d60/project.clj#L34-L38
Note that in a deps.edn
project, exclusions are rarely needed because the algorithm to pick versions is more predictable than Leiningen's and you can "force" a version by specifying it as a top-level dependency. Leiningen can essentially choose a "random" version when there are conflicts/ambiguities. In a deps.edn
project, you'll get the newest version in such situations. We used to have a lot of exclusions when we used Leiningen and we carried that over to Boot (which uses the same algorithm as Leiningen). We have hardly any exclusions now since we switched to deps.edn
(in 2018).
@U04V70XH6 oh, that's neat - I didn't realize that. Do you happen to know/seen a relatively recent leiningen -> tools.deps + tools.build migration guide? Not having to worry about exclusions and using the "blessed" stack is slowly convincing me to move off Lein
@U0JEFEZH6 I don't know of any such guide. So much will depend on how fancy your project.clj
is.
If it's basic and you aren't relying on lein
plugins in your user profile, it's really easy to migrate -- with the caveat that it also depends on how much of Leiningen's "included batteries" you use.
At this point it's dependency management, uberjar compilation and launching the nrepl server in development. We can do without the uberjar and leverage Docker image layers, so it's not a hard requirement.
HoneySQL used to be a Leiningen project but I switched it to deps.edn
as part of the 1.0.444 release and these days the whole CI pipeline is based on build-clj
-- and it runs both cljs and clj tests, runs Eastwood, runs all the documentation examples as tests, automatically builds and deploys its library JAR. So that might be a good project to delve into the history of.
Happy to answer Qs about that via DM if you want.
I'm still assessing if/when we're going to switch - we are managing ~50 Clojure projects (libraries and applications) so I want to make sure that the migration is a smooth as possible, especially in the ones where we have to inject special Maven configs etc. I'll keep your offer in mind though, I really appreciate it!
Are those separate repos???
We are migrating to a proper mono repo soon, that will help a bit. It's not as painful as it sounds as we have a lot of tooling around keeping everything up to date, share components etc and for the most part nobody has to run the whole stack locally
you mean clj -Stree
probably (or clj -X:deps list
)
I found that transients usually pay for themselves after about three operations. Since transients are not exactly mutable I think they could pay off, but it would heavily depend on the scenario and you'll need to program the solution carefully
I’ve tried this myself, but depending on the depth and width of the nesting, my experience has been that the cost of persisting them wasn’t worth it. I’ve been looking at using transients at the top levels, but I haven’t yet benchmarked this
transients and volatiles are separate ideas. transients are fast ways to collect many updates to a single data structure. volatiles are a way to have thread-local state. I'm curious how you used a volatile to replace a transient?
For clarity, you cannot perform mutating updates on transients. That is to say, in general you cannot expect the key :a
to be on the returned value in this expression:
(let [ret (transient {})]
(assoc! ret :a true)
(persistent! ret))
the issue I've always found with nested transients is the book keeping necessary to record which keys have been 'opened', kind of wish it was different as I can imagine a bunch of places I could use nested transients if it were a trivial 1/2 line optimisation like transient and persistent! are. I'd be surprised if it wasn't always faster beyond 2-3 operations and in some case significantly so if you keep banging on the same keys, but hey cache locality, tlabs/gc get in the way of my intuition all the time so who knows.
rewinding a little… I don’t see how volatiles could be used to replace the use case of a transient. That said, I’ve used volatiles to hold a transient, then passed that volatile down a stack with various functions adding to the transient and updating the volatile as it goes. It’s totally not functional, but it doubled the speed of the previous functional code, so it was worth it.
sorry yeah, I was trying to figure out how to structure my algorithm in a more performant way. one way was to keep the loop recur and use a transient. another was to do it in a purely mutable fashion and use a volatile. I ended up with the latter because I have a map of at least 2 depth I'm changing and doing that with a transient didn't seem fun
You can just work with a Java HashMap and in the end build a Clojure hash map from it
i'll do some profiling of the version that uses fast-zip. i would much rather this operation not be limited by the size of the stack
might not be worth it since i'm 90% sure it's just allocation of all the zipper objects, and the fact that zippers can't really be used (AFAICT) in a depth-first way
I'm wondering if something like using eduction can help
(defn eprocess
[o]
(cond
(map? o)
(->Eduction (mapcat (fn [[k v]] [k (eprocess v)])) o)
(coll? o)
(->Eduction (mapcat eprocess) o)))
You can also reify the stack
(defn process
[^Iterable o]
(loop [it (.iterator o)
stack ()
acc []]
(if (.hasNext it)
(let [x (.next it)]
(cond
(map? x)
(recur (.iterator ^Iterable x) (cons it stack) acc)
(coll? x)
(recur (.iterator ^Iterable x) (cons it stack) acc)
:else (recur it stack (conj acc x))))
(if (seq stack)
(recur (first stack) (rest stack) acc)
acc))))
the problem I'm running into w/ using my own stack is I need to maintain the parent->child relationship between nodes in the tree
I want to both index each entity, as well as edit any entities that appear in each subtree. i.e.
{:id 1
:foo [{:id 2
:foo [{:id 3}]}
{:id 4}]}
should be processed into the following:
[{:id 1
:foo [[:id 2]
[:id 4]]}
{:id 2 :foo [[:id 3]]}
{:id 3}
{:id 4}]
so to do it all in one pass, I need to both edit each subtree (replacing each entity with a reference tuple) and add each entity to the stackI'm not sure you lose it here. This is just a rough sketch but you can push whatever you like onto the accumulator, add whatever bindings you want, etc.
The idea was to first create a linearized enumeration, then iterate over it. Then you'd say those are two passes, true, but then translate the loop into an iterable
Hm, maybe you're right, but if we recognize it as a postwalk perhaps we can implement one ourselves with a reified stack
yeah I think I'm just struggling to wrap my head around the right way to transform the algo into a reified stack
Then just copy the translation from https://www.metosin.fi/blog/malli-regex-schemas/
@UK0810AQ2 CPS + trampoline ended doing really well after a lot of trial and error: https://github.com/lilactown/pyramid/pull/16/files
it's about 30% faster than what's in main, correct (AFAICT) and works up to a depth of 40,000 before it it overflows on my machine. much better than the measly 3000-4000 depth in my PR I posted in this conversation originally
Now, would it be faster to pass db and entities as arguments instead of binding them as volatiles?
couple of other opportunities to squeeze out some more perf too, like converting all the local build up of collections into transients
the current algorithm definitely depends on mutating the volatiles in the outer scope to work. i'd have to restructure it further to ensure i'm capturing entities as it descends
Definitely. There are three other opportunities for gain: use reduce instead of loop, unroll update-in, avoid merge. Churning merges is very slow
Are there any clojure libraries that allow you to have a callable inline function that doubles as a doc string ? e.g.
(defn add-to-numbers
"a function that adds two numbers
(add-to-numbers 1 2)
=>
3
"
[a b]
(+ a b)
)
It's tricky with escaping strings, i.e.
(defn say-hello
" say hello to username
(say-hello \"John\")
=>
Hello, John
"
[username]
(str "Hello, " username)
)
Unit tests are arguably better at that than escaped strings and they integrate better with automated test runners, so I don't see the appeal to be honest.
However:
1. the :test
https://clojure.org/reference/special_forms#def (TIL, this is actually neat!)
2. if you really need the examples to be in strings, there are a couple of small libraries that are inspired by python's doctest, e.g. https://github.com/liquidz/testdoc (no idea how good it is, I've never used anything like it)
which is different but also functions as documentation
You might like https://clojuredocs.org/clojure.test/with-test (built-in, via clojure.test
) that lets you put usage-examples-as-tests right there alongside your function definition.
(caveat: not all IDE/test tooling expects this!)