Fork me on GitHub
Jakub Holý (HolyJak)14:12:56

Hi! It seems that either or slurp cache the content of the file. If I change the file and run (slurp (io/resource "path")) again, I get the old content. How to fix w/o restarting the repl? 🙏

R.A. Porter14:12:54

It's not exactly that it's cached; rather, the resource has been loaded by the classloader already and you need to force a new loader into place. If you're working in your REPL, the easiest way to do that is to reload the namespace where you're slurping the resource. That'll start a new classloader up that will be able to rescan the file when you slurp it.

Jakub Holý (HolyJak)14:12:28

Ah, thank you very much!


oh wow TIL, I've never experienced this (and personally like/use resources over files quite a lot and also fiddle a lot with tooling)

☝️ 1

I'm looking for a little style assistance ... wondering if this is an idiomatic approach to update a complex data structure (without a new dependency)


(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (atom 0)]
      (fn [node]
        (let [return-node (if (and (map? node)
                                   (= index @index-count)
                                   (:params node))
                            (assoc-in node [:params] new-value)
          (when (and (map? node) (:params node))
            (swap! index-count inc))


this takes in a conformed data structure, finds the existing node and replaces it

Jakub Holý (HolyJak)14:12:08

I am not sure what the input is. Is it a sequence of maps and you want to update the index -th one?

Jakub Holý (HolyJak)14:12:03

Or just the index-th map anywhere in the nested structure that has the expected key?

Jakub Holý (HolyJak)14:12:02

Walk is quite expensive so it depends on what you need. I would just do the ugly nested update-in calls.


 {:defn-type defn,
  {:fn-name apply-theme,
   :docstring "Apply the theme to a given output environment",
   :meta {:api-version "0.1.0"},
     [{:params {:args [[:local-symbol output]]},
        {:prepost {:pre [(outputs output)]},
         :body [(apply-theme output :terminal)]}]}
      {:params {:args [[:local-symbol output] [:local-symbol device]]},
       [:body [(str " To be implemented " output " for " device)]]}],
     :attr-map {:stub true}}]}}
 {:args [[:local-symbol outputs]]} 


yeah, spec generates quite complex data structures and I couldn't figure out a way to do update-in


I do use assoc-in for fields such as :docstring where the path is clear


the data that conform generates varies according to do the arity of the form


so I feel like postwalk is probably the right thing to use but open to suggestions obvs ... but I kinda wanted some help on the postwalk itself

Jakub Holý (HolyJak)14:12:45

I see. I guess you are right in this case

Jakub Holý (HolyJak)14:12:51

This works for me (where d is your data): (update-in d [1 :defn-args :fn-tail 1 :bodies 0] (constantly :REPLACED)) You can of course nest these so the (constantly ...) could actually be #(update-in % ...)


sometimes it generates arity-1 rather than arity-n and then the :params are at a different nesting level


just a silly question @U04V5V0V4 ... are you sure that you want to do it to all nodes with a :params key? Or would it be preferable to be specific about the locations that you want to update. What if there's a :params key in the metadata (for example)? Will that throw off the index count? Maybe you'd be safer specifying the exact paths that you want? Something like

(defn set-params [conformed-data new-value index]
  (let [update-params (fn [p] (when p new-value))]
    (-> conformed-data
        (cond-> (zero? index) (update-in [1 :defn-args :fn-tail :arity-1 :params] update-params))
        (update-in [1 :defn-args :fn-tail 1 :bodies index :params] update-params))))
? (this is entirely untested and probably doesn't work in any way)


I like the point about making the path clearer (will adjust for that) though the logic to reach the various bodies is tricky cos the conformed data has a few variations ... must within :fn-tail though so that could be a good sentinel


after some testing, its not a problem cos I'm doing a postwalk - any extra :params always come after the action is finished


but it was definitely worth checking


after some advice from @U051N6TTC I went with this


(defn set-pre-post
  "Insert `new-value` for :prepost into the params at `index` (or 0) of the `conformed-data`"
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
      (fn [node]
          (and (= index @index-count)
               (vector? node)) (cond
                                 ;; There is an existing prepost property at the index
                                 (and (= (first node) :prepost+body))
                                 [:prepost+body (assoc (last node) :prepost new-value)]

                                 ;; There is not an existing prepost property at the index
                                 (and (= (first node) :body)
                                      (= (first (last node)) :body))
                                 [:body [:prepost+body {:prepost new-value
                                                        :body    (last (last node))}]]

                                 :default node)
          (and (map? node)
               (:params node)) (do (vswap! index-count inc)
          :default node))


actually that was the more complex function but you get the idea 🙂


for the simpler one I did this


(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
      (fn [node]
          (and (map? node)
               (:params node))
          (let [node' (cond-> node
                              (= index @index-count) (assoc-in [:params] new-value))]
            (vswap! index-count inc)
          :default node))


thanks to @U051N6TTC, @U0P0TMEFJ and @U0522TWDA for the suggestions

👍 1

most especially to @U051N6TTC in fairness 🙂

❤️ 2

I'm feeling awkward about the atom and also the style in which I update it so wondering if some fresh eyes could bring a more elegant approach


Maybe I need to take a further step back, but at first blush I’d probably just tweak it a little:

(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
      (fn [node]                   
        (if (and (map? node) (= index @index-count) (:params node))
            (vswap! index-count inc)
            (assoc node :params new-value))


i.e. switch to a volatile, rather than an atom, and only test once


oh… I missed that you increment either way. Hang on. Let me update it


(defn set-params
  [conformed-data new-value index]
  (let [index       (dec (or index 0))
        index-count (volatile! 0)]
      (fn [node]
        (if (and (map? node) (:params node))
            (vswap! index-count inc)
            (update node :params #(if (= index @index-count) new-value %)))


hah was just going to mention the condition specified by the if will mean that the inc only occurs once so .... cool update


The update runs either way, which implies a little work, even when the index doesn’t match, but when the old value is returned then the object doesn’t change


drat… I put the vswap in too soon! My brain is still sluggish today. Sigh


I think I would stick with the atom though cos it's not exactly critical perf


yeah, the inc should be after ... so not sure if that works


I see the choice between atom and volatile to be based on: • scope • threads Atoms used to be a default, but now that volatiles are here I think they make more sense to only use them when they are in a multithreaded environment, or when the scope is outside of a single function


ok , that's a good rule of thumb


Can dec the index in the let block. I think I’d rather do that than save a value as I called assoc on, update the index, and then return the saved value. It works, but it feels clumsy


(I edited that second code block)


on reflection I'll adopt the volatile and leave the rest as is cos it's starting to get difficult for my 🧠


At this point, the vswap! could be embedded in the update which would avoid the need for the do. But I don’t know that this is idiomatic. I hate the idea of a side-effect happening during the update


I have other cases with several conditions where I could not do the inc inside. Given that they will be read together, it makes more sense to me to adopt a more obvious if clunky approach


So, about the log4j / logback vulnerabilities. I don't have a dependency on any of them. We use timbre, is there anything I should update? tools.logging is a dependency

Alex Miller (Clojure team)14:12:37

tools.logging does not depend on log4j or logback so you probably don't need to do anything, but would be good to check your full dep tree to ensure you're not including log4j if you don't think you're using it


I did install the plugin com.livingsocial/lein-dependency-check that creates a report that shows all dependencies and there was no log4j or logback, just tools.logging.


It did show some vulnerabilities that we are figuring out how to deal with, but nothing on log4j. I was just wondering if there was some feature on Timbre that could have the same problem.


the double edged sword of using something niche- it probably exists, it isn't being exploited on a mass scale


btw, although I'm biased (as a maintainer), I recommend nvd-clojure over lein-dependency-check. Experience has shown us that this type of job is generally best not offered as a lein plugin, which is why it's a discouraged api in nvd-clojure (soon to be entirely removed)


will check, thanks for the pointer

🙂 1

Is the ordering of the keys in a destruction significant? The compiler says so:

(defonce !db (atom {:config {:bar "baz"}}))

(defn foo
  [{:keys [config !state]
    :or {!state !db
         config (:config @!state)}}]
;; => Syntax error compiling at (destruct.clj:8:17).
;;    Unable to resolve symbol: !state in this context
While, if I swap the keys :keys [!state config], the compiler is happy.

Alex Miller (Clojure team)16:12:37

it's undefined so don't rely on it

Alex Miller (Clojure team)16:12:00

:or should not depend on the state of possibly destructured keys like this

Ben Sless16:12:07

It also evaluates the or arguments unlike the or macro


What should I do instead?

Joshua Suskalo16:12:47

I would recommend doing this type of logic in a let binding inside the function

☝️ 3
🙏 1

Thanks. Makes sense. And what about the difference in the compiler’s behaviour? Is it the behaviour that is undefined?

Alex Miller (Clojure team)16:12:29

there is no defined order that the destructuring occurs or when the :or values are computed

👍 2

Gotcha. Thanks!


Is there a simple way to find which dependency on project.clj links to a specific java package jar dependency?


lein deps :tree ?


First, thanks. That is exactly what I wanted!


When I use that command, it shows some sugestions of :exclusions. Is it just to avoid having two paths to the same library?


Yes, you can setup global exclusions and add a specific version of a given dependency, or setup exclusions on per-direct dependency basis. Leiningen picks a specific version automatically for you if it find multiple artifacts with different versions, most of the time it's not what you want and it's best to ensure you pull in only one version in given project.


Can I use this to change the version of the dependency of a specific library?


For example, I want to upgrade the jetty package used by ring-jetty-adapter. So I could exclude it on the import of ring-jetty-adapter and put a specific version on my project as a top dependency?


Yes, you'd exclude jetty-server from ring-jetty-adapter, and then provide your own version of jetty-server as a direct dependency. here's an example:


Note that in a deps.edn project, exclusions are rarely needed because the algorithm to pick versions is more predictable than Leiningen's and you can "force" a version by specifying it as a top-level dependency. Leiningen can essentially choose a "random" version when there are conflicts/ambiguities. In a deps.edn project, you'll get the newest version in such situations. We used to have a lot of exclusions when we used Leiningen and we carried that over to Boot (which uses the same algorithm as Leiningen). We have hardly any exclusions now since we switched to deps.edn (in 2018).


Good to know!


@U04V70XH6 oh, that's neat - I didn't realize that. Do you happen to know/seen a relatively recent leiningen -> tools.deps + migration guide? Not having to worry about exclusions and using the "blessed" stack is slowly convincing me to move off Lein


@U0JEFEZH6 I don't know of any such guide. So much will depend on how fancy your project.clj is.


If it's basic and you aren't relying on lein plugins in your user profile, it's really easy to migrate -- with the caveat that it also depends on how much of Leiningen's "included batteries" you use.


At this point it's dependency management, uberjar compilation and launching the nrepl server in development. We can do without the uberjar and leverage Docker image layers, so it's not a hard requirement.


Nice, I'll add build-clj to my list, thank you!


HoneySQL used to be a Leiningen project but I switched it to deps.edn as part of the 1.0.444 release and these days the whole CI pipeline is based on build-clj -- and it runs both cljs and clj tests, runs Eastwood, runs all the documentation examples as tests, automatically builds and deploys its library JAR. So that might be a good project to delve into the history of.


Happy to answer Qs about that via DM if you want.


I'm still assessing if/when we're going to switch - we are managing ~50 Clojure projects (libraries and applications) so I want to make sure that the migration is a smooth as possible, especially in the ones where we have to inject special Maven configs etc. I'll keep your offer in mind though, I really appreciate it!


Are those separate repos???


Yes, for now


We are migrating to a proper mono repo soon, that will help a bit. It's not as painful as it sounds as we have a lot of tooling around keeping everything up to date, share components etc and for the most part nobody has to run the whole stack locally


hmmm... cool!


clj -Sdeps for deps.edn projects

Alex Miller (Clojure team)17:12:12

you mean clj -Stree probably (or clj -X:deps list)


are nested transients ever worth it?

Ben Sless18:12:14

I found that transients usually pay for themselves after about three operations. Since transients are not exactly mutable I think they could pay off, but it would heavily depend on the scenario and you'll need to program the solution carefully


I’ve tried this myself, but depending on the depth and width of the nesting, my experience has been that the cost of persisting them wasn’t worth it. I’ve been looking at using transients at the top levels, but I haven’t yet benchmarked this


I went with a volatile for now. seems fast enough

Joshua Suskalo19:12:03

transients and volatiles are separate ideas. transients are fast ways to collect many updates to a single data structure. volatiles are a way to have thread-local state. I'm curious how you used a volatile to replace a transient?

Joshua Suskalo19:12:24

For clarity, you cannot perform mutating updates on transients. That is to say, in general you cannot expect the key :a to be on the returned value in this expression:

(let [ret (transient {})]
  (assoc! ret :a true)
  (persistent! ret))


the issue I've always found with nested transients is the book keeping necessary to record which keys have been 'opened', kind of wish it was different as I can imagine a bunch of places I could use nested transients if it were a trivial 1/2 line optimisation like transient and persistent! are. I'd be surprised if it wasn't always faster beyond 2-3 operations and in some case significantly so if you keep banging on the same keys, but hey cache locality, tlabs/gc get in the way of my intuition all the time so who knows.


rewinding a little… I don’t see how volatiles could be used to replace the use case of a transient. That said, I’ve used volatiles to hold a transient, then passed that volatile down a stack with various functions adding to the transient and updating the volatile as it goes. It’s totally not functional, but it doubled the speed of the previous functional code, so it was worth it.


sorry yeah, I was trying to figure out how to structure my algorithm in a more performant way. one way was to keep the loop recur and use a transient. another was to do it in a purely mutable fashion and use a volatile. I ended up with the latter because I have a map of at least 2 depth I'm changing and doing that with a transient didn't seem fun

Ben Sless21:12:03

You can just work with a Java HashMap and in the end build a Clojure hash map from it

Ben Sless21:12:39

And you can always rethink the original algorithm

Ben Sless21:12:54

Is it for Pyramid?

Ben Sless21:12:40

Is it for term rewriting during the query or some other operation?


it's for normalizing

Ben Sless21:12:46

Do you have a flame graph?

Ben Sless22:12:26

It's always good to know what's eating your CPU


i'll do some profiling of the version that uses fast-zip. i would much rather this operation not be limited by the size of the stack


might not be worth it since i'm 90% sure it's just allocation of all the zipper objects, and the fact that zippers can't really be used (AFAICT) in a depth-first way

Ben Sless07:12:08

I'm wondering if something like using eduction can help

(defn eprocess
    (map? o)
    (->Eduction (mapcat (fn [[k v]] [k (eprocess v)])) o)

    (coll? o)
    (->Eduction (mapcat eprocess) o)))

Ben Sless07:12:59

You can also reify the stack

(defn process
  [^Iterable o]
  (loop [it (.iterator o)
         stack ()
         acc []]
    (if (.hasNext it)
      (let [x (.next it)]
          (map? x)
          (recur (.iterator ^Iterable x) (cons it stack) acc)

          (coll? x)
          (recur (.iterator ^Iterable x) (cons it stack) acc)

          :else (recur it stack (conj acc x))))
      (if (seq stack)
        (recur (first stack) (rest stack) acc)


the problem I'm running into w/ using my own stack is I need to maintain the parent->child relationship between nodes in the tree


I want to both index each entity, as well as edit any entities that appear in each subtree. i.e.

{:id 1
 :foo [{:id 2
        :foo [{:id 3}]}
       {:id 4}]}
should be processed into the following:
[{:id 1
  :foo [[:id 2]
        [:id 4]]}
 {:id 2 :foo [[:id 3]]}
 {:id 3}
 {:id 4}]
so to do it all in one pass, I need to both edit each subtree (replacing each entity with a reference tuple) and add each entity to the stack

Ben Sless15:12:24

I'm not sure you lose it here. This is just a rough sketch but you can push whatever you like onto the accumulator, add whatever bindings you want, etc.

Ben Sless15:12:52

The idea was to first create a linearized enumeration, then iterate over it. Then you'd say those are two passes, true, but then translate the loop into an iterable

Ben Sless15:12:26

Letting you consume it in a single pass with a hidden stack somewhere inside


hmm. not sure I understand yet but that sounds interesting

Ben Sless15:12:44

Can't you get the same result with a postwalk?


I thought clojure.walk uses the call stack

Ben Sless15:12:31

Hm, maybe you're right, but if we recognize it as a postwalk perhaps we can implement one ourselves with a reified stack

Ben Sless15:12:09

Can always cps transform and trampoline :)

😭 1

yeah I think I'm just struggling to wrap my head around the right way to transform the algo into a reified stack


shoulda took some more CS classes I guess

Ben Sless16:12:58

I think I took about 3 and I came out alright 🙃 Start with a cps transform?

Ben Sless16:12:59

And you don't have backtracking to worry about


@UK0810AQ2 CPS + trampoline ended doing really well after a lot of trial and error:


it's about 30% faster than what's in main, correct (AFAICT) and works up to a depth of 40,000 before it it overflows on my machine. much better than the measly 3000-4000 depth in my PR I posted in this conversation originally

Ben Sless21:12:42

Now, would it be faster to pass db and entities as arguments instead of binding them as volatiles?


couple of other opportunities to squeeze out some more perf too, like converting all the local build up of collections into transients


the current algorithm definitely depends on mutating the volatiles in the outer scope to work. i'd have to restructure it further to ensure i'm capturing entities as it descends


i might call this Good Enough:tm: for now

Ben Sless05:12:14

Definitely. There are three other opportunities for gain: use reduce instead of loop, unroll update-in, avoid merge. Churning merges is very slow


Are there any clojure libraries that allow you to have a callable inline function that doubles as a doc string ? e.g.

(defn add-to-numbers 
 "a function that adds two numbers
  (add-to-numbers 1 2) 
  [a b] 
  (+ a b) 
It's tricky with escaping strings, i.e.
(defn say-hello 
  " say hello to username 
   (say-hello \"John\") 
   Hello, John
  (str "Hello, " username) 


Unit tests are arguably better at that than escaped strings and they integrate better with automated test runners, so I don't see the appeal to be honest. However: 1. the :test (TIL, this is actually neat!) 2. if you really need the examples to be in strings, there are a couple of small libraries that are inspired by python's doctest, e.g. (no idea how good it is, I've never used anything like it)

Cora (she/her)22:12:31

which is different but also functions as documentation


You might like (built-in, via clojure.test) that lets you put usage-examples-as-tests right there alongside your function definition.


(caveat: not all IDE/test tooling expects this!)