Fork me on GitHub
#clojure
<
2021-12-15
>
Jakub Holý (HolyJak)14:12:56

Hi! It seems that either or slurp cache the content of the file. If I change the file and run (slurp (io/resource "path")) again, I get the old content. How to fix w/o restarting the repl? 🙏

1
R.A. Porter14:12:54

It's not exactly that it's cached; rather, the resource has been loaded by the classloader already and you need to force a new loader into place. If you're working in your REPL, the easiest way to do that is to reload the namespace where you're slurping the resource. That'll start a new classloader up that will be able to rescan the file when you slurp it.

Jakub Holý (HolyJak)14:12:28

Ah, thank you very much!

vemv13:12:33

oh wow TIL, I've never experienced this (and personally like/use resources over files quite a lot and also fiddle a lot with tooling)

☝️ 1
genRaiy14:12:02

I'm looking for a little style assistance ... wondering if this is an idiomatic approach to update a complex data structure (without a new dependency)

genRaiy14:12:09

(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (atom 0)]
    (walk/postwalk
      (fn [node]
        (let [return-node (if (and (map? node)
                                   (= index @index-count)
                                   (:params node))
                            (assoc-in node [:params] new-value)
                            node)]
          (when (and (map? node) (:params node))
            (swap! index-count inc))
          return-node))
      conformed-data)))

genRaiy14:12:50

this takes in a conformed data structure, finds the existing node and replaces it

Jakub Holý (HolyJak)14:12:08

I am not sure what the input is. Is it a sequence of maps and you want to update the index -th one?

Jakub Holý (HolyJak)14:12:03

Or just the index-th map anywhere in the nested structure that has the expected key?

Jakub Holý (HolyJak)14:12:02

Walk is quite expensive so it depends on what you need. I would just do the ugly nested update-in calls.

genRaiy14:12:28

[:conformed-data
 {:defn-type defn,
  :defn-args
  {:fn-name apply-theme,
   :docstring "Apply the theme to a given output environment",
   :meta {:api-version "0.1.0"},
   :fn-tail
   [:arity-n
    {:bodies
     [{:params {:args [[:local-symbol output]]},
       :body
       [:prepost+body
        {:prepost {:pre [(outputs output)]},
         :body [(apply-theme output :terminal)]}]}
      {:params {:args [[:local-symbol output] [:local-symbol device]]},
       :body
       [:body [(str " To be implemented " output " for " device)]]}],
     :attr-map {:stub true}}]}}
 :new-value 
 {:args [[:local-symbol outputs]]} 
 :index 
 nil]

genRaiy14:12:50

yeah, spec generates quite complex data structures and I couldn't figure out a way to do update-in

genRaiy14:12:07

I do use assoc-in for fields such as :docstring where the path is clear

genRaiy14:12:58

the data that conform generates varies according to do the arity of the form

genRaiy14:12:28

so I feel like postwalk is probably the right thing to use but open to suggestions obvs ... but I kinda wanted some help on the postwalk itself

Jakub Holý (HolyJak)14:12:45

I see. I guess you are right in this case

Jakub Holý (HolyJak)14:12:51

This works for me (where d is your data): (update-in d [1 :defn-args :fn-tail 1 :bodies 0] (constantly :REPLACED)) You can of course nest these so the (constantly ...) could actually be #(update-in % ...)

genRaiy15:12:42

sometimes it generates arity-1 rather than arity-n and then the :params are at a different nesting level

genRaiy15:12:59

ie no :bodies

Ed17:12:39

just a silly question @U04V5V0V4 ... are you sure that you want to do it to all nodes with a :params key? Or would it be preferable to be specific about the locations that you want to update. What if there's a :params key in the metadata (for example)? Will that throw off the index count? Maybe you'd be safer specifying the exact paths that you want? Something like

(defn set-params [conformed-data new-value index]
  (let [update-params (fn [p] (when p new-value))]
    (-> conformed-data
        (cond-> (zero? index) (update-in [1 :defn-args :fn-tail :arity-1 :params] update-params))
        (update-in [1 :defn-args :fn-tail 1 :bodies index :params] update-params))))
? (this is entirely untested and probably doesn't work in any way)

1
genRaiy17:12:20

I like the point about making the path clearer (will adjust for that) though the logic to reach the various bodies is tricky cos the conformed data has a few variations ... must within :fn-tail though so that could be a good sentinel

genRaiy19:12:11

after some testing, its not a problem cos I'm doing a postwalk - any extra :params always come after the action is finished

genRaiy19:12:21

but it was definitely worth checking

genRaiy19:12:40

after some advice from @U051N6TTC I went with this

genRaiy19:12:44

(defn set-pre-post
  "Insert `new-value` for :prepost into the params at `index` (or 0) of the `conformed-data`"
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
    (walk/postwalk
      (fn [node]
        (cond
          (and (= index @index-count)
               (vector? node)) (cond
                                 ;; There is an existing prepost property at the index
                                 (and (= (first node) :prepost+body))
                                 [:prepost+body (assoc (last node) :prepost new-value)]

                                 ;; There is not an existing prepost property at the index
                                 (and (= (first node) :body)
                                      (= (first (last node)) :body))
                                 [:body [:prepost+body {:prepost new-value
                                                        :body    (last (last node))}]]

                                 :default node)
          (and (map? node)
               (:params node)) (do (vswap! index-count inc)
                                   node)
          :default node))
      conformed-data)))

genRaiy19:12:11

actually that was the more complex function but you get the idea 🙂

genRaiy20:12:17

for the simpler one I did this

genRaiy20:12:20

(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
    (walk/postwalk
      (fn [node]
        (cond
          (and (map? node)
               (:params node))
          (let [node' (cond-> node
                              (= index @index-count) (assoc-in [:params] new-value))]
            (vswap! index-count inc)
            node')
          :default node))
      conformed-data)))

genRaiy20:12:42

thanks to @U051N6TTC, @U0P0TMEFJ and @U0522TWDA for the suggestions

👍 1
genRaiy20:12:02

most especially to @U051N6TTC in fairness 🙂

❤️ 2
genRaiy14:12:58

I'm feeling awkward about the atom and also the style in which I update it so wondering if some fresh eyes could bring a more elegant approach

quoll15:12:41

Maybe I need to take a further step back, but at first blush I’d probably just tweak it a little:

(defn set-params
  [conformed-data new-value index]
  (let [index       (or index 0)
        index-count (volatile! 0)]
    (walk/postwalk
      (fn [node]                   
        (if (and (map? node) (= index @index-count) (:params node))
          (do
            (vswap! index-count inc)
            (assoc node :params new-value))
          node))
      conformed-data)))

quoll15:12:41

i.e. switch to a volatile, rather than an atom, and only test once

quoll15:12:15

oh… I missed that you increment either way. Hang on. Let me update it

quoll15:12:03

(defn set-params
  [conformed-data new-value index]
  (let [index       (dec (or index 0))
        index-count (volatile! 0)]
    (walk/postwalk
      (fn [node]
        (if (and (map? node) (:params node))
          (do
            (vswap! index-count inc)
            (update node :params #(if (= index @index-count) new-value %)))
          node))
      conformed-data)))

genRaiy15:12:03

hah was just going to mention the condition specified by the if will mean that the inc only occurs once so .... cool update

quoll15:12:05

The update runs either way, which implies a little work, even when the index doesn’t match, but when the old value is returned then the object doesn’t change

quoll15:12:17

drat… I put the vswap in too soon! My brain is still sluggish today. Sigh

genRaiy15:12:23

I think I would stick with the atom though cos it's not exactly critical perf

genRaiy15:12:06

yeah, the inc should be after ... so not sure if that works

quoll15:12:46

I see the choice between atom and volatile to be based on: • scope • threads Atoms used to be a default, but now that volatiles are here I think they make more sense to only use them when they are in a multithreaded environment, or when the scope is outside of a single function

genRaiy15:12:25

ok , that's a good rule of thumb

quoll15:12:43

Can dec the index in the let block. I think I’d rather do that than save a value as I called assoc on, update the index, and then return the saved value. It works, but it feels clumsy

quoll15:12:20

(I edited that second code block)

genRaiy15:12:12

on reflection I'll adopt the volatile and leave the rest as is cos it's starting to get difficult for my 🧠

quoll15:12:36

At this point, the vswap! could be embedded in the update which would avoid the need for the do. But I don’t know that this is idiomatic. I hate the idea of a side-effect happening during the update

genRaiy16:12:01

I have other cases with several conditions where I could not do the inc inside. Given that they will be read together, it makes more sense to me to adopt a more obvious if clunky approach

hoynk14:12:21

So, about the log4j / logback vulnerabilities. I don't have a dependency on any of them. We use timbre, is there anything I should update? tools.logging is a dependency

Alex Miller (Clojure team)14:12:37

tools.logging does not depend on log4j or logback so you probably don't need to do anything, but would be good to check your full dep tree to ensure you're not including log4j if you don't think you're using it

hoynk15:12:04

I did install the plugin com.livingsocial/lein-dependency-check that creates a report that shows all dependencies and there was no log4j or logback, just tools.logging.

hoynk15:12:19

It did show some vulnerabilities that we are figuring out how to deal with, but nothing on log4j. I was just wondering if there was some feature on Timbre that could have the same problem.

noisesmith20:12:17

the double edged sword of using something niche- it probably exists, it isn't being exploited on a mass scale

vemv13:12:31

btw, although I'm biased (as a maintainer), I recommend nvd-clojure over lein-dependency-check. Experience has shown us that this type of job is generally best not offered as a lein plugin, which is why it's a discouraged api in nvd-clojure (soon to be entirely removed)

hoynk12:12:18

will check, thanks for the pointer

🙂 1
pez16:12:10

Is the ordering of the keys in a destruction significant? The compiler says so:

(defonce !db (atom {:config {:bar "baz"}}))

(defn foo
  [{:keys [config !state]
    :or {!state !db
         config (:config @!state)}}]
  config)
;; => Syntax error compiling at (destruct.clj:8:17).
;;    Unable to resolve symbol: !state in this context
While, if I swap the keys :keys [!state config], the compiler is happy.

Alex Miller (Clojure team)16:12:37

it's undefined so don't rely on it

Alex Miller (Clojure team)16:12:00

:or should not depend on the state of possibly destructured keys like this

Ben Sless16:12:07

It also evaluates the or arguments unlike the or macro

pez16:12:22

What should I do instead?

Joshua Suskalo16:12:47

I would recommend doing this type of logic in a let binding inside the function

1
☝️ 3
🙏 1
pez16:12:29

Thanks. Makes sense. And what about the difference in the compiler’s behaviour? Is it the behaviour that is undefined?

Alex Miller (Clojure team)16:12:29

there is no defined order that the destructuring occurs or when the :or values are computed

👍 2
pez16:12:00

Gotcha. Thanks!

hoynk16:12:20

Is there a simple way to find which dependency on project.clj links to a specific java package jar dependency?

lukasz16:12:14

lein deps :tree ?

hoynk16:12:52

First, thanks. That is exactly what I wanted!

hoynk16:12:36

When I use that command, it shows some sugestions of :exclusions. Is it just to avoid having two paths to the same library?

lukasz17:12:43

Yes, you can setup global exclusions and add a specific version of a given dependency, or setup exclusions on per-direct dependency basis. Leiningen picks a specific version automatically for you if it find multiple artifacts with different versions, most of the time it's not what you want and it's best to ensure you pull in only one version in given project.

hoynk17:12:36

Can I use this to change the version of the dependency of a specific library?

hoynk17:12:16

For example, I want to upgrade the jetty package used by ring-jetty-adapter. So I could exclude it on the import of ring-jetty-adapter and put a specific version on my project as a top dependency?

lukasz17:12:13

Yes, you'd exclude jetty-server from ring-jetty-adapter, and then provide your own version of jetty-server as a direct dependency. here's an example: https://github.com/nomnom-insights/nomnom.duckula/blob/34107b2927f3a5a3db45b0947cc9bd8f66418d60/project.clj#L34-L38

seancorfield20:12:17

Note that in a deps.edn project, exclusions are rarely needed because the algorithm to pick versions is more predictable than Leiningen's and you can "force" a version by specifying it as a top-level dependency. Leiningen can essentially choose a "random" version when there are conflicts/ambiguities. In a deps.edn project, you'll get the newest version in such situations. We used to have a lot of exclusions when we used Leiningen and we carried that over to Boot (which uses the same algorithm as Leiningen). We have hardly any exclusions now since we switched to deps.edn (in 2018).

hoynk20:12:43

Good to know!

lukasz21:12:59

@U04V70XH6 oh, that's neat - I didn't realize that. Do you happen to know/seen a relatively recent leiningen -> tools.deps + tools.build migration guide? Not having to worry about exclusions and using the "blessed" stack is slowly convincing me to move off Lein

seancorfield21:12:35

@U0JEFEZH6 I don't know of any such guide. So much will depend on how fancy your project.clj is.

seancorfield21:12:26

If it's basic and you aren't relying on lein plugins in your user profile, it's really easy to migrate -- with the caveat that it also depends on how much of Leiningen's "included batteries" you use.

lukasz21:12:19

At this point it's dependency management, uberjar compilation and launching the nrepl server in development. We can do without the uberjar and leverage Docker image layers, so it's not a hard requirement.

lukasz21:12:59

Nice, I'll add build-clj to my list, thank you!

seancorfield21:12:58

HoneySQL used to be a Leiningen project but I switched it to deps.edn as part of the 1.0.444 release and these days the whole CI pipeline is based on build-clj -- and it runs both cljs and clj tests, runs Eastwood, runs all the documentation examples as tests, automatically builds and deploys its library JAR. So that might be a good project to delve into the history of.

seancorfield21:12:47

Happy to answer Qs about that via DM if you want.

lukasz22:12:48

I'm still assessing if/when we're going to switch - we are managing ~50 Clojure projects (libraries and applications) so I want to make sure that the migration is a smooth as possible, especially in the ones where we have to inject special Maven configs etc. I'll keep your offer in mind though, I really appreciate it!

seancorfield22:12:07

Are those separate repos???

lukasz22:12:43

Yes, for now

lukasz22:12:20

We are migrating to a proper mono repo soon, that will help a bit. It's not as painful as it sounds as we have a lot of tooling around keeping everything up to date, share components etc and for the most part nobody has to run the whole stack locally

1
hoynk16:12:51

hmmm... cool!

emccue17:12:09

clj -Sdeps for deps.edn projects

Alex Miller (Clojure team)17:12:12

you mean clj -Stree probably (or clj -X:deps list)

lilactown17:12:45

are nested transients ever worth it?

Ben Sless18:12:14

I found that transients usually pay for themselves after about three operations. Since transients are not exactly mutable I think they could pay off, but it would heavily depend on the scenario and you'll need to program the solution carefully

quoll18:12:27

I’ve tried this myself, but depending on the depth and width of the nesting, my experience has been that the cost of persisting them wasn’t worth it. I’ve been looking at using transients at the top levels, but I haven’t yet benchmarked this

lilactown18:12:42

I went with a volatile for now. seems fast enough

Joshua Suskalo19:12:03

transients and volatiles are separate ideas. transients are fast ways to collect many updates to a single data structure. volatiles are a way to have thread-local state. I'm curious how you used a volatile to replace a transient?

Joshua Suskalo19:12:24

For clarity, you cannot perform mutating updates on transients. That is to say, in general you cannot expect the key :a to be on the returned value in this expression:

(let [ret (transient {})]
  (assoc! ret :a true)
  (persistent! ret))

wotbrew20:12:25

the issue I've always found with nested transients is the book keeping necessary to record which keys have been 'opened', kind of wish it was different as I can imagine a bunch of places I could use nested transients if it were a trivial 1/2 line optimisation like transient and persistent! are. I'd be surprised if it wasn't always faster beyond 2-3 operations and in some case significantly so if you keep banging on the same keys, but hey cache locality, tlabs/gc get in the way of my intuition all the time so who knows.

quoll20:12:08

rewinding a little… I don’t see how volatiles could be used to replace the use case of a transient. That said, I’ve used volatiles to hold a transient, then passed that volatile down a stack with various functions adding to the transient and updating the volatile as it goes. It’s totally not functional, but it doubled the speed of the previous functional code, so it was worth it.

lilactown21:12:03

sorry yeah, I was trying to figure out how to structure my algorithm in a more performant way. one way was to keep the loop recur and use a transient. another was to do it in a purely mutable fashion and use a volatile. I ended up with the latter because I have a map of at least 2 depth I'm changing and doing that with a transient didn't seem fun

Ben Sless21:12:03

You can just work with a Java HashMap and in the end build a Clojure hash map from it

Ben Sless21:12:39

And you can always rethink the original algorithm

Ben Sless21:12:54

Is it for Pyramid?

Ben Sless21:12:40

Is it for term rewriting during the query or some other operation?

lilactown21:12:54

it's for normalizing

Ben Sless21:12:46

Do you have a flame graph?

Ben Sless22:12:26

It's always good to know what's eating your CPU

lilactown22:12:54

i'll do some profiling of the version that uses fast-zip. i would much rather this operation not be limited by the size of the stack

lilactown22:12:28

might not be worth it since i'm 90% sure it's just allocation of all the zipper objects, and the fact that zippers can't really be used (AFAICT) in a depth-first way

Ben Sless07:12:08

I'm wondering if something like using eduction can help

(defn eprocess
  [o]
  (cond
    (map? o)
    (->Eduction (mapcat (fn [[k v]] [k (eprocess v)])) o)

    (coll? o)
    (->Eduction (mapcat eprocess) o)))

Ben Sless07:12:59

You can also reify the stack

(defn process
  [^Iterable o]
  (loop [it (.iterator o)
         stack ()
         acc []]
    (if (.hasNext it)
      (let [x (.next it)]
        (cond
          (map? x)
          (recur (.iterator ^Iterable x) (cons it stack) acc)

          (coll? x)
          (recur (.iterator ^Iterable x) (cons it stack) acc)

          :else (recur it stack (conj acc x))))
      (if (seq stack)
        (recur (first stack) (rest stack) acc)
        acc))))

lilactown15:12:54

the problem I'm running into w/ using my own stack is I need to maintain the parent->child relationship between nodes in the tree

lilactown15:12:21

I want to both index each entity, as well as edit any entities that appear in each subtree. i.e.

{:id 1
 :foo [{:id 2
        :foo [{:id 3}]}
       {:id 4}]}
should be processed into the following:
[{:id 1
  :foo [[:id 2]
        [:id 4]]}
 {:id 2 :foo [[:id 3]]}
 {:id 3}
 {:id 4}]
so to do it all in one pass, I need to both edit each subtree (replacing each entity with a reference tuple) and add each entity to the stack

Ben Sless15:12:24

I'm not sure you lose it here. This is just a rough sketch but you can push whatever you like onto the accumulator, add whatever bindings you want, etc.

Ben Sless15:12:52

The idea was to first create a linearized enumeration, then iterate over it. Then you'd say those are two passes, true, but then translate the loop into an iterable

Ben Sless15:12:26

Letting you consume it in a single pass with a hidden stack somewhere inside

lilactown15:12:03

hmm. not sure I understand yet but that sounds interesting

Ben Sless15:12:44

Can't you get the same result with a postwalk?

lilactown15:12:21

I thought clojure.walk uses the call stack

Ben Sless15:12:31

Hm, maybe you're right, but if we recognize it as a postwalk perhaps we can implement one ourselves with a reified stack

Ben Sless15:12:09

Can always cps transform and trampoline :)

😭 1
lilactown15:12:24

yeah I think I'm just struggling to wrap my head around the right way to transform the algo into a reified stack

lilactown15:12:36

shoulda took some more CS classes I guess

Ben Sless16:12:58

I think I took about 3 and I came out alright 🙃 Start with a cps transform?

Ben Sless16:12:59

And you don't have backtracking to worry about

lilactown20:12:21

@UK0810AQ2 CPS + trampoline ended doing really well after a lot of trial and error: https://github.com/lilactown/pyramid/pull/16/files

lilactown20:12:22

it's about 30% faster than what's in main, correct (AFAICT) and works up to a depth of 40,000 before it it overflows on my machine. much better than the measly 3000-4000 depth in my PR I posted in this conversation originally

Ben Sless21:12:42

Now, would it be faster to pass db and entities as arguments instead of binding them as volatiles?

lilactown21:12:18

couple of other opportunities to squeeze out some more perf too, like converting all the local build up of collections into transients

lilactown22:12:44

the current algorithm definitely depends on mutating the volatiles in the outer scope to work. i'd have to restructure it further to ensure i'm capturing entities as it descends

lilactown22:12:29

i might call this Good Enough:tm: for now

Ben Sless05:12:14

Definitely. There are three other opportunities for gain: use reduce instead of loop, unroll update-in, avoid merge. Churning merges is very slow

jdkealy20:12:53

Are there any clojure libraries that allow you to have a callable inline function that doubles as a doc string ? e.g.

(defn add-to-numbers 
 "a function that adds two numbers
  (add-to-numbers 1 2) 
  => 
  3 
  "
  [a b] 
  (+ a b) 
)
It's tricky with escaping strings, i.e.
(defn say-hello 
  " say hello to username 
   (say-hello \"John\") 
   => 
   Hello, John
  "
  [username]
  (str "Hello, " username) 
)

pavlosmelissinos21:12:06

Unit tests are arguably better at that than escaped strings and they integrate better with automated test runners, so I don't see the appeal to be honest. However: 1. the :test https://clojure.org/reference/special_forms#def (TIL, this is actually neat!) 2. if you really need the examples to be in strings, there are a couple of small libraries that are inspired by python's doctest, e.g. https://github.com/liquidz/testdoc (no idea how good it is, I've never used anything like it)

Cora (she/her)22:12:31

which is different but also functions as documentation

seancorfield22:12:23

You might like https://clojuredocs.org/clojure.test/with-test (built-in, via clojure.test) that lets you put usage-examples-as-tests right there alongside your function definition.

seancorfield22:12:37

(caveat: not all IDE/test tooling expects this!)