Fork me on GitHub
#clojure
<
2022-02-28
>
Kris C11:02:50

I have a problem understanding what is happening here:

(defn prepare-specter-fn [specterPathStr]
  #(select-first (load-string specterPathStr) %))

(defn prepare-sort-fns [sortAttrs]
  (mapv #(prepare-specter-fn (:sortExpression %)) sortAttrs))

(def pathExpStr "[:some-attr]")

(def sortAttrs [{:sortExpression "[:some-attr]"}])

(def ptest #(select-first (load-string pathExpStr) %))

(def ptest2 (first (prepare-sort-fns sortAttrs))
select-first is a specter macro why is ptest2 not the same as ptest? And what should I do to make it so?

Kris C11:02:32

ptest2 and ptest behave the same, but ptest2 is much slower...

p-himik12:02:05

How did you measure?

p-himik12:02:15

Also, it helps having the full code - including imports and tests. Apart from helping with reproducing the issue, what you think is the most important part can easily end up being completely irrelevant.

Kris C12:02:59

I am using those functions to sort a list of nested maps (count about 900)

Kris C12:02:47

What I find "weird" is that if I evaluate ptest and ptest2 in the REPL I get the following:

user> ptest
#function[user/ptestF]
user> ptest2
#function[user/prepare-specter-fn/fn--445128]

Kris C12:02:43

Does that ring any bells?

p-himik12:02:03

That's irrelevant - those are just automatically generation function names.

Kris C12:02:30

ok... for the rest, I think the example should provide enough context for figuring out the issue?

Kris C12:02:23

in my very limited clojure knowledge, the resulting functions (`ptest` and ptest2) should be the same, but I am obviously missing something

Ferdinand Beyer13:02:52

What does load-string do?

Kris C13:02:03

reads and evaluates a form

p-himik13:02:15

A minimal reproducible example would be perfect, yes.

Ferdinand Beyer13:02:55

Ooops, sorry, thought it was some custom implementation 😄

Ferdinand Beyer13:02:11

Specter’s macros try to compile the given argument for optimal performance. Macros are evaluated at read time. For ptest, all information is available at macro read time:

(def ptest #(select-first (load-string pathExpStr) %))
It will resolve pathExpStr, load it, and produce an optimised function. For ptest2, this cannot be done:
(defn prepare-specter-fn [specterPathStr]
  #(select-first (load-string specterPathStr) %))
When this Clojure code is compiled, specterPathStr is unknown. So the select-first macro cannot compile it yet. That’s why it is slower.

p-himik13:02:56

The above statement is incorrect. pathExprStr is a string in run time, not compile time. load-string is not a macro. select-first is a macro but it doesn't do any job - it just replaces itself with a couple of function calls that are done in run time. From that point of view, ptest and ptest2 are completely equivalent.

Kris C13:02:55

Same thing if I don't use specter:

(defn prepare-fn [exp]
  #(get-in % (load-string exp) ))

(defn prepare-sort-fns [sortAttrs]
  (mapv #(prepare-fn (:sortExpression %)) sortAttrs))


(def ptest (sort-by #(get-in % [:some-attr]) col))


(def sortAttrs [{:sortExpression "[:some-attr]"}])

(def ptest2 (first (prepare-sort-fns sortAttrs))

Kris C13:02:32

if I do sort-by with ptest it's fast, while if I do it with ptest2 it's very slow

Ferdinand Beyer13:02:41

@U2FRKM4TWselect-first expands to a form that uses another macro: path. This one looks into &env for whatever smart thing specter does. I suspect that this can do optimisations when the path is known at macro-compile time that it cannot do otherwise.

Kris C13:02:49

@U031CHTGX1T it doesn't matter, I have given an example without specter (using get-in instead) and it's the same problem...

Ferdinand Beyer13:02:57

No, it’s not the same. One example uses load-string, the other one uses the parsed result. How can you expect these to perform the same?

p-himik13:02:50

@U031CHTGX1T Huh, you're right, thanks. One more reason for me to dislike Specter, heh.

👍 1
Ferdinand Beyer13:02:36

@U013100GJ14 In your get-in example, try loading the string only once and see how it compares:

(defn prepare-fn [exp]
  (let [p (load-string exp)]
    #(get-in % p)))
For specter, try using comp-paths and compiled-select-first in the same way.

Kris C13:02:48

uh, thanks, @U031CHTGX1T, that seems to be the problem!

Kris C13:02:33

hmm, but still...

Kris C13:02:28

I will post a complete example without specter in the main thread, please reply there. And thank you so much for your help!

Kris C14:02:55

I have deleted the other post, does not contribute because of the bug @U031CHTGX1T quickly found 🙂

Kris C14:02:48

@U031CHTGX1T thanks again!!!

👍 1
Kris C13:02:46

The complete source for the previous problem I have posted (no specter lib usage, vanilla clojure):

(defn prepare-test [exp]
  (let [p (load-string exp)]
    #(get-in % p)))

(defn prepare-sort-fns [sortAttrs]
  (mapv #(prepare-test (:sortExpression %)) sortAttrs))

(def col (shuffle (mapv #(hash-map :attr %) (range 100000))))

(time (def a (sort-by #(get-in % [:attr]) col)))

(def pfn (prepare-sort-fns [{:sortExpression "[:attr]"}]))
(time (def b (sort-by (first pfn) col)))
Elapsed times:
"Elapsed time: 42.302409 msecs"
"Elapsed time: 481.693445 msecs"
Why is the second one so much slower?

Ferdinand Beyer14:02:47

Fix your first example from [:encounter/id] to [:attr] and you will see they perform about the same

Kris C14:02:01

deleting question, does not contribute to anything due to a bug in source...

hanDerPeder19:02:47

given:

(ns foo
  (:require [bar :as b]))

(ns bar
  (:require [foo :as-alias f]))
you get a circular dependency error, but shouldn’t this be fine? all it’s doing is enabling me to write ::f/my-key in bar instead of :foo/my-key (which works).

hiredman19:02:26

when you say you get a circular dependency error, what error are you getting?

hiredman19:02:17

you could be seeing an error from tooling that doesn't understand as-alias or you could be using an older version of clojure that doesn't understand as-alias

hanDerPeder19:02:42

ah, tools.namespace. thanks

hiredman19:02:56

you can monkey patch tools.namespace to make it work

hiredman19:02:31

(in-ns 'clojure.tools.namespace.parse)

(defn- deps-from-libspec [prefix form]
  (cond (prefix-spec? form)
          (mapcat (fn [f] (deps-from-libspec
                           (symbol (str (when prefix (str prefix "."))
                                        (first form)))
                           f))
                  (rest form))
          (and (sequential? form) (some #{:as-alias} form))
          nil
        (option-spec? form)
          (deps-from-libspec prefix (first form))
        (symbol? form)
          (list (symbol (str (when prefix (str prefix ".")) form)))
        (keyword? form)  ; Some people write (:require ... :reload-all)
          nil
        (string? form) ; NPM dep, ignore
          nil
        :else
          (throw (ex-info "Unparsable namespace form"
                          {:reason ::unparsable-ns-form
                           :form form}))))


(in-ns 'user)

👍 1
andy.fingerhut05:03:38

Is there already a proposed patch/issue for tools.namespace for this?

hiredman05:03:22

there is, the ask question links to the ticket, the proposed patch is slightly different

👍 1
hiredman05:03:17

that monkey patch is just what I ended up with when I was having trouble with tools.namespace, I didn't dig into it a ton and I am not sure what behavior differences would result in the above vs the patch in jira

zendevil.eth23:02:14

what’s the best way to check that a map contains only certain keys and not others?

zendevil.eth23:02:05

Let’s say I have all the necessary and sufficient keys in a list:

[::x ::y ::z]
How do I make the check that the map only contains those keys?
{::x 1 ::y 2 ::z 3}

hiredman23:02:19

(= (set (keys m)) #{...})

Alex Miller (Clojure team)23:02:05

I think you can do better with something like (every? (set [::x ::y ::z]) (keys %))

Alex Miller (Clojure team)23:02:50

esp if the necessary set is much smaller than the key set (which I assume is the case)

hiredman23:02:53

that ensures that all the keys in the map are in the set, but not that all the keys in the set are in the map

Alex Miller (Clojure team)23:02:52

I think the intent here is the former based on the other conversation in spec