Clojurians Log v2

Micro-benchmarks aside, find + current implementations of maps allocate. It can be that it isn't faster to use get in some narrow scope, but find adds gc pressure where avoidable.

I'm just a bit disappointed that CLJ-1789 was merged so early. Could have waited until Bastille day 🇫🇷

This observation of the allocations incurred by find is not detached from reality either, it is based on real high throughput scenarios where we tried to reduce allocations as much as possible. It may seem redundant when using persistent collections, but it made a measurable impact.

+1 for allocations, would be nice to have fewer of those

How does :select behave when :or is specified? Given the wording: > Binds a name to a subset of the map ... > I infer that anything in :or is just ignored. Something that I've seen affects to new clj developers and it's not ergonomic (IMO) is that :or bindings don't get applied if the key exist in the map with nil. Would be nice to pay attention to this now that destructuring is being extended. I'd say this has been an issue at least once in every Clojure team I've worked with, so it's probably generally confusing people.

@ulises.ssb506 Can you make an example?

A couple of comments. I'm trying now to participate more here. But seeing a thread like this devolve from talking about the how and why of new features to marginal optimization appeals is not great. Might I suggest the right place and time for that is on the tickets before they are screened/oked? There will always be places where you can trim a cycle or an allocation, and taken to the nth degree you are back to programming in C++.

@ulises.ssb506 The point of :or is to supply a value for bindings when the key is not present in the map. So there's no reason to mention a key only in :or and not in a binding, and the binding will include it in :select. As far as mapping keys to nil, that is in general an anti-pattern in idiomatic Clojure. If we don't have information in Clojure we leave the key out of the map. This is more ergonomic than place/field-oriented records with missing/optional values since you can now talk about what you do/don't know using keys and not have to inquire of every "field". Thus all the logic in :or and the new checked directives is based on the presence of a key and not its value. A key mapped to nil is not a missing key.

@robert-stuttaford the way to think about & is bound (before) or not (after). Being able to talk about things you wouldn't otherwise bind at this scope lets you define 'boundary' interfaces that check/document the needs of nested scopes. It interacts with :select but the purpose is not (just) driving :select.

Does a value provided by :or end up in the :select map?

@richhickey I agree the discussion had outgrown the purpose of an announcement thread. I am happy to see you participating more. As for the place for discussions, IIUC you mean the jira tickets, or ask.clojure, or both. Since the :select ticket came to life 5 hours before being released and is already OK'd, what would be the correct discussion protocol here? For me personally, I'd be more interested in early level discussions, but I didn't know keys! and :select were even coming. What is the level of involvement you expect from the community here, and at which stage(s)?

This seems a good time for people to (re)watch https://www.youtube.com/watch?v=YR5WdGrpoug 🙂 This is where a lot of that thinking landed. The connection to spec is a logical next step (no promises about that for 1.13 though we keep working on it)

@xifi We look at both ask and the Jira tickets. The https://clojure.atlassian.net/browse/CLJ-1789 has been around for a while. You can see things move into work by looking at the Fix Version field in Jira (maybe @alexmiller can share the Jira incantation to query those) I'm not trying to dictate behavior on the part of the community, nor make promises of my own. I'm just trying to keep myself from being overwhelmed :)

@xifi I will say that sometimes a feature lands in an alpha that is a result of a lot of hammock time and internal-to-core-team discussion, without a specific jira/ask, but as a result of one or more jira/ask issues—sometimes as a stepping stone toward addressing other jira/ask issues. That's why we've adopted alpha builds so quickly at work, going back to the 1.3 alphas in 2011, so we can test/explore the feature and provide feedback based on actual use. Sometimes we get a great new feature we hadn't even thought to ask for 🙂 @jason.bullers Fire up a REPL with 1.13 Alpha 3 and try it—that will answer your question.

@richhickey Funnily enough, I'd gone back and re-watched Maybe Not last night, because of this thread!

@seancorfield yeah, I understand the status quo. I was asking around to see if it's changing in any way. As I understand it's not. While I can say I dislike the status quo I'm also accepting it as it is since it's none of my business. I am happy to see Rich around though 😉

@seancorfield will try later when I'm at my computer. Just thought I'd ask in the meantime 🙂

And I just found my answer reading the tests in the patch: or does not appear in the select map

@jason.bullers What do you mean? That if you default a not-present binding using :or but it otherwise isn't declared after the ampersand it won't be in the select map?

:or binds separately and does not affect the :as map or the :select map.

I see. So you can't use :or to default missing entries inside the select map.

:select is a subset of the incoming map, which of course does not include :or defaults (same as :as)

@didibus I was wondering about something like: {:keys [verbose?] :or {verbose? false} :select debug-opts} Would debug-opts always contain verbose?? The alternative I guess is to use merge somewhere, but then you don't have it in the params list and have to repeat keys. It sounds like :or is specifically for bindings though, so doesn't apply to selecting out from the input map (which would not have the key to select out)

an idea I've had in this area (but not discussed with Rich) is to add a :defaults map which would be merged before the incoming map and before all the rest of destructuring is applied.

I think that addresses what people seem to constantly want (but not get) out of :or in their use cases

#clojure probably?

either #clojure or #clojure-dev if it's a question for dev team

:or does not bind. It supplies a not-found value to the get created by a binding.

Ya I see, it makes sense to think of select as selecting from the input map and not selecting from the bindings.

did we know about :select coming? The JIRA ticket is 5 hours old

At least part of it was committed right after alpha2 dropped 🙂

I knew about it

Better phrasing for :select - binds a name to a subset of the map being destructured containing only the keys mentioned (anywhere) in the binding form

via select-keys

I told Chouser about it at the last Conj

that's how long this stuff cooks

@borkdude Should I open an issue for :select support in clj-kondo or are you already on it? 🙂

this is a minified example from the tests

(def M
  {:a 1
   :b 2
   :c 3
   :d 4})

(let [{:keys   [a b & c z]
       :keys!  [d]
       :select keys-sel} M]
  keys-sel)
;;=> {:a 1, :b 2, :c 3, :d 4}

It's mentioned in :keys. It just isn't bound as a symbol.

@seancorfield wait, what? clojure is developing faster that than I can keep up now!

I haz a red squiggly 🙂

point_up::skin-tone-2 Nice that we can :as the whole map and :select just the named keys!

@seancorfield yes, issue welcome!

I wonder if this somehow relates to spec2 stuff

https://github.com/clj-kondo/clj-kondo/issues/2874

@richhickey I understand that, I was wondering if this was public knowledge I missed

@seancorfield so treat :select as :as but what would be "gravy" on top of that in your mind?

I think Alex mentioned Rich had committed part of the :select implementation for Alpha 3, somewhere in the Alpha 1 or 2 discussions

we can use the. knowledge that the alias only contains certain keys and this can integrate with the "type" system

@borkdude Oh, I just meant that if you figure out any flow analysis, like you did with :keys! and spotting a missing key statically... Perhaps if you see (:foo thing) and you know :select thing cannot have a :foo key.

right

well (:foo m) doesn't warn currently on maps that clj-kondo know it doesn't have :foo since it's not required ;)

but I get your idea better now

I noticed there's an edge case with double eval/expansion that might be worth looking into. (let [{a (compute) :select m} ...]) now calls/expands (compute) twice.

"Now" as in, there wasn't an opportunity to duplicate computed expressions like this before AFAIK.

Does (compute) produce a keyword? I'd never even considered that might be legal syntax?!?

i'm surprised the spec doesn't reject this

Which of these do folks prefer? And which is faster (I assuming the first version is faster, since the second version does some destructuring and also calls select-keys under the hood)?

fields  (select-keys (:params req) [:country :zip :city])
        {:select fields :keys [& country zip city]} (:params req)

for normal code I'd prefer the first, but for function "signatures" the destructuring is probably nicer since you get this info back in multiple places (docstrings, etc)

(let [{binding expression} map]) is equivalent to (let [binding (get map expression)])

Would be nice to consider doing some perf improvements to select-keys now that it’s “easier” to use? seq -> reduce and find -> get give 8-40% perf improvement depending on map / keyseq sizes

https://clojure.atlassian.net/browse/CLJ-1789 is implemented in Alpha 3.

Nice, though, find->get (with sentinel not-found) also gives a perf boost (7-13% faster)

It's clojure.lang.RT/find, not clojure.core/find tho'...

yes, it’s still faster to use get

it wasn't when we measured it

in this context

Presumably because using get with a sentinel means an additional check on the return value (instead of just if-let on the entry) and then having to conj! a key/value that has to be constructed, instead of using the entry.

https://gist.github.com/vlaaad/1e97f818cc66d0225f6e76f4c424ee13 hmm, on my machine: small is same, large is 19% faster

sentinel with identical check shouldn’t be slower than null check I think…

if you want to create an https://ask.clojure.org and share further info (w/ benchmark code, timings, Java version, etc), happy to review it

ideally comparisons would include compiled / direct-linked versions ala core so that it's apples to apples

I shared benchmark code above, is it fine for ask?

yes, please also include java version

ok (25)

we are moving to 17 as a baseline, so 17, 21, and 25 are most interesting to check

I really enjoy seeing this fast turnaround between core and the community!

FWIW, on JDK 27, I got similar results to Vlad, both with and without direct linking, running his benchmark.

@alexmiller We have Alpha 3 on our QA cluster, so it'll probably get promoted to production tomorrow.

@borkdude Confirmed :select is recognized in latest nightly LSP build (with clj-kondo master in it).

also try passing the select map to a function that requires a certain key, without that key - should be a warning

@vlaaad I don't think those large results are relevant to real life. As I said, I already ran that exact test.

What's the downside to using get instead of find? Large maps sound like an uncommon real world use case, but if the implementation is changing already for performance, why not pick the one that's faster in more scenarios? Is there some downside or risk to the get approach over RT.find? Or is it more that it's not worth reopening for the benefit?

So & for map destructuring is new? How does it work if you don't use the shorthand form?

@didibus what are you asking?

@jjttjj Showed an example from the test in the patch like: (let [{:keys [a b & c d]} m] ...) And I don't remember the ampersand syntax inside :keys before, unless I just didn't know. So is this new ?

That's new

In :keys! etc, it marks required keys that do not get bound. In :keys etc it has no effect, serves only as documentation of optional keys

@jason.bullers I really don't like spending time on pointless optimizations. The patch had find, I thought get might be faster and it isn't for real-world cases, so the screened patch was used and we're going to get on with our lives.

@didibus I have this stuff in production already... Ya gotta keep up man 😁

confirming: & there to specify keys for :select to use, but not for destructuring. if i also want a key destructured, it should precede &? (and ofc :keys! will throw if such a key is missing)

@robert-stuttaford yes, but if you don't want it to be required, you can use regular :keys and those will also be present in :select

very cool!