Fork me on GitHub
#clojure
<
2020-06-10
>
lilactown00:06:33

fun bug I just tracked down, caused by a fact I didn't know: sorted-set-by uses the comparator to assert identity

lilactown00:06:41

I had something akin to:

(sorted-set-by :order #{:order 2 :foo "bar"} #{:order 2 :foo "baz"})

lilactown00:06:01

couldn't figure out why one of those wasn't showing up!

andy.fingerhut00:06:09

The comparators guide article on http://Clojure.org mentions this and many other useful facts about using comparator functions in Clojure

andy.fingerhut00:06:19

Without that behavior, there would be no way to maintain a consistent order between elements considered equal by your comparator function, unless perhaps one created another kind of sorted collection that had a “tie breaker” comparator for things considered equal by the primary comparator fan

lilactown00:06:18

yeah. for my case, I don't need a stable order between elements whose orders are equal

craftybones04:06:42

on the Clojure website at https://clojure.org/reference/metadata, where it describes vary-meta , it reads

Returns an object of the same type and value as obj, with (apply f (meta obj) f & args) as its metadata.
Shouldn’t it be
(apply f (meta obj) args)

seancorfield17:06:32

@srijayanth Alex merged my PR so vary-meta is now correct on that page. Thank you for spotting that and mentioning it, so the docs can be improved!

seancorfield04:06:33

Looks like a typo, the docstring says

user=> (doc vary-meta)
-------------------------
clojure.core/vary-meta
([obj f & args])
  Returns an object of the same type and value as obj, with
  (apply f (meta obj) args) as its metadata.
nil
user=>

seancorfield04:06:11

The arguments are listed incorrectly too.

seancorfield04:06:48

@srijayanth Good catch on that. I submitted a PR to fix it.

hindol06:06:31

Since we have namespaces keywords now, is this acceptable or just a hack?

(defprotocol Service
  (start [this])
  (stop [this]))

(extend-protocol Service
  APersistentMap
  (start [this]
    (if-let [start-fn (::start-fn this)] ; Note: namespaced key
      (start-fn this)
      (throw (ex-info "This map does not satisfy protocol 'Service'"
                      {:map this}))))
  (stop [this]
        ;; Similar to start, omitted
        ))
Then all maps having the right namespaced keys "automatically" satisfy the protocol without using with-meta.

vlaaad06:06:00

what's wrong with with-meta ?

vlaaad06:06:15

this looks fine 🙂

hindol06:06:08

Nothing wrong with with-meta except it is per map. In the above approach, you write the rule only once.

vlaaad06:06:24

it is still per map, you just have to put your start function in a different place

hindol06:06:08

You are right, it is still per map. Every map still needs to add the right key. with-meta is slightly longer I think.

seancorfield06:06:59

@hindol.adhya I'm not sure what problem that solves. ::start-fn resolves to a specific qualified keyword, based on the namespace that code is in. How is that better than adding metadata to a map that refers to protocols?

seancorfield06:06:58

You've introduced a convention (that might be different for each library) compared to a standard way to interact with every protocol.

hindol06:06:48

I meant, the ns that is using this will have the :<right ns>/start-fn key. :: is just for this namespace where I have the protocol.

seancorfield06:06:14

Right, but that is a convention that is separate from the protocol itself.

seancorfield06:06:46

You've called it start-fn (and, presumably, stop-fn) in your service -- but other protocols in other namespaces could do something differently.

seancorfield07:06:09

It's much better to use the fully-qualified names of the protocol functions themselves.

seancorfield07:06:14

You've also restricted it to just a hash map -- protocols can be satisfied via metadata for any IObj

seancorfield07:06:36

(so, in answer to your question, I'd say it's a horrible hack!)

hindol07:06:40

I agree with the "deviation from standard convention is a bad thing" bit. But I would argue a fully qualified protocol fn is equivalent to a fully qualified keyword.

hindol07:06:30

🙂 Yeah, probably it is a bad hack. That's why I am seeking opinions.

seancorfield07:06:39

Protocols can be extended to a lot more types than a qualified keyword can participate in.

seancorfield07:06:30

(with-meta [1 2 3] {'com.stuartsierra.component/start (fn [v] ...)}) for example

hindol07:06:25

Yes, and the protocol is still available for every other type. I meant this just for maps.

seancorfield07:06:30

@hindol.adhya I think it's a mistake to take a generalized feature like this and add a convention that restricts it to a narrower type...

hindol07:06:03

I am kind of convinced now that this is a bad idea. Mainly because with-meta is more explicit and the above approach is more implicit.

4
borkdude09:06:49

$ clj -e '(clojure.lang.PersistentQueue/EMPTY)'
#object[clojure.lang.PersistentQueue 0x25ddbbbb "clojure.lang.PersistentQueue@1"]
$ clj -e 'clojure.lang.PersistentQueue/EMPTY'
#object[clojure.lang.PersistentQueue 0x2c104774 "clojure.lang.PersistentQueue@1"]
Calling a field like a method with no args is valid in Java interop?

hindol09:06:59

Does PersistentQueue implement IFn? Curious. EDIT: It does not.

p-himik09:06:23

Heh.

user=> (macroexpand '(clojure.lang.PersistentQueue/EMPTY))
(. clojure.lang.PersistentQueue EMPTY)
But I don't think I've seen it documented anywhere.

borkdude09:06:51

According to the docs it should be (. clojure.lang.PersistentQueue -EMPTY)

borkdude09:06:56

to access the field

p-himik09:06:56

Oh, right.

p-himik09:06:05

Huh. Given this line

maybeField = Reflector.getMethods(c, 0, munge(sym.name), true).size() == 0;
and what follows it in Compiler.java, it seems that (. A x) will call x if it's a method and will resolve x if it's an attribute.

p-himik09:06:40

Actually, it is documented at https://clojure.org/reference/java_interop#_the_dot_special_form

If the second operand is a symbol and no args are supplied it is taken to be a field access - the name of the field is the name of the symbol, and the value of the expression is the value of the field, unless there is a no argument public method of the same name, in which case it resolves to a call to the method.

robert-stuttaford09:06:01

has anyone perhaps used https://github.com/xsc/rewrite-clj to dissoc keys from maps in defs in clj source? i'm trying but i don't grok zippers all that well, or i don't get how this lib works 😅 perhaps you, @borkdude?

borkdude09:06:28

yes, clj-kondo is based on this lib so I know it quite well

borkdude09:06:25

You don't have to use the zipper API, you can also just process the nodes manually

borkdude09:06:28

which I do in clj-kondo

robert-stuttaford09:06:31

ok i'll dig some more, thanks

robert-stuttaford09:06:01

could i zip to find, and then drop back to basic node ops from there?

borkdude09:06:27

yes, you can can find the node and then use z/replace and transform the node using some function

robert-stuttaford09:06:46

ok nice, i think i got it

borkdude09:06:21

is your use case to call this from a Java server process or from a script?

robert-stuttaford09:06:49

a java server. i got it right. culling unused namespaces from our i18n dictionaries 😆

helios11:06:23

is there a way to have edn parsing "leave alone" a tag literal? I.e. {:foo #something "here" #another "here"} i want to specify a custom data-reader for the #something tag, but #another should be left alone (i.e. i need to manipulate a edn file but ignore some tag literals)

p-himik11:06:23

Specify identity as a reader for that tag?

helios11:06:22

No because that would chuck away the tag and only get the value

helios11:06:56

A potential solution is to use {:default (fn [t v] (tagged-literal t v))} as options for edn/read

p-himik11:06:19

What is tagged-literal?

p-himik11:06:13

When you read EDN, you get a Clojure data structure as an output. The reader tags intrinsically cannot be preserved there as they exist only in the scope of reading stuff (thus the name, "reader tag").

p-himik11:06:05

> What is tagged-literal? Ah, found it.

p-himik11:06:35

Ah, I see now that I misunderstood that statement in parentheses in OP. Seems like your solution should works, yes. But only if you handle the resulting object explicitly. E.g. if you want to process them in any way, you will have to unwrap and wrap the values in tagged-literal.

helios11:06:04

In this particular case, as I am merely manipulating a edn file with babashka, i wanted to only do something with some of the tags, and keep the others as tag literals (to be eventually read by another program somewhere else down the pipeline)

p-himik11:06:50

Yeah, I see now, sorry for the confusion. I didn't realize initially that you're writing the file back.

🙂 3
👍 3
borkdude11:06:06

@U0AD3JSHL yes, :default tagged-literal should take care of this

Ykä12:06:58

https://twitter.com/andrestaltz asked on Twitter > What’s the software or tool or language or framework where you’ve felt the most productive? Clojure seems to be pretty popular aswer. 🎉 If you want to answer his poll, it’s still open for 6 hours: https://twitter.com/andrestaltz/status/1270438843880259590

William Skinner14:06:16

Hi, I'm new here! I wanted to get a few recommendations for must have books that teach systems architecture and design. I became interested in clojure back in 2012 after starting my first programming job but only got the chance to use it for toy projects. Now I've come a long way but still don't feel have a true reference for large scale system design. Specifically at my job I've been tasked with rewriting a large batch transaction processing system and would like to read about the state of the art in this field.

lilactown16:06:47

“Designing Data-Intensive Applications” has been recommended to me multiple times and it is currently sitting at the top of my to-read pile 😛

8
lilactown16:06:58

I am in need of an “ordered” set: a set that I can conj objects onto and use a custom comparator so that when I retrieve them, they are in sorted order. but multiple objects can have the same order, e.g.:

(ordered-set
 (comp compare :height)
 {:height 0 :foo "bar"}
 {:height 3 :foo "asdf"}
 {:height 0 :foo "baz"})
any thoughts on what a good data structure might be for this?

lilactown16:06:50

sorted-set treats objects that have the same key as equivalent

lilactown16:06:28

(sorted-set-by
 (comp compare :height)
 {:height 0 :foo "bar"}
 {:height 3 :foo "asdf"}
 {:height 0 :foo "baz"})
;; => #{{:height 0 :foo "baz"}
;;      {:height 3 :foo "asdf"}}

bortexz16:06:42

bar != baz

Derek16:06:41

Isn’t a priority queue the solution here?

lilactown16:06:06

what would that look like, Derek?

Derek16:06:08

or is the uniqueness of the elements essential?

lilactown16:06:32

I never want to process the same element twice

Derek16:06:20

Are you in JVM land or JS land?

lilactown16:06:51

JS, but if there’s a good example of this on the JVM I would appreciate that as well

Derek16:06:20

but this won’t dedupe

p-himik16:06:22

@lilactown (def key (juxt :height identity))? Or what would "the same order" mean?

Derek16:06:57

and it’s mutable

bortexz16:06:09

@lilactown

(sorted-set-by
  (comp < :height)
  {:height 0 :foo "bar"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"})
This for me returns a set of the 3 elements

lilactown16:06:31

@U6CN6JQ22 that’s because < isn’t a valid compare function

👍 4
lilactown16:06:32

@p-himik i’m not sure exactly what you mean. can you elaborate?

p-himik16:06:10

@lilactown If "same order" in your case doesn't care about the order of the items with the same height, then you can just add the item itself to be a part of the key, no?

p-himik16:06:02

BTW (comp compare :height) is wrong.

p-himik16:06:32

Remember how keywords are 1- and 2-arity functions? :) Well, a comparator is a 2-arity function...

lilactown16:06:46

ack right. I do have it correct in my impl 😄

raspasov16:06:45

@lilactown look at the “second-<-with-tie-break” example here https://clojuredocs.org/clojure.core/sorted-set-by

raspasov16:06:02

It says:

;; Here is one way to write a good comparison function.  When the two
;; objects are equal in the parts we care about, use the tie-breaker
;; 'compare' on the whole values to give them a consistent order that
;; is only equal if the entire values are equal.

p-himik16:06:07

Heh, pretty much the same as what I offered, only with a more manual/granular approach.

✌️ 3
bortexz16:06:15

If the comparator returns numbers, no longer removes the ones with same keys

lilactown16:06:21

awesome this is very confidence building. thank you @raspasov @p-himik!

👍 3
bortexz16:06:52

Passing a function that returns positive or negativer numbers seems to not remove same keys

bortexz16:06:54

(sorted-set-by
  (fn [a b] (if (> (:height a) (:height b))
              1
              -1))
  {:height 0 :foo "bar"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"})
=> #{{:height 0, :foo "baz"} {:height 0, :foo "bar"} {:height 3, :foo "asdf"}}

p-himik16:06:48

That's because your comparator is not a comparator.

p-himik16:06:51

You never return 0.

p-himik16:06:23

It may seem fine, but it will most definitely result in unpredictable behavior given enough data, simply because in your case among two maps with the same height, the first one is always smaller than the second one. Even if you swap them.

bortexz16:06:34

thanks for the explanation

bortexz16:06:51

When you say unpredictable, is that for the same elements, it might not always return the same order?

bortexz16:06:52

Now I understand raspasov comment

bortexz16:06:03

I am trying this example, is it possible to do compare on 2 maps?

p-himik16:06:25

Yes, absolutely.

raspasov16:06:48

Hmm… this actually does not work:

raspasov16:06:54

(compare {:height 0 :foo "bar"} {:height 3 :foo "asdf"})
Execution error (ClassCastException) at (REPL:1).

raspasov16:06:09

vectors work

raspasov16:06:09

(compare [0 “bar”] [0 “bar”]) => 0

raspasov16:06:16

(compare [0 “bar”] [3 “asdf”]) => -1

raspasov16:06:32

So… what would be the correct way to compare maps? I am actually not sure

p-himik16:06:35

Oh, huh... My bad then, sorry. I was sure it was possible to compare them.

raspasov16:06:50

Yea… no examples here… either https://clojuredocs.org/clojure.core/compare I guess you can say “this map is equal to that”; but hard to say when a map is “less” or “bigger than” another map

lilactown16:06:16

ah, when I tried p-himik’s example at I actually used hash

👍 4
lilactown16:06:31

I think that might be stable enough

p-himik16:06:08

Not sure what you mean. Where would you use it? Instead of identity?

lilactown16:06:28

user=> (def height (juxt :height hash))

user=> (defn compare-height [a b]
         (compare (height a) (height b)))

user=> (sorted-set-by
        compare-height
        {:height 0 :foo "bar"}
        {:height 0 :foo "baz"}
        {:height 3 :foo "asdf"}
        {:height 2 :foo "jkl"})
#{{:height 0, :foo "bar"} {:height 0, :foo "baz"} {:height 2, :foo "jkl"} {:height 3, :foo "asdf"}}

raspasov16:06:29

@lilactown possibly, yes… if you care about same instance only (I think)

p-himik16:06:50

Well, in case of a hash collision you will lose an item. Not fun.

p-himik16:06:58

I'd just use identity.

lilactown16:06:58

in my case, I care about same instance only. I’m not actually using maps, but deftype objects

p-himik16:06:49

It doesn't solve the issue of hash collisions.

lilactown16:06:09

identity is probably better, you’re right

bortexz16:06:33

How would identity be used instead of hash? hash on maps make them comparable, but identity doesn’t

raspasov17:06:08

One possible way I’m thinking is str… not sure how consistent is that, but that is comparable

raspasov17:06:53

(defn height-with-tie-break [x y]
  (let [c (compare (get x :height) (get y :height))]
    (if (not= c 0)
      c
      ;; Otherwise we don't care as long as ties are broken
      ;; consistently.
      (compare (str x) (str y)))))
(def s3 (sorted-set-by
          height-with-tie-break
          {:height 0 :foo "bar"}
          {:height 3 :foo "asdf"}
          {:height 0 :foo "baz"}))

p-himik17:06:06

Oh, wow. I must be sleeping or something - I treated the topic of comparing maps and the topic of hashes as two completely different topics and didn't realize that they're closely related.

p-himik17:06:45

@lilactown If I understand "I care about same instance only" correctly, then you should be able to use System/identityHashCode. Although I'm not sure how portable it is since there are no explicit guarantees about it.

lilactown17:06:16

yeah, I only need it to be stable across different “height”, intra-height ordering I don’t think I care about

raspasov17:06:41

@lilactown another way is to turn the maps into vectors for comparison; I believe this is stable; but you should double-check:

(defn height-with-tie-break [x y]
  (let [c (compare (get x :height) (get y :height))]
    (if (not= c 0)
      c
      ;; Otherwise we don't care as long as ties are broken
      ;; consistently.
      (compare (vec (seq x)) (vec (seq y))))))

(def s3 (sorted-set-by
          height-with-tie-break
          {:height 0 :foo "bar"}
          {:height 3 :foo "asdf"}
          {:height 0 :foo "baz"}
          {:height 2 :foo "baz"}))

raspasov17:06:53

s3 => #{{:height 0, :foo “bar”} {:height 0, :foo “baz”} {:height 2, :foo “baz”} {:height 3, :foo “asdf”}}

raspasov17:06:30

(that should work across JS/JVM…)

lilactown17:06:53

this is a pretty tight loop, I’d like to avoid allocating as much as possible. I’ll need to fiddle with it

lilactown17:06:01

I’ll post what I end up going with

raspasov17:06:08

But I’m not sure about “equivalence” across both… but independently should work on each

raspasov17:06:31

@lilactown yea this is not the most performant option I think; wasn’t sure what’s your perf requirement here

raspasov17:06:08

I actually got curious about (str …) on maps; asking in the main channel

andy.fingerhut18:06:39

This was probably stated clearly above already, but one way to use sorted sets as already built into Clojure is to make a comparator function that is "more specific" than you actually care about, if you want to keep "duplicates", i.e. if you really only care about sorting by height, but you have some other value you can compute from, or extract from, the objects you want to sort, and it is unique across all of them, e.g. an id field, a name, or some vector that is a combination of multiple such things that you know must be unique, and it is comparable, e.g. strings, numbers, keywords, symbols, etc., then you can use those tie-breaker values in your comparator function, and you will never "eliminate duplicates", because according to the comparator function, there won't be any.

andy.fingerhut18:06:34

Clojure's compare function does not work on maps, because for those values, and sets, there is no good way I can think of to define a total order among all possible maps (nor between sets).

andy.fingerhut18:06:02

You can make a comparator function that compares two maps, but they will be custom for the kind of maps you want to compare, and you get to define the field or fields that are significant for comparing them.

lilactown18:06:38

that makes sense

lilactown18:06:06

I think I can reframe this to be, what I want is really a “partial order” w.r.t. some field

lilactown18:06:27

in which case it might be worth just using a vector of vectors

lilactown18:06:50

or since I can safely mutate in my case, an array of lists

andy.fingerhut20:06:27

Clojure's sorted sets and maps work with comparators that give a total order. Comparator functions cannot represent partial orders.

andy.fingerhut20:06:12

They can represent total orders with bugs 🙂. But in order to represent a partial order, there would have to be a way to return a value meaning "no ordering relationship exists in the partial order between the two values given as parameters"

andy.fingerhut20:06:36

I believe what you want is a total order, but with many things that are different according to clojure.core/=, but equal according to your comparator, to be stored independently in the collection, with "duplicates" allowed ("duplicates" meaning equal according to your comparator function, but different according to clojure.core/=). Clojure's built in sorted sets and maps don't support that, as you have already learned.

andy.fingerhut20:06:17

My suggestion above was to "make up" a total order, where things you want to be equal according to your comparator function, have some "I don't care so much what it is, but it is determinstic" total ordering between items you don't care what order they occur in.

andy.fingerhut20:06:05

The main reason I suggest that is: you then can use the existing sorted set and map implementations, rather than creating a new custom one.

lilactown21:06:09

I think I would rather create my own custom data structure than bastardize the sorted-sets

lilactown21:06:36

the code that uses this data structure lives at the center of a framework for doing reactive computations. I’m trying to be very sensitive to both time and space when it comes to how I build this, even if it is just a first pass

lilactown21:06:58

to me, an array of sets seems like a clear win over trying to construct a total order on top of what I know to be partial order

andy.fingerhut21:06:36

Or perhaps a sorted-map where the values are sets of things you want to consider as equal?

lilactown21:06:00

FWIW I rewatched this video again https://youtu.be/G6a5G5i4gQU after the discussion here, which is basically someone explaining how they implemented in OCaml exactly what I’m trying to build. at https://youtu.be/G6a5G5i4gQU?t=2852 they explicitly talk about moving away from a total order to a partial order of the nodes and what data structure they used (an array of lists) 😄

lilactown16:06:28

the way I need to use this structure is I’m taking the “first” element from the set, doing an operation, and then adding a bunch of new elements. so the ordering is rebalanced frequently

William Skinner16:06:58

Thanks @lilactown! This looks promising

Alex Miller (Clojure team)16:06:12

I would use a queue and a separate "seen" set to avoid re-processing

4
lilactown16:06:05

is your intuition that it would be faster than using a sorted set?

Alex Miller (Clojure team)16:06:11

I just find it to be clearer in my head. I doubt the perf difference matters as it's probably dominated by the processing

Alex Miller (Clojure team)16:06:31

general shape:

(loop [q (conj (PersistentQueue/EMPTY) root)
       seen #{}]
  (when-let [item (peek q)]
    (if (seen item)
      (recur (pop q) seen)
      (do
        (process item)
        (recur (pop q) (conj seen item))))))

Alex Miller (Clojure team)16:06:18

if you need a custom "ordering" for processing

bhauman16:06:43

a general question about practice here: it seems safe to generate and distribute a selfsigned certificate pem and keystore.jks that only refers to localhost and say www.localhost to use for a local devserver

bhauman16:06:10

where developers can import and trust the pem or p12 cert

bhauman16:06:07

any auth attack would have to come from localhost 127.0.0.1???

martinklepsch17:06:05

Does anyone know of examples using datafy to wrap an HTTP API? Is this a bad idea? :thinking_face:

lilactown17:06:30

i’m waiting for the day that someone builds a datafy / nav interface for HATEOAS / REST 😄

p-himik17:06:45

"What do you mean I can't use datafy on this API because it's not REST and just HTTP? It says RESTful right there!"

lilactown17:06:40

lol yeah. the utility of such a thing would be close to the number of APIs that are actually REST - aka approaching 0 😛

lilactown17:06:16

but maybe it’s the incentive we need! Make it REST, and you get datafy / nav support!

p-himik17:06:09

Let's first persuade the majority to at least look at Clojure.

p-himik17:06:24

It may sound like a story about unicorns, but I actually know a guy that has implemented a perfectly fine REST API with a client for it. Later he said something along the lines of "never again" and "better go ad-hoc".

lilactown17:06:55

I imagine it’s a PITA w/o first-class support in some sort of framework

raspasov17:06:44

This is from another discussion, but I was wondering if somebody knows: how consistent is (str {:a 1}), aka (str …) on maps… does it produce the same string every time for the same map (I assume yes)? What about across Clojure vs ClojureScript?

Alex Miller (Clojure team)17:06:28

for the same instance, yes, seq order is stable

Alex Miller (Clojure team)17:06:55

for clj vs cljs, not guaranteed and I wouldn't rely on them being the same

raspasov17:06:30

@alexmiller thanks! That’s actually the important part; I assume same instance, you mean like consistent version of Clojure? Or that can change across JVM instances of the same version?

noisesmith17:06:53

if you mean same as in identical? yes, but it will definitely print equal small maps differently from one another

user=> (let [a {:a 0 :b 1}
             b {:b 1 :a 0}]
          [(= a b) (= (str a) (str b))])
[true false]

Alex Miller (Clojure team)17:06:02

I mean if you have the same object in the same vm and print it twice, it will be the same

Alex Miller (Clojure team)17:06:19

no guarantee across jvm versions, clojure versions, jvm executions, etc

Alex Miller (Clojure team)17:06:27

maps are unordered, don't rely on order

raspasov17:06:52

@alexmiller I see, makes sense; thanks again!

Alex Miller (Clojure team)17:06:05

the actual order is based on map impl, key hash, and possibly on the history of the map. hashing has changed in the past (in Clojure 1.6 most recently)

🎯 4
raspasov17:06:32

yea… hm… that stemmed from another discussion… what’s the best way to to (compare …) maps; is that even recommended to do?

noisesmith17:06:25

as in comparator, for sorting?

raspasov17:06:50

rf.shared.nutrients=> (compare (vec (seq {:a 1 :b 2})) (vec (seq {:a 1 :b 2}))) 0

noisesmith17:06:57

you can compare by hash

(ins)user=> (hash {:a 0 :b 1})
1561470772
(ins)user=> (hash {:b 1 :a 0})
1561470772

raspasov17:06:59

rf.shared.nutrients=> (compare (vec (seq {:a 1 :b 2})) (vec (seq {:b 2 :a 1}))) -1

noisesmith17:06:11

of course hash is lossy...

raspasov17:06:24

Hash collisions potentially?

noisesmith17:06:38

right, a collision would not be sorted in a stable way

noisesmith17:06:50

and don't use hash alone to remove dups, obviously

noisesmith17:06:58

but other than that, the sorting will work

raspasov17:06:03

Basically, how to stably (is that a word) sort maps

raspasov17:06:34

Sort on, say :a, and if that is equal, sort on the whole map (somehow)

raspasov17:06:38

consistently

noisesmith17:06:49

you could compare hash, and when not identical, compare the set of hashes of entries

raspasov17:06:50

Without losing data

noisesmith17:06:24

but this is half baked, it's an interesting problem

dpsutton17:06:27

i've been down this road and its not fun

🎩 3
dpsutton17:06:49

you're going to get to a point too where you want to ignore some differences and then you're really sunk

raspasov17:06:38

@dpsutton thanks for the input 🙂

noisesmith17:06:44

@raspasov if you have some domain logic that lets you select keys and be sure of types, you could compare the vector of "interesting" keys

raspasov17:06:06

@noisesmith It was somebody else’s problem 🙂 I just got curious…

ghadi17:06:11

to compare maps, it's as simple as =

dpsutton17:06:27

are you saving state and checking to see if it is unchanged?

raspasov17:06:27

@ghadi not quite… for sorted-sets

ghadi17:06:56

I said maps 🙂

raspasov17:06:13

See my next comment 🙂 It’s in the context of comparators… so in addition to “is something equal”, figure out a way to say a map is “less than” or “bigger than” another map

raspasov17:06:22

@dpsutton

(defn height-with-tie-break [x y]
  (let [c (compare (get x :height) (get y :height))]
    (if (not= c 0)
      c
      ;; Otherwise we don't care as long as ties are broken consistently
      (compare (vec (seq x)) (vec (seq y))))))

(def s3 (sorted-set-by
          height-with-tie-break
          {:height 0 :foo "bar"}
          {:height 3 :foo "asdf"}
          {:height 0 :foo "baz"}
          {:height 2 :foo "baz"}))
Sort by :height, if that is equal, how to tie break consistently

raspasov17:06:42

(I don’t think this is fully correct)

raspasov17:06:52

Possibly, this is the wrong approach about this… a map cannot be fundamentally “bigger than” or “less than” another map because it is, like @alexmiller said, un-ordered

raspasov17:06:57

so trying to use consistent order where there isn’t one by definition is probably the wrong way of solving that problem

dpsutton17:06:55

clojure's persistent maps are unordered. but you're talking about ordering the set of maps which contain keys :height and :foo. that doesn't seem fundamentally impossible

craftybones17:06:04

There was a great talk many years ago about how to help a team migrate to clojure(or just new languages in general, but using Clojure to illustrate it)

craftybones17:06:22

I can’t remember whose talk it was and I can’t find it any more. Did I hallucinate the whole thing?

raspasov17:06:37

@dpsutton right, if you limit your domain to those two keys :height and :food - yes… I was trying to invent a general way (which seems like a bad idea)

andy.fingerhut21:06:09

If you want a general way, then you need a general way to create a total order on all possible immutable values that can be map keys, which is a lot of values whose types we know now, and many that haven't been created yet.

andy.fingerhut21:06:29

The http://clojure.org guide on comparators has some ramblings and code near the end where I started trying to come up with a general comparator that works for arbitrary pairs of immutable values, which if it isn't one of Clojure's built in collection types or a handful of others, compares them by their JVM class name first, but if you get a JVM object of an unknown class, that has no compareTo method, then you are stuck

andy.fingerhut21:06:37

Pick a reasonable subset of values that you can write a "nearly fully general" comparator function for, and delimit/document what those are, and you can then use that to devise a comparator function for two arbitrary maps

andy.fingerhut21:06:56

There just isn't that much need for such a thing, that I am aware of.

dpsutton17:06:53

gotcha. i thought you were going down a different path 🙂

dpsutton17:06:09

i think this should work

(sorted-set-by
  #(let [f (juxt :height :foo)]
     (compare (f %1) (f %2)))
  {:height 0 :foo "bar"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"}
  {:height 2 :foo "baz"})

raspasov17:06:22

I guess the problem becomes… in the case of map… what if those two keys are equal again, :height and :foo - how do you tie break then in the case of a map? 🙂 It’s like a never ending story

raspasov17:06:09

@dpsutton right… I guess if your maps are guaranteed to NOT have any other keys except for :height and :foo, you can do it

raspasov17:06:30

Seems a bit at odds with the Clojure’s philosophy of open-data

raspasov17:06:57

(sorted-set-by
  #(let [f (juxt :height :foo)]
     (compare (f %1) (f %2)))
  {:height 0 :foo "bar"}
  {:height 0 :foo "bar" :c "HERE I COME"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"}
  {:height 2 :foo "baz"})

raspasov17:06:00

{:height 0 :foo "bar" :c "HERE I COME"}
Is gone 🙂

raspasov17:06:20

My personal conclusion is, “don’t do it” lol… unless you are really sure you just want those two keys (:height and :foo)

isak17:06:19

(sorted-set-by
  (fn [a b]
    (let [f (juxt :height  (fn [m] (into [] (apply sorted-map m))))]
      (compare (f a) (f b))))
  {:height 0 :foo "bar"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"}
  {:height 2 :foo "baz"})
would this work?

noisesmith17:06:47

it would if matching keys always shared matching vals, and I think (sort m) would be the same as that anonymous fn in behavior, with less complexity

3
raspasov17:06:34

(sorted-set-by
  (fn [a b]
    (let [f (juxt :height  (fn [m] (into [] (apply sorted-map m))))]
      (compare (f a) (f b))))
  {:height 0 :foo "bar"}
  {:height 0 :foo "bar" :c "HERE I COME"}
  {:height 3 :foo "asdf"}
  {:height 0 :foo "baz"}
  {:height 2 :foo "baz"})
No value supplied for key: [:c “HERE I COME”]

raspasov17:06:04

@isak potentially… that might be a decent idea

raspasov17:06:39

@isak thanks for the input though 🙂

isak17:06:02

hm, is it because the maps don't have the same keys? if so, could add a step to add missing keys @raspasov

raspasov17:06:33

@isak I think, fundamentally, trying to apply order where there isn’t one is just not a very good approach; a comparator requires not only “equal” semantics, but also “less than” and “greater than”

raspasov17:06:15

Extracting “less than” and “greater than” semantics from a generalized un-ordered collection sounds like a mathematical impossibility; maybe a certain heuristic approach can work reasonably well, but I don’t think a general and consistent solution is possible

raspasov17:06:04

That’s why if you start out with vectors, it’s much easier, they have order built-in

raspasov17:06:12

Which you can rely on

isak18:06:43

I think you can - worst case you just add more things to the comparison, like .getType, but I won't argue. Imo the real problem would be to make a generalized solution that is also efficient

✌️ 4
lilactown18:06:13

I think I figured out what I want: an array of sets 😄

lilactown18:06:01

here’s a POC using vectors of sets:

(defn poset [order-fn & els]
  (loop [v (vec (repeat (count els) #{}))
         els els]
    (if-some [el (first els)]
      (recur (update v (order-fn el) conj el)
             (rest els))
      (with-meta v {:order-fn order-fn}))))

(defn conj-poset [poset el]
  (let [order-fn (:order-fn (meta poset))
        order (order-fn el)
        ;; grow the vector in case order > size of poset
        poset' (into poset (repeat (- order (count poset)) #{}))]
    (update poset' order conj el)))

(defn disj-poset [poset el]
  (let [order-fn (:order-fn (meta poset))
        order (order-fn el)]
    (update poset order disj el)))

(defn first-poset [poset]
  (ffirst poset))

p-himik19:06:34

Hard to understand at a glance - what is it for, what's the use-case?

lilactown19:06:40

replying in the channel

p-himik19:06:12

v has the size of (count els), but later you index it by order-fn. So order-fn has to always return something less than (count els). Given the previous example with :height, it seems strange.

lilactown19:06:07

yeah that’s probably an error, it just so happened that my REPL testing always had :height < (count els)

lilactown19:06:50

I would want to either pass in an arg for how much to allocate or get the max order of the els passed in and allocate that much

p-himik19:06:34

Also, you may end up with a vector that's mostly empty. E.g. one call order-fn returns 0 and another one returns 1,000,000. Not great. And what if order-fn returns something that's not a non-negative integer?

p-himik19:06:37

Lots of questions. :)

lilactown19:06:17

yeah I’m not convinced that a vector is the best data structure

p-himik19:06:27

I would probably use a sorted map of the values of order-fn to the sets of the elements.

lilactown19:06:39

I think that a sparse array, or a sorted map, would be better

p-himik19:06:57

Sorted map is better because keys could then be (almost) arbitrary.

lilactown19:06:03

a vector or array works better for my case because my order is always integers

lilactown19:06:12

less overhead

lilactown19:06:31

but less general

p-himik19:06:39

Do I sense some bitmasks coming soon? :)

lilactown19:06:48

ahahahaha 🤓

lilactown18:06:28

I might distill this into a lib that implements the appropriate interfaces/protocols

ghadi19:06:01

a picture docstring is worth 1000 words sexprs

🙂 4
dominicm20:06:53

You're going to be pinned in my company slack :)

lilactown19:06:51

ahaha, right. sorry, this was a solution to a problem I posted about above: https://clojurians.slack.com/archives/C03S1KBA2/p1591805338491800

lilactown19:06:28

we went around and around w/ how we could hack ordered-set-by to get good behavior

lilactown19:06:33

the problem is that sorted-set-by is built for a total ordering, so you end up having to hack on to the comparison fn how to deal with when a piece of data that your ordering is = but the elements are actually distinct

lilactown19:06:00

which rolled into the “how do you compare maps” convo

avi21:06:25

👋 I’ve been starting up a socket repl and specifying the port as 0, which IIRC is a unix convention for “find some available port”. This has been working well for me, but then I have to find out what the actual port is before I can connect. Just in case it might be helpful to anyone, here’s what I’m running when I start my REPL: (println "Socket REPL listening on port" (.getLocalPort (get-in @#'clojure.core.server/servers ["repl" :socket])))

Alex Miller (Clojure team)21:06:44

why don't you just tell it a port to use?

avi21:06:53

Because I’m sometimes working on multiple projects simultaneously, each with its own socket repl. If I hard-code the port numbers, then they’ll either conflict, or I’ll have to remember which port number I’d assigned to which project. Which I don’t think I can do…

seancorfield21:06:16

(and perhaps worth noting there can be multiple Socket REPLs with names corresponding to the JVM option name used to start each, and so the get-in would need a different string from "repl")

👍 4
avi21:06:22

Also, I don’t know, I kinda like the semantics of specifying port 0 — it’s neat

avi21:06:53

(good point Sean, thanks!)

seancorfield21:06:27

It's interesting, I hadn't even thought to try port 0. That is neat. But discoverability is kind of ugly.

dominicm21:06:52

I think port 0 is a Java thing.

avi21:06:35

I don’t think so; or at least, not now — it works in Python too, IIRC.

dominicm21:06:21

https://docs.oracle.com/javase/7/docs/api/java/net/ServerSocket.html#ServerSocket(int) > A port number of 0 means that the port number is automatically allocated, typically from an ephemeral port range. This port number can then be retrieved by calling getLocalPort.

dominicm21:06:52

Python probably also special cases it

avi21:06:47

Yeah I saw that in the Java docs, but it doesn’t say where the convention came from. The docs don’t claim it originated in Java. (Not saying that that’s proof that it did not.)

andy.fingerhut21:06:20

I am pretty sure the port=0 choosing a local free port on the system originated with the BSD socket API, at the C level.

👍 8
dominicm21:06:32

Oh, I'm not trying to suggest that. Just that it's unix agnostic :)

avi21:06:32

From https://www.grc.com/port_0.htm → > The designers of the original Berkeley UNIX “Sockets” interface, upon which much of the technology and practice we use today is based, set aside the specification of “port 0" to be used as a sort of “wild card” port. When programming the Sockets interface, the provision of a zero value is generally taken to mean “let the system choose one for me”. Programmers who specify “port 0” know that it is an invalid port. They are asking the operating system to pick and assign whatever non-zero port is available and appropriate for their purpose.

noisesmith21:06:34

yeah, it's a Posix thing for sure

seancorfield21:06:02

seanc@DESKTOP-QU2UJ1N:~/clojure$ clj -J-Dclojure.server.repl='{:port,0,:accept,clojure.core.server/repl}' -e "(.getLocalPort,(get-in,@#'clojure.core.server/servers,[\"repl\",:socket]))" -r
60438
user=>
seanc@DESKTOP-QU2UJ1N:~/clojure$ clj -J-Dclojure.server.repl='{:port,0,:accept,clojure.core.server/repl}' -e "(.getLocalPort,(get-in,@#'clojure.core.server/servers,[\"repl\",:socket]))" -r
60439
user=>
(that's WSL 1 on Windows, BTW).

notbad 3
🤯 3
dominicm21:06:03

Ah, the berkeley socket api is closely followed by windows too, makes sense that it's pretty universal then :)

avi21:06:38

I only learned of it recently myself, via Simon Willison, when he created this ticket: https://github.com/encode/uvicorn/issues/530

noisesmith21:06:39

s/closely followed/copy pasted/ :D

avi21:06:41

@seancorfield can you think of some trick to get my project to automatically print out the port every time I start my repl with a particular tools.deps alias? I tried to naïvely add it to :main-opts as a -e arg but ran into the usual issues with escaping. And then I discovered that -r and -e seem to be mutually exclusive, so if I specify -e then the process exits after the expression is evaluated, and if I specify -r along with the -e then the repl starts but the expression doesn’t seem to be evaluated.

avi21:06:22

oh wait, it seems to have worked for you in the snippets you posted above — I’ll try again 😅

avi21:06:18

Aha, the trick is to specify the -e first and the -r second!

seancorfield21:06:22

Confirmed that it works on Powershell (Windows) too -- I added :socket-zero to my dot clojure file, ran clj -A:socket-zero -r on Powershell, then attached to it from WSL:

$ telnet 127.0.0.1 60468
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
user=> (System/getProperty "user.dir")
"C:\\Users\\seanc\\clojure"
user=>

seancorfield21:06:37

Here's where I start it in Powershell:

PS C:\Users\seanc\clojure> clj -A:socket-zero -r
60468
user=>

avi21:06:24

Yup, I’ve got nearly the exact same thing set up for myself now, working great in bash on MacOS

seancorfield21:06:54

I off work today so I'm on my ancient little laptop: Windows with WSL 1 (since it can't run any virtualized stuff).

seancorfield21:06:33

I'll try it on macOS when I'm back at work tomorrow. I should update the README in my dot clojure project to describe the new alias.

Alex Miller (Clojure team)21:06:33

you could also -e and start the server manually (via clojure.core.server/start-server), which returns the ServerSocket

👍 8
avi21:06:03

Good point, that might be less brittle, in that maybe it wouldn’t require a specific name

Alex Miller (Clojure team)21:06:45

clj  -e "(.getLocalPort ((requiring-resolve 'clojure.core.server/start-server) {:port 0, :accept 'clojure.core.server/repl :name \"repl\"}))" -r

seancorfield22:06:31

The downside to this is that it won't work on Clojure 1.8 or 1.9 (which the combined JVM option/`-e` approach allows). May not matter to anyone but wanted to mention it.

Alex Miller (Clojure team)21:06:03

needs the commas etc to go in an alias

Alex Miller (Clojure team)21:06:13

$ clj  -e "(.getLocalPort ((requiring-resolve 'clojure.core.server/start-server) {:port 0, :accept 'clojure.core.server/repl :name \"repl\"}))" -r
57739
user=>

seancorfield21:06:59

@aviflax Still needs to specify the name in there (right, @alexmiller)

Alex Miller (Clojure team)21:06:11

if you want a random name (str (gensym))

avi21:06:59

I’m not sure how/where the name is used, so I’m unclear on whether it matters

Alex Miller (Clojure team)21:06:43

doesn't unless you're starting multiple

Alex Miller (Clojure team)21:06:27

name is used in the thread names and to key the per-server state

deactivateduser21:06:26

Would it be possible to put the port # in the name, somehow? I know about the trick of asking for port 0 (to get a randomly available port), immediately unbind from that port, then rebind it directly by number…

dominicm21:06:49

@deactivateduser10790 probably better to rename than to try and claim & unclaim a port

dominicm21:06:58

but then the thread needs renaming... yuk :)

deactivateduser21:06:13

Yeah - that was where I started getting stuck too. 😉

deactivateduser21:06:34

I know that elsewhere the “bind twice” pattern is somewhat common / accepted. Just don’t know if there’s a better way in Clojure.

Alex Miller (Clojure team)21:06:36

the code could do so, but it doesn't right now

deactivateduser21:06:02

(and of course it has a race condition)

Alex Miller (Clojure team)21:06:04

I'm not sure why that would be usfeul

Alex Miller (Clojure team)21:06:16

thread names are mutable

Alex Miller (Clojure team)21:06:23

so it's quite easy to do

deactivateduser21:06:27

So that if you’re looking at thread names (e.g. in a dump) it’s more obvious which thread is bound to which port.

Alex Miller (Clojure team)21:06:39

well that's why it's named :)

Alex Miller (Clojure team)21:06:58

you started the server and gave it a name, so ...

deactivateduser21:06:14

But you don’t know the actual port number until after the thread is initially named (though thread renaming could get around that, of course).

Alex Miller (Clojure team)21:06:37

if you start it explicitly, then you have the server socket and know the port and can do whatever you want

👍 4
deactivateduser21:06:13

Right. Either you double bind (as is done in some other languages), or you mutate the thread name. They’re both a bit icky (though not terrible). 😉

dpsutton21:06:45

What’s icky about changing a thread name?

dominicm21:06:46

Mutating the thread name seems OK. But really you just need to compose the pieces together yourself at this point. The abstraction doesn't hold any more, so compose the smaller pieces (SocketServer & starting a thread)

deactivateduser21:06:11

@dpsutton how do you retrospectively change the thread name in things like log files? I mean it’s possible, but situations like these are icky.

Alex Miller (Clojure team)21:06:15

abusing mutable thread names has a long tradition in Java

Alex Miller (Clojure team)21:06:25

it's a great hack to actually use the thread name to leak a state variable (like a system wide tap>)

😱 6
Alex Miller (Clojure team)21:06:00

you can then watch it in a mbean or profiler via the debug apis

deactivateduser21:06:45

Yeah - or a correlation id of some sort, to track a single logical “transaction” through multiple threads / JVMs / whatever.

deactivateduser21:06:42

Anyhoo, seems like the simplest thing would be to just rename the thread to “repl-on-port-<concrete port number>” after the socket is bound, and try not to do anything in that thread before the rename (i.e. to avoid logging with the wrong thread name, or other side-effecty shenanigans).

isak21:06:56

Little warning when you compare lines of code between Clojure and other languages - clojure docstrings count as lines of code, not a comment (tested with tokei)

jacekschae04:06:57

That is correct, cloc thinks that doc strings are LoC. In the article, or rather the repo — there are no docstrings, so the reason why CLJS is not #1 is not the docstrings. I believe it’s because people created DSLs, such as Imba to reduce LoC

isak14:06:13

Ah I see

avi21:06:07

have you tested with cloc ? That’s the tool I’ve used for years…

seancorfield22:06:31

The downside to this is that it won't work on Clojure 1.8 or 1.9 (which the combined JVM option/`-e` approach allows). May not matter to anyone but wanted to mention it.

isak22:06:19

@aviflax just tested with cloc via WSL, same problem

💪 3
avi13:06:36

ah, too bad. Thanks for checking tho! 💪