Fork me on GitHub
#clojure
<
2022-03-23
>
gavin02:03:10

Hey folks… need some quick help with Clojure and Protobufs. Particularly with the “Any” type. It is not clear to me how to unmarshal such an object via Clojure. One of you mavens I am sure knows. Feel free to poke at me directly. Thanks in advance.

mbragg12:03:14

Hi, i’ve come across the following when updating to Clojure 1.11.0. In our application we have some places where we create V5 (name based, SHA1 hash) uuids, using the https://github.com/danlentz/clj-uuid library. After the upgrade the v5 uuid’s produced are different. An example being:

(def ^:const +namespace+ #uuid "50d94d91-a1cf-422d-9586-4ddacf6df176")

(clj-uuid/v5 +namespace+ :some-keyword) 

;; Clojure 1.10.3
=> #uuid "d30e9c3c-ced2-534e-a6b8-ecf784fb0785"

;; Clojure 1.11.0
=> #uuid "a16f6719-952a-55b9-b71b-b15dd263665b"
After some trial an error It seems it is the local part argument i.e :some-keyword that is causing the difference, as within the clj-uuid/v5 fn it converts the keyword Object to a ByteArray, which now appears to be different. If I use a String instead of a keyword for the local part argument then the uuid produced is consistent before and after the Clojure upgrade. The uuids produced are used in downstream systems and It would be quite difficult to have them handle the change. Is there anyway that I could achieve the exact same previous uuids?

👀 1
😱 1
Alex Miller (Clojure team)14:03:04

At a glance it's not clear to me how anything added in 1.11 could affect this (just a couple functions that wrap Java uuid methods) but happy to take more than a glance later

mbragg14:03:04

Thanks @U064X3EF3 > Could you report this on https://ask.clojure.org ? yup sure thing: https://ask.clojure.org/index.php/11658/clj-uuid-v5-uuids-inconsistent-after-clojure-1-11-0 It looked to me that it may be here where it calls writeObject https://github.com/danlentz/clj-uuid/blob/master/src/clj_uuid.clj#L565 , and as the signature of clojure.lang.Keyword has changed with https://github.com/clojure/clojure/commit/bd4c42dc7946cb015b8d0699596662aa68bcdc89, then the ByteArray used to construct the final UUID has changed(?)

fogus (Clojure Team)14:03:35

That seems like a promising line of thinking. If that's the case then maybe the implementation of as-byte-array for Objects relies too much on implementation details?

mbragg15:03:04

Yes indeed. We probably should have used plain string in the first place 😉 But as it stands, I can’t see how we will be able to generate the exact same byte-array/the same uuids as before.

fogus (Clojure Team)15:03:30

Yeah this one is a very sticky problem. 😞

Alex Miller (Clojure team)16:03:55

so the problem is that the serialized byte array has changed for keywords?

Alex Miller (Clojure team)16:03:13

if so, then setting the serialversionuid on Keyword would be a fix for that (would need testing), but really I would not say we guarantee anything about the serialized form of clojure objects

Alex Miller (Clojure team)16:03:36

it might be useful to file this as an issue on clj-uuid as well as this inherently unreliability might be better doc'ed (and knowing Dan, I'm sure he would be happy to help with this)

Michael Gardner21:03:04

@U0HJAJ64E what is the downstream system doing with these UUIDs, out of curiosity?

mbragg07:03:45

Thanks @U064X3EF3 good idea, ill raise an issue on clj-uuid. @U01R1SXCAUX after searching our internal GitHub i’ve found a few different examples where this will cause a problem; one being where the v5 uuid is generated and then used as the key when publishing to a Kafka topic. Having new/different keys on these topics could result in the records sent to different partitions, which could effect ordering guarantees, and potentially joining behaviour

kirill.salykin12:03:54

hi is there way how i can cast nil to some java class? eg how i can reproduce this java code in clojure

Message msg = new MimeMessage((Session)null);
thanks UPD
(MimeMessage. ^Session nil)
Doesnt work, because nil doesnt have IMeta

Ferdinand Beyer12:03:43

Not sure if there is a better way, but you can use let:

(let [^Session session nil]
  (MimeMessage. session))

kirill.salykin12:03:20

seems working, thanks!

👍 1
kirill.salykin15:03:58

hi again another question how can i produce array of Address from clojure? Needed by https://javaee.github.io/javamail/docs/api/javax/mail/internet/MimeMessage.html#setRecipients-javax.mail.Message.RecipientType-javax.mail.Address:A- signature:

public void setRecipients(Message.RecipientType type,
                          Address[] addresses)

erwinrooijakkers15:03:08

Hi all, we noticed some interesting behaviour coming from a library, where the result differed based on if a sequence is of type ChunkedSeq or of type IndexedSeq

erwinrooijakkers15:03:15

So basically, depending on the length of the seq

erwinrooijakkers15:03:06

Namely:

(require '[graphql-query.core :refer [graphql-query]])

(def indexed-seq
  (seq (mapv (comp keyword str) (take 10 (range)))))
;; => (:0 :1 :2 :3 :4 :5 :6 :7 :8 :9)

(type indexed-seq)
;; => cljs.core/IndexedSeq

(graphql-query
 {:queries [[:foo {:bar indexed-seq}]]})
;; => "{foo(bar:[0,1,2,3,4,5,6,7,8,9])}"

(def chunked-seq
  (seq (mapv (comp keyword str) (take 40 (range)))))
;; => (:0 :1 :2 :3 :4 :5 :6 :7 :8 :9 :10 ... :39)

(type chunked-seq)
;; => cljs.core/ChunkedSeq

(graphql-query
 {:queries [[:foo {:bar chunked-seq}]]})
;; => "{foo(bar:(:0 :1 :2 :3 :4 :5 :6 :7 :8 :9 :10 ... :39))}"
So when the sequence is longer and thus Clojure internally handles it as a ChunkedSeq instead of an IndexedSeq, the collection in the arguments is processed to become a list with keywords, instead of a vector with symbols. In the ChunkedSeq case the query generated is invalid GraphQL.

erwinrooijakkers15:03:19

Any ideas why this could be?

p-himik15:03:30

A bug in the graphql-query library.

1
p-himik15:03:48

At least, that's where I'd start researching it.

erwinrooijakkers15:03:54

Thing is I see nothing strange, maybe the dynamics

erwinrooijakkers16:03:24

That the dynamic definition does something different depending on the type

ghadi16:03:34

there is a protocol extended to IndexedSeq, but not ChunkedSeq

erwinrooijakkers16:03:35

(def ^:dynamic *kw->gql-name* name)

ghadi16:03:52

extending to concretes is a smell, you're gonna miss some

👍 1
erwinrooijakkers16:03:59

thanks a lot it was nagging me why things were going wrong and did not really know where to look, but now it’s clear as day (bit strange I overlooked that protocol in the middle of the page)

erwinrooijakkers16:03:18

And I agree that it’s a code smell, better to find some abstractions that cover all . The library creator did do so in the Clojure version, but that maybe did not work in the ClojureScript one

Carlo17:03:48

what predicate could I use to determine if something accepts metadata?

andy.fingerhut17:03:35

user=> (instance? clojure.lang.IObj {1 2})
true
user=> (instance? clojure.lang.IObj [])
true
user=> (instance? clojure.lang.IObj nil)
false

delaguardo17:03:38

you could check if argument is an instance of IObj #(instance? clojure.lang.IObj %)

dpsutton17:03:52

interesting. (source meta) checks for clojure.lang.IMeta and not IObj

dpsutton17:03:20

ah.

public interface IObj extends IMeta {

    public IObj withMeta(IPersistentMap meta);

}

dpsutton17:03:58

i wonder what things implement IMeta without IObj

Alex Miller (Clojure team)18:03:56

there are several things like that

Alex Miller (Clojure team)18:03:12

IMeta is "metadata read", IObj is "metadata modify"

Alex Miller (Clojure team)18:03:36

so checking whether something "accepts metadata" sounds like the latter to me, going back to the original question

Carlo18:03:53

thank you all, I solved the problem that prompted this question checking if the value is an instance of IObj 😍

Carlo17:03:30

like, I want to do something like (with-meta nil {:hey true}) but this fails for nil , so I'd like to do a check before

Sam Ritchie18:03:40

what are people’s thoughts on when to upgrade to clojure 1.11 as a library maintainer? I want to use some goodies from clojure.math in #sicmutils of course, but that will force anyone depending on the library to upgrade as well.

dpsutton18:03:43

i’m a fan of “quick on application code, slow on library code” for upgrades like that

dpsutton19:03:39

also, there might be a really subtle issue that can prevent people from upgrading to 1.11 so i’d wait a bit anyways

👍 2
Sam Ritchie19:03:46

that’s my default, I think that is the right move

Noah Bogart19:03:45

Is it possible to conditionally require or conditionally expose code, depending on the clojure version? Something like (if (= 1.11 *clojure-version*) (def abs clojure.core/abs) (def abs ...)), and then use that definition in your library code?

p-himik19:03:16

What you write is already possible, no?

Noah Bogart19:03:37

Oh is it? Hah I'm on my phone so can't test that in a repl at the moment

p-himik19:03:38

Except that 1.11 would be a proper map with the version.

👍 1
p-himik19:03:24

And I would put the if inside the def of course.

ghadi19:03:20

eval is a helluva drug

😂 2
Noah Bogart19:03:14

If that's the solution, then I'm sorry I asked haha.

1
borkdude06:03:01

You can do it using a macro in which you check the Clojure version, it's a little cleaner than eval

👍 1
mars0i23:03:41

I'm using [clj-async-profiler](https://github.com/clojure-goes-fast/clj-async-profiler) to identify bottlenecks in my code. The [flamegraphs](http://clojure-goes-fast.com/blog/clj-async-profiler-040) are very helpful. In some cases I'm puzzled by the output, though. I understand it as trying to show at the sampled moments what calls were on the stack. Then if one of the bars in the flamegraph is very wide near the top, that means that that function is taking up a lot of time. What is confusing to me is that I've got a flamegraph that seems to show that a function I wrote is calling things it couldn't possibly cal (afaics). In particular, consider this function:

(defn perc-foodspots-exactly
  [^Continuous2D env perc-radius [x y]]
  (let [foodspots-bag (.getNeighborsExactlyWithinDistance env
                                                          (Double2D. x y)
                                                          perc-radius)]
    (if (.isEmpty foodspots-bag) nil foodspots-bag)))

mars0i23:03:11

Notice that aside from the if that decides whether or not to return nil, everything is done by calls to Java. Those Java classes are in a library written solely in Java [https://cs.gmu.edu/~eclab/projects/mason/}. So they cannot be calling Clojure. But according to the flamegraph produced by clj-async-profiler, perc-foodspots-exactly seems to call clojure.lang.RT.nth.

mars0i23:03:42

Meta: I'm not sure this is the best place for this question. Feel free to suggest another forum--thanks.

p-himik23:03:56

[x y] - this is the source of nth.

p-himik23:03:39

=> (destructure '[[x y] data])
[vec__147 data x (clojure.core/nth vec__147 0 nil) y (clojure.core/nth vec__147 1 nil)]

mars0i23:03:58

Ohh .... Thank you @U2FRKM4TW. I can't believe how much time it's taking to pull that apart! I'll try passing x and y as regular arguments.

p-himik23:03:29

Are you passing a vector as the last argument or something else? It should be rather fast for a vector.

mars0i01:03:39

It is a vector. It's newly constructed by the function that calls perc-foodspots-exactly. (I wouldn't think this would matter, but fwiw, the actual function to which the vector pair is passed is (partial perc-foodspots-exactly env perc-radius). That's what sees [x y].) Let's see what happens when I have a chance to run clj-async-profiler with the changed code.

mars0i03:03:36

In case anyone is interested, passing x and y as [x y] was indeed the cause of the call to nth, and getting rid of that made a significant difference. That's an important lesson. Thanks @U2FRKM4TW! (Also grateful to Alexander Yakushev for clj-async-profiler, which I just started using.) (The time used up by nth was not as much as I thought, btw. I think I might not have been running a sufficient number of iterations to get an accurate result from async-profiler.)

👍 1