Fork me on GitHub

Is it possible to exclude part of my project from being included in the uberjar depending how I call leiningen? nevermind


in Clojure, is it possible to construct looping (but non lazy) values x, y such that (= x y) infinite loops?


Both infinite and non-lazy?


(defn loop-forever [number] (do (println number) (recur (+ 1 number)))) use at your own risk 😛


somehow it put full load on 2 cores when used


“Somehow.” I’m sure with a little work it could tie up more cores.


following on from that, and just for jest, would it be possible, just thinking, of using clojure reducers, to pretty much cause heat death by having each processor core sum an infinite sequence of numbers...

👍 4

would it make sense for clojure itself to enable some sponsporing options? via Github you can now enable several sponsporing links, e.g. patreon, opencollective. if most of the Clojure users would chip in for a couple of dollars a month, would that make a difference to the development of Clojure?


hey everyone, got a super weird bug i’m kind of stuck on. i have a collection of 6000 32-byte arrays that i’m sorting with a function compare-bytes. (sort compare-bytes coll). when the comparison function hits a particular element, it throws an exception Comparison method violates its general contract!


if i generate a separate collection of 6000 randomized 32-byte arrays, it works fine. even beyond 50,000 elements, no exception is thrown


it’s only with this specific collection. i can see nothing odd about the element on which the comparison function breaks down


also, (= 6000 (count (distinct (map bytes->hex coll))))


can you post your compare-bytes fn?


oh yeah, should have mentioned: it’s not mine, it’s the one in ztellman/byte-streams


i’ll get a link

Alex Miller (Clojure team)14:05:50

wrap it in your own comparator function that catches and prints the inputs


i tried this earlier, but the exception comes from the sort and not the comparison, so i’m not sure how to grab the breaking elements


i guess i could just print all of them. one sec

Alex Miller (Clojure team)14:05:43

I believe you get this exception in Java when the transitive property doesn't hold (that is, e1 > e2 and e2 > e3 but NOT e1 > e3), which is presumably something the sort relies on


yeah, that’s what google seems to say


must the function only return something ε [-1, 0, 1]? i noticed it can return other values


Ooh! Debugging why a comparator is not a total order. Looking at it ...


The JVM can handle negative, 0, positive fine.


i mean they seem correct (negative when l<r etc)

Alex Miller (Clojure team)14:05:24

given that this is a property, it's a perfect candidate for a generative test with test.check

Alex Miller (Clojure team)14:05:39

that's not a transitivity test


oh, guess it’s not really checking transitivity

Alex Miller (Clojure team)14:05:52

but could easily be extended to do so! :)

Alex Miller (Clojure team)14:05:34

just gen a / b / c, and verify that if a>b and b >c, that a>c

Alex Miller (Clojure team)14:05:06

test.check will spit you back a failing example


sweet! on it


Something that seems odd to me in cmp-bufs, which the function you linked calls in most/all cases - it calls a ByteBuffer .getInt method in some cases, or .get. From my reading of Java API, those mutate a "position" member of a ByteBuffer object. I don't yet see anything that initializes this position back to the beginning of the ByteBuffer, which would be a problem if the same ByteBuffer was compared more than once.

👍 4

Digging a bit further, I don't think that is it. cmp-bufs appears to be using the "absolute" variants of the .getInt and .get methods, which might not mutate the position field at all. It didn't on my first attempt trying it, to see if it did modify the position field.


Next guess to investigate -- cmp-bufs has an optimization where for buffers that contain at least 4 bytes, it gets 4 bytes at a time to compare, until it gets to the end, then does 1 byte at a time. It uses p/- to subtract 2 32-bit integers, which can wrap around when the result is a 32-bit integer, too, and cause a JVM comparator to behave in undesired ways, as described in "Beware using subtract" section of this article:


quite the rabbit hole..


I will let you know if I find anything, or give up on it. Haven't given up yet. My current suspicion is that most of the calculations and the return value are longs (64-bit signed integer), but Java comparator functions expect int (32-bit signed integer), and Clojure takes any value returned from a comparator function and coerces it to int via the .intValue method, which can be a different sign than the long value.


I suspect I can find a 4-byte ByteBuffer content for three byte buffers such that their comparison results are consistent if you look at the sign of the long, but inconsistent if you look at the sign of the coerced values returned by .intValue on those longs.


thanks a lot! this is definitely out of my league at this point. i ran a test.check for transitivity and it passed, as i kind of expected


This may be a case where it requires having an "extreme" combination of 4-byte values, e.g. those that when looked at as a 32-bit int, are either at or near the min possible value, max possible value, or near 0.


unless your random test case generation artificially stresses such values, it could take a very long time to find such corner cases.


would it help if i provided you with the actual collection i was using?

g15:05:26 (6000 32-byte strings, not delimited)


fails on 5632nd element


maybe can email info to me at <mailto:[email protected]|[email protected]> if data is not secret or anything


it’s in that pastebin link


I have a pair of 4-byte ByteBuffer values such that (compare-bytes b1 b2) and (compare-bytes b2 b1) both return a negative value when converted to 32-bit int, so that looks wrong. I don't think it causes the exception, but shows that the comparator is not good.


I have a proposed change to function cmp-bufs in that library that I think should correct the problem.


Would you be interested in trying it? The change is replacing the first of two occurrences of (p/* sign cmp) in the file byte_streams.clj in function cmp-bufs with (p/* sign (if (pos? cmp) 1 -1))


yeah, sure!


I'd like to find my own smaller sequence than yours that also causes the exception to be thrown, but I may not generate the necessary enthusiasm for that if that change fixes it for you, and just submit a PR with that change, plus some other tests that give wrong results without the change


In my limited testing, that change fixes the problem I found with a pair of values where (compare-bytes x y) and (compare-bytes y x) both return a negative value, meaning "x < y" and also "y < x". I suspect it will also eliminate the exception you saw thrown.


Created a PR here, if you want to follow along:


looks like this got merged in, great! should be really easy to check if this worked, but i’m not sure how to tell if the jar has been updated. the version number is the same, so i assume not yet


doesn’t look like this fixed my issue. :<


You can see when new releases are made on Clojars here:


Same data set still throws the same exception?


If so, I will try testing with it, then, and stop assuming I have found the last problem 🙂


that’s right, same issue


I tried with those 6000 32-byte sequences, converted with some little bit of code I wrote to convert those hex strings into 32-byte long java.nio.ByteBuffer objects, and sorting them with the updated compare-bytes caused no exception to be thrown.


I do get an exception thrown when I try to sort using the original definition of compare-bytes that you reported.


Probably a good idea to double check what method you are using to get the new version rather than the one currently published on Clojars.


Doing 'git clone' on byte-streams, then 'lein install' should install the modified version in your local $HOME/.m2 directory, where lein and other tools should be able to find it if you give the proper version number, which I think is 0.2.5-alpha2


You can also change the version string in the first line of the project.clj file to something that has never been released on if you want to be sure your code is getting that version, instead of one from clojars. You can search for byte-streams on to see what versions have been released there, and the dates. The last release date is in 2018, so cannot include the latest fix.


thanks! you’re right, i was probably using the wrong version. will check this again


case anyone is interested in the issue @gtzogana was asking about, but doesn't want all the gory details in the thread, I do not know if I have fixed the bug he found, but I have a proposed fix that definitely improves the result of compare-bytes for a bad case I found where it can return (x < y) and also (y < x) for many pairs of byte streams x and y. is the PR if you want to track it.

👍 4

Is this another example where clojures compare-machinery sometimes returns the wrong result because of truncating longs to int?


Well, wrong if you do not know that fact and plan for it in advance, yes 🙂


There may be another problem I haven't found and fixed yet, but the first problem I found with the comparator function was that it was returning what appeared to be a correctly signed Long value, but when truncated to 32-bit int would sometimes change the sign and not be the desired result.


I thought datafy could transform a Java object into Clojure data, am I wrong?


Like say I want to extract all values from a Java object into a Clojure datastructure instead like say EDN


And I want that to be deep


in the sense that datafy by default doesn't do anything


So, when I call datafy, it just prints the object memory address


(defprotocol Datafiable
  :extend-via-metadata true

  (datafy [o] "return a representation of o as data (default identity)"))


if you have something you want turned in to data in a particular way, you have to extend the protocol to do it


Oh. Okay I guess I assumed that Java objects had a smarter default then identity


I am also not sure how deep the 'datafy' should go


from my understanding a intended use case is kind of a cycle from datafy to nav and back again


you datafy something, then nav to something it references, then datafy that, etc


Oh right, so each datafy is one level. And to go deeper you have to recursively datafy. Makes sense. So it handles circular graphs


or things that are method calls returning data under that object, which may be newly generated or may just be a getter for existing contents


Think of datafy/`nav` as an extended, explicit form of laziness.


I still wished there was a default datafy for objects.


Don't really want to implement my own reflection field parsing


for things that use normal getter methods, there's bean


and there's definitely other reflection libs for making data out of objects that satisfy a few assumptions, IIRC TimMC made one...


Ya, bean is there, but not deep. It's okay, I'll just use GSON


And then Cheshire to pull that into Clojure


Hum... there seems to be

Alex Miller (Clojure team)18:05:18

yes, works too and makes different choices


does cognitect's aws-api library support signing CloudFront URLs? i inspected the ops of but don't see anything related. same with


it may be worth having an add-on library that does presigned URLs for cloudfront + s3


out of curiosity, what would you name this function?

(defn- foo
  "Take `n` random elements from `coll`, choosing those that satisfy `pred` first."
  [n pred coll]
  (let [[preferred rem] ((juxt filter remove) pred coll)]
    (take n (concat (shuffle preferred) (shuffle rem)))))
e.g. (foo 3 even? [1 2 3 4 5 7 9]) => (4 2 1)


extract or pluck would be my suggestions


I was thinking something like take-rand-preferring or take-rand-biased but bias might suggest probability


I like the word draw (as in drawing a hand of cards), but that would probably be more confusing due to double meaning


naming things is hard. 😅




random-sample is in clojure.core


it’s hard but I find it weirdly enjoyable


sample-randomly 😆

😂 4

I accidentally wrote random-sample on IRC, and someone on the channel was like "I know you are joking but that's actually useful" and it ended up in core


it's too simple to take much credit for, but I found the whole process amusing