This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-05-23
Channels
- # announcements (12)
- # beginners (225)
- # calva (7)
- # cider (45)
- # clj-kondo (1)
- # cljdoc (1)
- # cljsrn (3)
- # clojure (112)
- # clojure-dev (45)
- # clojure-europe (6)
- # clojure-finland (2)
- # clojure-india (1)
- # clojure-nl (27)
- # clojure-spec (37)
- # clojure-uk (171)
- # clojurescript (39)
- # core-async (9)
- # cursive (22)
- # datascript (8)
- # datomic (50)
- # emacs (12)
- # figwheel-main (17)
- # fulcro (42)
- # garden (2)
- # hoplon (27)
- # jobs (4)
- # kaocha (8)
- # klipse (2)
- # luminus (2)
- # off-topic (9)
- # perun (33)
- # planck (2)
- # re-frame (9)
- # reagent (48)
- # reitit (5)
- # remote-jobs (1)
- # rum (2)
- # shadow-cljs (23)
- # slack-help (3)
- # spacemacs (18)
- # sql (7)
- # tools-deps (24)
- # unrepl (9)
- # vim (30)
Is it possible to exclude part of my project from being included in the uberjar depending how I call leiningen? nevermind
in Clojure, is it possible to construct looping (but non lazy) values x, y such that (= x y)
infinite loops?
(defn loop-forever [number] (do (println number) (recur (+ 1 number))))
use at your own risk 😛
following on from that, and just for jest, would it be possible, just thinking, of using clojure reducers, to pretty much cause heat death by having each processor core sum an infinite sequence of numbers...
would it make sense for clojure itself to enable some sponsporing options? via Github you can now enable several sponsporing links, e.g. patreon, opencollective. if most of the Clojure users would chip in for a couple of dollars a month, would that make a difference to the development of Clojure?
hey everyone, got a super weird bug i’m kind of stuck on. i have a collection of 6000 32-byte arrays that i’m sorting with a function compare-bytes
. (sort compare-bytes coll)
. when the comparison function hits a particular element, it throws an exception Comparison method violates its general contract!
if i generate a separate collection of 6000 randomized 32-byte arrays, it works fine. even beyond 50,000 elements, no exception is thrown
it’s only with this specific collection. i can see nothing odd about the element on which the comparison function breaks down
wrap it in your own comparator function that catches and prints the inputs
i tried this earlier, but the exception comes from the sort and not the comparison, so i’m not sure how to grab the breaking elements
I believe you get this exception in Java when the transitive property doesn't hold (that is, e1 > e2 and e2 > e3 but NOT e1 > e3), which is presumably something the sort relies on
Ooh! Debugging why a comparator is not a total order. Looking at it ...
can be other values
The JVM can handle negative, 0, positive fine.
given that this is a property, it's a perfect candidate for a generative test with test.check
ha, seems it’s been done https://github.com/ztellman/byte-streams/blob/b5f50a20c6237ae4e45046f72367ad658090c591/test/byte_streams_simple_check.clj
that's not a transitivity test
but could easily be extended to do so! :)
just gen a / b / c, and verify that if a>b and b >c, that a>c
test.check will spit you back a failing example
Something that seems odd to me in cmp-bufs, which the function you linked calls in most/all cases - it calls a ByteBuffer .getInt method in some cases, or .get. From my reading of Java API, those mutate a "position" member of a ByteBuffer object. I don't yet see anything that initializes this position back to the beginning of the ByteBuffer, which would be a problem if the same ByteBuffer was compared more than once.
Digging a bit further, I don't think that is it. cmp-bufs appears to be using the "absolute" variants of the .getInt and .get methods, which might not mutate the position field at all. It didn't on my first attempt trying it, to see if it did modify the position field.
Next guess to investigate -- cmp-bufs has an optimization where for buffers that contain at least 4 bytes, it gets 4 bytes at a time to compare, until it gets to the end, then does 1 byte at a time. It uses p/- to subtract 2 32-bit integers, which can wrap around when the result is a 32-bit integer, too, and cause a JVM comparator to behave in undesired ways, as described in "Beware using subtract" section of this article: https://clojure.org/guides/comparators
I will let you know if I find anything, or give up on it. Haven't given up yet. My current suspicion is that most of the calculations and the return value are longs (64-bit signed integer), but Java comparator functions expect int (32-bit signed integer), and Clojure takes any value returned from a comparator function and coerces it to int via the .intValue method, which can be a different sign than the long value.
I suspect I can find a 4-byte ByteBuffer content for three byte buffers such that their comparison results are consistent if you look at the sign of the long, but inconsistent if you look at the sign of the coerced values returned by .intValue on those longs.
thanks a lot! this is definitely out of my league at this point. i ran a test.check for transitivity and it passed, as i kind of expected
This may be a case where it requires having an "extreme" combination of 4-byte values, e.g. those that when looked at as a 32-bit int, are either at or near the min possible value, max possible value, or near 0.
unless your random test case generation artificially stresses such values, it could take a very long time to find such corner cases.
https://pastebin.com/Hh3EVYXy (6000 32-byte strings, not delimited)
maybe can email info to me at <mailto:[email protected]|[email protected]> if data is not secret or anything
I have a pair of 4-byte ByteBuffer values such that (compare-bytes b1 b2) and (compare-bytes b2 b1) both return a negative value when converted to 32-bit int, so that looks wrong. I don't think it causes the exception, but shows that the comparator is not good.
I have a proposed change to function cmp-bufs in that library that I think should correct the problem.
Would you be interested in trying it? The change is replacing the first of two occurrences of (p/* sign cmp)
in the file byte_streams.clj in function cmp-bufs with (p/* sign (if (pos? cmp) 1 -1))
I'd like to find my own smaller sequence than yours that also causes the exception to be thrown, but I may not generate the necessary enthusiasm for that if that change fixes it for you, and just submit a PR with that change, plus some other tests that give wrong results without the change
In my limited testing, that change fixes the problem I found with a pair of values where (compare-bytes x y) and (compare-bytes y x) both return a negative value, meaning "x < y" and also "y < x". I suspect it will also eliminate the exception you saw thrown.
Created a PR here, if you want to follow along: https://github.com/ztellman/byte-streams/pull/39
looks like this got merged in, great! should be really easy to check if this worked, but i’m not sure how to tell if the jar has been updated. the version number is the same, so i assume not yet
You can see when new releases are made on Clojars here: https://clojars.org/byte-streams
Same data set still throws the same exception?
If so, I will try testing with it, then, and stop assuming I have found the last problem 🙂
I tried with those 6000 32-byte sequences, converted with some little bit of code I wrote to convert those hex strings into 32-byte long java.nio.ByteBuffer objects, and sorting them with the updated compare-bytes caused no exception to be thrown.
I do get an exception thrown when I try to sort using the original definition of compare-bytes that you reported.
Probably a good idea to double check what method you are using to get the new version rather than the one currently published on Clojars.
Doing 'git clone' on byte-streams, then 'lein install' should install the modified version in your local $HOME/.m2 directory, where lein and other tools should be able to find it if you give the proper version number, which I think is 0.2.5-alpha2
You can also change the version string in the first line of the project.clj file to something that has never been released on http://clojars.org if you want to be sure your code is getting that version, instead of one from clojars. You can search for byte-streams on http://clojars.org to see what versions have been released there, and the dates. The last release date is in 2018, so cannot include the latest fix.
case anyone is interested in the issue @gtzogana was asking about, but doesn't want all the gory details in the thread, I do not know if I have fixed the bug he found, but I have a proposed fix that definitely improves the result of compare-bytes for a bad case I found where it can return (x < y) and also (y < x) for many pairs of byte streams x and y. https://github.com/ztellman/byte-streams/pull/39 is the PR if you want to track it.
Is this another example where clojures compare-machinery sometimes returns the wrong result because of truncating longs to int?
Well, wrong if you do not know that fact and plan for it in advance, yes 🙂
There may be another problem I haven't found and fixed yet, but the first problem I found with the comparator function was that it was returning what appeared to be a correctly signed Long value, but when truncated to 32-bit int would sometimes change the sign and not be the desired result.
Like say I want to extract all values from a Java object into a Clojure datastructure instead like say EDN
(defprotocol Datafiable
:extend-via-metadata true
(datafy [o] "return a representation of o as data (default identity)"))
if you have something you want turned in to data in a particular way, you have to extend the protocol to do it
from my understanding a intended use case is kind of a cycle from datafy to nav and back again
Oh right, so each datafy is one level. And to go deeper you have to recursively datafy. Makes sense. So it handles circular graphs
or things that are method calls returning data under that object, which may be newly generated or may just be a getter for existing contents
Think of datafy
/`nav` as an extended, explicit form of laziness.
for things that use normal getter methods, there's bean
and there's definitely other reflection libs for making data out of objects that satisfy a few assumptions, IIRC TimMC made one...
yes, java.data works too and makes different choices
does cognitect's aws-api library support signing CloudFront URLs? i inspected the ops of com.cognitect.aws/cloudfront
but don't see anything related. same with com.cognitect.aws/signer
.
thanks again, @U050ECB92
out of curiosity, what would you name this function?
(defn- foo
"Take `n` random elements from `coll`, choosing those that satisfy `pred` first."
[n pred coll]
(let [[preferred rem] ((juxt filter remove) pred coll)]
(take n (concat (shuffle preferred) (shuffle rem)))))
e.g.
(foo 3 even? [1 2 3 4 5 7 9]) => (4 2 1)
I was thinking something like take-rand-preferring
or take-rand-biased
but bias might suggest probability
I like the word draw (as in drawing a hand of cards), but that would probably be more confusing due to double meaning
I accidentally wrote random-sample on IRC, and someone on the channel was like "I know you are joking but that's actually useful" and it ended up in core
it's too simple to take much credit for, but I found the whole process amusing