Fork me on GitHub
#clojure
<
2019-05-23
>
Nazral01:05:34

Is it possible to exclude part of my project from being included in the uberjar depending how I call leiningen? nevermind

todo09:05:01

in Clojure, is it possible to construct looping (but non lazy) values x, y such that (= x y) infinite loops?

penryu10:05:44

Both infinite and non-lazy?

gklijs10:05:15

(defn loop-forever [number] (do (println number) (recur (+ 1 number)))) use at your own risk 😛

gklijs10:05:09

somehow it put full load on 2 cores when used

penryu10:05:44

“Somehow.” I’m sure with a little work it could tie up more cores.

dharrigan10:05:56

following on from that, and just for jest, would it be possible, just thinking, of using clojure reducers, to pretty much cause heat death by having each processor core sum an infinite sequence of numbers...

👍 4
borkdude10:05:53

would it make sense for clojure itself to enable some sponsporing options? via Github you can now enable several sponsporing links, e.g. patreon, opencollective. if most of the Clojure users would chip in for a couple of dollars a month, would that make a difference to the development of Clojure?

g14:05:43

hey everyone, got a super weird bug i’m kind of stuck on. i have a collection of 6000 32-byte arrays that i’m sorting with a function compare-bytes. (sort compare-bytes coll). when the comparison function hits a particular element, it throws an exception Comparison method violates its general contract!

g14:05:36

if i generate a separate collection of 6000 randomized 32-byte arrays, it works fine. even beyond 50,000 elements, no exception is thrown

g14:05:53

it’s only with this specific collection. i can see nothing odd about the element on which the comparison function breaks down

g14:05:35

also, (= 6000 (count (distinct (map bytes->hex coll))))

dpsutton14:05:54

can you post your compare-bytes fn?

g14:05:29

oh yeah, should have mentioned: it’s not mine, it’s the one in ztellman/byte-streams

g14:05:46

i’ll get a link

Alex Miller (Clojure team)14:05:50

wrap it in your own comparator function that catches and prints the inputs

g14:05:55

i tried this earlier, but the exception comes from the sort and not the comparison, so i’m not sure how to grab the breaking elements

g14:05:37

i guess i could just print all of them. one sec

Alex Miller (Clojure team)14:05:43

I believe you get this exception in Java when the transitive property doesn't hold (that is, e1 > e2 and e2 > e3 but NOT e1 > e3), which is presumably something the sort relies on

g14:05:08

yeah, that’s what google seems to say

g14:05:41

must the function only return something ε [-1, 0, 1]? i noticed it can return other values

andy.fingerhut14:05:47

Ooh! Debugging why a comparator is not a total order. Looking at it ...

andy.fingerhut14:05:05

The JVM can handle negative, 0, positive fine.

g14:05:10

i mean they seem correct (negative when l<r etc)

Alex Miller (Clojure team)14:05:24

given that this is a property, it's a perfect candidate for a generative test with test.check

Alex Miller (Clojure team)14:05:39

that's not a transitivity test

g14:05:39

oh, guess it’s not really checking transitivity

Alex Miller (Clojure team)14:05:52

but could easily be extended to do so! :)

Alex Miller (Clojure team)14:05:34

just gen a / b / c, and verify that if a>b and b >c, that a>c

Alex Miller (Clojure team)14:05:06

test.check will spit you back a failing example

g14:05:17

sweet! on it

andy.fingerhut14:05:18

Something that seems odd to me in cmp-bufs, which the function you linked calls in most/all cases - it calls a ByteBuffer .getInt method in some cases, or .get. From my reading of Java API, those mutate a "position" member of a ByteBuffer object. I don't yet see anything that initializes this position back to the beginning of the ByteBuffer, which would be a problem if the same ByteBuffer was compared more than once.

👍 4
andy.fingerhut14:05:41

Digging a bit further, I don't think that is it. cmp-bufs appears to be using the "absolute" variants of the .getInt and .get methods, which might not mutate the position field at all. It didn't on my first attempt trying it, to see if it did modify the position field.

andy.fingerhut15:05:45

Next guess to investigate -- cmp-bufs has an optimization where for buffers that contain at least 4 bytes, it gets 4 bytes at a time to compare, until it gets to the end, then does 1 byte at a time. It uses p/- to subtract 2 32-bit integers, which can wrap around when the result is a 32-bit integer, too, and cause a JVM comparator to behave in undesired ways, as described in "Beware using subtract" section of this article: https://clojure.org/guides/comparators

g15:05:45

quite the rabbit hole..

andy.fingerhut15:05:26

I will let you know if I find anything, or give up on it. Haven't given up yet. My current suspicion is that most of the calculations and the return value are longs (64-bit signed integer), but Java comparator functions expect int (32-bit signed integer), and Clojure takes any value returned from a comparator function and coerces it to int via the .intValue method, which can be a different sign than the long value.

andy.fingerhut15:05:26

I suspect I can find a 4-byte ByteBuffer content for three byte buffers such that their comparison results are consistent if you look at the sign of the long, but inconsistent if you look at the sign of the coerced values returned by .intValue on those longs.

g15:05:26

thanks a lot! this is definitely out of my league at this point. i ran a test.check for transitivity and it passed, as i kind of expected

andy.fingerhut15:05:47

This may be a case where it requires having an "extreme" combination of 4-byte values, e.g. those that when looked at as a 32-bit int, are either at or near the min possible value, max possible value, or near 0.

andy.fingerhut15:05:08

unless your random test case generation artificially stresses such values, it could take a very long time to find such corner cases.

g15:05:05

would it help if i provided you with the actual collection i was using?

g15:05:26

https://pastebin.com/Hh3EVYXy (6000 32-byte strings, not delimited)

g15:05:27

fails on 5632nd element

andy.fingerhut15:05:16

maybe can email info to me at <mailto:[email protected]|[email protected]> if data is not secret or anything

g15:05:18

it’s in that pastebin link

andy.fingerhut17:05:02

I have a pair of 4-byte ByteBuffer values such that (compare-bytes b1 b2) and (compare-bytes b2 b1) both return a negative value when converted to 32-bit int, so that looks wrong. I don't think it causes the exception, but shows that the comparator is not good.

andy.fingerhut17:05:12

I have a proposed change to function cmp-bufs in that library that I think should correct the problem.

andy.fingerhut17:05:33

Would you be interested in trying it? The change is replacing the first of two occurrences of (p/* sign cmp) in the file byte_streams.clj in function cmp-bufs with (p/* sign (if (pos? cmp) 1 -1))

g17:05:14

yeah, sure!

andy.fingerhut17:05:43

I'd like to find my own smaller sequence than yours that also causes the exception to be thrown, but I may not generate the necessary enthusiasm for that if that change fixes it for you, and just submit a PR with that change, plus some other tests that give wrong results without the change

andy.fingerhut17:05:50

In my limited testing, that change fixes the problem I found with a pair of values where (compare-bytes x y) and (compare-bytes y x) both return a negative value, meaning "x < y" and also "y < x". I suspect it will also eliminate the exception you saw thrown.

andy.fingerhut17:05:00

Created a PR here, if you want to follow along: https://github.com/ztellman/byte-streams/pull/39

g02:05:43

looks like this got merged in, great! should be really easy to check if this worked, but i’m not sure how to tell if the jar has been updated. the version number is the same, so i assume not yet

g16:05:05

doesn’t look like this fixed my issue. :<

andy.fingerhut16:05:48

You can see when new releases are made on Clojars here: https://clojars.org/byte-streams

andy.fingerhut16:05:03

Same data set still throws the same exception?

andy.fingerhut16:05:36

If so, I will try testing with it, then, and stop assuming I have found the last problem 🙂

g18:05:16

that’s right, same issue

andy.fingerhut18:05:45

I tried with those 6000 32-byte sequences, converted with some little bit of code I wrote to convert those hex strings into 32-byte long java.nio.ByteBuffer objects, and sorting them with the updated compare-bytes caused no exception to be thrown.

andy.fingerhut18:05:29

I do get an exception thrown when I try to sort using the original definition of compare-bytes that you reported.

andy.fingerhut18:05:04

Probably a good idea to double check what method you are using to get the new version rather than the one currently published on Clojars.

andy.fingerhut18:05:50

Doing 'git clone' on byte-streams, then 'lein install' should install the modified version in your local $HOME/.m2 directory, where lein and other tools should be able to find it if you give the proper version number, which I think is 0.2.5-alpha2

andy.fingerhut18:05:38

You can also change the version string in the first line of the project.clj file to something that has never been released on http://clojars.org if you want to be sure your code is getting that version, instead of one from clojars. You can search for byte-streams on http://clojars.org to see what versions have been released there, and the dates. The last release date is in 2018, so cannot include the latest fix.

g21:05:03

thanks! you’re right, i was probably using the wrong version. will check this again

andy.fingerhut17:05:41

case anyone is interested in the issue @gtzogana was asking about, but doesn't want all the gory details in the thread, I do not know if I have fixed the bug he found, but I have a proposed fix that definitely improves the result of compare-bytes for a bad case I found where it can return (x < y) and also (y < x) for many pairs of byte streams x and y. https://github.com/ztellman/byte-streams/pull/39 is the PR if you want to track it.

👍 4
slipset18:05:48

Is this another example where clojures compare-machinery sometimes returns the wrong result because of truncating longs to int?

andy.fingerhut18:05:44

Well, wrong if you do not know that fact and plan for it in advance, yes 🙂

andy.fingerhut18:05:16

There may be another problem I haven't found and fixed yet, but the first problem I found with the comparator function was that it was returning what appeared to be a correctly signed Long value, but when truncated to 32-bit int would sometimes change the sign and not be the desired result.

didibus18:05:56

I thought datafy could transform a Java object into Clojure data, am I wrong?

didibus18:05:42

Like say I want to extract all values from a Java object into a Clojure datastructure instead like say EDN

didibus18:05:50

And I want that to be deep

hiredman18:05:17

in the sense that datafy by default doesn't do anything

didibus18:05:23

So, when I call datafy, it just prints the object memory address

hiredman18:05:44

(defprotocol Datafiable
  :extend-via-metadata true

  (datafy [o] "return a representation of o as data (default identity)"))

hiredman18:05:20

if you have something you want turned in to data in a particular way, you have to extend the protocol to do it

didibus18:05:39

Oh. Okay I guess I assumed that Java objects had a smarter default then identity

hiredman18:05:53

I am also not sure how deep the 'datafy' should go

hiredman18:05:26

from my understanding a intended use case is kind of a cycle from datafy to nav and back again

hiredman18:05:16

you datafy something, then nav to something it references, then datafy that, etc

didibus18:05:48

Oh right, so each datafy is one level. And to go deeper you have to recursively datafy. Makes sense. So it handles circular graphs

noisesmith18:05:07

or things that are method calls returning data under that object, which may be newly generated or may just be a getter for existing contents

seancorfield18:05:20

Think of datafy/`nav` as an extended, explicit form of laziness.

didibus18:05:35

I still wished there was a default datafy for objects.

didibus18:05:58

Don't really want to implement my own reflection field parsing

noisesmith18:05:20

for things that use normal getter methods, there's bean

noisesmith18:05:05

and there's definitely other reflection libs for making data out of objects that satisfy a few assumptions, IIRC TimMC made one...

didibus18:05:11

Ya, bean is there, but not deep. It's okay, I'll just use GSON

didibus18:05:26

And then Cheshire to pull that into Clojure

didibus18:05:49

Hum... there seems to be clojure.java.data

Alex Miller (Clojure team)18:05:18

yes, java.data works too and makes different choices

joshkh19:05:05

does cognitect's aws-api library support signing CloudFront URLs? i inspected the ops of com.cognitect.aws/cloudfront but don't see anything related. same with com.cognitect.aws/signer.

ghadi19:05:25

it may be worth having an add-on library that does presigned URLs for cloudfront + s3

ccann19:05:21

out of curiosity, what would you name this function?

(defn- foo
  "Take `n` random elements from `coll`, choosing those that satisfy `pred` first."
  [n pred coll]
  (let [[preferred rem] ((juxt filter remove) pred coll)]
    (take n (concat (shuffle preferred) (shuffle rem)))))
e.g. (foo 3 even? [1 2 3 4 5 7 9]) => (4 2 1)

Mno19:05:51

extract or pluck would be my suggestions

ccann19:05:12

I was thinking something like take-rand-preferring or take-rand-biased but bias might suggest probability

eval-on-point19:05:53

I like the word draw (as in drawing a hand of cards), but that would probably be more confusing due to double meaning

Mno19:05:51

naming things is hard. 😅

Mno19:05:32

random-sample

ccann20:05:13

random-sample is in clojure.core

ccann20:05:30

it’s hard but I find it weirdly enjoyable

Mno20:05:13

sample-randomly 😆

😂 4
noisesmith22:05:31

I accidentally wrote random-sample on IRC, and someone on the channel was like "I know you are joking but that's actually useful" and it ended up in core

noisesmith22:05:51

it's too simple to take much credit for, but I found the whole process amusing