Fork me on GitHub
#clojure
<
2019-04-08
>
aryyya00:04:49

Can someone help me understand if I have found an error in Brave Clojure? It gives an implementation of partial:

(defn my-partial
  [partialized-fn & args]
  (fn [& more-args]
    (apply partialized-fn (into args more-args))))
I wrote this function to use with it:
(defn print-args
  [& args]
  (loop [index 0]
    (println (format "arg #%d: %s" index (nth args index)))
    (if (< index (dec (count args)))
      (recur (inc index)))))
But it doesn’t seem to behave like the built-in partial and gives arguments out of order:
=> ((my-partial print-args "Today" "was" "a") "good" "day")
arg #0: day
arg #1: good
arg #2: Today
arg #3: was
arg #4: a
nil
As opposed to:
=> ((partial print-args "Today" "was" "a") "good" "day")
arg #0: Today
arg #1: was
arg #2: a
arg #3: good
arg #4: day
nil
I don’t understand why this happens because the implementation of my-partial seems correct to me. I don’t understand why the arguments are given out of order. The relevant line is (into args more-args).

lilactown00:04:01

into uses the semantics of whatever collection it’s collecting into. & args is a seq

aryyya00:04:44

@lilactown And seqs have stuff pushed onto them?

lilactown00:04:51

so when elements are added to the args seq, it adds them to the head

aryyya00:04:34

@lilactown Hmm. So the implementation in the book is actually wrong?

aryyya00:04:42

@lilactown Here is the usage example in the book:

(def add20 (my-partial + 20))
(add20 3) 
; => 23
This example works because addition is commutative, right? So either the author flubbed it or purposefully didn’t complicate the implementation and purposefully used a simple example where order didn’t matter.

lilactown00:04:58

also they’re not using many args

lilactown00:04:14

but yeah, pretty much

lilactown00:04:31

I’m guessing it’s a mistake

aryyya00:04:16

Yeah I would hope so because I’ve sank way too much time into trying to understand why I don’t get it. It’s probably a mistake.

lilactown00:04:57

I think the easiest fix is to change into -> concat

aryyya00:04:14

Yes, that does the trick!

aryyya00:04:20

Thanks for the help.

aryyya00:04:22

@lilactown I was about to submit a message to the repository of the book but it has dozens of issues and pull requests that haven’t been addressed in years, doesn’t seem worth the time. Now I’m starting to doubt the quality of this resource. But it’s been helpful so far so I guess I’ll stick with it a bit longer.

the2bears03:04:30

@aryyya.xyz it's a good resource, stick with it.

👍 4
borkdude08:04:28

I’m trying to use print-stack-trace with GraalVM, but that doesn’t work since it has reflection issues (which can be resolved using type hints)

borkdude08:04:25

I’ll post a JIRA

oyakushev09:04:00

Does anybody know what is the motivation for having single-argument = arity in Clojure which always returns true? Is this a technical detail or is there a mathematical reason for it?

yuhan11:04:36

I suppose it's so you can do things like (apply = coll) to mean "are all elements in coll equal?" without having to treat the trivial 1-element case differently

4
oyakushev11:04:54

However, you still have to handle 0-element case explicitly, so I don't know if the victory is huge here.

yuhan11:04:53

yeah, I was about to say that "mathematically" it makes sense to have a 0-argument arity that always returns true also

oyakushev11:04:18

I'm not sure it is obvious what 0 or 1-argument = should return.

yuhan11:04:53

but it might be problematic to have = not be the complement of not=

oyakushev11:04:30

not= is derivative from =, it just follows suit

yuhan11:04:52

I think of = as saying "are my arguments all equal?"

oyakushev11:04:54

However, (not= 1) => false makes even less sense to me.

oyakushev11:04:08

> I think of = as saying "are my arguments all equal?" Are equal to what?

yuhan11:04:14

to each other

oyakushev11:04:21

What if there is no other?

yuhan11:04:43

then it's trivially true

yuhan11:04:08

same reason why < has a 1-argument arity

yuhan11:04:35

which should make even less sense unless you think of it as saying "are my arguments all in ascending order?"

oyakushev11:04:08

What you describe is actually different, it's "Doesn't the collection contain distinct elements?"

oyakushev11:04:38

There is a place for a function that behaves like that, but it's a stretch to link that and =

oyakushev11:04:16

> are my arguments all in ascending order? That interpretation of < makes sense, so I don't question it.

yuhan11:04:04

well, the alternative would be to throw an exception, which makes things harder to reason about

yuhan11:04:00

I think it's also convention among other lisps to define = in this way

oyakushev12:04:06

In contrast, it would catch mistakes like (= value #_missing-comparison).

oyakushev12:04:28

> I think it's also convention among other lisps to define = in this way Yeah, that is a valid argument.

oyakushev12:04:12

However, all from eq/eql/equal variety expect exactly 2 arguments.

quadron11:04:36

how to spec a hashmap, whose keys are not specs themselves? say I need to provide spec for a hashmap whose keys are string literals and not keywords? should I be using something other than spec/keys?

quadron11:04:29

I am using a multimethod that dispatches based on deriving the spec of an incoming message.

quadron11:04:39

should I just roll my own spec (with contains?) or is there something in the std library or a simple way of working with spec/keys that i am not aware of?

Alex Miller (Clojure team)12:04:21

You could use s/conformer to keywordize, then use s/keys

✔️ 4
Alex Miller (Clojure team)12:04:52

Right now, we don’t have anything to specifically support keys as strings

vemv14:04:43

> (-> '[^int? ^some? a] first meta)
{:tag int?}
Is ^some? swallowed (lost) here? (I realise it is not expected to pass functions as metadata, but I'm developing a little sth)

bronsa14:04:16

well, ^foo bar is the same as ^{:tag 'foo} bar, and ^foo bar baz is the same as ^{:tag 'foo} ^{:tag 'bar} baz, which is semantically (with-meta (with-meta baz {:tag 'bar}) {:tag 'foo})

bronsa14:04:29

so it's not "lost", you're overriding it

bronsa14:04:52

(same as doing (assoc (assoc {} :foo 1) :foo 2))

👍 4
vemv14:04:05

Great expl! Thanks

Mike C15:04:45

I’m trying to pass a custom deftype over a core.async channel, and it seems to coming out of the other side as a seq, losing its type information and structure.

Mike C15:04:59

Is this a known thing, or is there something I’m missing?

Alex Miller (Clojure team)15:04:47

user=> (require '[clojure.core.async :as a])
nil
user=> (deftype Foo [x])
user.Foo
user=> (def c (a/chan 1))
#'user/c
user=> (a/>!! c (->Foo 1))
true
user=> (def c2 (a/<!! c))
#'user/c2
user=> (class c2)
user.Foo
user=> (ancestors (class c2))
#{clojure.lang.IType java.lang.Object}
user=> (.-x c2)
1

Alex Miller (Clojure team)15:04:53

looks fine to me with a simple test

Mike C15:04:12

Yeah, I think I’ve worked it out

Mike C15:04:27

this particular deftype implements ISeq

Mike C15:04:56

That seems to be enough for it to treated as one — if I remove that implementation it makes it through intact

bronsa15:04:24

I think I know what's going on

bronsa15:04:06

can you give me a small repro for this?

Mike C15:04:27

try that as is, and then comment out the iseq implementation

bronsa15:04:58

yeah so this is not what it appears to be ;)

bronsa15:04:05

user=> (->Segment 0 :foo [1 2 3])
(1 2 3)
user=> (class (->Segment 0 :foo [1 2 3]))
user.Segment

bronsa15:04:10

nothing at all with core.async

bronsa15:04:27

it's just that print-dup gives priority to ISeq over IType

bronsa15:04:34

so the object is printed as if it was a seq

bronsa15:04:03

print-dup or whatever is responsible for printing here, I can never remember what does what

bronsa15:04:12

one of those multimethods anyway

bronsa15:04:36

which is weird because I thought there were appropriate prefer-methods

bronsa15:04:38

let me check

Mike C15:04:41

this actually cropped up because I’m trying to match on it (using core.match) on the other side of the channel

bronsa15:04:17

looks like the prefer-methods is just for records, not types

ghadi15:04:29

printing what you took out of the channel can make it look like types are being lost

ghadi15:04:03

but they're not

Mike C15:04:59

yeah, it does still have the correct type

Mike C15:04:08

so something else is being weird

bronsa15:04:26

well you've told clojure that Segment is a seq so it's printing it using the seq printer

bronsa15:04:39

arguably not the best behavior but that's how it works now

bronsa15:04:11

if the type is definitely getting lost maybe you have something that tries to pr-str/read-string the value?

Mike C15:04:49

it’s possible that it’s actually to do with the match treating it as a seq

bronsa15:04:21

you should probably make that type seqable instead of a seq directly if you can

Mike C15:04:01

I think I can live without it being a seq; it was mostly for completness

Mike C15:04:07

Kind of a weird interaction though

dpsutton16:04:14

I guess that explains why a lot of things implement a “to seq” style rather than directly implement seq

dpsutton16:04:00

I remember a comment in core match about destructing problems since the types implemented both seq and assoc protocols

bronsa17:04:43

yes, the general advice is to make types ISeqable instead of ISeq, when you care about them being more than simple seqs

bronsa17:04:38

which doesn't really make the type less powerful since most collections operate on seqables instead of just on seqs

👍 4
oyakushev20:04:26

Did anyone ever have a problem that JVM doesn't flush SoftReferences even when the heap is 99% occupied, and everything is spinning in FullGC most of the time? I'm using hot Java recompilation heavily which inflates DynamicClassLoader's classCache, and it never clears.

hiredman21:04:48

the simplest explanation is your soft refs are being held strongly somewhere

oyakushev21:04:11

Nope, they are not. Explicitly triggering OutOfMemoryError clears them out.

Alex Miller (Clojure team)21:04:28

well DCL only clears cache when you define a new class

Alex Miller (Clojure team)21:04:39

due to Java recomp are you never defining new classes?

Alex Miller (Clojure team)21:04:53

and thus never triggering a clear?

oyakushev21:04:14

> due to Java recomp are you never defining new classes? It's the opposite, I define new classes all the time with defineClass.

oyakushev21:04:15

In fact it might not even be related to Virgil/Java compilation as I'm using tools.namespace too, and the main contributors to inflated heap seem to be some big data objects that are referenced by Clojure function classes, which are in turn kept around by classCache.

oyakushev21:04:32

I took a heap dump of the application, and MAT couldn't find any live paths to GC roots from those big objects. FullGC was doing nothing. However, as soon as manually triggered OOM, those softrefs successfully disappeared.

hiredman21:04:21

when you say "explicitly triggering an outofmemoryerror" what mechanism are you using to do that?

oyakushev21:04:37

@alexmiller Interestingly enough, right now the DynamicClassLoader.rq is empty, but those SoftReferences are still there.

hiredman21:04:57

and on what thread are you raising that exception, and what happens if you raise a different exception on that thread?

oyakushev21:04:43

Just the REPL thread. Other exceptions won't do anything.

hiredman21:04:48

it sounds like you are allocating small enough amounts of memory that the full gc recovers a very small amount about that seems to be enough so refs aren't cleared, and then you allocate again, so full gc runs again, etc

hiredman21:04:12

either increase or decrease your heap size

oyakushev21:04:16

Yep, it looks exactly like what is happening.

hiredman21:04:55

(or fiddle with gc tuning params)

oyakushev21:04:10

I doubt changing the heap size would work here, it's currently set to 7g for devtime, and the leak manifests itself very slowly, it takes hundreds of reloads before the heap becomes occupied enough to cause lags and hangups. And it never goes past it, I was at 100% OldGen util, and it still refused to give up those SoftRefs.

oyakushev21:04:25

> (or fiddle with GC tuning params) That one could possibly work, yet it is still strange to me that such behavior could be the default. Maybe switching a GC would help, but I don't want to move away from ParGC during development.

hiredman21:04:10

if it is what I described the issue is in the relationship between the gc cycles and the mutator cycles, it isn't an issue of heap size per se, but changing the heap size will change the gc cycle and hopefully break the relationship

hiredman21:04:37

you may want to try g1

oyakushev21:04:58

I prefer a predictable GC in development, but this is subjective and another story. Thanks for the help, anyway.

hiredman21:04:59

as of java 9 g1 is the default collector

Alex Miller (Clojure team)21:04:59

in java 12, I know the new ZGC actually has support for parallel class unloading, which might be helpful for you

Alex Miller (Clojure team)21:04:32

in general though, I'd say this is a level of gc tuning that I am aware of, but not an expert in :)

oyakushev21:04:07

Interesting, I didn't link this problem to class unloading. They could indeed be related. I gonna study the GC logs, see if there's a mention of it there.

oyakushev21:04:12

On a different note, I had a huge success running Clojure with ShenandoahGC, hope to write about it someday.

parrot 4
Alex Miller (Clojure team)22:04:41

would love to read that

16
Alex Miller (Clojure team)22:04:56

would make a great conj talk ;)

😅 4
4