Fork me on GitHub
#clojure
<
2016-07-03
>
Doug Kirk00:07:50

anybody know who would be an admin for the clojurians Slack team?

akiva00:07:53

@gjnoonan started it but I haven’t seen him around for awhile now.

Doug Kirk00:07:30

thanks very much! I'll try him

seancorfield00:07:08

I’m an admin @kirked — you can PM me if you want

Doug Kirk00:07:09

Oh, thanks Sean! I need to have my 2FA disabled for Clojurians, please! It went away with Google Authenticator on my last phone.

seancorfield00:07:23

I don’t know that admin’s have that power but I’ll take a look...

Doug Kirk01:07:11

The Slack docs (and @ slack on Twitter) say a team admin can turn it off.

seancorfield01:07:43

…Yup, done. Looks like a relatively new feature? I hadn’t seen that before.

Doug Kirk01:07:01

Thanks very much!

seancorfield01:07:19

For future reference (to everyone), Slack-related assistance is often easier to get in #C04V84U7G — and the admins are all listed on the Code of Conduct for this Slack.

Doug Kirk01:07:04

oh, thanks. I looked through the channel list, but it's easy to miss one when there are so many!

Doug Kirk01:07:05

(and now back to your regularly scheduled programming...)

seancorfield01:07:16

(I was going to add a link to the CoC — https://goo.gl/VsJ8q8 — since this is the second most popular channel behind the announce-to-everyone unpartable channel)

maxim04:07:29

@seancorfield: you could use ctrl-o to do that, like in emacs

seancorfield05:07:26

Doesn't work for me, but thanks for updating it like that. Cool.

myguidingstar15:07:04

sometimes some types share the same implementation for a protocol. Is there anything like this:

myguidingstar15:07:21

`(extend-protocol Protocol AType BType (foo [x] ...) (bar [x y] ...) CType (foo [x] ...) (bar [x y] ...))`

myguidingstar15:07:56

(without using hash-maps + merge)

lvh15:07:07

Hm, I’m seeing a really interesting problem where a call only succeeds with type hints (reflector can’t find appropriate method with). Is that necessarily a reflector bug?

lvh15:07:19

This is extra exciting because it’s making my tests fail, but only under cloverage.

bronsa15:07:48

@lvh: I don't know what your case is without looking at the code, but sometimes there's ambiguity that can't be figured out at runtime. Say you have 2 interfaces, A and B, and a class C that implements both. Now say you have two a static method overloaded to take either an instance of A or an instance of B, and you invoke it with an instance of C. both methods might apply, which one should the reflector pick?

lvh15:07:49

@bronsa: hm, so, the code I have now annotates with ^ByteBuffer or ^bytes, and the interface supports a bunch of different permutations for those arguments, so at least that particular problem doesn’t apply

bronsa15:07:57

without knowing exactly the signatures of the method you're trying to invoke and how you're invoking it I can only make guesses

bronsa15:07:52

relying on runtime reflection is a bad idea anyway as it's orders of magnitude slower than a call resolved at compile time

bronsa15:07:18

(I wouldn't be surprised if it turned out to be a bug in Reflector tho, the method matching algorithm can be non-deterministic and non-consistent and there are a number of edge-cases that don't behave as you'd expect I've found over the years)

lvh16:07:28

@bronsa: So, I’m still debugging this further (and I don’t know if you’re asking for the code or not; I just feel bad imposing on your time) and I’m 99% sure it’s about DirectByteBuffer

lvh16:07:58

also, long

lvh16:07:09

and yes, I definitely don’t want reflected calls here

lvh16:07:17

the tests run with reflection warnings

lvh16:07:22

it’s just that cloverage is breaking reflection 😄

lvh16:07:45

One of the warnings I’m getting under cloverage is:

Reflection warning, caesium/crypto/secretbox.clj:1:146 - call to method crypto_secretbox_easy on caesium.binding.Sodium can't be resolved (argument types: [B, [B, unknown, [B, [B).

Reflection warning, caesium/crypto/secretbox.clj:1:146 - call to method crypto_secretbox_easy on caesium.binding.Sodium can't be resolved (argument types: java.nio.ByteBuffer, [B, unknown, [B, [B).

Reflection warning, caesium/crypto/secretbox.clj:1:146 - call to method crypto_secretbox_open_easy on caesium.binding.Sodium can't be resolved (argument types: [B, [B, unknown, [B, [B).

bronsa16:07:48

@lvh: if you have a small reproducible example I wouldn't mind taking a look at it

lvh16:07:53

That “unknown” is long

lvh16:07:11

nope, sorry, just convoluted and crazy

lvh16:07:22

Also is it just me or is type hinting for primitives super weird

lvh16:07:27

I know ^long silently fails

lvh16:07:40

^{:tag ‘long} works … sometimes

lvh16:07:54

Long/TYPE isn’t resolvable for reasons I don’t understand

bronsa16:07:16

depending on where you're placing the type hint, it might or might not get evaluated

bronsa16:07:29

an unevaluated Long/TYPE doesn't make sense as type hint

lvh16:07:18

Why not? (Or is it because it’s not a symbol?)

lvh16:07:48

^ByteBuffer seems like a valid type hint; does that just not do enough evaluation to resolve static variables, or is there some weirdness around primitives I’m missing

bronsa16:07:56

it's.. complex.. clojure resolves type hints via Class/forName + special cases long/`longs`/`object`/..

bronsa16:07:12

(Class/forName "Long/TYPE") isn't going to work

lvh16:07:13

long the fn, or ’long the symbol?

lvh16:07:26

because calling long seems to fix the problem, but I can’t figure out how to type hint that

lvh16:07:30

(it’s already a long)

bronsa16:07:30

>Also is it just me or is type hinting for primitives super weird type hinting in general is super weird and inconsistent

bronsa16:07:42

'long the symbol

lvh16:07:02

OK, so I have a fn:

(defn secretbox-open-easy-to-buf!
  "Decrypt with `crypto_secretbox_open_easy` into the given byte array.

  You only want this to manage the output byte array yourself. Otherwise,
  you want [[secretbox-open-easy]]."
  [^bytes out ^bytes ctext ^bytes nonce ^bytes key]
  (let [clen (long (buflen ctext))
        res (.crypto_secretbox_open_easy sodium out ctext clen nonce key)]
    (if (= res 0)
      out
      (throw (RuntimeException. "Ciphertext verification failed")))))
buflen is a fn from a protocol, which I ended up annotating ^Long because I couldn’t get any permutation of ^’long to work

lvh16:07:20

Here’s the protocol and the two impls:

(defprotocol BufLen
  (^Long buflen [this]))

(extend-protocol BufLen
  (Class/forName "[B")
  (buflen [this] (alength ^bytes this))

  ByteBuffer
  (buflen [this] (.remaining ^ByteBuffer this)))

bronsa16:07:45

right, so protocols can't return primitive types, that ^Long cast is useless and so would ^long

lvh16:07:48

It’s just an abstraction for saying “how many bytes can I throw at this buffer”)

lvh16:07:06

wait, why doesn’t ^Long help?

lvh16:07:18

I thought the entire point of Long is that it’s not a primitive

bronsa16:07:58

using the long function there is the right way to do it -- you don't want to type-hint (buflen ctext) as long because that would be lying to the compiler (it's a boxed Long, not a primitive long)

bronsa16:07:13

(long ..) actually unboxes the Long into a long

lvh16:07:46

apparently alength gives me an Integer

lvh16:07:42

(.remaining ^ByteBuffer buf) returns int though

bronsa16:07:00

it's going to be boxed by the buflen call

lvh16:07:31

so the only reason I have a protocol is because I’m expecting all calls to be hinted, so I wanted them to be resolved at compile time

bronsa16:07:45

protocol functions don't return primitives

lvh16:07:48

It’s also going right into a jnr-ffi call, so

lvh16:07:55

it’s going to be a primitive pretty soon again

bronsa16:07:59

if you want to return primitives, use definterface

bronsa16:07:18

protocol functions return Object no matter how you type hint them

bronsa16:07:18

also remember that type hints should not be used to cast, so if you have an int and type-hint it as long weird things might happen

lvh16:07:53

yeah, that was a mistake

lvh16:07:55

I was just messing around with it trying to get the refl warning to go away at all in an attempt to understand hitning

bronsa16:07:05

your best option is to either not use protocols and just use a function & switch on the argument class, or wrap the protocol function in a type-hinted function returning long, but you're going to pay the price of boxing+unboxing in that case

bronsa16:07:49

which might not be a big overhead, don't know how performance sensitive your code has to be

bronsa16:07:56

definitely way faster than using reflection anyway

bronsa16:07:49

(defprotocol BufLen (-buflen [this])
(defn buflen ^long [this] (long (-buflen this)))

lvh17:07:47

hmhm, I’ll measure it

lvh17:07:57

I don’t know how good the JIT is at making that go away

lvh17:07:01

I mean, I guess there’s the distinction between Integer as returned by alength and the int returned by ByteBuffer/remaining

lvh17:07:08

so I’m calling long no matter what

lvh17:07:21

I’m guessing there’s a weird method in Arrays or something that will tell me the array length as a primitive

lvh17:07:33

@bronsa: You’ve given me a lot to go on, thanks a lot!

Doug Kirk18:07:57

@lvh I can tell you that generally Java developers don't worry about un/boxing of primitives from a performance perspective. If your application is that time-sensitive you'll want to make a thin interface over native code. Usually other things will dominate when you profile.

bronsa18:07:18

@kirked: boxed vs unboxed math can make huge performance differences in hot code paths

bronsa18:07:45

sure if you're doing tons of IO you're probably not going to notice much of a difference, but in algorithmic code boxed vs unboxed does matter

bronsa18:07:04

as an example, tools.reader is about twice as slow as the clojure reader written in java because clojure can't pass around unboxed chars between functions

Doug Kirk19:07:24

@bronsa: Yes, I know all about boxing, having been a Java developer for 15 years. As I said, it depends upon what you're doing whether it matters or not. I've found performance in most applications to be dominated by string manipulation or IO, not numeric algorithms. Thanks for the info about tools.reader. I do find some of the compilation strategies in Clojure to be strange, and think that I would've made some different choices. Then again, I never worked with traditional lisps (so wouldn't carry any baggage from that), and I'm currently happy enough to not even think about performance most of the time. I leave that to Intel, Apple, and Google.

Doug Kirk19:07:57

Generally speaking, if you're running on the HotSpot JVM, it's good enough to optimise all of the hot code paths for you.

bronsa19:07:10

@kirked: I wasn't questioning your knowledge of java/boxing FWIW if that's how it came across

Doug Kirk19:07:15

@bronsa: np, I just wanted to let you know some of my background. I've done some amazing things on the JVM in that time (some of which "couldn't be done"). 🙂 I find it a bit strange that I keep seeing performance coming up in Clojure discussions, tho. My overall perception is that Clojure is plenty fast enough. But then again, I'm not controlling medical or nuclear devices 😄

bronsa19:07:32

generally I'd agree with you that focusing too much on performance of algorithmic code might be a premature/useless optimization -- if we were talking about application code. But @lvh's use case is an algorithmic library code as far as I can tell, and as a rule of thumb I tend to write my libraries to be as fast as they can. Sure the difference between .4ms and 2ms might be completely irrelevant to most use cases, but for others (which as a library writer, you can't foresee!) might mean the difference between your library being usable or not

bronsa19:07:12

(OT with this discussion, but I'd like to know what compilation strategies of clojure you find strange, as that's an argument that interests me quite a bit)

Doug Kirk19:07:29

Hah! Ironically enough, my perception of that has to do with performance optimisation! (I do think it matters for compiler output, for one example). Just on cursory glance (I haven't made any deep study), I would've chosen to emit functions differently. For pure functions, there's no reason not to emit them as JVM static methods, which dispatch faster than instance methods. I'd also emit a module as a single JVM class.

bronsa19:07:33

functions are emitted as static methods when possible since 1.8 FWIW

Doug Kirk19:07:40

If you're interested in this topic, there's a really nice old book Programming for the Java Virtual Machine (Joshua Engel, Addison Wesley) that gives good coverage of the topic. It's too old to include some of the bytecode additions they've made since 1.5 (e.g., frames), but I believe nearly everything in the book is still applicable to today's VMs.

frozenlock19:07:44

I'm trying to connect to a remote nREPL with a given ip/port, but cider-connect is trying to open an SSH connection. Any ideas how to make it connect directly?

seancorfield19:07:06

@frozenlock: Which version of CIDER are you using? That sounds like an issue that was resolved a few versions back...

frozenlock19:07:00

0.11 on the server side.

frozenlock19:07:24

I was on 0.11 in Emacs, but just updated and I still have the same issue 😞

frozenlock19:07:37

seancorfield: I think it's still present in the latest version https://github.com/clojure-emacs/cider/issues/1544

seancorfield20:07:01

Having read all of that (both tickets) I'm trying to think what we did when we ran into this problem...

frozenlock20:07:39

I read them both, but I don't see any obvious solution.

seancorfield20:07:22

Yeah, me neither... Sorry 😐

seylerius21:07:03

How do I parse nested structures in instaparse? Like, foo *bar /baz/ bar* /baf/

lvh21:07:55

In my context, this is crypto code, so I do particularly care about perf and especially about consistent, predictable perf.

lvh21:07:21

I mostly know the relevant parts of the JLS; I’m just confused by the Clojure-specific parts 🙂

seylerius21:07:24

(Someone talked about this with me earlier, but it passed the 10,000 msg threshold)

lvh21:07:39

bronsa: Does the suggestion with Interfaces work for existing types? I’m counting the length of byte arrays (as in [B) and java.nio.ByteBuffers; my understanding was that JVM classes are immutable, so you can’t teach them a new interface (which is part of the reason protocols are cool)

bronsa21:07:30

yeah, interfaces won't do it then

lvh21:07:44

Also, do those type hints cross fn borders? So:

(defprotocol BufLen (-buflen [this])
(defn buflen ^long [this] (long (-buflen this)))

(defn f [^ByteBuffer b]
  (-buflen b) #_hinted
  (buflen b) #_hinted? )

lvh21:07:01

I guess *warn-on-reflection* will tell me 🙂

lvh21:07:05

maybe I just want two fns if it’s still slow

lvh21:07:09

it’s probably good enough though

lvh21:07:40

part of the problem is that I’m implementing new crypto using these tools, and, for better or for worse, the crypto community doesn’t care much unless you get reasonable cpb (cycles per byte)

bronsa21:07:43

they don't, no

bronsa21:07:02

you'll have to type hint f aswell

lvh21:07:19

hm? you mean buflen?

bronsa21:07:18

the -buflen call isn't hinted, only the buflen one will return a long -- if you want to return a primitive long from f you'll have to type-hint f aswell

lvh21:07:28

ah, I see

bronsa21:07:43

setting *unchecked-math* to :warn-on-boxed should also be helpful

lvh21:07:46

fns can return primitives, other fns just can’t take primitives, right?

bronsa21:07:04

they can take primitive longs and doubles if you type hint them

lvh21:07:15

Ah, OK, cool, awesome

bronsa21:07:24

the only way to avoid the boxing+unboxing is not to use a protocol though

lvh21:07:30

but that does mean that all callers aren’t allowed to lie

lvh21:07:35

yeah, but I already have ByteBuffer and [B and I can’t extend them

bronsa21:07:01

so (defn buflen ^long [b] (long (case (class b) .. )) will avoid the boxing + unboxing

bronsa21:07:08

but you don't get the open-dispatch of protocols

lvh21:07:22

fortunately I don’t really care about that

lvh21:07:38

or rather, I care more about the boxed math and slow dispatch 🙂

lvh21:07:26

@bronsa: I’m assuming that explicit type dispatch like that, while still slower than compile-time dispatch, is a lot faster than Clojure’s Reflector?

bronsa21:07:12

definitely

bronsa21:07:55

but yeah, I'd go with the protocol + wrapping fn & profile if the additional boxing is an issue, the overhead might be negligible

lvh21:07:27

OK, I’ll measure it

lvh21:07:53

this is just 1 parameter though, I have some that (at a C level) take 6 or 8

lvh21:07:20

maybe case + map type is appropriate as well, because I know not all of them really exist, it’s just a handful

bronsa21:07:38

primitive support in clojure fns is limited to 4 arguments unfortunately, if you need more than that I don't think you're going to have much luck avoiding boxing

lvh21:07:48

yeah, I ran into that already

lvh21:07:56

That’s fine, I can live with that

lvh21:07:26

Oh, do you need to import type hinted classes, or is it really just Class/forName

bronsa21:07:45

there are a bunch of tricks involving creating a deftype + using an interface that can get you around that but it's not pretty and often unnecessary unless you really need no boxing at all

lvh21:07:46

because slamhound wants to delete that import and I can’t tell if it’s because it doesn’t understand type hinting or if I don’t understand type hinting 😄

lvh21:07:53

and I don't

bronsa21:07:09

depends, some classes are auto-imported by clojure

lvh21:07:34

OK, so, remove it, run tests, see what happens

lvh21:07:32

I’m hoping to do some autogeneration for this because a lot of the code is very repetitive

bronsa21:07:51

thankfully we have macros for that

lvh21:07:28

yeah but type hinting has proven confusing enough before trying to do it in a macro and remembering why ^bytes doesn’t work and you really wanted vary-meta or something 😄

lvh21:07:34

but it appears that maybe jnr-ffi + type hinting + macros are too many chainsaws to juggle at once

bronsa21:07:00

yeah emitting type-hints from macros can be tricky

bronsa21:07:03

anyway, I have to shut off now, hope that helped

lvh21:07:48

@bronsa: That was super helpful all over, thanks a lot 🙂

lvh21:07:09

I am at some point also going to try and figure out why cloverage breaks these type hnts

lvh21:07:17

and also why clojure’s reflector can’t figure this out

lvh21:07:32

I’ve found one problem before where I’m pretty sure it’s because DirectByteBuffer

lvh21:07:41

here it’s probably an unboxed long, we’ll see

bronsa21:07:31

AFAICR cloverage does some weird code-rewriting thing, which might be dropping the type-hints

aengelberg21:07:43

@seylerius I don't think I wrote my example down anywhere other than here, but I can whip up another example for you when I get home tonight

lvh22:07:31

Caesium is up to 100% coverage again :

lvh22:07:38

I mean, it always was, but the coverage build works now

aengelberg22:07:31

<S> = token*
<token> = b | i | u | text
b = <'*'> token* <'*'>
i = ...
u = ...
<text> = #'[^*_/]+'
@seylerius something like that

aengelberg22:07:41

Also see: #instaparse channel

lvh23:07:13

How can I run the “real” lein test if I want to shadow it with an alias?