clojure 2020-05-19 | Slack Archive

thelittlesipper01:05:26

Due to certain constraints, the program that needs a representation of the map isn't one of Clojure origins - but it is concerned with Clojure code, and I'm using babashka as a middleman, haha. Thanks guys, I got something working.

zzzzpoi02:05:36

Can someone help me. Why multithread sval of lazy-seq will make the cpu to 100% useage？

dpsutton02:05:46

got some code?

valerauko04:05:10

in performance-conscious code i often see stuff like (though obviously more complicated)

(let [local-constant (computation)]
  (defn foo []
    local-constant))

is let faster than using def ?

Alex Miller (Clojure team)04:05:45

yes, but probably the scoping is actually the higher concern than performance

Alex Miller (Clojure team)04:05:53

def is global, let is local

Alex Miller (Clojure team)04:05:11

so using the let there means only foo will have access to local-constant

hiredman05:05:03

I just assume anything like that encountered in the wild is a premature optimization

💯 4

valerauko05:05:43

i see, that makes sense! thanks!

hiredman05:05:52

Or scoping wise someone being a jerk, because I can still get at local constant if I want by using reflection, but they just want to make my life harder

potetm12:05:22

you can get a local using reflection?

potetm12:05:00

#Also sent to the channel

But, to be clear, I wouldn’t consider it a “jerk move.” It tells you something about the utility of local-constant. It says, “this cannot be used outside of this function.” It can be more informative than a var.

👆 4

potetm12:05:54

I usually assume if you’re looking at source, you can grab forms and run them at your leisure, even if scoping makes it hard 🙂

hiredman15:05:43

I spent a good chunk of the week before last using reflection to monkey patch part of a library we use (counting utf8 characters where the correct thing was counting utf16). I have definite views about about people making decisions about what parts of a program I have access to.

hiredman15:05:31

Closed over data is stored in instance fields on fn objects, so you can reflect and get it

dominicm06:05:37

@hiredman I'm just not promising it'll be there later.

dominicm06:05:47

Otherwise I have to name it

restenb12:05:59

anybody aware of a relatively lightweight clojure library for performing mathematical interpolation?

noisesmith14:05:52

I wouldn't get my hopes up for something clojure specific, clojure is a poor platform for numerics generally. But if there's a java lib, this is the sort of thing that's easy to use without a wrapper.

noisesmith14:05:40

there's an interpolator abstract class in javafx https://docs.oracle.com/javase/10/docs/api/javafx/animation/Interpolator.html

noisesmith14:05:40

clearly designed for animation smoothing rather than curve smoothing, though I imagine the math can be made to cross over

simongray19:05:59

Maths is not my strong suit, but maybe https://github.com/thi-ng/math ? It’s from the prolific mathematician/artist behind https://thi.ng/

💯 4

restenb12:05:11

like linear, quadratic etc. for curve-fitting

potetm12:05:00

replied to a thread:Or scoping wise someone being a jerk, because I can still get at local constant if I want by using reflection, but they just want to make my life harder

👆 4

Eliraz15:05:18

for those of you who uses emacs (evil-mode), which package do you use to handle s-expression manipulation and smart parentheses?

kenny16:05:38

Anyone know what a "Not enough space" message from jmap means?

jmap -dump:format=b,file=heapdump.bin 9
Not enough space

Command exits with status 0. Not sure if the space it's talking about is disk or memory. I'm guessing it has something to do with memory... Trying to load the heapdump in visualvm gets stuck on the "Loading heap dump..." screen. Likely due to a malformed heapdump because of the Not enough space error.

noisesmith16:05:38

not enough heap space is much more likely than not enough disk space (unless you are using docker with a small volume...)

kenny16:05:56

The / fs is at 27% used :thinking_face:

kenny16:05:22

df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G  5.4G   15G  27% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/nvme0n1p1   20G  5.4G   15G  27% /dumps
shm              64M     0   64M   0% /dev/shm
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

^ Doesn't appear to be close to hitting any storage limits. Wonder if this is some weird thing about docker not being able to create "large" files.

noisesmith16:05:10

yeah, this is most likely a memory thing - can you do this again with twice the max heap limit in the jvm?

kenny16:05:47

Perhaps. I was trying to avoid that since I don't have a repro for this issue yet.

kenny16:05:19

The potentially malformed/incomplete heap dump file is a bit over 62mib.

noisesmith16:05:41

you could try

jcmd <pid> GC.heap_dump <file-path>

kenny16:05:44

Ooo, that went through. What's the difference in commands?

noisesmith16:05:19

I'm not absolutely certain, it's supposed to end up doing the same thing, but it might not allocate memory where jamp would(?)

kenny16:05:24

> It is recommended to use the latest utility, `jcmd` instead of `jmap` utility for enhanced diagnostics and reduced performance overhead. > - https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks004.html TIL

noisesmith16:05:08

clearly there's some improvement if only one of them worked :D

kenny16:05:01

I was a bit amused by this problem if it was a memory issue with getting a heap dump. In order to debug a memory leak, I need a heap dump. In order to get a heap dump, I need memory. 😆

noisesmith16:05:35

this reminds me I want to try jdb again now that I have a better understanding of how clojure maps to the vm...

noisesmith16:05:04

it would be interesting to have a clojure-friendly wrapper(s) for the various jdk cli tools

kenny16:05:35

That could be interesting. What makes them Clojure unfriendly? The Clojure guts often surfacing?

noisesmith16:05:18

for jdb, it's the fact that one clojure function call is indirected through multiple levels of interface methods and inner classes

noisesmith16:05:38

I've looked at enough stack traces now that I'd be more likely to know what to do with it

noisesmith16:05:22

also, jdb lets you evaluate arbitrary expressions in a scope, this would be much more interesting if those expressions could be piped into the clojure compiler instead of just being java pseudocode :D

dominicm16:05:29

How can I intentionally trigger a clojure.lang.LispReader$ReaderException? I seem to just get RuntimeExceptions around EOF.

noisesmith16:05:06

what if you close the source stream then read?

Alex Miller (Clojure team)16:05:22

::: should do it

noisesmith16:05:57

that gives a Exceptioninfo, with ReaderExecption as the cause, yeah

Alex Miller (Clojure team)16:05:33

you're always going to get wrapping unless you're calling the reader directly

dominicm16:05:06

(read-string ":::") gives me a runtime exception. Does read-string work differently?

Alex Miller (Clojure team)16:05:19

shouldn't be

noisesmith16:05:46

@dominicm look directly at *e - the cause is a ReaderException

noisesmith16:05:59

same with (read-string ")") actually

dominicm16:05:08

Clojure 1.10.1
(try
  (read-string ":::")
  (catch Exception e
    [(type e) (ex-data e) (ex-cause e)]))
[java.lang.RuntimeException nil nil]
user=>

dominicm16:05:37

Same for ")".

noisesmith16:05:54

try just returning the exception:

noisesmith16:05:54

user=> *e                                                                                                                                                                                                                                                               
#error {                                                                                                                                                                                                                                                                
 :cause "Unmatched delimiter: )"                                                                                                                                                                                                                                        
 :via                                                                                                                                                                                                                                                                   
 [{:type clojure.lang.ExceptionInfo                                                                                                                                                                                                                                     
   :message nil                                                                                                                                                                                                                                                         
   :data #:clojure.error{:phase :read-source}                                                                                                                                                                                                                           
   :at [clojure.main$repl$read_eval_print__9086$fn__9087 invoke "main.clj" 433]}                                                                                                                                                                                        
  {:type clojure.lang.LispReader$ReaderException                                                                                                                                                                                                                        
   :message "java.lang.RuntimeException: Unmatched delimiter: )"                                                                                                                                                                                                        
   :data #:clojure.error{:line 1, :column 2}                                                                                                                                                                                                                            
   :at [clojure.lang.LispReader read "LispReader.java" 314]}                                                                                                                                                                                                            
  {:type java.lang.RuntimeException                                                                                                                                                                                                                                     
   :message "Unmatched delimiter: )"                                                                                                                                                                                                                                    
   :at [clojure.lang.Util runtimeException "Util.java" 221]}]                                                                                                                                                                                                           
 :trace                                                                                                                                                                                                                                                                 
 [[clojure.lang.Util runtimeException "Util.java" 221]

dominicm16:05:30

user=> (try (read-string ":::") (catch Exception e e))
#error {
 :cause "Invalid token: :::"
 :via
 [{:type java.lang.RuntimeException
   :message "Invalid token: :::"
   :at [clojure.lang.Util runtimeException "Util.java" 221]}]
 :trace
 [[clojure.lang.Util runtimeException "Util.java" 221]
  [clojure.lang.LispReader interpretToken "LispReader.java" 412]
  [clojure.lang.LispReader read "LispReader.java" 305]
  [clojure.lang.LispReader read "LispReader.java" 216]
  [clojure.lang.LispReader read "LispReader.java" 205]
  [clojure.lang.RT readString "RT.java" 1878]
  [clojure.lang.RT readString "RT.java" 1873]
  [clojure.core$read_string invokeStatic "core.clj" 3815]
  [clojure.core$read_string invoke "core.clj" 3805]
  [user$eval5 invokeStatic "NO_SOURCE_FILE" 1]
  [user$eval5 invoke "NO_SOURCE_FILE" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7177]
  [clojure.lang.Compiler eval "Compiler.java" 7132]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
  [clojure.main$repl$fn__9095 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl_opt invokeStatic "main.clj" 522]
  [clojure.main$main invokeStatic "main.clj" 667]
  [clojure.main$main doInvoke "main.clj" 616]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.RestFn applyTo "RestFn.java" 132]
  [clojure.lang.Var applyTo "Var.java" 705]
  [clojure.main main "main.java" 40]]}

Same sort of stacktrace for *e too

noisesmith16:05:33

I get the same result for your try/catch, but the prn of the object itself in the repl does show ReaderException

dominicm16:05:00

If I type ::: in the REPL, and check *e then I get something that looks like that you provided.

noisesmith16:05:29

odd, I'm using 1.10.1

Alex Miller (Clojure team)16:05:52

oh, it only wraps in ReaderException if using a LineNumberingPushbackReader, which readString does not

dominicm16:05:53

Same, just in a clj - nothing fancy going on

dominicm16:05:02

Ah :)

Alex Miller (Clojure team)16:05:06

another possibly confusing bit is that ReaderException is also an ex-info (implements IExceptionInfo, but is not an ExceptionInfo instance)

dominicm17:05:10

Yeah, that's why I'm interested in it 😈

dominicm17:05:45

Although I've broken something really badly...

Execution error (VerifyError) at java.lang.Class/getDeclaredConstructors0 (Class.java:-2).                                                                        
Catch type is not a subclass of Throwable in exception handler 232                                                                                                
Exception Details:                                                                                                                                                
  Location:                                                                                                                                                       
    io/dominic/try_PLUS_meander/core$eval19804.invokeStatic()Ljava/lang/Object; @232: astore_1                                                                    
  Reason:                                                                                                                                                         
    Type 'clojure/lang/IExceptionInfo' (constant pool 140) is not assignable to 'java/lang/Throwable'                                                             
  Bytecode:                                                                                                                                                       
    0x0000000: b200 0fb6 0014 c000 16bb 0018 59bb 001a                                                                                                            
    0x0000010: 5912 1cc0 001e b700 21c0 0023 b700 26b9                                                                                                            
    0x0000020: 002a 0200 57b2 002d b600 14c0 0016 122f                                                                                                            
    0x0000030: b200 33b9 0036 0300 c000 38bf 00bf 0000                                                                                                            
    0x0000040: 00bf 0000 bf4c b200 424b a701 754c b200        
...

Quite impressive!

dominicm17:05:31

I guess IExceptionInfo can't be caught on. You can't generally treat ReaderException and ExceptionInfo.

Alex Miller (Clojure team)17:05:37

yeah

Alex Miller (Clojure team)17:05:04

Java depends on the exception class hierarchy

Alex Miller (Clojure team)17:05:20

so you can't really do things you'd want interface wise

dominicm17:05:03

I'm not sure I really get java, but my first thought is that ReaderException should extend ExceptionInfo instead. But I'm betting it's not possible.

Alex Miller (Clojure team)17:05:37

it could

Alex Miller (Clojure team)17:05:00

pretty sure I looked at that option though when I added this in 1.10

Alex Miller (Clojure team)17:05:24

I don't remember now why I didn't do that

dominicm17:05:23

It does extend RuntimeException, I'm betting you can't extend multiple classes like that in Java (is that multiple inheritance?).

Alex Miller (Clojure team)17:05:04

you can't, but you wouldn't need to

Alex Miller (Clojure team)17:05:25

ReaderException < ExceptionInfo < RuntimeException would be same effect

Alex Miller (Clojure team)17:05:34

this patch seems like it would have been an alternative

Alex Miller (Clojure team)17:05:57

Alex Miller (Clojure team)17:05:03

I would have to scrounge through some old notes to see if I tried that and if so why I didn't do that. I do recall trying several things in this area.

Alex Miller (Clojure team)17:05:08

would really need a much more thorough review to see if there are any catch sites that catch both ExceptionInfo and ReaderException to ensure those are still properly differentiated

craftybones17:05:31

Is there a way to write a reporter for cognitect/test-runner? I want to spit out a JSON report

craftybones17:05:50

I could simply switch to kaocha…but wondering if there is a way to extend it

Alex Miller (Clojure team)17:05:41

it just runs clojure.test, which does have some notion of being able to customize reporting

craftybones17:05:58

So I can simply implement the multimethod and it should work?

Alex Miller (Clojure team)17:05:22

like https://github.com/clojure/clojure/blob/d9f3f83182e146525a78cf638f0613487d7e18c6/src/clj/clojure/test/junit.clj#L182 implements junit test output by rebinding some stuff

craftybones17:05:14

Cool, got it. Yeah I’ve tinkered with it before, should be able to do so. Thanks

Alex Miller (Clojure team)17:05:16

or https://github.com/clojure/clojure/blob/d9f3f83182e146525a78cf638f0613487d7e18c6/src/clj/clojure/test/tap.clj is a namespace that outputs a different format

craftybones17:05:17

Thanks Alex. Both are good starting points

Alex Miller (Clojure team)17:05:34

you might have to wrap test-runner to make this runnable via clj though

craftybones17:05:14

Sure, I’ll have to plug intot he test-runners run-test call

Alex Miller (Clojure team)17:05:14

if you can get to the point where you have a more precise idea of what you need, would be happy to look at changing test-runner - I know there's already at least one pr towards this

craftybones17:05:04

An easy thing that can be done is to simply provide an option that takes a namespace/fn which is the override multimethod and run all tests binding clojure.test/report with this provided namespace/fn

Alex Miller (Clojure team)17:05:21

well, would be more interested in evaluating options than doing the first easy thing we can think of :)

craftybones17:05:29

Fair 🙂

craftybones17:05:29

My general use case tends to be running a bunch of tests against source code that interns turn in and generating a JSON based report that I can then turn into either an html page or an email or something. I do this for JS and C and so on. I hacked together a solution with Kaocha, but there’s a bit more there than I need

souenzzo21:05:56

if I (def my-map {:a {} :b {} :c {}}) is guaranteed that (for [[k _] my-map] k) will return (:a :b :c) ? as far as i understood deps.edn use the fact that :deps {...} preserve the order.

noisesmith21:05:34

if so, they are using an alternative ordered implementation when reading hash-maps

noisesmith21:05:40

maps are explicitly not ordered

dpsutton21:05:12

i don't believe so. which is i suppose why deps.edn uses top level and newest rather than first version it finds? since first found isn't well defined

souenzzo21:05:43

as far I understand tools.deps behavior: if a depends on c@v1 and b depends on c@v2 - clj -Sdeps "{:deps {a {...} b {...}}}" will use c@v1 - clj -Sdeps "{:deps {b {...} a {...}}}" will use c@v2 (pick first algorithm)