Fork me on GitHub
#clojure
<
2017-10-05
>
cfleming01:10:14

Since clojure.test is compiled with direct linking, am I correct that I should be able to recompile it without direct linking using:

(binding [*compiler-options* (assoc *compiler-options* :direct-linking false)]
  (load "/clojure/test"))
?

Alex Miller (Clojure team)01:10:59

load will just reload the class file. You'd need to do this and call compile to recompile the aot class without direct linking.

cfleming01:10:09

Is what I ended up doing.

cfleming01:10:46

Fugly, but works.

ghadi01:10:24

I don't think so @cfleming - what are you actually trying to do?

cfleming01:10:04

@ghadi So for Cursive’s test integration (looks sort of like this: https://cursive-ide.com/userguide/testing.html, slightly out of date but you get the idea) I with-redefs do-report, because report gives no file/line info for :pass events.

cfleming01:10:25

I have a simple test:

(defn patched-report [original]
  (fn [report]
    (println (pr-str report))
    (clojure.stacktrace/print-cause-trace (Throwable.))
    (flush)
    (original report)))

(let [original-do-report do-report
      original-report report]
  (with-redefs [do-report (patched-report original-do-report)]
    ;(binding [report (patched-report original-report)]
      (run-tests *ns*)))

cfleming01:10:44

So, prints the event map and a thread dump of how we got there.

cfleming01:10:16

Using this, I only get :pass and :fail events in Clojure 1.8, but not e.g. :begin-test-ns

cfleming01:10:43

I can’t figure out why I even get those, TBH.

cfleming01:10:30

I figured I should be able to reload clojure.test with direct linking disabled, but AFAICT it doesn’t work.

cfleming01:10:10

Obviously it wouldn’t work with 3rd party code adding custom assert-exprs

ghadi01:10:23

Yeah maybe we should mark it ^:redef

ghadi01:10:43

I don't think the reload will work

ghadi01:10:59

It's aot'ed

cfleming01:10:04

I think the idea is that report should be dynamic, but needing file/line info is kind of an esoteric use case.

cfleming01:10:27

Why does that matter? Oh, you mean that load will pick up the AOT’ed version?

cfleming01:10:18

Perhaps I should grab the contents of /clojure/test.clj off the classpath and explicitly load that into the REPL.

cfleming01:10:02

(binding [*compiler-options* (assoc *compiler-options* :direct-linking false)]
  (-> "clojure/test.clj"
      ( (-> (class list) .getClassLoader))
      ()
      (load-reader)))

cfleming01:10:36

I’m off for a shower and a good scrub now.

cfleming01:10:44

That’s the see-no-eval monkey 🙂

qqq02:10:47

I need to generate a gaussian sampled float-array of 100 elements. Is there a builtin for this? If not, what library should I look into?

noisesmith02:10:43

@qqq

user=> (let [r (java.util.Random. 42)] (repeatedly 100 #(.nextGaussian r)))
(1.1419053154730547 0.9194079489827879 -0.9498666368908959 -1.1069902863993377 0.2809776380727795 0.6846227956326554 -0.8172214073987268 -1.3966434026780434 -0.19094451307087512 1.4862133923906502 0.8023071496873626 -0.12151292466549345 1.4105062239438624 -0.6402327822135738 -1.2096444592532913 0.35375769787202876 -0.4903496491990076 0.5507215382743629 -1.2035510019650835 0.3210160806416416 1.5511476388671834 0.43853028624710505 0.4815980608245389 1.5196310789680683 -0.2768317291873249 -0.08393897849486337 1.255833005788796 -0.3252727938665772 -0.17329033306108363 -1.8585851445864527 1.4238069456328435 -1.363726024075023 -1.964666098753878 -0.9185948439341892 -2.548887393384806 -1.6309606578419305 -0.12200477461989162 1.289159071801577 -0.2691388556559934 0.2574914085090889 -0.3199143760045327 -1.7684998592513064 -0.4834503128592458 -0.5099904653893699 1.1166733769661994 -0.04094720151728288 -1.1076715169200795 1.8623214176471945 1.1457411377091524 -1.0586772048930921 1.0725991339400673 -1.9317441520296659 0.30102521611534994 0.2475231582804265 1.406156849249087 -1.5202207203569256 0.2709294126920897 0.561249284813777 -0.5298295780368607 0.5390221914988275 2.2123402141787243 -0.6329335687728442 -1.8831759122084633 0.3865659853763343 0.32582927090649455 -0.9013043195000002 -0.002680308907617573 -0.4739592549853249 -0.5479781547659026 -0.01910014847196348 1.6468163882596327 -1.107062592215791 0.5938103926672539 -0.15566462108511642 0.6632872929286855 1.226793360688623 0.8839698437730904 0.22172454670212935 0.9197020859698617 -0.7393758185888677 0.803517749531419 -0.2539417447630359 -0.7638388605060555 -1.8645567427274516 -1.861306200027518 -0.576599881116305 -0.40899380621224757 0.24846093761654187 -0.48091295490277447 0.44621205735391023 -0.4465888888803913 0.045638687865053575 0.7045663273135641 -0.2718240183671583 0.08074877915238832 1.2590964696340183 0.7635098382407334 1.7220810801509723 0.14595005405372477 -0.9946630124621867)

noisesmith02:10:00

the "42" there is the seed

qqq02:10:59

@noisesmith 1. nice, this answered the question I initially asked 2. is this making a function call per random number? as it turns out, I need to generate lots of random numbers (simulating noise for machine learning) -- is there a 'vectorized' approach where I specify the # of random numbers I want, make one call, and it all happens inside some highly optimized java code ?

noisesmith02:10:18

I'd assume hotspot would likely be able to optimize it, but we'd need to test

noisesmith02:10:10

you might be able to get better results with filling a mutable array instead of using repeatedly, depending on how much of a bottleneck this is

noisesmith02:10:59

and no, as far as I know there's no vectorized version - I don't know of good RNGs that parallelize well (not that I'm an expert on that)

qqq02:10:29

I think technically, one can run AES(key, i) all in parallel for different values of i, but running an entire round of AES just to generate a random number might be overkill

noisesmith02:10:54

I've benchmarked the first version (15 microsecs) and I'm benchmarking the array version now

noisesmith02:10:30

user=> (let [r (java.util.Random. 42) b (double-array 100)] (dotimes [i 100] (aset b i (.nextGaussian r))) (seq b))
(1.1419053154730547 0.9194079489827879 -0.9498666368908959 -1.1069902863993377 0.2809776380727795 0.6846227956326554 -0.8172214073987268 -1.3966434026780434 -0.19094451307087512 1.4862133923906502 0.8023071496873626 -0.12151292466549345 1.4105062239438624 -0.6402327822135738 -1.2096444592532913 0.35375769787202876 -0.4903496491990076 0.5507215382743629 -1.2035510019650835 0.3210160806416416 1.5511476388671834 0.43853028624710505 0.4815980608245389 1.5196310789680683 -0.2768317291873249 -0.08393897849486337 1.255833005788796 -0.3252727938665772 -0.17329033306108363 -1.8585851445864527 1.4238069456328435 -1.363726024075023 -1.964666098753878 -0.9185948439341892 -2.548887393384806 -1.6309606578419305 -0.12200477461989162 1.289159071801577 -0.2691388556559934 0.2574914085090889 -0.3199143760045327 -1.7684998592513064 -0.4834503128592458 -0.5099904653893699 1.1166733769661994 -0.04094720151728288 -1.1076715169200795 1.8623214176471945 1.1457411377091524 -1.0586772048930921 1.0725991339400673 -1.9317441520296659 0.30102521611534994 0.2475231582804265 1.406156849249087 -1.5202207203569256 0.2709294126920897 0.561249284813777 -0.5298295780368607 0.5390221914988275 2.2123402141787243 -0.6329335687728442 -1.8831759122084633 0.3865659853763343 0.32582927090649455 -0.9013043195000002 -0.002680308907617573 -0.4739592549853249 -0.5479781547659026 -0.01910014847196348 1.6468163882596327 -1.107062592215791 0.5938103926672539 -0.15566462108511642 0.6632872929286855 1.226793360688623 0.8839698437730904 0.22172454670212935 0.9197020859698617 -0.7393758185888677 0.803517749531419 -0.2539417447630359 -0.7638388605060555 -1.8645567427274516 -1.861306200027518 -0.576599881116305 -0.40899380621224757 0.24846093761654187 -0.48091295490277447 0.44621205735391023 -0.4465888888803913 0.045638687865053575 0.7045663273135641 -0.2718240183671583 0.08074877915238832 1.2590964696340183 0.7635098382407334 1.7220810801509723 0.14595005405372477 -0.9946630124621867)
same answer, theoretically faster method

noisesmith02:10:03

that's just over twice as fast

noisesmith02:10:33

user=> (crit/bench (let [r (java.util.Random. 42)] (doall (repeatedly 100 #(.nextGaussian r)))))
Evaluation count : 3887760 in 60 samples of 64796 calls.
             Execution time mean : 15.584700 µs
    Execution time std-deviation : 244.518601 ns
   Execution time lower quantile : 15.212346 µs ( 2.5%)
   Execution time upper quantile : 16.101522 µs (97.5%)
                   Overhead used : 1.570108 ns

Found 1 outliers in 60 samples (1.6667 %)
	low-severe	 1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil
+user=> (crit/bench (let [r (java.util.Random. 42) b (double-array 100)] (dotimes [i 100] (aset b i (.nextGaussian r)))))
Evaluation count : 9223620 in 60 samples of 153727 calls.
             Execution time mean : 6.688612 µs
    Execution time std-deviation : 115.068935 ns
   Execution time lower quantile : 6.514699 µs ( 2.5%)
   Execution time upper quantile : 6.917354 µs (97.5%)
                   Overhead used : 1.570108 ns
nil

qqq02:10:59

(map identity (let [r (java.util.Random.)] (as-> r (.ints 0 100) (.limit 100) (.toArray )))) this isn't gaussian yet, but there is limit / toArray

noisesmith02:10:21

wait, where is that ints method comingfrom? I don't see it in the docs

noisesmith02:10:41

also you can use seq instead of map identity

noisesmith02:10:42

oh I see it now, it's new in java 8

qqq02:10:12

1. thanks for info on seq 2. I found .ints on stack overflow 🙂

qqq02:10:12

alright, so there's also a .doubles

qqq02:10:21

but it's not clear if we can get gaussian from these

noisesmith02:10:00

next question is whether 6.8 microsecs per 100 numbers is too slow

noisesmith02:10:22

(or whatever that translates to on your target of course)

qqq02:10:37

I'm probably prematurely optimizing -- but I'm translating over numpy code, and the one thing that numpy has hammered into my thought process is "vector ops" instead of "for loops"

qqq02:10:42

anyway, thanks for your help / benchmarking -- back to translating numpy/tensorflow code 🙂

noisesmith02:10:09

yeah, I think for actual vectorized stuff you need to use native libs with the jvm

noisesmith02:10:22

there's bindings to the fortran libs right?

qqq02:10:41

I'm using java/tensorflow bindings

qqq02:10:59

but those bindings don't have linspace / rand (as far as my grepping skills show)

qqq02:10:10

so I think the situation is: there are java tensorflow bindings, but there are NOT java numpy bindings

qqq02:10:24

so numpy util functions like linspace / random generation are not available

blueberry08:10:30

@qqq This is one of the simplest things that Bayadera does, and it can give you billions of such numbers (not only Gaussian) in very short time (AMD GPU only for now, Nvidia support is comming).

mseddon08:10:35

@blueberry oh, i had not run into that library, neat. Although afaik, tensorflow is nvidia only (or at least was), so that's an annoying gpu support mismatch for qqq's problem. 😞

Oliver George09:10:44

Is there a reason that for doesn't have an implied do in it's body? It's an inconvenience when I want to add a debug print statement. (for [] x) has to become (for [] (do (print...) x))

Oliver George09:10:03

Perhaps there's a performance hit or some optimisation issue. I'm just curious really.

pesterhazy09:10:19

just guessing but doseq is for imperative code and for is for pure collection processing - the fact that doseq has implicit do but for doesn't reflects that difference

Oliver George09:10:17

I agree the logic is there. Just wondering if it's pedantic or pragmatic. Print statements are kind of important when you want to know lazy things happened.

pesterhazy09:10:34

I agree, I've definitely run into this before

pesterhazy09:10:10

pro tip: (for [x xs] (doto (do-something x) prn))

tomaas10:10:58

hi, (resolve (symbol "prn")) resolves well to function that I can call, but if i require a function called export-csv then (resolve (symbol "export-csv")) does not work anymore, resolves to`nil`

rauh10:10:18

@tomaas how do you require it?

tomaas10:10:54

(ns lala.x (:require [some.lib :refer [export-csv]))

qqq10:10:52

I have a vector of (vector of (vector of float))) representing a 3d array of height width depth what is the best way to flatten this into a "float-array" ?

delaguardo10:10:37

(flatten your-vector-of-vector-of-vector-of-float)

mchampine12:10:22

Why not (float-array (flatten [[[1.0 2.0] [3.0 4.0]] [[5.0 6.0] [7.0 8.0]] [[9.0 10.0] [11.0 12.0]]])) ?

bronsa14:10:51

evaluation of function objects works in some cases but you should not rely on that

bronsa14:10:36

in particular, it doesn't work when the function object closes over some values

bronsa14:10:49

in this case, the function returned by (comp inc inc) closes over inc

sparkofreason14:10:30

Interesting, thanks. Is there a workaround?

bronsa14:10:32

if you need to make it work you can use (fn [x] (inc (inc x))) manually

bronsa14:10:44

this version doesn't close over any local binding

bronsa14:10:02

just don't do this :)

tdantas14:10:42

what you guys are using to communicate with aws ? s3 / cognito / sqs …

sparkofreason14:10:48

Yeah, it was definitely a hack in the first place.

erwinrooijakkers15:10:52

Will there be specs for functions in clojure.core? Are they needed? Or is it too difficult to make?

sparkofreason15:10:28

Huh. Interestingly leaky abstraction.

bronsa15:10:34

how is it leaky?

sparkofreason15:10:18

Just that it works in some cases and not others, presumably has something to do the underlying implementation.

bronsa16:10:19

it has to do with the fact that eval works on clojure code/data structures

bronsa16:10:25

and function objects are neither

sparkofreason16:10:07

I see. Good tip, actually was able to get my hack to work with that in mind.

leontalbot15:10:36

I am trying to migrate from clj to cljc some ns but I get java.io.FileNotFoundException: Could not locate... on classpath

leontalbot15:10:12

Apparently something is renaming my cljc file extension to cljc.clj

leontalbot15:10:20

at least emacs displays the file with this weird extension

leontalbot15:10:30

Anyone has an idea?

leontalbot15:10:41

My file system still sees .cljc but emacs tells me otherwise when browsing the file tree

dpsutton15:10:57

so i'll take a wild guess. you have clj-refactor along with cider. your buffers are in clojure-mode and the refactoring is ensuring that the file suffix is appropriate. i'll bet that if you change to clojurec-mode (note the c) you will end this algorithmic tug of war

leontalbot15:10:33

Thank you @dpsutton 🙂 How to change to clojurec-mode?

dpsutton15:10:58

M-x clojucec-mode

leontalbot16:10:53

Not sure. I am in a clojure file then I type M-x clojurec-mode

leontalbot16:10:10

i see clojure-mode but not clojurec-mode

leontalbot16:10:45

not sure if this get me in a clojure buffer. I am not an emacs pro 😞

leontalbot16:10:35

Ok I reinstalled clojure-mode and I see clojurec-mode

leontalbot16:10:50

i select it, but still see .cljc.clj …

leontalbot16:10:01

Even if I reboot emacs

lvh16:10:51

Does anyone use Jupyter Notebooks with Clojure? There are three Clojure kernels and I have no idea which are best.

leontalbot16:10:56

And even if I close emacs and have proper .cljc extension, I get java.io.FileNotFoundException: Could not locate

leontalbot16:10:30

I seems to be two problems

dpsutton16:10:53

maybe my suggestion was misleading. sorry

leontalbot16:10:09

I got Could not locate foo/bar__init.class or foo/bar.clj on classpath. instead of Could not locate foo/bar__init.class, foo/bar.clj, or foo/bar.cljc on classpath.

leontalbot16:10:24

I seams cljc in not there

leontalbot16:10:53

I have clojure 1.8.0

leontalbot16:10:17

and i got :source-paths ["src/clj" "src/cljc"] in my projet.clj

leontalbot16:10:32

Thank you anyway for your help

Alex Miller (Clojure team)16:10:32

there’s a ticket for that - the error message is just missing it. it is actually looking for foo/bar.cljc though

leontalbot16:10:36

Reloading Clojure file "/foo/bar.cljc.clj" failed.

leontalbot16:10:04

even without emacs, I get .cljc.clj file naming issue

leontalbot16:10:06

Compiling ClojureScript...
Reloading Clojure file “/foo/bar.cljc.clj” failed.
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Conditional read not allowed, compiling:(foo/bar.cljc.clj:3:6)

Alex Miller (Clojure team)16:10:01

that ends in .clj so will be treated as a Clojure file, not a cljc file

Alex Miller (Clojure team)16:10:43

but maybe that’s an error message error. That “Reloading” message is not part of Clojure core, not sure where that’s from.

leontalbot16:10:38

On my mac, if I do CMD + i on file, I see the file has .cljc.clj extension. When I rename it from there, it compiles

leontalbot16:10:51

Seems really an Emacs issue

leontalbot16:10:58

Sorry about that

Alex Miller (Clojure team)17:10:22

np, too many things that need to work together!

kaosko17:10:29

If I'm invoking a static operation in a java class in repl, is the static initializer block in the class guaranteed to be executed? because it looks to me it's not. it only seems to be normally executed if I add an extra level of indirection, i.e. have another class on java side that invokes the static operation

hiredman17:10:23

how are you testing that?

kaosko17:10:38

well I noticed a very odd behavior when my unit test in clj failed even though it executed the same code as the corresponding junit class. they invoke static operation but results depend on the static initializer block to be run before

kaosko17:10:43

if I add the indirection with an extra java class, the clj test succeeds

hiredman17:10:58

have you been able to directly observe effects of the static initializer not happening?

kaosko17:10:57

I can't explain the result otherwise. there are no other code changes involved, but no, not directly. I suppose it'd be simple to write a standalone test for it. but sounds like this is not a known issue?

hiredman17:10:11

the static initiliazer isn't usually something you have to invoke directly, but there are ways to load a class without having it invoked, and clojure has had some changes over the years to delay when the static initializers are run

hiredman17:10:35

are you invoking the static methods reflectively?

kaosko17:10:10

eh no, how can I invoke it reflectively from clj? I can try that

hiredman17:10:37

if the compiler can't determine the types to resolve the method at compile time it will insert the reflection for you

hiredman18:10:08

you can (set! *warn-on-reflection* true) at the top of your source file and then recompile it to get warnings about it

plins18:10:17

hello everybody, is anyone aware of some webserver in cljs? something like a port of compojure to run with nodejs or something like that

noisesmith18:10:41

compojure is a router

noisesmith18:10:32

@plins anyway you might find macchiato interesting https://github.com/macchiato-framework

noisesmith18:10:00

just be aware that if you care about performance or resources jvm clojure is going to be the better bet

plins18:10:04

ill take a look thx 🙂

didibus21:10:24

What would be the best way to have an experimental namespace? So maybe one day the fns inside it move to another namespace, but I wouldn't want people to need to change their require when that happens. So the experimental should still point to the now none experimental fn.

tanzoniteblack21:10:01

Potemkin's import-vars https://github.com/ztellman/potemkin#import-vars does what you want. Personally, rather than doing it that way though, I'd probably just tag the function as experimental in the final intended namespace, rather than having an NS of experimental things imported all over the place

didibus16:10:41

Thanks for both suggestions. You might be right about the last bit. I'll look into just tagging things as experimental.