Fork me on GitHub
#clojure
<
2017-11-05
>
mingp00:11:55

@zignd I haven't worked with Datomic, so I can't speak to that. But, in the past when I've encountered ClassNotFoundException, it's because I had my classpath set up incorrectly one way or another.

mingp00:11:03

@zignd What build tool are you using? Have you declared all the appropriate dependencies as required by the documentation that you're following?

zignd00:11:00

@mingp I think it's somehow related to my dependencies, some sort of conflict between versions, because I just created a new leiningen project and it worked there. I have also found some Stack Overflow questions and old conversations from the #datomic channel where another person were facing the same problem

zignd00:11:03

It seems to be related to ring as I'm also using it in my project and the fact that it uses Jetty on a certain version...

mingp00:11:43

@zignd I see. In that case, you may need to bump up some versions. Although, of course, that has other effects on the project as a whole.

zignd00:11:52

@mingp I don't think I can, my dependencies are at latest versions

mingp00:11:47

@zignd That is a bit unexpected, then. Sorry, but it doesn't match what I've seen, so I don't think I know how to help you, unfortunately.

zignd00:11:51

@mingp I have an idea, I'm inspecting the dependency tree of a new leiningen project from which the connection works and then I will compare it to the dependency tree of my project with lein deps :tree

mingp00:11:33

And, if it turns out a transitive dependency is resolving to a different version than what works, you can always explicitly pin it.

zignd00:11:58

@mingp Oh, by the way no problem, I appreciate your attention :)

mingp00:11:34

The problem in, for example, NodeJS land is that, sometimes, there does not exist a set of mutually compatible versions for some sets of dependencies. For what it's worth, Java land I feel like has been better about this on average.

zignd00:11:35

I still have to learn how that thing you mentioned works

mingp00:11:01

If there is a transitive dependency (not directly declared in your dependencies, but pulled in as a result of what was directly declared) that needs to be at a specific version, you can pin it (declare it explicitly, at the version needed, in your dependencies).

zignd00:11:05

@mingp Oh, now it makes sense. Merging what you said with what I'm seeing in the comparison of both dependency trees. It seems that the ring/ring-jetty-adapter dependency in my project contains a transitive dependency as you said to org.eclipse.jetty/jetty-io which is also used by com.datomic/clj-client, but because it was already available for ring it was not downloaded for datomic.

mingp00:11:42

Hopefully, you can find a version that makes everything happy.

zignd00:11:02

Yup, I had to add a :exclusions modifier for org.eclipse.jetty/jetty-io and it worked fine. Thanks for the help @mingp!

mingp00:11:53

You're welcome. Actually, thank you for explaining, because I learned something new there, also.

noisesmith01:11:08

@qqq instead of aset in a doseq, use amap or areduce for performance

qqq01:11:18

@noisesmith: in the full problem. I am modifying an existing float-array -- amap is out of the question; also, areduce only returns one item right?

noisesmith01:11:48

an array is one item, but areduce will walk an array input faster than doseq will

noisesmith01:11:24

have you tried using your same array as the ret in amap?

noisesmith01:11:30

it should work

noisesmith01:11:03

generalized seq-oriented code is not going to be as fast as array specific code for the kind of thing you are doing

qqq01:11:51

@noisesmith: this is my fault, my general problem is: I have matrices A, B, C I have a function f I want each eleme of A[i, j] to be repalced with f(A[i-1, j], A[i, j-1], B[i, j], C[i, j]) I don't think areduce / amap is right for this I posted a much simpler questino (for which areduce/amap works fine) mainly to learn how to do 'numeric unboxing stuff' in clojure

noisesmith01:11:35

honestly beyond some very simple levels of complexity, just write some java

noisesmith01:11:44

if unboxing matters that much, it's easier

qqq01:11:21

yeah; I'm seriously considering doing part clojure + part scala (this problem arises from dealing with a scala library in particular)

noisesmith01:11:29

also a multi-dimensional array is an array of arrays right? so you just need nested array ops

qqq01:11:52

the library I'm using has blas / vectorized tensor ops

noisesmith01:11:02

OK - what's the actual type?

qqq01:11:07

unfortunately, I'm doing something dynamic-programmikng ish, where A[i,j] dedpends on A[i-1, j] A[i, j-1]

qqq01:11:46

"actual type?' <-- I don't get what you are asking

noisesmith01:11:36

like what does (type A) return. I would assume [[F

noisesmith01:11:18

so that you would use (aget A 0 0) to get the first item, etc.

imetallica03:11:35

Hey folks, I'm trying to understand how to wrap ServerSocketChannel with core.async... can someone point me towards the right direction?

qqq05:11:25

for side-effecty looping, does clojure have something faster than

(doseq [i (range n)]
...)
or is that the standard, optimal way to od it?

rickmoynihan11:11:21

I’d expect loop/`recur` to be the fastest way available, as it should be closer to the metal than other options built on reduce etc… If performance matters so much measure it, it might be negligible in practice, and reduce/run! etc will probably be nicer.

qqq05:11:38

I can't believe https://clojuredocs.org/clojure.core/run!is a real function -- somehow I've managed to never hear of it all these years

kaosko17:11:30

any compojure api users here? what's the url supposed to be if you want to send a query param set? specifying {id :- [Long] []} for "query?id=0&id=1=3" works fine but expecting a set, i.e. {id :- #{Long} #{}} gives me "{:id (not (set? ["0" "1" "3"]))}

kaosko17:11:54

the documentation on that is ridiculously thin or I just can't find the right place

qqq18:11:51

if I'm doing tensor math, should I just accept that any pure clj+java solution is going to be about 10x slower than C ?

qqq18:11:59

(cc/quick-bench 
 (let [n (* 10 1000 1000) 
       z (float-array n)]
   (doseq [^long i (range 1 n)]
     (aset-float z i
                 (float i)))))

results in:
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 514.302217 ms
    Execution time std-deviation : 1.078130 ms
   Execution time lower quantile : 512.321577 ms ( 2.5%)
   Execution time upper quantile : 515.160140 ms (97.5%)
                   Overhead used : 2.944295 ns

does this seem right 10M aset-float takes 500ms ==> each aset-float takes 50ns ? this seems a bit high, as I expect, on a GHz CPU, for each aset-float to take about 1ns

noisesmith18:11:58

@qqq remember what I said about amap? on my machine that benches 405ms, and this benches 26 ms

(let [n (* 10 1000 1000)                                                        
        z (float-array n)                                                       
        indexes (range 1 n)]                                                    
    (cc/quick-bench                                                             
     (amap z i r                                                                
       (float i))))

noisesmith18:11:28

@qqq it does the same thing, 20 times faster, you rejected what I was saying without even trying it

noisesmith18:11:26

notice that I don't need to use r or z - for your math where you need to look at other indexes of the array, r and z are there (plus i to do math on for index math) - it's much more general than you think

qqq18:11:40

@noisesmith: I tried it just now; $&@($#* amap is fast

noisesmith18:11:49

maybe you'll listen next time

phronmophobic18:11:06

not sure if it matters, but the two different snippets aren’t measuring the same thing

phronmophobic18:11:18

one is creating the float array in the benchmark, while the other is not

noisesmith18:11:26

@smith.adriane it's a silly refactor I can undo 😄

phronmophobic18:11:41

yea, i don’t think it would change the results too much

noisesmith18:11:43

I also benchmarked that refactor on the original, it makes it slower

noisesmith18:11:55

or more likely ,the difference was less than epsilon haha

noisesmith18:11:38

yeah, my quick bench says pulling the quick-bench to the outside again adds .2 ms to the run time

qqq18:11:00

great oracle of @noisesmith, can you show me how to make this code faster too? (note that computing a[i] depends on a[i-1] which makes me unsure how to use amap)

(cc/quick-bench 
 (let [n   (* 100 1000 1000)
       dst (float-array n)
       out (float-array n)]
   (aset out 0 (aget ^floats dst 0))
   (aset out 1 (aget ^floats dst 1))
   (aset out 2 (aget ^floats dst 2))
   (doseq [^long i (range 3 n)]
     (aset out i  
           (min 

            (+ (aget ^floats dst (- i 0))
               (aget ^floats out (- i 2)))

            (+ (aget ^floats dst (- i 1))
               (aget ^floats out (- i 1)))

            (+ (aget ^floats dst (- i 2))
               (aget ^floats out (- i 0))))))))
my current bench output is:
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 1.578544 sec
    Execution time std-deviation : 21.633538 ms
   Execution time lower quantile : 1.548456 sec ( 2.5%)
   Execution time upper quantile : 1.599705 sec (97.5%)
                   Overhead used : 2.944295 ns

noisesmith18:11:36

@qqq like I said, the bindings i, r, z allow you to access index, original, output respectively - you can do math on i and do an array lookup on r or z

noisesmith18:11:46

the weird part is starting on item index 3, you'll need a conditional clearly (or maybe just copy the input back to the first 4 indexes of the output, to skip the conditional on each iteration)

qqq18:11:36

@noisesmith: amap just sped up my code by a factor of 10. I'm not sure what you could have done to get me to take amap seriously earlier, but I'm glad you kept repeating it.

phronmophobic18:11:54

i thought part of the problem was that it doesn’t modify the array in place

noisesmith18:11:19

you can do that inside amap - if you want to

noisesmith18:11:34

maybe areduce is better if that is what you are trying to do actually

noisesmith18:11:46

nobody says the accumulated value maintained by areduce can't be the input

phronmophobic19:11:12

alternatively amap is only like 10 lines of code

phronmophobic19:11:15

(defmacro amap
  "Maps an expression across an array a, using an index named idx, and
  return value named ret, initialized to a clone of a, then setting 
  each element of ret to the evaluation of expr, returning the new 
  array ret."
  {:added "1.0"}
  [a idx ret expr]
  `(let [a# ~a
         ~ret (aclone a#)]
     (loop  [~idx 0]
       (if (< ~idx  (alength a#))
         (do
           (aset ~ret ~idx ~expr)
           (recur (unchecked-inc ~idx)))
         ~ret))))

phronmophobic19:11:28

it’d be pretty easy to make an amap-inplace that just doesn’t create a clone

qqq19:11:47

@smith.adriane: why is it faster than doseq, is it the loop+unchecked-inv vs range ?

phronmophobic19:11:03

the main reason is aset-float

qqq19:11:15

aset-float is slower than a regular aset ?

phronmophobic19:11:27

I think i mislead you there

qqq19:11:28

I thought giving it more info (hey, this is a float) would make it faster

phronmophobic19:11:32

if you do (aset z i ^float i)

phronmophobic19:11:40

it runs faster

phronmophobic19:11:47

but not as fast as amap

phronmophobic19:11:57

(let [n (* 10 1000 1000) 
        z (float-array n)]
    (doseq [^long i (range 1 n)]
      (aset z i ^float i)))

phronmophobic19:11:19

> (macroexpand-1 '(doseq [^long i (range 1 n)]
                                       (aset ^floats z ^long i ^float i)))
(clojure.core/loop
 [seq_75583 (clojure.core/seq (range 1 n)) chunk_75584 nil count_75585 0 i_75586 0]
 (if
  (clojure.core/< i_75586 count_75585)
  (clojure.core/let
   [i (.nth chunk_75584 i_75586)]
   (do (aset z i i))
   (recur seq_75583 chunk_75584 count_75585 (clojure.core/unchecked-inc i_75586)))
  (clojure.core/when-let
   [seq_75583 (clojure.core/seq seq_75583)]
   (if
    (clojure.core/chunked-seq? seq_75583)
    (clojure.core/let
     [c__4685__auto__ (clojure.core/chunk-first seq_75583)]
     (recur
      (clojure.core/chunk-rest seq_75583)
      c__4685__auto__
      (clojure.core/int (clojure.core/count c__4685__auto__))
      (clojure.core/int 0)))
    (clojure.core/let
     [i (clojure.core/first seq_75583)]
     (do (aset z i i))
     (recur (clojure.core/next seq_75583) nil 0 0))))))

qqq19:11:29

interesting, aset-float -> aset changed runtime from "500ms" to "90ms"

phronmophobic19:11:57

but the doseq version is also doing a bunch of other stuff that makes it slower than the amap version as seen in the macroexpand

phronmophobic19:11:44

it does seem crazy that aset-float is slower than aset

phronmophobic19:11:02

but those are similar results to what i saw on my computer as well

qqq19:11:17

@smith.adriane this is so insightful: aset-float -> aset = 500ms -> 90 ms doseq -> loop + unchecked-inc = 90ms -> 20ms which basically matches amap

qqq19:11:21

now I'm happy

qqq19:11:39

I feel like I've made every $&@(#$ possible numeri-clojure mistake there is to make :-)

phronmophobic19:11:10

i’m learning a lot from this too

noisesmith19:11:01

@qqq nah there's lots of numeri-clojure mistakes to make, you're still making the one where you write clojure instead of java when it has to be fast for numerics for example

qqq19:11:54

@noisesmith: now you're ignoring what I'm saying ... see "numeri-clojure" 🙂

qqq19:11:34

@smith.adriane: showing the source of the amap macro was especially useful; as I actually need a 2d version

noisesmith19:11:31

why wouldn't nested amap work for 2d?

qqq19:11:48

because it returns an array of array, instead of a flat float-array

noisesmith19:11:28

and what I meant is that if you care about your time, writing it java in the first place is more expedient, no matter how much we all prefer writing clojure

qqq19:11:53

actually, most of the code is running the GPU using tensor libraries

noisesmith19:11:58

why would anything give you a flat float array out of a 2d array? in java a 2d array is an array of arrays

qqq19:11:02

there just happens to be some $@#($ ETL type work I need to do beforehand

phronmophobic19:11:19

i have jump to source setup and it’s really useful for clojure. either looking at source in clojure core or in libraries I consume

qqq19:11:20

@noisesmith BufferedImage -> getRaster -> getData is an flat array

noisesmith19:11:58

sounds like you could skip a bunch of trouble and use the flat array and do some simple indexing math instead of pretending it's 2d

noisesmith19:11:22

and I hear you saying that the GPU is doing most of the work, but if you'd written this in java in the first place you would have saved a lot of time, and that's 100% of my experience when numeric performance is an issue in clojure

noisesmith19:11:40

my workflow is write in clojure, if it's slow figure out the bottleneck, if the bottleneck is math related write a small java class for it

qqq19:11:37

is float-array guarnateed to be all 0? https://clojuredocs.org/clojure.core/float-array does not mention it, but I ran (seq (float-array 20)) a bunch of times, and it's all 0 all the time

qqq20:11:50

@noisesmith: I see, so float-array just creates a java float array, but then according to https://stackoverflow.com/questions/2154251/any-shortcut-to-initialize-all-array-elements-to-zero that defaults to 0

zignd21:11:41

Regarding Datomic, I'd like to have my unit tests running in isolated environments just like what was demonstrated in this video, [Test-driven Development with Datomic](https://www.youtube.com/watch?v=JaZ1Tm6ixCY), but using Datomic's Client API (`datomic.client`). The author demonstrates a technique in which he creates an in memory database for every unit test in his suite, but it only seems to be possible to do using the Datomic API (`datomic.api`) because it has function create-database (http://docs.datomic.com/clojure/index.html#datomic.api/create-database) that provides some facilities for creating in memory databases whenever you want to. I've been thinking about a work around for his technique using the Client API and the only thing that comes to my mind is executing from within my test code a peer server and then stopping it after the test finishes its execution. Something like: bin/run -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic: It sounds like an overkill solution for me and I believe it's a pretty common use case to execute your unit tests against a fresh database instance. Does anyone have an easier solution that I could use?

csm00:11:00

FWIW, this is pretty much exactly what we do for tests in our services that use the client API. It’s a little heavyweight, but not overly so. Running tests works fine with this procedure.

zignd00:11:33

@U1WMPA45U Good to know I'm not entirely mad in doing so xD Thanks for the letting me know!

gonewest81823:11:10

On the one hand, you can create a fixture that deletes the database and reloads the necessary test data between every test. I found this example on SO: https://stackoverflow.com/questions/35005669/wiping-out-a-datomic-db-for-test-environment