This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-11-05
Channels
- # bangalore-clj (5)
- # beginners (77)
- # boot (29)
- # cider (11)
- # clojure (110)
- # clojure-dev (5)
- # clojure-greece (1)
- # clojure-poland (1)
- # clojure-russia (11)
- # clojure-spec (1)
- # clojurescript (143)
- # clojurex (1)
- # core-async (1)
- # css (1)
- # cursive (40)
- # data-science (1)
- # datomic (6)
- # figwheel (4)
- # fulcro (45)
- # immutant (5)
- # off-topic (4)
- # onyx (8)
- # portkey (9)
- # re-frame (112)
- # shadow-cljs (30)
- # spacemacs (3)
@zignd I haven't worked with Datomic, so I can't speak to that. But, in the past when I've encountered ClassNotFoundException
, it's because I had my classpath set up incorrectly one way or another.
@zignd What build tool are you using? Have you declared all the appropriate dependencies as required by the documentation that you're following?
@mingp I think it's somehow related to my dependencies, some sort of conflict between versions, because I just created a new leiningen project and it worked there. I have also found some Stack Overflow questions and old conversations from the #datomic channel where another person were facing the same problem
It seems to be related to ring as I'm also using it in my project and the fact that it uses Jetty on a certain version...
@zignd I see. In that case, you may need to bump up some versions. Although, of course, that has other effects on the project as a whole.
@zignd That is a bit unexpected, then. Sorry, but it doesn't match what I've seen, so I don't think I know how to help you, unfortunately.
@mingp I have an idea, I'm inspecting the dependency tree of a new leiningen project from which the connection works and then I will compare it to the dependency tree of my project with lein deps :tree
And, if it turns out a transitive dependency is resolving to a different version than what works, you can always explicitly pin it.
The problem in, for example, NodeJS land is that, sometimes, there does not exist a set of mutually compatible versions for some sets of dependencies. For what it's worth, Java land I feel like has been better about this on average.
If there is a transitive dependency (not directly declared in your dependencies, but pulled in as a result of what was directly declared) that needs to be at a specific version, you can pin it (declare it explicitly, at the version needed, in your dependencies).
@mingp Oh, now it makes sense. Merging what you said with what I'm seeing in the comparison of both dependency trees. It seems that the ring/ring-jetty-adapter
dependency in my project contains a transitive dependency as you said to org.eclipse.jetty/jetty-io
which is also used by com.datomic/clj-client
, but because it was already available for ring it was not downloaded for datomic.
Yup, I had to add a :exclusions
modifier for org.eclipse.jetty/jetty-io
and it worked fine. Thanks for the help @mingp!
You're welcome. Actually, thank you for explaining, because I learned something new there, also.
@qqq instead of aset in a doseq, use amap or areduce for performance
@noisesmith: in the full problem. I am modifying an existing float-array -- amap is out of the question; also, areduce only returns one item right?
an array is one item, but areduce will walk an array input faster than doseq will
have you tried using your same array as the ret in amap?
it should work
generalized seq-oriented code is not going to be as fast as array specific code for the kind of thing you are doing
@noisesmith: this is my fault, my general problem is: I have matrices A, B, C I have a function f I want each eleme of A[i, j] to be repalced with f(A[i-1, j], A[i, j-1], B[i, j], C[i, j]) I don't think areduce / amap is right for this I posted a much simpler questino (for which areduce/amap works fine) mainly to learn how to do 'numeric unboxing stuff' in clojure
honestly beyond some very simple levels of complexity, just write some java
if unboxing matters that much, it's easier
yeah; I'm seriously considering doing part clojure + part scala (this problem arises from dealing with a scala library in particular)
also a multi-dimensional array is an array of arrays right? so you just need nested array ops
OK - what's the actual type?
unfortunately, I'm doing something dynamic-programmikng ish, where A[i,j] dedpends on A[i-1, j] A[i, j-1]
like what does (type A) return. I would assume [[F
so that you would use (aget A 0 0)
to get the first item, etc.
Hey folks, I'm trying to understand how to wrap ServerSocketChannel
with core.async
... can someone point me towards the right direction?
for side-effecty looping, does clojure have something faster than
(doseq [i (range n)]
...)
or is that the standard, optimal way to od it?I’d expect loop
/`recur` to be the fastest way available, as it should be closer to the metal than other options built on reduce etc… If performance matters so much measure it, it might be negligible in practice, and reduce/run! etc will probably be nicer.
I can't believe https://clojuredocs.org/clojure.core/run!is a real function -- somehow I've managed to never hear of it all these years
any compojure api users here? what's the url supposed to be if you want to send a query param set? specifying {id :- [Long] []} for "query?id=0&id=1=3" works fine but expecting a set, i.e. {id :- #{Long} #{}} gives me "{:id (not (set? ["0" "1" "3"]))}
if I'm doing tensor math, should I just accept that any pure clj+java solution is going to be about 10x slower than C ?
(cc/quick-bench
(let [n (* 10 1000 1000)
z (float-array n)]
(doseq [^long i (range 1 n)]
(aset-float z i
(float i)))))
results in:
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 514.302217 ms
Execution time std-deviation : 1.078130 ms
Execution time lower quantile : 512.321577 ms ( 2.5%)
Execution time upper quantile : 515.160140 ms (97.5%)
Overhead used : 2.944295 ns
does this seem right 10M aset-float takes 500ms ==> each aset-float takes 50ns ?
this seems a bit high, as I expect, on a GHz CPU, for each aset-float to take about 1ns@qqq remember what I said about amap? on my machine that benches 405ms, and this benches 26 ms
(let [n (* 10 1000 1000)
z (float-array n)
indexes (range 1 n)]
(cc/quick-bench
(amap z i r
(float i))))
@qqq it does the same thing, 20 times faster, you rejected what I was saying without even trying it
notice that I don't need to use r or z - for your math where you need to look at other indexes of the array, r and z are there (plus i to do math on for index math) - it's much more general than you think
@noisesmith: I tried it just now; $&@($#* amap is fast
maybe you'll listen next time
not sure if it matters, but the two different snippets aren’t measuring the same thing
one is creating the float array in the benchmark, while the other is not
@smith.adriane it's a silly refactor I can undo 😄
yea, i don’t think it would change the results too much
I also benchmarked that refactor on the original, it makes it slower
or more likely ,the difference was less than epsilon haha
yeah, my quick bench says pulling the quick-bench to the outside again adds .2 ms to the run time
great oracle of @noisesmith, can you show me how to make this code faster too? (note that computing a[i] depends on a[i-1] which makes me unsure how to use amap)
(cc/quick-bench
(let [n (* 100 1000 1000)
dst (float-array n)
out (float-array n)]
(aset out 0 (aget ^floats dst 0))
(aset out 1 (aget ^floats dst 1))
(aset out 2 (aget ^floats dst 2))
(doseq [^long i (range 3 n)]
(aset out i
(min
(+ (aget ^floats dst (- i 0))
(aget ^floats out (- i 2)))
(+ (aget ^floats dst (- i 1))
(aget ^floats out (- i 1)))
(+ (aget ^floats dst (- i 2))
(aget ^floats out (- i 0))))))))
my current bench output is:
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 1.578544 sec
Execution time std-deviation : 21.633538 ms
Execution time lower quantile : 1.548456 sec ( 2.5%)
Execution time upper quantile : 1.599705 sec (97.5%)
Overhead used : 2.944295 ns
@qqq like I said, the bindings i, r, z allow you to access index, original, output respectively - you can do math on i and do an array lookup on r or z
the weird part is starting on item index 3, you'll need a conditional clearly (or maybe just copy the input back to the first 4 indexes of the output, to skip the conditional on each iteration)
@noisesmith: amap just sped up my code by a factor of 10. I'm not sure what you could have done to get me to take amap seriously earlier, but I'm glad you kept repeating it.
i thought part of the problem was that it doesn’t modify the array in place
you can do that inside amap - if you want to
maybe areduce is better if that is what you are trying to do actually
nobody says the accumulated value maintained by areduce can't be the input
alternatively amap
is only like 10 lines of code
(defmacro amap
"Maps an expression across an array a, using an index named idx, and
return value named ret, initialized to a clone of a, then setting
each element of ret to the evaluation of expr, returning the new
array ret."
{:added "1.0"}
[a idx ret expr]
`(let [a# ~a
~ret (aclone a#)]
(loop [~idx 0]
(if (< ~idx (alength a#))
(do
(aset ~ret ~idx ~expr)
(recur (unchecked-inc ~idx)))
~ret))))
it’d be pretty easy to make an amap-inplace that just doesn’t create a clone
@smith.adriane: why is it faster than doseq, is it the loop+unchecked-inv vs range ?
the main reason is aset-float
I think i mislead you there
if you do (aset z i ^float i)
it runs faster
but not as fast as amap
(let [n (* 10 1000 1000)
z (float-array n)]
(doseq [^long i (range 1 n)]
(aset z i ^float i)))
> (macroexpand-1 '(doseq [^long i (range 1 n)]
(aset ^floats z ^long i ^float i)))
(clojure.core/loop
[seq_75583 (clojure.core/seq (range 1 n)) chunk_75584 nil count_75585 0 i_75586 0]
(if
(clojure.core/< i_75586 count_75585)
(clojure.core/let
[i (.nth chunk_75584 i_75586)]
(do (aset z i i))
(recur seq_75583 chunk_75584 count_75585 (clojure.core/unchecked-inc i_75586)))
(clojure.core/when-let
[seq_75583 (clojure.core/seq seq_75583)]
(if
(clojure.core/chunked-seq? seq_75583)
(clojure.core/let
[c__4685__auto__ (clojure.core/chunk-first seq_75583)]
(recur
(clojure.core/chunk-rest seq_75583)
c__4685__auto__
(clojure.core/int (clojure.core/count c__4685__auto__))
(clojure.core/int 0)))
(clojure.core/let
[i (clojure.core/first seq_75583)]
(do (aset z i i))
(recur (clojure.core/next seq_75583) nil 0 0))))))
but the doseq version is also doing a bunch of other stuff that makes it slower than the amap
version as seen in the macroexpand
it does seem crazy that aset-float is slower than aset
but those are similar results to what i saw on my computer as well
@smith.adriane this is so insightful: aset-float -> aset = 500ms -> 90 ms doseq -> loop + unchecked-inc = 90ms -> 20ms which basically matches amap
i’m learning a lot from this too
@qqq nah there's lots of numeri-clojure mistakes to make, you're still making the one where you write clojure instead of java when it has to be fast for numerics for example
@noisesmith: now you're ignoring what I'm saying ... see "numeri-clojure" 🙂
@smith.adriane: showing the source of the amap macro was especially useful; as I actually need a 2d version
why wouldn't nested amap work for 2d?
and what I meant is that if you care about your time, writing it java in the first place is more expedient, no matter how much we all prefer writing clojure
why would anything give you a flat float array out of a 2d array? in java a 2d array is an array of arrays
i have jump to source setup and it’s really useful for clojure. either looking at source in clojure core or in libraries I consume
@noisesmith BufferedImage -> getRaster -> getData is an flat array
sounds like you could skip a bunch of trouble and use the flat array and do some simple indexing math instead of pretending it's 2d
and I hear you saying that the GPU is doing most of the work, but if you'd written this in java in the first place you would have saved a lot of time, and that's 100% of my experience when numeric performance is an issue in clojure
my workflow is write in clojure, if it's slow figure out the bottleneck, if the bottleneck is math related write a small java class for it
is float-array
guarnateed to be all 0?
https://clojuredocs.org/clojure.core/float-array does not mention it, but I ran (seq (float-array 20))
a bunch of times, and it's all 0 all the time
@qqq https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Numbers.java#L1126
@noisesmith: I see, so float-array just creates a java float array, but then according to https://stackoverflow.com/questions/2154251/any-shortcut-to-initialize-all-array-elements-to-zero that defaults to 0
so that is using new float[]
yeah https://stackoverflow.com/a/3426854
Regarding Datomic, I'd like to have my unit tests running in isolated environments just like what was demonstrated in this video, [Test-driven Development with Datomic](https://www.youtube.com/watch?v=JaZ1Tm6ixCY), but using Datomic's Client API (`datomic.client`).
The author demonstrates a technique in which he creates an in memory database for every unit test in his suite, but it only seems to be possible to do using the Datomic API (`datomic.api`) because it has function create-database
(http://docs.datomic.com/clojure/index.html#datomic.api/create-database) that provides some facilities for creating in memory databases whenever you want to.
I've been thinking about a work around for his technique using the Client API and the only thing that comes to my mind is executing from within my test code a peer server and then stopping it after the test finishes its execution. Something like: bin/run -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic:
It sounds like an overkill solution for me and I believe it's a pretty common use case to execute your unit tests against a fresh database instance. Does anyone have an easier solution that I could use?
FWIW, this is pretty much exactly what we do for tests in our services that use the client API. It’s a little heavyweight, but not overly so. Running tests works fine with this procedure.
@U1WMPA45U Good to know I'm not entirely mad in doing so xD Thanks for the letting me know!
On the one hand, you can create a fixture that deletes the database and reloads the necessary test data between every test. I found this example on SO: https://stackoverflow.com/questions/35005669/wiping-out-a-datomic-db-for-test-environment