Fork me on GitHub
#clojure
<
2017-08-14
>
schmee07:08:29

@lepistane it can be done automatically using something like https://github.com/ngrunwald/ring-middleware-format

juhoteperi07:08:22

@lepistane @schmee Or https://github.com/metosin/muuntaja, which is faster, supports async and is extensible

schmee07:08:58

@juhoteperi cool, haven’t seen that!

andrea.crotti11:08:03

does anyone use Newrelic with Clojure?

gonewest81814:08:02

Have you looked at http://OpenTracing.io? It's a vendor neutral spec for collecting distributed tracing data with multiple language bindings and multiple backends including Jaeger (Uber), Zipkin (Twitter), Appdash, LightStep and others.

gonewest81814:08:56

I wrote a clojure wrapper around the opentracing-java API which was relatively easy to do. Needs work though, because the opentracing spec is evolving quickly and they made a significant change to the way trace information is propagated within a threaded java program. And I just haven't gotten back to look at that.

andrea.crotti14:08:33

ah no I didn't know about that

andrea.crotti14:08:38

looks interesting thanks

andrea.crotti14:08:13

for this project to be fair I don't really need it, but since with Heroku it would have been more or less free it would have been nice to have

gonewest81815:08:01

You're welcome. On my project, we discovered we had access to unused appdynamics licenses from another team, not exactly free but we just had to pick up the annual maintenance. So we did that as a starting point. But opentracing is on our radar for a distributed, streaming raytracer we have.

andrea.crotti11:08:23

it supports java but not sure if it would be actually useful

andrea.crotti11:08:27

or other similar alternatives?

henrik11:08:56

This seems quite interesting. Will we see this work reflected in Clojure eventually? https://michael.steindorfer.name/publications/phd-thesis-efficient-immutable-collections.pdf Speedups Compared to Clojure’s Maps. In every runtime measurement CHAMP is better than Clojure. CHAMP improves by a median 72 % for Lookup, 24 % for Insert, and 32 % for Delete. At iteration and equality checking, CHAMP significantly outperforms Clojure. Iteration (Key) improves by a median 83 %, and Iteration (Entry) by 73 %. Further, CHAMP improves on Equality (Distinct) by a median 96 %, and scores several magnitudes better at Equality (Derived). Speedups Compared to Clojure’s Sets. The speedups of CHAMP for sets are similar to maps across the board, with exception of insertion and deletion where it scores even better. CHAMP reduces the footprint of Clojure’s maps by a median 16 % (32-bit) and 23 % (64-bit), ranging from 3 % to 48 %. The median reductions for sets are 30 % (32-bit) and 41 % (64-bit) respectively, ranging from 19 % to 60 %.

Aron12:08:11

probably same content just in video format

henrik13:08:20

@ashnur Not yet, but it’s now on the to-see list!

henrik13:08:09

So there’s been some work in the community on these things it seems.

bronsa13:08:04

those speedups don't take into account that clojure's collections use a different hashing/equality than the default hashCode/equals

jeff.terrell13:08:45

Does that mean that Clojure collections are doing hashing/equality better than this CHAMP thing?

bronsa13:08:28

they're doing it differently

bronsa13:08:16

in particular, clojure uses murmur3 on longs/ints, java uses the value itself

jeff.terrell13:08:04

Ah, so CHAMP might have more hash collisions for maps keyed by a collection, is that the idea?

bronsa13:08:25

no, my point is that the hash calculation used by clojure adds overhead

bronsa13:08:32

that overhead is ignored in the CHAMP perf measurment

bronsa14:08:21

that benchmark is not just benchmarking CHAMP vs HAMT, it's benchmarking CHAMP w/hashCode vs HAMT w/ hasheq (the clojure hashing that uses murmur3)

bronsa14:08:13

it is possible that CHAMP is still faster than clojure's HAMT impl even after using the same hashing/equivalence checks that clojure's coll use, but I suspect the margin might be significantly reduced

jeff.terrell14:08:52

Ah, interesting. Thanks for explaining, @bronsa.

henrik14:08:05

@bronsa They seem to believe that Scala’s implementation can be improved with this, it seems: https://github.com/scala/collection-strawman

bronsa14:08:25

i'm not arguing that there's no performance benefits to be had

henrik14:08:38

Oh sorry, I didn’t mean to imply that you were

henrik14:08:54

Just throwing this in there

bronsa14:08:55

i'm just saying that the order of magnitude figures they give don't take into account other factors in the clj impl

henrik14:08:32

What exactly makes Clojure special in this case? That is, what does the overhead consist of and why would it be necessary in Clojure, while being unnecessary in other implementations?

bronsa14:08:23

>>>Clojure’s hashing strategy for numbers, sequences/vectors, sets, and maps mimics Java’s. In Clojure, however, it is far more common than in Java to use longs, vectors, sets, maps and compound objects comprised of those components (e.g., a map from vectors of longs to sets) as keys in other hash maps. It appears that Java’s hash strategy is not well-tuned for this kind of usage. Clojure’s hashing for longs, vectors, sets, and maps each suffer from some weaknesses that can multiply together to create a crippling number of collisions.

ghadi14:08:43

@cgrand and I have separately looked at implementing CHAMP on the java side. It's a lot more complicated than 'free performance'

cgrand15:08:04

And @ztellman reproduced my measurements that CHAMP vs Clojure differences are hash algorithms differences.

ghadi15:08:24

so the perf side is a wash; did you ever quantify whether the memory improvements were worth?

cgrand16:08:12

No. There should be memory gains, but not those claimed.

cgrand16:08:57

I also expect iteration to be faster but not for reasons evoked in the paper.

henrik14:08:48

The by far most common key in Clojure maps must surely be the keyword. I don’t know what situation would demand that I use “a map from vectors of longs to sets” as a key in a map.

henrik14:08:19

@ghadi I bet it is. I barely comprehend it.

henrik14:08:34

clojure.spec apparently thinks non-keyword keys is such an uncommon thing that it doesn’t even support them.

lvh15:08:26

Is there some weirdness with *ns* while running tests? I’m seeing a failure that I can only explain if ns can somehow be a different test ns

lvh15:08:22

(ns bleeker.io-test
  (:require [ :as bio]
            [clojure.test :as t]))

(t/deftest ns-keyword-tests
  (try
    (in-ns 'bleeker.io-test)
    (t/is (= :bleeker.io-test/a (bio/ns-keyword *ns* "a")))
    (t/is (= :bleeker.io-test/a (bio/ns-keyword *ns* :a)))))
That test only passes with lein test when I add the in-ns. (I know the try is superfluous)

Sam H15:08:03

I’ve noticed *ns* evaluates to a higher level namespace. Possibly what ever lein test calls under the hood. I don’t really fully understand it tho.

lvh15:08:16

I’d grok that but it’s just another test namespace

lvh15:08:29

not even the first one or the previous one

urbank15:08:00

@henrik Hm, I use vectors for keys where vectors are coordinates. [0] [0 0] [0 0 0]

henrik15:08:05

@urbank Sure, i definitely won’t say it doesn’t happen, far from it. In any case, there’s nothing stopping us from adding different implementations with different performance characteristics and using them as appropriate.

urbank15:08:19

Wasn't aware that clojure spec doesn't support it though. In that s/keys only allows keywords, I assume? (Haha, guess I just gave away that I didn't try to spec it.)

henrik15:08:19

Yeah, unfortunately. I discovered this precisely because I had a non-keyword map. It may be that it gets covered in the future. After all, spec is just alpha.

urbank15:08:27

Hm, that is unfortunate. Though I guess in my case the exact keys aren't determined in advance so s/keys wouldn't be applicable anyway.

hmaurer15:08:27

Hi. I have a basic clojure spec for a map, e.g. (s/keys :req [:a :b] :opt [:c]) and I would like to get the list of required and optional keys back out; how can I do this?

hmaurer15:08:07

s/describe* on the spec returns (clojure.spec.alpha/keys :req [:a :b] :opt [:c]) (a list) which I could then parse to get what I want

hmaurer15:08:33

but is there an easy way to convert something like this (clojure.spec.alpha/keys :req [:a :b] :opt [:c]) to something like this {:req [:a :b] :opt [:c]}?

tbaldridge16:08:06

@henrik the problem with having multiple hashmap impls is then you get killed by callsites becoming megamorphic. That was one of the problems with using tuples for small vectors. Using just tuples was fast, using just vectors was fast, using both caused the callsites to to megamorphic and the code got slower than just using one or the other.

henrik16:08:58

Gotcha. That makes a lot of sense. Could spec be leveraged for some kind of static analysis? I.e., if I describe my data structure in one way, one implementation could be used, and if I describe it in another, a different one is invoked?

tbaldridge17:08:42

The JVM already does that, the problem is when the implementation changes from one invocation of a function to the next

noisesmith16:08:54

oh, wow, that’s a big gotcha

richiardiandrea16:08:01

Is there a way with the clj tool to dump/cache the classpath (resolved) without launching anything? Pure resolution?

noisesmith16:08:17

@richiardiandrea that’s a hard thing to google, who provides this “clj” binary?

noisesmith16:08:00

if it can run arbitrary clojure code, you can always do (println (System/getProperty “java.class.path”))

richiardiandrea16:08:41

yes but that always starts a repl if I understood correctly

richiardiandrea16:08:21

I wanted something that just dumps like lein classpath > cp

ghadi16:08:57

@richiardiandrea ${PROJECT_DIR}/.cpcache/default/default.cp

ghadi16:08:19

that's where it gets cached

richiardiandrea16:08:51

yes I see that it is there, but I don't want to start any repl here, because I want to launch lumo instead...

ghadi16:08:38

i don't follow

ghadi16:08:49

lumo is cljs -- tools.deps is clj

ghadi16:08:09

do you want to use lumo to connect to a socket repl?

noisesmith16:08:24

how would you get the info created by a clojure library without running a clojure process?

dominicm16:08:18

The idea (as I understand) is to use tools.deps to: - fetch missing dependencies into m2 - generate a classpath string And then pass that to the lumo cli

noisesmith16:08:15

but how would lumo use your jvm deps?

richiardiandrea17:08:41

@noisesmith currently there is jvm cljs and bootstrapped cljs, the deps can be resolved by maven means

richiardiandrea17:08:56

if they work or not...that depends on the library

noisesmith16:08:21

@richiardiandrea btw lein cp starts a clojure process (only one, instead of two like lein repl does…)

richiardiandrea16:08:16

@noisesmith that's good, but I guess this is the fastest I can get:

$ time lein trampoline cp > cp
cp did not run any project code for trampolining.

real	0m8.430s
user	0m28.768s
sys	0m0.420s
$ time lein trampoline classpath > cp
classpath did not run any project code for trampolining.

real	0m8.453s
user	0m28.892s
sys	0m0.396s

richiardiandrea16:08:28

I guess they are the same 😄

dadair17:08:54

What's the preferred way of using Kafka in clojure? pyr/kinsky? Raw java interop?

noisesmith17:08:29

I use a combo of clj-kafka and interop and that works fine

qqq17:08:26

are multimethods / protocols Clojure's take on abstract-data-types ? the point being "people program this interface / set of functions, instead of the direct underlying implementation"

noisesmith18:08:05

yes but I’d express it differently - ADT and prototypes and multimethods are each ways of doing parametric polymorphism, or more generally handling the expression problem

bfabry18:08:54

@qqq multimethods/protocols are clojure's take on polymorphism (useful!), in clojure land we tend to think of language like "ADT" as data-hiding (not useful)

bfabry18:08:27

ie, you implement protocols using records, but records are still mostly just data, there's no requirement for getters setters etc

qqq18:08:38

but I think data hiding is useful. without data hiding, if I change the layout of a structure, I have no idea where all the pieces of code I have to change are. with data hiding, if I change the layout of a structure, I just reimplement the protocol / multimethods its involved in

noisesmith18:08:57

implementing a protocol / multimethod to access data isn’t a popular thing in clojure though - we just use the regular data structures

bfabry18:08:59

@qqq you don't need data hiding for that, you just need common functions and discipline

bfabry18:08:47

in general though, you're not going to have a great time if you're trying to do data hiding in clojure, the community and the creators are pretty explicit about thinking it's a bad idea

bfabry18:08:35

https://clojure.org/about/rationale#_object_orientation_is_overrated <-- specifically ""It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." - Alan J. Perlis" and "Many functions defined on few primary data structures (seq, map, vector, set)."

shader18:08:30

is there an idiom or function for wrapping an object in a vec or list?

shader18:08:07

so, basically #(do [%])?

shader18:08:22

some reason I keep forgetting about that, since I see vec so much and get disappointed that it just does type coercion

alice18:08:45

Hello local geniuses, I just forked this small clojure text editor because I saw a cool opportunity to explore an idea. Let's say I wanted to have a plugin directory where you could drop a .jar or a .clj file and have it loaded, I know you can load .clj files, but if someone wants to make a big plugin (wouldn't happen, but if), a jar is just how it comes packaged. tl;dr clojure plugins for clojure projects?

shader19:08:37

Another possibly silly pattern question: I have a reducer function (fn [acc v] (conj v acc)) — are there any facilities for converting a function into one of the opposite argument order?

ghadi19:08:52

you just wrote one though @shader

misha19:08:13

is there such thing as "branch prediction" in clojure? e.g. is there any performance benefits/penalties based on if branches order? I like to have short then branches, and long else ones. Do I pay for this anything, other than extra (not) call in predicate?

alice19:08:27

@shader you could just use the function with the -> or ->> macro depending on which order

misha19:08:37

@shader there is also #(-> [%]), if you parse literals and threading macros faster than recall function signatures

misha19:08:51

@alice plugins as jars were discussed here like last week, try searching log (I did not follow the discussion though)

tbaldridge19:08:55

@misha no, and if it did it would probably be abused a lot. The CPU branch predictor has something like a 95% correct rate

tbaldridge19:08:21

Once the code gets JIT-ited the CPU branch prediction will kick in, and it will do a good enough job.

tbaldridge19:08:39

That being said, if you can remove if statements and replace them with math, that's almost always a good idea.

ghadi19:08:31

java does branch profiling before it JITs

tbaldridge19:08:42

yeah, that too

ghadi19:08:51

then the CPU will do branch prediction

hlolli19:08:24

What would be a good way to chain/thread update calls so that the intermediate values are accessable (the following code is just wishful thinking)

(-> {:index 0 :dummy 0}
    (update :index inc)
    (update :dummy (fn [dummy] (+ dummy (:index %)))))
the wishfullness here is that I want to access :index when Im updateing :dummy so that dummy accesses :index after application, there is to say it would have the number 1.

noisesmith19:08:26

@hlolli (-> foo (bar baz) (as-> x (update x :dummy + (:index x))))

noisesmith19:08:52

as-> is really just a let block that keeps assigning to the same symbol, but formatted to work inside -> nicely

hlolli19:08:06

ah! as-> eye opener

noisesmith19:08:13

yeah, it’s great

hlolli19:08:35

up to this point I dismissed it blindly for some syntactical sugar. Thanks!!

Aleh Atsman20:08:48

Hi guys, i am trying to test my macros. Basically i want to test param validation. In macros definition i have some asset calls. In repl everything works fine. But with lein test not. I do not have any ideas what is wrong. Console output is lile this:

Testing lambada.core-test

FAIL in (def-lambda-fn-test) (core_test.clj:10)
should throw assert error if name is not a symbol

expected: (thrown?
           java.lang.AssertionError
           (macroexpand
            (quote (l/def-lambda-fn "" [in out context] (println "hello world")))))
  actual: nil
Test code.
(deftest def-lambda-fn-test
  (testing "should throw assert error if name is not a symbol"
    (is (thrown? java.lang.AssertionError 
                 (macroexpand 
                  '(l/def-lambda-fn ""
                    [in out context]
                    (println "hello world"))))))) 
Source of the macros:
(ns lambada.core
  (:require [clojure.spec.alpha :as s])
  (:import [com.amazonaws.services.lambda.runtime RequestStreamHandler]))

(s/def ::name symbol?)
(s/def ::args vector?)
(s/def ::body list?)
(s/def ::arguments (s/cat :name ::name :args ::args :body ::body))

(defn- assert-args [args]
  (assert (s/valid? ::arguments args)
          (s/explain ::arguments args)))

(defmacro def-lambda-fn
  "Create a named class that can be invoked as a AWS Lambda function."
  [name args body]
  (assert-args [name args body])
  (let [prefix (gensym)
        handleRequestMethod (symbol (str prefix "handleRequest"))]
    `(do
       (gen-class
        :name ~name
        :prefix ~prefix
        :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler])
       (defn ~handleRequestMethod
         ~(into ['this] args)
         ~@body))))

Aleh Atsman20:08:57
replied to a thread:Hi guys, i am trying to test my macros. Basically i want to test param validation. In macros definition i have some asset calls. In repl everything works fine. But with lein test not. I do not have any ideas what is wrong. Console output is lile this: Testing lambada.core-test FAIL in (def-lambda-fn-test) (core_test.clj:10) should throw assert error if name is not a symbol expected: (thrown? java.lang.AssertionError (macroexpand (quote (l/def-lambda-fn "" [in out context] (println "hello world"))))) actual: nil Test code. (deftest def-lambda-fn-test (testing "should throw assert error if name is not a symbol" (is (thrown? java.lang.AssertionError (macroexpand '(l/def-lambda-fn "" [in out context] (println "hello world"))))))) Source of the macros: (ns lambada.core (:require [clojure.spec.alpha :as s]) (:import [com.amazonaws.services.lambda.runtime RequestStreamHandler])) (s/def ::name symbol?) (s/def ::args vector?) (s/def ::body list?) (s/def ::arguments (s/cat :name ::name :args ::args :body ::body)) (defn- assert-args [args] (assert (s/valid? ::arguments args) (s/explain ::arguments args))) (defmacro def-lambda-fn "Create a named class that can be invoked as a AWS Lambda function." [name args body] (assert-args [name args body]) (let [prefix (gensym) handleRequestMethod (symbol (str prefix "handleRequest"))] `(do (gen-class :name ~name :prefix ~prefix :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler]) (defn ~handleRequestMethod ~(into ['this] args) ~@body))))

Any ideas?

hiredman20:08:11

you should use syntax quote for the macro form you are expanding

hiredman20:08:18

almost certainly for (long chain of reasons) macroexpand is failing to resolve l/def-lambda-fn to what you think it should resolve to, so it doesn't know it is a macro and does nothing

hiredman20:08:58

but if you use syntax quote l/def-lambd-fn will be resolved at read time to what you expect

Aleh Atsman20:08:20

but it does so in repl… How should i quote it? Because i have quoted it already

hiredman20:08:34

with syntax quote

hiredman20:08:01

the compilation and runtime contexts for tests run via a tool are often different, but for the repl they are almost always the same

Aleh Atsman20:08:08

‘(l/def-lambda-fn, here i have a quote symbol, or should i call “quote” explicitly?

hiredman20:08:34

do you now know what syntax quote is? because the def-lambda-fn macro is using it

Aleh Atsman20:08:36

i understand what ` symbol do, but i am not sure what do u mean

ghadi20:08:24

Warning, the current releases of leiningen and boot will fail on that without modifications to them ^

ghadi20:08:48

clojure, however is fine

seancorfield21:08:53

What did they end up doing with Jigsaw then?

seancorfield21:08:26

(and how come Clojure runs but Leiningen/Boot are broken?)

Aleh Atsman20:08:25

i got it. i used ' instead of `

Aleh Atsman20:08:39

@hiredman thank you for answers

misha21:08:47

@tbaldridge can you give an example of replacing if with math?

misha21:08:36

(my question comes from day-to-day application/ui code, not from some low level or algorithmic one, if that makes any difference)

tbaldridge21:08:33

@misha sure, in Clojure sometimes you have to count the number of set bits in an int

hmaurer22:08:21

lein repl takes about 50s to start on my laptop for a small project with no so many dependencies. Is that to be expected?

seancorfield23:08:03

FWIW, on my iMac desktop, lein repl outside a project has these timings 4.24 real 3.53 user 0.33 sys

noisesmith23:08:30

@hmaurer how about java -jar clojure.jar (you can get the jar from http://clojure.org or find it under ~/.m2/repository/org/clojure/clojure/

seancorfield23:08:07

@hmaurer What do you have in ~/.lein/profiles.clj? Maybe something there is causing the problem?

seancorfield23:08:41

Else perhaps share your project.clj?

ghadi23:08:12

clojure starts in 3.8 seconds on a Raspberry Pi 3

ghadi23:08:14

... while Running Java 9. 4.37 seconds on java 8.

ghadi23:08:08

# raspberry pi 3
[alarm@alarmpi ~]$ time ./jdk-9/bin/java -jar clojure-1.8.0.jar -e :foo
:foo

real	0m3.876s
user	0m4.170s
sys	0m0.140s
[alarm@alarmpi ~]$ time ./jdk1.8.0_144/bin/java -jar clojure-1.8.0.jar -e :foo
:foo

real	0m4.379s
user	0m3.840s
sys	0m0.150s

ghadi23:08:11

Not bad, right?

ghadi23:08:15

they must have made some ARM backend improvements for Java 9, because Java 9 startup has regressed a smidge on x86-64

seancorfield23:08:33

(for comparison, 2012 iMac, Clojure startup is around 0.750s so, yeah, Go Pi!)

ghadi23:08:05

It improves to 3.55s on java 9 when Clojure is compiled with JDK8 level bytecode.

ghadi23:08:24

A Raspberry Pi is a totally sufficient remote REPL if you are fine with a 920MB of usable heap.

ghadi23:08:30

You, however, would not enjoy running lein on it. tools.deps will soon be sufficient for bootstrapping a JVM.

ghadi23:08:22

I'm going to try to measure lein startup on it. It'll take a moment because I don't have it installed

ghadi23:08:47

[alarm@alarmpi ~]$ time JAVA_CMD=$PWD/jdk1.8.0_144/bin/java ./lein run -m clojure.main -e :foo
Java HotSpot(TM) Client VM warning: TieredCompilation is disabled in this release.
:foo

real	0m6.555s
user	0m7.670s
sys	0m0.260s

ghadi23:08:44

not as bad as I thought, but still double clojure.main. lein currently fails 💥 outright on Java 9 because it puts Clojure on the special bootclasspath and there are more restrictions on that in Java 9