Fork me on GitHub
#clojure
<
2018-02-13
>
hiredman00:02:47

I am sort of surprised that clara would be calling eval at all

hiredman00:02:57

surprised because I've written a rules engine and it never occurred to me to call eval, not because I know anything about clara-rules

hiredman00:02:54

in general, if you are calling eval over and over again, the better thing to do is to call eval once to generate a function, then invoke that function over and over again

aengelberg00:02:53

=> (distinct #{1 2 3})
UnsupportedOperationException nth not supported on this type: PersistentHashSet
neat

mikerod00:02:20

@hiredman It calls eval to compile functions around the provided code. It supports fairly arbitrary forms in the rule syntax, so it isn’t something as straightforward as some basic interpreter. Also, that may suffer in terms of performance even if it was. As far as rules engines on the JVM go, Drools has similar rule “eval” going on as in Clara. I agree that ideally there’d be less things evaled. Clara actually does attempt (and could do more) to avoid eval of equivalent forms. What is striking me as odd right now though is just that many calls to eval on separate forms is slower than doing a batch of them via eval

mikerod00:02:54

eg

(eval f1) (eval f2) (eval f3) .... (eval f-n)
;; is slower than

(eval [f1 f2 f3 ... f-n])
Where each f is an un-eval’ed form

bronsa00:02:15

how is making n classes taking less than making 1 class surprising?

mikerod00:02:45

I think now that you say that, I get it

mikerod00:02:29

I think I have been failing to see the number of classes needed in the first example vs the second

mikerod00:02:01

Each eval makes a class then I guess. And when you put forms together, you reduce that count?

hiredman00:02:21

in order to run code on the jvm you need to stick it in a class

hiredman00:02:37

evaluating clojure code first compiles it to jvm bytecode

mikerod00:02:37

Yeah, I get that part. I think that’s where I was just not thinking it through.

mikerod00:02:49

I was thinking that the class count was about the fs

mikerod00:02:55

not about the calls to eval itself

mikerod00:02:15

I had something like (eval '(fn [x] (+ x 2)))

mikerod00:02:24

So I was just thinking of the class made for that fn

bronsa00:02:26

(eval x) is roughly doing ((eval (fn [] x)))

mikerod00:02:13

I saw in profiling that there was notably more time spent in ObjExpr class defining

mikerod00:02:20

Which I think is making sense now.

mikerod00:02:34

I think I can easily see this distinction just by experimenting with the cardinality of classes created via eval using *compile-files* = true in the REPL

mikerod00:02:21

Thanks @bronsa and @hiredman for setting me straight. I guess it was a fairly dumb question afterall 😛

macrobartfast01:02:55

so, I have some perhaps misguided notion of making docker smaller services with clojure and running them with kubernetes... it seems that I am doing too much with individual clojure projects, mainly because they become weird multipurpose uberjars and are hence harder to reason about... any thoughts and is this an anti-pattern?

macrobartfast02:02:47

as a side note, I found https://github.com/Quantisan/docker-clojure and in there it is said "Compiling an uberjar and running it via java -jar or using lein run trampoline would be much more memory-efficient." ... this is a bit confusing to me...

macrobartfast02:02:21

is memory a primary concern? Isn't building smaller things and dockerizing them a plus?

macrobartfast02:02:59

I guess I'm looking for pointers on how to make my larger projects more 'composable' themselves, and docker comes to mind, but I may not be in touch with the best approach.

qqq02:02:46

Does Java have any builtin kv store? I'm looking for something besides "use the file system" to store a bunch of kv pairs

gonewest81802:02:51

@macrobartfast I think you’re really asking about the pros and cons of building microservices versus monoliths.

noisesmith02:02:18

@macrobartfast of how would a docker image containing all your code plus a jvm use less memory than a jvm plus a jar containing all your code?

gonewest81802:02:18

For which there are pros and cons, and in the specifics it will depend what you’re building and how large you intend it to scale.

Matt Phillips02:02:19

@qqq not that I’m aware of. I’ve been using H2's MVStore with great success so far (http://www.h2database.com/html/mvstore.html)

macrobartfast02:02:07

@noisesmith well, I guess each docker image has the overhead of a small os of sorts, Alpine in this case.

gonewest81802:02:31

No, that’s really not how docker works.

noisesmith02:02:44

point being that anything that can be done with N docker images can be done with N fat jars

gonewest81802:02:47

It’s kernel namespace mechanisms to isolate processes.

noisesmith02:02:27

the advantage docker gives you (if any) isn't about resource usage - it's about automation and simplicity of deployment

macrobartfast02:02:39

which is conceptually appealing

macrobartfast02:02:54

though every time I get into docker I rabbit hole into networking issues etc

macrobartfast02:02:26

which makes me wonder if java had the container thing going all along in application servers and so on.

macrobartfast02:02:00

that is, the jar as a sort of unit that docker now provides in cases

macrobartfast02:02:51

@gonewest818 it is probably not a discussion for here.

noisesmith02:02:00

what's missing with a jar is the deployment and OS tooling (keeping the service running, automated startup and shutdown) - that's not extremely hard (I do it with jsvc / commons daemon), but I see how docker has a similar appeal

macrobartfast02:02:41

although I was just sanity checking to make sure there isn't a commonly used alternative strategy clojure devs use for bigger apps before I waste 2 weeks learning about docker/kubernetes/etc.

noisesmith02:02:41

but you'll also simplify things in the docker world if you make a docker image that uses a jvm and a jar, instead of leiningen plus jvm plus resolving deps and building on startup etc.

macrobartfast02:02:31

well, I think https://github.com/Quantisan/docker-clojure has that underway, from the looks of it.

noisesmith02:02:54

like, do people who write C code make docker images that contain gcc, that install all the deps and compile the app from scratch on startup? I guess they could do that but I'd suggest they are making things more complicated than needed

macrobartfast02:02:51

I think I'll go the docker route for now because then I can pull down containers that do things I need, and they can use whatever language they want to.

macrobartfast02:02:15

anyways, off topic, as said, I guess.

macrobartfast02:02:34

thanks for the input!

gonewest81802:02:02

Yes, you’ll spend some time scratching your head about Docker but worth the effort in my experience. Ask questions on #docker as you need…

gonewest81802:02:16

@qqq in memory, or on disk? How big is the kv data?

itruslove05:02:45

@qqq single machine? if so I've seen RocksDB used to pretty good effect

wei05:02:34

I'm trying to split a range if the difference is greater than one. is there a good generic function for this? e.g.

(partition-with #(> (- %2 %1) 1) [1 2 3 6 7 8 9 15 16 17 20]) => [[1 2 3] [6 7 8 9] [15 16 17] [20]]
I'm aware of partition-by but it doesn't quite do what I want, though it's close

Björn Ebbinghaus11:02:03

(defn partition-between [pred? coll] 
  (let [switch (reductions not= true (map pred? coll (rest coll)))] 
    (map (partial map first) (partition-by second (map list coll switch)))))
https://stackoverflow.com/a/23221442/3616102

wei22:02:44

thanks!!

jumar10:02:48

Let's say I have a naive apply implementation:

(defn my-apply [f args]
  (loop [f f
         args args]
    (if (seq args)
      (recur (partial f (first args))
             (rest args))
      (f))))
(my-apply + (range 10000))
;;=> Throws StackOveflowError
I'm wondering why this throws StackOverflowError. Is this something related to partial?

schmee10:02:26

the f you output will be nested as many times as there are arguments, so with 10000 args you’ll have a call stack 10000 layers deep, which will overflow

erhardtmundt12:02:37

guys, I'd need to compare 2 vectors and I thought of clojure.data/diff, but I want to compare the elements in a custom way

erhardtmundt12:02:45

what would you suggest?

fmind12:02:29

@erhardtmundt do they have the same size or are they ordered somehow ? If so, map can accept multiple collections. Otherwise, you can convert them into set and use clojure.set

erhardtmundt12:02:27

@fmind the 2 vectors are not guaranteed to have the same size and their elements are hashes

erhardtmundt12:02:25

I'd like to use one of their values to identify them

dominicm12:02:52

@erhardtmundt I think you need to be more clear about the comparison you want to do before we can advise.

erhardtmundt12:02:31

I have something like [{:id 1 :foo "foo" :bar "bar"}] [{:id 1 :baz "baz"}]

erhardtmundt12:02:07

I want to compare the hashes I have in each vector and just check for id

erhardtmundt12:02:03

so I would expect diff to return (nil, nil, any of the 2 )

dominicm12:02:45

@erhardt.mundt compared by :id? Or compared by position in the vector?

erhardtmundt12:02:06

compared by :id

dominicm12:02:14

@erhardt.mundt group-by is probably your first port of call, so something like: (map (fn [[k v]] (map hash v)) (group-by :id (concat coll-a coll-b)))

dominicm12:02:27

Actually, even better would be to do vals before map, so like (map #(map hash v) (vals (group-by :id (concat coll-a coll-b))))

dominicm12:02:53

Essentially you're grouping the maps by their :id, and then doing the operation on them in a known state.

erhardtmundt12:02:13

:thinking_face:

sundarj12:02:26

could also do (map (comp #(map hash %) val) (group-by ,,,))

erhardtmundt12:02:53

what is that supposed to do?

sundarj13:02:51

same thing as (map #(map hash v) (vals (group-by :id (concat coll-a coll-b))))

erhardtmundt13:02:03

sorry if I'm a bit slow, but I'm not too experienced

sundarj13:02:36

no need to apologise 🙂 by the way, you might find the #beginners channel more appropriate

erhardtmundt13:02:57

(vals (group-by :id (concat coll-a coll-b)) should return a vector of elements which are in turn a vector and contain either 1 or 2 hashes (it depends if it's shared or not)

sundarj13:02:00

here is what the respective parts return:

sundarj13:02:04

=> (group-by :id [{:id 1 :foo 2} {:id 2 :foo 3}])
{1 [{:id 1, :foo 2}], 2 [{:id 2, :foo 3}]}

=> (vals (group-by :id [{:id 1 :foo 2} {:id 2 :foo 3}]))
([{:id 1, :foo 2}] [{:id 2, :foo 3}])

sundarj13:02:20

vals returns a sequence (not a vector) containing the values of the map returned by group-by

erhardtmundt13:02:26

yeah cool, so the sequence has 2 1-sized vectors inside

erhardtmundt13:02:47

just because they have no shared "record"

erhardtmundt13:02:59

otherwise there would be 2-sized vectors, correct?

sundarj13:02:29

ah right, yes. you are correct

erhardtmundt13:02:23

what puzzles me is the (map #(map hash v) (vals...)) part

erhardtmundt13:02:56

what is v bound to?

sundarj13:02:07

yeah that was a typo, it should be %

erhardtmundt13:02:04

and why do you suggest to hash the elements of the 1-or-2-sized vectors?

sundarj13:02:13

ah i think @U09LZR36F was misguided by the use of the word 'hash'

sundarj13:02:21

since there is a hash function in Clojure

sundarj13:02:16

note that there is a hash-map function too, which is how you programmatically create a map

sundarj13:02:45

=> (hash-map :id 1 :foo 2)
{:foo 2, :id 1}

dominicm13:02:11

@erhardtmundt if you can guarantee there will only ever be 2 items in a group, you could call clojure.data/diff on it instead.

erhardtmundt13:02:11

@U09LZR36F which group are you talking about?

dominicm13:02:34

@erhardt.mundt after you've done group-by

erhardtmundt13:02:34

yeah, it's gonna be 2 vectors, so I guess the maximum size of the groups is 2

dominicm13:02:28

Try the code: (map #(apply clojure.data/diff %) (vals (group-by :id (concat coll-a coll-b)))) 🙂

rauh13:02:44

@erhardtmundt What do you mean with hashes in "I want to compare the hashes I have in each vector..."?

rauh13:02:00

Do you just mean the maps?

rauh13:02:08

Or actually the hash number of the map?

rauh13:02:28

I'm asking since some programming languages use "hashes" to name map-style types

erhardtmundt13:02:38

I come from a ruby world so that's the case

erhardtmundt13:02:41

apologies 🙈

rauh13:02:43

Ok, so that explains that 🙂

rauh13:02:11

So: In clojure we'd call them maps. Never hashes. Hence you got the answer with using hash function. Which is probably not what you wanted

erhardtmundt13:02:04

good thinking, I should've probably called them maps

jarohen13:02:41

(defmacro defthing [name ,,,]
  `(def ~(-> name (with-meta {:arglists '([{:keys [k1 k2]}])}))
     ,,,))

(defthing foo
  ,,,)
fails on the call to defthing with 'can't resolve symbol k1' couple of questions: - (probably more pragmatic) is there a cleaner way to attach arglists to the generated var? - (probably more interesting): why does it work when I change it to ''([{:keys [k1 k2]}])? (as in, two single quotes)

bronsa13:02:21

yeah the compiler expects a double quote there

bronsa13:02:32

and removes it at analysis time

bronsa13:02:33

a bit weird

bronsa13:02:36

but that’s how it works

jarohen13:02:50

my intuition was that it'd only need one because the below works:

(def ^{:arglists '([{:keys [k1 k2]}])} foo
  ,,,)

bronsa13:02:54

this is because metadata is evaluated

bronsa13:02:00

right but that’s not what you’re doing :)

bronsa13:02:16

your quote is nullified by the unquote it’s wrapped in

bronsa13:02:14

macroexpand and prn with *print-meta* set to true

jarohen13:02:45

interestingly, it puts some line/col info in there, but again, not where I expected

jarohen13:02:48

(def ^{:arglists ^{:line 118, :column 42} ([{:keys [k1 k2]}])} foo)

jarohen13:02:06

this from

(binding [*print-meta* true]
  (prn (macroexpand '(defthing foo
                       ,,,))))

bronsa13:02:35

you’ll see that what you’#re emitting is (def ^{:arglists ([{:keys ..}])} ..)

bronsa13:02:58

but what you want is (def ^{:arglists (quote ..)} ..)

jarohen13:02:22

's mind is boggling

jarohen13:02:33

so when is the metadata evaluated?

jarohen13:02:38

the unquote returns a symbol with the metadata quoted - does that mean it's eval'd as part of the def?

bronsa13:02:17

right so you have (ignoring line/col) (def ^{:arglists ([{:keys [k1 k2]}])} foo)

jarohen13:02:43

ah, right, ok

bronsa13:02:52

metadata on vars is evaluated at the same time as when the var init is evaluated

bronsa13:02:14

so that’s not going to work just like (def foo ([{:keys [k1 k2]}) wouldn’t

jarohen13:02:14

ok - think the crucial step I'm missing here is why

`(def ~(-> 'foo (with-meta {:arglists '([{:keys [k1 k2]}])})))
unquotes the metadata?

jarohen13:02:34

oh, hang on

jarohen13:02:41

:thinking_face:

bronsa13:02:53

that quote is only there to prevent evaluating the list as a function call at macroexpansion time

jarohen13:02:05

have called (meta (second ...)) on the result of that, let's see what that does

bronsa13:02:15

but then it needs the quote at runtime

bronsa13:02:27

well, post macroexpansion time

bronsa13:02:08

same as why

`(do ~(do 'a))
is incorrect if you want to return the symbol 'a from a macro

jarohen13:02:08

cheers @bronsa - sorry, that was probably quite painful 🙂

tdantas15:02:30

is it possible, passing a string and getting back the function ? something like

(defn ns->fn [ns] (some-way ns))
((ns->fn "clojure.core/+") 1 2)

sundarj15:02:30

=> (deref (resolve (symbol "map")))
#object[clojure.core$map 0x203dd56b "clojure.core$map@203dd56b"]

tdantas20:02:14

thank you so much

firstclassfunc15:02:24

Guys need some help with some java interop calls I need to provide a byte array as in [B into the constructor but can't seem to figure out the right signature

(ns Response

  (:gen-class
    :implements [some.class]
    :state state
    :init init
    :constructors {[String]   []}))

(defn -init
  [message]
  [[] message])

schmee16:02:01

@firstclassfunc do you need a new class or just a regular byte array?

schmee16:02:34

ohh, nevermind, I guess you’re asking how to provide the byte array as an argument to the gen-class

firstclassfunc16:02:21

@schmee Yea I am trying to extend an existing Java class but probably have to find another way to do this.

schmee16:02:37

@firstclassfunc I think this is the right syntax: :constructors {[String] ["[B"]}

schmee16:02:37

but unless you’re doing something out of the ordinary you often don’t need to use gen-class. if you’re implementing an interface, use reify, if you need a concrete class, use proxy

schmee16:02:58

but I’m not sure how those work with primitive arguments, so maybe gen-class is called for in this case

firstclassfunc16:02:50

@schmee you are awesome that worked! thanks so much.. I am working with a Java library that I want to extend with Clojure so I need to create a new class..

justinlee18:02:31

i’m told it’s a known issue and is being worked on. the logging is apparently still happening but the searchable website isn’t

gonewest81817:02:42

Is there a way to get lein test to tell you what tests have been selected if a selector is in effect, or what :all consists of if no selector is in effect?

noisesmith19:02:59

@gonewest818 you can calculate this is a repl, by filtering all vars from all namespaces based on which ones have a :test metadata, then out of those filtering the ones that also have your selector as a metadata

noisesmith19:02:11

I don’t think there’s anything in leiningen to do this for you though

noisesmith19:02:30

(oh NB you also need to ensure you require all test files first too)

noisesmith19:02:50

user=> (->> (all-ns) (mapcat ns-publics) (filter (comp :deprecated meta val)))
([agent-errors #'clojure.core/agent-errors] [clear-agent-errors #'clojure.core/clear-agent-errors] [add-classpath #'clojure.core/add-classpath] [replicate #'clojure.core/replicate])
(showing deprecated instead of test, since this repl has no tests loaded)

gonewest81819:02:17

thanks for the example!

noisesmith19:02:36

I had no idea agent-errors was deprecated…

noisesmith19:02:10

oh, agent-error instead of agent-errors, haha

xiongtx22:02:34

Is there a neat way to apply some-> to a sequence? Like (some-> foo [f1 f2 f3]). some-> is a macro so apply doesn’t…apply.

noisesmith22:02:48

you could make a macro that does that, but it would require the sequence to be known at macro expansion time

noisesmith22:02:57

you could make a reduce with the same behavior (reduce (fn [prev f] (if (nil? prev) (reduced prev) (f prev)) input fns)

noisesmith22:02:44

@xiongtx

(ins)user=> (defn reduce-some
              [v fs]
              (reduce (fn [prev f]
                        (if (nil? prev)
                          (reduced prev)
                          (f prev)))
                      v
                      fs))
#'user/reduce-some
(ins)user=> (reduce-some 0 [inc inc inc inc])
4
(ins)user=> (reduce-some 2 [* * * *])
2
(ins)user=> (reduce-some 2 [inc :foo inc])
nil

xiongtx22:02:15

Alright, that’s what I figured