Fork me on GitHub
#clojure
<
2018-01-20
>
kaosko03:01:14

so a pretty simple problem here... I have a nested structure from which I need to produce some sequential data, however I also need to derive some accumulated information from the outer "loop" for the data. I can solve the problem with a loop, reduce or even with an atom but none feel very clojury. should I just go with a loop and two arguments and an inner for, or is there some other function that captures these aspects? something like a list comprehension with loop bindings

noisesmith03:01:08

if you need updated bindings as you consume a nested structure, you could consider (reduce f {} (tree-seq coll? seq input)) where f can update whichever keys of the hash-map you might need to represent a composite state

noisesmith03:01:51

that's the most general - it could be you could have a simpler value for an accumulator instead of a hash map, or a simpler form like (apply concat ...) instead of tree-seq depending on your task

noisesmith03:01:21

but if you are consuming something from start to end, reduce beats loop because it avoids the boiler plate of navigation

noisesmith03:01:15

for works for consuming one collection in order and generating a lazy collection from it, or working on all combinations of items from N inputs, but it's not a loop

noisesmith03:01:14

I wouldn't advise an atom if you aren't coordinating mutable states from multiple threads, and it doesn't sound like that's called for here at all

kaosko04:01:44

thanks @noisesmith. I love tree-seq! just couldn't find any mention of it when I was googling for nested structures

noisesmith04:01:47

it's kind of clumsy but it's great if you don't have control of a weirdly shaped data structure as long as you can make a predicate that detects and filters the leaves or subtrees you need

noisesmith04:01:24

obligatory: even better is producing the best data to work with in the first place (if that's under your control)

kaosko04:01:07

sure, but seems much higher-order than the loop. the data is fully in my control and works for other things. I could easily solve the problem by producing cached values beforehand but trying not to

kaosko05:01:51

in the end, I went with reduce though. tree-seq is great but I need to produce the info during walking of the tree, not only at leafs, and at the end, I'm producing sequences of sequences no matter what, so might just as well apply concat the final result

qqq06:01:19

is there a better way t owrite a clojure test that says: "this code does not throw an exception?" the best I have s ofar is: (ct/is (do ... true))

cddr12:01:32

Whenever I want to write a test like this, I stop to think "can I assert something that did happen instead?"

qqq13:01:08

no; I am literally testing for "no assertion was thrown"

qqq13:01:21

like I want to test (assert (= 2 2)) does not throw an assertion

seancorfield06:01:07

@qqq Using Expectations, I'd probably say (expect any? (some-code))

seancorfield06:01:38

So with clojure.test I guess that would be (ct/is (any? (some-code)))...

andy.fingerhut06:01:55

If you wanted to make it kinda self-documenting, you could do something like (ct/is (:no-exception (try ... :no-exception (catch Exception e :exception))))

cddr12:01:32

Whenever I want to write a test like this, I stop to think "can I assert something that did happen instead?"

lxsameer13:01:42

hey folks, how can i change the ::start-fn spec to validate the return value of the function against ::component spec. http://dpaste.com/09FZ438 (line 6)

lxsameer13:01:17

::start-fn is a part of ::component spec, So it's like a recursive spec. The above code isn't working, basically spec can't exercise the ::start-fn

bitti16:01:15

That is strange, when I call twitter.api.restful/account-verify-credentials during the course of my program, it won't exit even when I put (shutdown-agents) at the end

bitti16:01:52

I would like to avoid calling (System/exit 0) in order to make it still runnable from the repl without killing it

noisesmith18:01:34

calling shutdown-agents is just as bad as (System/exit 0) for a long running process - it will make any usage of the agent thread pools fail

bitti16:01:27

is there some other threading system which needs to be shut down explicitly?

bitti16:01:28

it doesn't halt at (shutdown-agents) or even (await) btw, so there don't seem to be any agents running

Jan Drake16:01:35

Anybody using clojure for high speed analytics?

bitti16:01:40

anyone knows how I can interpret this from the thread dump?

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f173c14a800 nid=0x106a in Object.wait() [0x00007f17126a9000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000006c70422b0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x00000006c70422b0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

bitti16:01:50

does that mean the thread is waiting on its own lock?

jgh17:01:07

not super familiar with how java mutexes work, but sure looks like it’s deadlocked.

andrea.crotti17:01:51

I think I haven't fully grasped how logging works in Clojure

andrea.crotti17:01:35

I'm trying to use Timbre now with a ring server, and I add this simple config

(defn -main [& args]
  (tm/merge-config!
   {:level :info
    :ns-blacklist ["org.eclipse.jetty.*"]})

andrea.crotti17:01:57

but it doesn't seem to care at all and I still see all these logs

17:31:44.964 [qtp1581040670-27] DEBUG org.eclipse.jetty.io.AbstractEndPoint - onClose SelectChannelEndPoint@58ca7ef1{/0:0:0:0:0:0:0:1:57244<->3001,CLOSED,ISHUT,OSHUT,-,-,27903/200000,HttpConnection}{io=0,kio=-1,kro=-1}
`

andrea.crotti17:01:25

however if I log directly with timbre functions they get filtered out correctly

andrea.crotti17:01:41

so maybe I need one of the various adapters to make the two logging systems talk together?

schmee08:01:16

I think you need to add something like this: https://github.com/fzakaria/slf4j-timbre

schmee08:01:53

Timbre doesn’t work out of the box with Java logging libs

andrea.crotti11:01:28

@U3L6TFEJF uhm I added all the dependencies declared there but still see all the DEBUG logs

andrea.crotti11:01:44

I guess I need a bit more initialization but can't see it documented anywhere, any hint?

schmee14:01:24

that’s all I’ve had to do when I’ve dealt with it

schmee14:01:46

just watch out that you don’t have any messages like this when you start a REPL:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/<foo>/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/<foo>/.m2/repository/com/fzakaria/slf4j-timbre/0.3.4/slf4j-timbre-0.3.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See  for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

schmee14:01:23

if you do, you might need to use :exclusions in your project.clj to remove the conflicting dependencies

damien19:01:45

Does anyone have experience or know of prior art using core.logic for probabilistic logic (e.g. markov logic networks)?

damien20:01:46

Very interesting, thanks @alexmiller 👍

misha23:01:10

is there a way to write a function/macro, which will qualify arg's symbol in an application ns and return a qualified keyword?

(ns user.utils)
(defn ->kw [x])

(ns user.foo
  (:require
   [[user.utils :as u]
    [clojure.string :as str]]))
(defn f []
  (u/->kw str/join))

(ns user.bar
  (:require [user.foo :as foo])
  (foo/f)) ;; => :clojure.string/join

misha23:01:10

I got it working for args which are functions, but another use case is a map. The global goal here is to avoid hardcoding fn names in Onyx jobs, which is prone to refactoring errors

misha23:01:52

All is good for task descriptions (where every interesting keyword is a function name), but for lifecycles, e.g. for :read-from-kafka task, you need to pass :onyx.kafka.plugin/write-messages-calls, which is a hashmap harold

misha23:01:12

I have a macro, able to do that, but it relies on application-site ns passed in as an arg (not even a *ns*), so this gets ugly af from the get go

misha23:01:13

the solution for functions is:

(defn fn->kw [f]
  {:pre [(fn? f)]}
  (->> (-> f               ;; str/ends-with?
         str               ;; "clojure.string$ends_with_QMARK_@65576657"
         repl/demunge      ;; "clojure.string/ends-with?@65576657"
         (str/split #"@")) ;; ["clojure.string/ends-with?" "65576657"]
    (butlast)              ;; ["clojure.string/ends-with?"]
    (str/join "@")         ;; "clojure.string/ends-with?"
    keyword))              ;; :clojure.string/ends-with?

noisesmith23:01:27

why would a macro need the application site value of *ns* to be passed in?

noisesmith23:01:41

the value of ns from inside a macro is always the value of *ns* as bound in the application site

misha23:01:05

there are 3 distinct places: ns with defmacro ns where macro is applied ns where function-inside-which-macro-is-applied is called

noisesmith23:01:57

and you need the last one?

misha23:01:48

the way I wrote macro 1st time, it resolved passed using aliases map in the *ns*, so the aliases map were not from the second place, but from 3rd

misha23:01:48

yes, I thought I got it, until I found out, that unqualified args get resolved not as "ns.2/arg", but as "ns.3/arg"

misha23:01:55

implementation is

(defn -impl [ns sym]
  (let [aliases (ns-aliases ns)
        ns* (or (some-> sym namespace symbol aliases)
              ns)]
    (keyword
      (.getName ^clojure.lang.Namespace ns*)
      (name sym))))

(defmacro ->kw [ns sym]
  `(-impl ~ns '~sym))

misha23:01:35

this works, I think, but requires

(def ns *ns*)
(defn f []
  (->kw ns x))
in the second (macro application site) ns

misha23:01:21

is there some introspection "trick" to see which ns fn or macro is applied in?

misha23:01:59

oh, I just comprehended what you said about *ns* and macro application side.

misha23:01:24

I just need to pass *ns* in defmacro to -impl

noisesmith23:01:30

the trick is making it see any ns other than the one which is using the macro

misha23:01:37

(defmacro ->kw [sym]
  `(-impl ~*ns* '~sym))

noisesmith23:01:52

that should work

misha23:01:15

it does 😄

noisesmith23:01:45

it looks weird, but you can replace that macro body to one that doesn’t use syntax-quote - (-impl *ns* sym) (edit - that first ns/ wasn’t needed because you aren’t trying to emit impl)

noisesmith23:01:35

oh wait, never mind, you do want to emit a call to impl don’t you

misha23:01:09

yeah, everything is a :onyx.kafka.plugin/sym now opieop

misha23:01:52

instead of :onyx.kafka.plugin/read-messages

noisesmith23:01:41

oh - I edited to get rid of the ' which shouldn’t be there

misha23:01:48

(-impl *ns* sym) works. thank you

misha23:01:46

(I did not reload things in repl properly first time)

misha23:01:06

this means I can get rid of impl, and just put everything in macro, doesn't it?

noisesmith23:01:04

if that clarifies things, I think it would make sense - I mean, it kind of looks like you can just replace defn -impl with defmacro ->kw in the form and that’s just the thing

noisesmith23:01:23

though having the logic in a defn does improve things like being able to unit test