This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-21
Channels
- # arachne (1)
- # aws-lambda (50)
- # beginners (10)
- # boot (59)
- # capetown (4)
- # cider (9)
- # cljsjs (27)
- # clojure (249)
- # clojure-berlin (8)
- # clojure-finland (7)
- # clojure-germany (1)
- # clojure-italy (6)
- # clojure-nl (7)
- # clojure-russia (91)
- # clojure-spec (100)
- # clojure-uk (61)
- # clojureremote (2)
- # clojurescript (171)
- # core-async (11)
- # cursive (31)
- # data-science (1)
- # datascript (2)
- # datomic (11)
- # dirac (2)
- # emacs (16)
- # events (1)
- # hoplon (142)
- # juxt (4)
- # lein-figwheel (9)
- # leiningen (10)
- # luminus (7)
- # lumo (44)
- # mount (3)
- # off-topic (150)
- # om (18)
- # onyx (5)
- # perun (12)
- # planck (12)
- # protorepl (13)
- # re-frame (28)
- # reagent (8)
- # ring (1)
- # ring-swagger (10)
- # spacemacs (2)
- # specter (11)
- # sql (14)
- # untangled (99)
- # vim (18)
- # yada (2)
Is there a way to generally control the number of threads that could possibly be created by calls to (future)
etc? I'm running into some OOME's, unable to create new native thread, and the machine I'm stuck running on has a hard ulimit for the number of user processes.
@damionjunk I've never tried this but it looks like you could (set! clojure.lang.Agent/soloExecutor ...)
or whatever is the correct syntax for setting a static field
Ah yes.. I just found these functions:
(set-agent-send-off-executor!
(java.util.concurrent.Executors/newFixedThreadPool 5))
Seems to be getting the job done!
oh hey look at that
It's just:
(defn set-agent-send-off-executor!
"Sets the ExecutorService to be used by send-off"
{:added "1.5"}
[executor]
(set! clojure.lang.Agent/soloExecutor executor))
clojure.core: there's always something else you didn't know was there
Totally! I remember stumbling across as->
and being like WAT!
can anyone explain the behavior of this to me: (clojure.walk/postwalk #(if (vector? %) (println %)) {0 [1 2 3], 1 [4 5 6], 2 [7 8 9]})
? the contents of the vectors are printed as nil and additionally there are three extra vectors containing two nil elements i'm not sure why are there...
nil
is the return value of println
and the extra vectors are probably map entries
the fact that a map entry is a vector is also relevant here
yeah, i get the exact same output if i try: (clojure.walk/postwalk #(doall (if (and (not (map-entry? %)) (vector? %)) (println %))) {0 [1 2 3], 1 [4 5 6], 2 [7 8 9]})
(clojure.walk/postwalk #(if (vector? %) % (type %)) {0 [1 2 3], 1 [4 5 6], 2 [7 8 9]})
<- a clue
@sophiago and that is (I believe) because the intermediate representation of map entries in the context of postwalk is a vector that's not a map entry, just for extra confusion
a two-element vector can pretend to be a map entry sometimes
@gfredericks huh. i tried prewalk as well, but it wasn't outputing anything. how would you recommend getting just the values that are vectors?
if you want a list of vectors at every level I'd use tree-seq
(->> coll (tree-seq coll? seq) (filter vector?))
oh, interesting. the actual implementation is to filter for only vectors of nested vectors (i.e. (vector? (first %))
) and call swap
to mutate them. not sure if i could use tree-seq
like that
call swap on a vector...?
is the atomic hash-map your top-level collection or are there lots of them sprinkled around something else?
but there's just one of them?
okay so you can find that with tree-seq
(->> coll (tree-seq coll? seq) (filter atomic-hash-map?) first)
or just pull it out directly if you know the path to it
and then treat it like a mutable data structure instead of trying to do functional things with it
what's an atomic hash-map?
an atom with a map in it?
okay, I got confused because I thought you were talking about ConcurrentHashMap
in either case, because you have that stateful part you can't do it all as a functional algorithm, you have to transition between the two worlds
postwalk, prewalk, and tree-seq are functional things, so they can only get you halfway there
i don't see how that's the issue. it's easy to mutate parts of an atom using swap
. for example, how i was trying to do this with postwalk: (defn map-map [m f] (swap! m (partial clojure.walk/postwalk (fn [value] (if (and (not (map-entry? value)) (vector? (first value))) (f value) value)))))
the only issue with that is it doesn't seem to be possible to filter postwalk for solely values that are vectors
you just want to call f
on each element at the second level down in the vector of vectors?
if so I'd just call mapv
in a nested way rather than using postwalk
post/pre-walk and tree-seq really fit best with recursive data structures
Oh forgot about the map; map-vals + mapv + mapv then
Where map-vals is the function that's been defined in a million utility libraries but never in clojure.core
It's a one liner
(defn map-vals [f m] (into {} (for [[k v] m] [k (f v)])))
this is not quite right: (defn map-map [m f] (swap! m (for [[k v] m :when (vector? (first v))] [k (f v)])))
you're not passing a function to swap!
is one problem
I was imagining you'd call (swap! m (partial map-vals (fn [vv] (mapv #(mapv f %) vv))))
but it depends on what level f
is supposed to operate at
this works, except not on nested maps: (defn map-map [m f] (swap! m #(for [[k v] % :when (vector? (first v))] [k (f v)])))
these are nonessential functions for this library...might be time to put them aside if there's no elegant solution
@gfredericks thanks for your help anyway! infinitely nested fmap doesn't seem possible, but it was really helpful to have clojure.walk explained a bit
sophiago: sorry if i missed important details, but basically you're trying to recursively apply a function to a map, but only if the first element of the value is a vector?
Yup. That's p much it. Although it's the least important part of what I'm finishing up now so I've pretty much decided to ditch those functions unless you have a flash of insight. I suppose I would like to know for the future, anyway.
doesn't this do what you want?
(defn fmap
[f m]
(into {} (for [[k v] m
:when (vector? (first v))]
[k (f v)])))
(defn fmap-recursive
[f m]
(fmap #(if (map? %)
(fmap-recursive f %)
(f %))
m))
it's not TCO so it will blow the stack if you have a super large input, but i think it does what you want. i'm assuming here that if the first element is not a vector, that you don't even want to include it in the result (at least that's how you wrote your version that you said works)
you know i appreciate the help, but i've already decided to nix these functions and am short on time with this
question
did anyone have experience with setting up drift with reagent-template folder structure / project.clj? Have some issues with generated migrations namespace, would appreciate hints/pointers in how to debug / fix the issue. https://gist.github.com/sudodoki/98b422931a79301ed0f543df29306fb7
hello. I have a function that just inserts averages between values of a list. Itβs pretty simple, but rewritten in python with just going through indices looks much simpler. Is there a way to rewrite this function in clojure to make it a bit more readable?
;; insert-beats-between-beats [0 1 2 3 4 5]) => [0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5]
(defn insert-beats-between-beats [coll]
(let [average #(/ (+ %1 %2) 2)]
(loop [[x & xs] coll
result []]
(if (nil? x)
result
(recur xs (apply conj result
(if xs
[x (average x (first xs))]
[x])))))))
@negaduck itβs usually better to use pre-made recursion schemes, like reduce
than write your own recur:
(reduce (fn [acc [a b]] (concat acc [a (/ (+ a b) 2) b])) [] (partition 2 1 inp))
or, with bindings:
(defn insert-beats-between-beats [coll]
(let [average #(/ (+ %1 %2) 2)
pairs (partition 2 1 coll)
reduce-step (fn [acc [a b]] (concat acc [a (average a b) b]))]
(reduce reduce-step [] pairs)))
(interleave coll (map #(/ (+ %1 %2) 2) coll (drop 1 coll)))
edit: sorry, itβs wrong - loses the last element of a collOkay. Iβm tired of feeling dumb. If you were trying to recreate these lines of java
in clojure
, what would they be?
import com.google.i18n.phonenumbers.PhoneNumberUtil;
import com.google.i18n.phonenumbers.PhoneNumberUtil.PhoneNumberFormat;
private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
phoneUtil.format(number, PhoneNumberFormat.INTERNATIONAL)
(taken from Googleβs libphonenumber demo: https://github.com/googlei18n/libphonenumber/blob/master/java/demo/src/com/google/phonenumbers/PhoneNumberParserServlet.java)
I canβt even get the packages to import.
(ns foo
(:import com.google.i18n.phonenumbers.PhoneNumberUtil
com.google.i18n.phonenumbers.PhoneNumberUtil.PhoneNumberFormat))
should work...?http://stackoverflow.com/questions/7140843/how-to-access-static-inner-java-class-via-clojure-interop
The INTERNATIONAL
part is an enum, which means it getβs turned into a class?
CompilerException java.lang.RuntimeException: No such namespace: PhoneNumberFormat, compiling:(/tmp/form-init562821416926576309.clj:1:1)
@seantempesta maybe you wrote (import ...) instead of (:import ...) ?
@mrchance no, it should be keyword version inside the ns
form
yes, I was suggesting that he might have gotten it wrong, might have been badly worded π
(ns kidlink.specs
(:import (com.google.i18n.phonenumbers PhoneNumberUtil)
(com.google.i18n.phonenumbers.PhoneNumberUtil.PhoneNumberFormat)))
(.format (PhoneNumberUtil/getInstance) "" PhoneNumberFormat/INTERNATIONAL)
Right?
Is :foo.bar/baz
a valid keyword or should we expect problems when we start using them this way?
well, like I said, if the NumberFormat is a nested class, you need ...Util$PhoneNumberFormat
According to this spec keywords should not contain dots: https://clojure.org/reference/reader
Sorry, Iβm still not getting it. (.format (PhoneNumberUtil/getInstance) "
also gives an No such namespace: PhoneNumberUtil$PhoneNumberFormat
error.
@seantempesta You have to change the import line too
@borkdude No, I think they mean the non-namespace part of the keyword can't contain dots
@seantempesta Maybe you are better off using https://github.com/vlobanov/libphonenumber altogether? π
Unless the goal is to learn about interfacing with all the kinks and corners of Java π
Oh hell yeah. Iβve wasted 2 hours on this. I just didnβt think java interop could be that hard. But then again, I donβt really know java...
Yeah, it's a lot easier with well designed libraries, but several things like needing to implement an actual subclass, accessing nested classes and the need for singletons require some more acrobatics
Incidentally, this also shows the right way to do it, in case you are wondering π
@negaduck nailed it
(defn interleave-all [s & seqs]
(lazy-seq
(if-let [s (seq s)]
(cons (first s) (apply interleave-all (concat seqs [(rest s)])))
(when seqs (apply interleave-all seqs)))))
(let [average #(/ (+ %1 %2) 2)
coll (range 5)]
(interleave-all coll (map average coll (drop 1 coll))))
btw, there is partition-all
in core (regular version failed me once)
@residentsummer, thank you. Isnβt it easier to add the last element to the one liner above?
well, itβs easier, but 1. looks suspicious 2. appending to an end is expensive for many collections (except for e.g. vectors)
and 3. you can reuse interleave-all
laterβ¦ sometimeβ¦ let it just hang on in (ns utilβ¦
for a while π
@negaduck Just gist it, I frequently search github+gists first when i look for a function. (
+
)
@negaduck how about
(rest (into []
(fn [rf]
(let [p (atom 0)]
(fn
([] [])
([acc] acc)
([acc v]
(let [p0 @p]
(reset! p v)
(rf acc (/ (+ p0 v) 2))
(rf acc v)
)))))
(range 6)))
=> (0 1/2 1 3/2 2 5/2 3 7/2 4 9/2 5)
wrap it in triple backticks
@benha I like it, probably much faster, though I'd do it like this:
(defn compute-between
[f]
(fn [rf]
(let [rf ((drop 1) rf)
p (volatile! 0)]
(completing
(fn [acc v]
(let [p0 @p]
(vreset! p v)
(rf (rf acc (f p0 v)) v)))))))
(into [] (compute-between #(/ %2 2)) (range 6))
@benha Btw, you can't bash the rf
into place. Your code is lucky to work since conj!
on vectors works while bashing.
ah silly me; You mean
(-> acc
(rf (/ (+ p0 v) 2))
(rf v))
instead of
(rf acc (/ (+ p0 v) 2))
(rf acc v)
x = () OR x = (1 2 3)
I want to pattern match x
via core.match
into either
()
or (h & rst)
so I try:
(match [x]
[()] :foo
[(h & rst)] :bar
)
^^ apparently the above is wrong; how do I fix it?@seantempesta you were likely having issues due to inner classes. java interop is very simple, just have to do it once or twice and it becomes very straightforward. first, you import. I searched google for the class you mentioned, found the maven info, and put that in dependencies in project.clj:
[com.googlecode.libphonenumber/libphonenumber β8.3.0β]
Next, import the classes you need:
(ns your.namespace
(:import [com.google.i18n.phonenumbers
PhoneNumberUtil
PhoneNumberUtil$PhoneNumberFormat
Phonenumber$PhoneNumber]))
next, just use standard java interop:
(def number (.setNationalNumber (Phonenumber$PhoneNumber.) 5553331212))
(.format (PhoneNumberUtil/getInstance)
number
PhoneNumberUtil$PhoneNumberFormat/INTERNATIONAL)
Is someone with mathematic knowledge here?
Circle calculation, radians and such stuff
which ring middleware should I use, so that my javascripts would not be cached by the browser (chrome)?
@danielgrosse : what's youre question?
confucius say, better to ask question and receive no answer, than to ask question about asking question, receive question about question, answer question about question, receive answer, then leave question unasked π
@joshjones I see. I'll give it a shot. Thanks for digging into it.
@seantempesta no worries mate β at first i hated java interop because i could not remember when a βdotβ went in front, at the end, when to use slashes, etc., itβs frustrating when you just want to get the darn thing to work.
hi. any pointers here? trying ^^ to use clojure.inspector. am I perhaps doing som n00b mistake here, something i haven't installed for instance or what do you think?
Hi Is it possible (and how) to use if-let inside -> macro?
@kirill.salykin you need to be more specific, but the answer is probably 'no'
I was thinking about something like this:
`
(-> params
validate
(if-let [user (find-user)]
β¦
...
))
but I guess there it is no way to pass args to find-user
this might work
thanks!
Hey, does anyone have experience with using the new ring async handlers with compojure?
@kirill.salykin While some->
is probably what you want there, it may be helpful to know that you can use common macros such as if-let
, if
, and
in a threading macro chain, but you have to wrap it in a function first. for example:
(-> params
validate
(#(if-let [user (find-user %)]
(...))))
an embedded as->
can also do that sort of thing
@joshjones thanks!
One of my teammates is looking for a utility that can be used in a testing context to stub specific function invocations (based on supplied parameters) and either conditionally dispatch to the original function or return a value appropriate to the testing scenario. I believe that Midje offers something close to this, but the need is for application in clojure.test
tests. It feels to me like something that core.match
could be part of, but I wanted to ask on this list in case a utility already exists that I'm unaware of.
where I used to work we had this macro with-var-roots, which pre-dated with-redefs, and had a lot of overlapping functionality
one feature that with-var-roots had that with-redefs did not is, you could do something like (with-var-roots [^{:o original} some-function (fn [& args] ...)] ...)
and it would bind the original value of the some-function var to original in the scope of the value
you can do the same thing with with-redefs, (with-redefs [some-function (let [original some-function] (fn [& args] ...))] ...)
but that is so verbose and it is such a common thing
@marioaquino - stubadub removes some of the pain of using with-redefs
and adds a couple small features. You'd have to add the conditional dispatch logic yourself, but that'd get you halfway there maybe: https://github.com/magnars/stubadub
@marioaquino another option is to not stub (or mock) functions. IMO this tests the wrong thing. Most of the time you don't really care if a function was called with certain args or not...you really care about side effects.
Think of it this way: if the function is pure, test the function or test the output of the calling function. If that doesn't allow you to test what you want to test, then you're trying to assert something about side-effects. In that case you should try to test the effect of the function.
So for example: write a record to the DB then assert that the record is there. Write a IMessageQueue protocol, then use that via your DI system of choice to replace the actual queue.
All this with-redefs stuff is a fantastic way to introduce some really hard to find bugs.
cough breaks in multithreading cough
Thanks for the feedback and recommendations, @hiredman, @jeff.terrell, and @tbaldridge!!
e.g. if you have a test that spins up some multithreaded system, and does stuff, and then tears down the multithread system, and wrap the whole thing in with-redefs, then everything will see the redefs, which may or may not be broken depending on your intent
I don't like systems that "may or may not be broken" π
everything may or may not be broken depending on your intent, so it is a bound I am comfortable with
I just hate the idea of taking a language like Clojure that's designed to handle concurrency, and then ruining it by introducing global mutability. IMO we should be building code that could work just as well if clojure.test ran tests in parallel.
@tbaldridge - I agree in general about avoiding stubbing, but sometimes you only want to test up to a certain boundary. I was recently testing an implementation of an OAuth client, and I wanted to be sure it was sending the requests I expected using clj-http/get
etc. I didn't want to actually make network requests during my tests. Seems like stubbing is the right approach there, no? I'm definitely open if there's a better wayβthat code was not fun to write. :-)
so in that respect anything that introduces global mutability is straight out for the code I write.
assuming the URL for your requests is a parameter to the function (as it should be) you could also pass in a HTTP client instance. Or use some sort of functional DI system, like Component, or Integrant.
Or just test the code that constructs the request. If you could have a test against (construct-request "user-name") => {:url ... :params ...}
That's really what you're trying to test, right? That a call to your business logic resulted in the proper construction of the request structure.
Yeah, that's true. Good point. I kinda don't want to introduce an "http client" abstraction, but weighing that against the pain of stubbing, stubbing is probably worse. And separating request generation from request sending is even better. Thanks! Will be thinking about this...
I very much like component, and I am using it with new stuff, and haven't missed with-var-root or with-redefs too much
@jeff.terrell in addition to what @tbaldridge said, you can also check out mount: and use mount/start-with
to swap an implementation: https://www.dotkam.com/2016/01/17/swapping-alternate-implementations-with-mount/ during testing.
I have written and revised a bunch of clojure projects by this point and without question, the ones that have been easiest to modify have been those using component.
@danboykis does that work with multithreading?
Nope! It stores the state in the global atom. How is this better than with-redefs?
@tbaldridge this question needs more context. can application states be used by multiple threads? absolutely. can you run tests in the exact same REPL you develop? most likely, depending on your app, no. but I prefer not to do that. And I actually like that I am forced to run tests within a different REPL (with a watcher for example): it is much cleaner.
Mount is just like with-redefs with another layer of indirection. If one thread is starting/stopping the system it will mess with other threads trying to do the same.
speaking of multithreaded tests https://github.com/weavejester/eftest
nice!
Fun fact, if you test your code multithreaded and it works, it does not mean it works all the time...
@sveri that's when I hook it up to test.check and run stuff randomly in parallel until I find what two tests conflict π
In practice though if this sort of stuff isn't threadsafe it's going to blow up pretty quickly.
I agree, usually it does that, oh man, when I look at my work and see how much code, that can be run multithreaded is actually never tested multithreaded...
@tbaldridge
> Mount is just like with-redefs with another layer of indirection
this is you opinion, not a fact. I created at least 3 large apps + several small ones with Mount for the last year and a half. All of them are highly concurrent.
I tested them without problems, some with lein, some with boot (i.e. in a different REPL: boot watch speak test
). I used Component before that. It's a framework: "it calls me", I don't like that in Clojure, which is also an opinion.
@danboykis no, it's not my opinion, it's a fact. Mount stores its state in a private global atom. A call to mount state does a (:state @global-atom)
. with-redefs
is exactly the same. All vars are stored in a global atom, when you with-redef
a var you mutate the root of the atom that is inside the global atom (the Namespace).
I didn't make a judgement call on Mount. I said with-redefs mutates global data, which I find sub-par. And I stated that Mount does the same thing.
@tbaldridge does this stance on global state extend to the implementation of MultiFn too?
asking because I seem to remember doing some nasty reflecting to be able to clone MultiFns for the purposes of testing
@bja what was the cloning for?
@thheller use Quicktime. Go to File->New Screen Recording. Nice thing is, it comes with OSX
I use Screenflow: http://www.telestream.net/screenflow/overview.htm
rgdelato: i'll add a plug for screenflow. amazing caps. not free, but at $99 pretty close.
@tbaldridge wow that was indeed simple, thanks π
anyone know the best way to merge vectors and preserver order?
Have an example of what you're trying to get to?
hello, any good suggestion how to deal with defs re-used on different namespaces, e.g. same db connection for source code and test code?
@ejelome ya make them ENV vars
thanks @josh_tackett and @jr, will look into these, esp. the envs π
@josh_tackett "merge vectors" .... given [1 2 3]
and [4 5 6]
, what does a merge look like?
is there any way within a REPL to get the documentation for a reader macro? For example, I can (doc ->>)
, but not (doc ^)
there is some special support for special forms, but nothing like that for reader macros
doing that would be hard as the reader interprets them before you could invoke doc on it
But it just kicks you over to the right heading on http://clojure.org/reference/reader
Does anyone have a suggestion for a clojure probability distribution library? Something similar to numpy or http://math.net would be great. I need to sample poisson, lognormal, etc distributions.
@genec: there's hy + https://github.com/tobgu/pyrsistent if you want to stick with numpy
Incanter seems to be well-regarded http://incanter.github.io/incanter/index.html
@qqq @donaldball @alexmiller Thanks very much for the links. I've played around with incanter, haven't heard of watershed - but it looks like it's just what I need. Not sure I understand how to use pyrsistent from clojure though.