This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-06-07
Channels
- # aleph (19)
- # aws (1)
- # beginners (75)
- # boot (28)
- # cider (1)
- # cljs-dev (12)
- # cljsrn (20)
- # clojure (350)
- # clojure-argentina (1)
- # clojure-chicago (2)
- # clojure-dev (2)
- # clojure-russia (5)
- # clojure-spec (2)
- # clojure-uk (14)
- # clojure-ukraine (3)
- # clojurescript (68)
- # component (87)
- # core-async (25)
- # core-logic (13)
- # cursive (4)
- # data-science (72)
- # datascript (59)
- # datomic (15)
- # defnpodcast (7)
- # emacs (33)
- # hoplon (5)
- # immutant (73)
- # jobs (21)
- # klipse (6)
- # lumo (14)
- # off-topic (26)
- # om (23)
- # onyx (6)
- # parinfer (37)
- # protorepl (4)
- # re-frame (13)
- # ring (2)
- # rum (3)
- # spacemacs (2)
- # specter (22)
- # sql (47)
- # uncomplicate (10)
- # unrepl (79)
- # untangled (66)
- # vim (47)
- # yada (17)
Hello all, if I use doall
with pmap
to threading
some functions, if any of thread fails, fails the result for doall?
you will get an exception from doall, but pmap may have processed some items "beyond" that point
Is it that bad to do something like this?
(defn function-long-run [ip conc message]
(try
(swap! message assoc (keyword ip) (icmp ip))
(catch Exception e (swap! message assoc (keyword ip) :fail))
(finally (swap! conc inc))))
(let [ conc (atom 0)
output (atom {})
ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]
limit (count ips)]
(pmap #(function-long-run % conc output) ips)
(while (< @conc limit)
(do (Thread/sleep 2000)))
(println @output))
I'd expect a lot of retries on that atom
and pmap only uses a limited number of threads, it's not optimized for io bound tasks
you eliminate the retries if you return [ip result] from each function call, and conj them all into a map
there's nothing here that requires an atom
pmap returns a specific number of results, when they all come back, you are done
any other alternative will offer something similar easily
also, pmap is lazy, so not consuming results guarantees that you lock up
isn't the try catch sufficient for the icmp failure issue?
but in that code you are not using doall, it will lock up
unless the number of items handed to pmap is small enough that there's no backlog
but the pmap results don't happen because you aren't using pmap's return value
it's lazy
the result won't happen because pmap is lazy
you don't even understand what this code is doing
it's not even a smell, it's an error
because nobody looked at pmap's return value
+user=> (do (pmap (fn [_] (swap! done inc)) (range 100000)) nil)
nil
+user=> @done
32
fixed
so 32 items out of 100k get done
illustrating what I was saying about retries:
+user=> (do (pmap (fn [_] (swap! done (fn [x] (print (str \space x \space)) (inc x)))) (range 100000)) (println))
0 0 0 0 1 1 1 2 2 3 3 4 4 4 4 4 5 5 5 5 6 6 6 7 7 7 8 8 8 9 9 10 11 12 13 13 13 14 14 14 15 14 15 13 16 16 16 17 17 17 18 18 18 19 19 20 20 21 21 22 22 22 22 22 23 23 23 23 24 24 24 0 25 25 25 25 26 26 26 27 27 27 28 28 28 29 29 30 31
nil
notice how many times it needs to run to get 32 results
the problem goes away, and the code becomes simpler when you stop using an atom, and just put the return values from pmap into a hash map directly
if try/catch isn't fixing the errors you were seeing before, you weren't using try/catch properly
so this
(defn function-long-run [ip conc message]
(try
{(keyword ip) (icmp ip)}
(catch Exception e {(keyword ip) :fail})))
(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
(println (pmap #(function-long-run %) ips)))
will be nice?#(function-long-run %)
is just a long way of writing function-long-run
then
I assume your list of IP addresses would be large? Much bigger than 32 elements?
Except you need to (map deref ...)
over that... and it's chunked, so beware.
hmm, maybe a (into [] (comp (map #(future (function-long-run %))) (map #(deref %))) ips)
will it really? hmm. Did you try it? Maybe the eagerness in the pipeline is running the first deref before the second future...
Well, transducers nicely translate into thrushes. Does this work? (into [] (->> (map #(future (function-long-run %)) (map #(deref %))) ips)
Yeah, that'll spawn thousands of futures, I believe. Futures use clojure's agent send-off pool. Not sure if there's any rate limiting or maximums on that pool.
As @seancorfield kindly pointed out, a future sorta wraps threads in an atom-like derefable interface
I can either return a pending status (I think) or the final value of the thread's return value. And thereafter always returns that value on deref.
@f
will block until the future
is completed.
pmap
is a very blunt instrument and is nearly always the wrong solution -- unless you really want something quick'n'dirty.
Using map
like this for side-effecting functions has some... subtleties... that you really ought to avoid.
If a future
expression takes "forever" to execute, then a deref
on it will block "forever".
@fabrao I missed the beginning of this thread (yes, I know I could scroll back) but are you just trying to do something throwaway or is this meant to be "production" quality?
It probably isn't a good idea to fire off 1000 parallel ping requests under any circumstances.
If you want timeouts, use core.async.
In fact, if you want any real level of control over this, use core.async.
production @seancorfield
Then don't use pmap
and don't use map future
.
See above
You should be rate limiting your pings. Create batches of a certain size, send them out in parallel. It's not good etiquette to DOS networks with ICMP packets.
There's a #core-async channel if that helps.
But, yeah, it's definitely got a learning curve.
right, so you might be okay with rate limiting a stream of futures... but yeah, once you get into fine grained concurrency management, there's core.async. If you're pinging thousands of hosts with super long response times and other long term / session oriented connections for monitoring, etc, managing those using core.async can save you from a lot of wasted thread management.
Clojure makes concurrency a lot easier than with many other languages but this sort of stuff is never trivial.
FWIW, when we were starting out with Clojure, I wrote a process that scanned our database and ran searches against a custom search engine and then generated and sent HTML emails. When we first powered it up, it could do about 250K emails a day. A pmap
later, it brought down our search engine... ahem
So I tweaked things and expanded our search engine cluster and got it up to 750K a day.
Eventually I got it up to 3M a day but pmap
was not the right solution at any level.
Moral: be very careful with pmap
and just doing everything in a future
...
I´m learning now that core.logic to replace somethig like http://www.clara-rules.org/
I want to perform a operation on a series of values, each operation feeding the return value as a argument to the next. Kind of like The threading macros, but i must work on a sequence. Is there a core function like this?
@fabrao Clojure changes the way you work -- we started using it for "small" stuff back in 2011 and it became our primary language in 2015 and we've been rolling out all sorts of pure Clojure apps in the last year.
We're at close to 60K lines of Clojure these days...
Nice!
hum, having interesting classpath problem:
dev=> (require ‘[compojure.core :as c])
CompilerException java.lang.RuntimeException: No such var: response/send, compiling:(compojure/core.clj:160:6)
ps auxf | grep java
shows that compojure 1.6.0 is on the classpath, as requested in :dependencies
by
[ring “1.6.1”]
[compojure “1.6.0" :exclusions [commons-codec
medley]]
viesti: Use lein deps :true
to see if unexpected older versions of libraries are coming in.
Er, :tree
sorry, typing with one hand -- beer in the other
10:30pm for me 🙂
fabrao: More likely to be a classpath / version issue -- nothing to do with :refer
I need to read a file that contains psql queries in clojure with
and I am facing issues with \COPY
\COPY
is a postgres command but it confuses the reader
I probably need to escape the backslash. But how?
@viebel what are the contents of the file?
pretty should that io/reader is agnostic vis-a-vis file contents
my guess is the issue lies elsewhere?
\COPY commodity_details_tmp FROM 'path/to/aa.csv' DELIMITER ',' CSV;
you should be able to slurp that file (or the reader). You could try that and print the output
Reading https://github.com/juxt/bidi#wrapmiddleware and don’t fully understand what it suggest instead of ->WrapMiddleware
. Do they mean writing a postwalk function which will replace handlers to fns with applyed middleware to the handler? or something else?
@razum2um can't you just use replace your handler
with (wrap-params handler)
?
@razum2um the postwalk refers to a situation like:
[["/"
["admin/" [["index" handler-A] ["logout" handler-B]]
["" homepage]]]
If you wanted to add middleware to all of the admin/*
routes, you should use a postwalk. Does this answer your question?so that would be something like this
["admin/" (postwalk #(if (fn? %) (wrap-params %) %) [["index" handler-A] ["logout" handler-B]])]
?@pesterhazy yeah, exactly something like that.
that's not too bad
the fact that bidi uses simple data structures enables so many hacks
yeah when all you have is data, the line between hack and feature is blurred
data enables you to make decisions for yourself. You're not dependent on a library author to say "Okay, I've thought about it. Having middleware apply to sections of routes is okay, you can do that now".
what's a feature if not a hack that you're stuck with?
A really unique feature is route-seq
, you could build a site-map from that for example.
data also enables you to pretty much immediately estimate how hard implementing something would be: "oh, that's just a postwalk away"
postwalk is something we use a lot. Yada + Bidi is a pretty powerful combo in this way.
those four letter libs are growing on me, bit by bit
true quip, mate
what is the idiomatic way for mapping [{:id 1, :value 10}, {:id 2 :value 20}, {:id 1 :value 15}]
=> {:id 1, :values [10, 15], id: 2 :values [20]}
.
I can envision doing this with a recursive function starting with an empty map, but I would assume there is a much more elegant way of doing this in clojure. Any tips would be much appreciated.
@graemenelson maybe merge-with
@graemenelson that's not a valid map
@graemenelson - usually reduce
would be preferred to a recursive function. Although it's possible there's something more elegant you could do here.
You are right @bronsa — sorry haven’t had my coffee yet. I meant output to [{:id 1 :values [10 15]}, {id: 2 :values [20]}]
https://github.com/clojure/tools.nrepl/blob/master/src/main/clojure/clojure/tools/nrepl.clj#L115-L135 might be worth reading, but I'm not so sure now
I was looking into to reduce
and into
, maybe I am just not groking it yet. I will keep looking
I might try something like this:
(letfn [(join-values [maps]
{:id (-> maps first :id)
:values (map :value maps)})]
(->> seq-of-id-val-maps
(group-by :id)
(join-values)))
Not sure that that would actually work, but something like that would.Thanks
@graemenelson You might want to start by doing a (group-by :id)
, that should make it easier to merge values of the same kind together. Then you can reduce over each value in the map using a function similar to the one I linked above 🙂
hi all, is it possible to create my own webapp with ssl support (https) without a signed certificate by any entity such as Godaddy? I just need to show a proof that it is possible to use ssl with clojure!!!
@alricu it's possible to use ssl with clojure. https://github.com/SevereOverfl0w/aleph-yada-ssl I made this the other day
it has self-signed certs in already, so you can use those, or generate your own using the instructions I documented
Thanks @dominicm I will take a look
Thanks group-by
might work
with reduce-kv
graemenelson:
(def data [{:id 1, :value 10}, {:id 2 :value 20}, {:id 1 :value 15}])
(map (fn [[_ v]]
(reduce (fn [m [k v]]
(case k
:id (assoc m k v)
:value (update m k conj v)))
{}
(apply concat v)))
(group-by :id data))
Thanks for the help!
Thanks @dominicm !!! Is it possible without using yada and aleph?? problem is that I only need to use http-kit and jetty
@alricu It is possible, screw SSL on the clojure side and upload the cert to your cloud-hosted loadbalancer or if you don't use LBs, make your nginx use it.
No one I know actually bothers with SSL on the app side.
yep, I have seen those options, my problem is that I have a lot of restrictions with regards to the libraries I can use
https://github.com/http-kit/http-kit/issues/32 It doesn't look like http-kit supports https in the server.
Hello @john I observe that the (into [] (->> ips (map #(future (function-long-run %)) (map #(deref %))))
serialize when function-long-run is doing network communication, how can it be?
@not-raspberry sometimes not possible, sometimes all data in transmission must be encrypted for example
@fabrao I'd suggest putting up an example of function-long-run
so some folks can test out what you're working with.
(defn- disable-certificate []
(doto
(java.util.Properties.)
(.put "StrictHostKeyChecking" "no")))
(defn- session [host port user password]
(doto
(.getSession
(com.jcraft.jsch.JSch.) user host port)
(.setPassword password)
(.setConfig (disable-certificate))
(.connect)))
(defn available? [ip port username password & {:keys [timeout] :or {timeout 60000}}]
(try
(let [sess (session ip port username password)]
(try
(catch Exception e (throw e))
(finally
(.disconnect sess))))
(catch Exception e (throw e))))
(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
(into [] (->> ips
(map #(future (available? % 22 "someuser" "somepassword"))) (map #(deref %)))))
You're launching the future immediately before dereffing it: (map (deref (future blah...)) collection)
if you want truer concurrency you could launch all the futures first & eagerly. Then in a separate step dereference the future
(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
(into [] (->> ips
(mapv #(future (available? % 22 "someuser" "somepassword"))) (mapv #(deref %)))))
?You could still use pmap for this. Just know that once your last available thread is waiting on your icmp response, and all others are hanging, all other resources that leverage that pool will also wait for those icmp responses to return. So all those extra cores are going to waste waiting on IO.
But, if you weren't using thread pools anywhere else, you could use pmap's queue to throttle your thread creation.
Hey wise people... offhand, does anyone know offhand which types one has to extend to have a protocol work on all maps? Poking around, I seem to be able to turn up at least clojure.lang.PersistentTreeMap, clojure.lang.PersistentArrayMap, clojure.lang.PersistentHashMap --- I'm thinking I might have to extend them all...?
As to your question, why mapv worked instead of map, I guess because map is lazy, the (map #(deref...
isn't pulling the next invocation of the upstream map function until its own deref operation completes.
@gowder check out https://github.com/ztellman/potemkin/blob/master/src/potemkin/PersistentMapProxy.java
@tsulej @fabrao yeah, you no longer need the (into []
as that was only necessary in the previous transducer example.
not saying to use it but he lists all the interfaces you need to implement and then he does as much as he can and tells you which ones are left to you
also mark engleberg wrote a priority map https://github.com/clojure/data.priority-map and you can see all that he did to make it "mappy"
my understanding is that defrecord makes a class but if you dissoc from it you lose the custom ness and it reverts back to a standard map
and you wouldn't be able to add special behavior, which would be the reason to make a special map in the first place
but i cut off @ghadi who probably had much more correct and interesting things to say
I have some problems with lazy evaluation, maybe one could give me some hints on this Question: https://stackoverflow.com/questions/44417831/swap-value-doesnt-resolve
hi, i think you should better precise your question, what exactly you call, what is returned and what you expect
and look on this library https://github.com/kyleburton/clj-xpath
BTW I am working now on some code to transform XML into vector of maps like convert XML orders into Clojure data
Thank you in advance
@tbaldridge hey man, I just saw your talk about core.async on clojure west. first of all kudos. I'm really interested in that design topic and patterns which you were talking about, can you please introduce me to a resource to learn more on this topic ?
@kyle.burton i found your library and it looks very useful https://github.com/kyleburton/clj-xpath . I was thinking about write something like that 🙂 Is it parse XML always lazy? How it look with GB XML files compare to https://github.com/clojure/data.xml . You did great job.
@lxsameer Most of those patterns were ones I discovered in the wild. And they're pretty broad. Books on distributed systems, Functional Reactive Programming or Dataflow are good places to start
I've heard great things about this book (even if all the examples are in Scala 🙂 ) https://www.manning.com/books/reactive-design-patterns
@tbaldridge thanks man
http://blog.cognitect.com/blog/2017/6/5/repl-debugging-no-stacktrace-required Nice post from @stuarthalloway
plexus: depends on what you need to do. you need gen-class if you have a container (e.g. servlet container) that needs to find bytecode on disk. use :impl-ns and think of it as delegation. then you get sth close to repl-ability for the delegate ns, which is good enough in my experience.
hey @mobileink sorry for the late reply, somehow I missed your response. I really appreciate you giving me a real answer, instead of all the non-answers that I got earlier.
np. dunno why :gen-class gets so much grief. the docs aren’t so great, I admit, but it works just fine, used correctly. oh well.
java -cp foo-0.1.1-standalone.jar clojure.main -m foo.core
you don’t need it totally, you can run your app like this
gen-class is a specific tool for a specific task, which you almost always don't need
sometimes you do need it, and in that case, beyond some work arounds here and there, you are stuck with its limitations
@plexus I do not use gen-class ever. I've found every situation where we could use it ended up simpler/more understandable with just a java shim class
bfabry: i use an absolutely minimal gen-class with :impl-ns referring to a delegate ns that does all the work. works just great, at least for servlets. what could be simpler?
Hi all, I have been trying to add ssl support to a webapp in Clojure; but my server never starts, I am using jetty I create my server certificate files and this is the code: (ns restserver.core (:require [ring.adapter.jetty :as jetty] [ring.util.response :refer [response]) (defn -main "A very simple web server using Ring & Jetty" [& args] (jetty/run-jetty (-> app wrap-json-response) {:port (Integer. "8123") :join? false :ssl? true :ssl-port 9999 :keystore "sitepkcs12.keystore" :key-password "qwerty"})) (defroutes app .... Do I need to do something more??? I cannot figure out what is wrong!!! I run boot run It tries to start but then nothing
I know ':keystore' can take either a string path or a Keystore object, but I think I have only ever used it with a Keystore object
is ther ea standard way for using docker to a) build uberjar b) put that uberjar in a docker image that just has a JRE?
(the standard clojure docker base image is several times larger than openjdk:8-jre-alpine)
lvh: don’t know if you got a response, maybe this article will be helpful: http://blog.alexellis.io/mutli-stage-docker-builds/
so a docker that builds a jar and makes a new docker that just runs that jar?
(I have no idea but that sounds useful if so)
IOP - image oriented programming
java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:(aws.clj:176:3)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:(aws.clj:176:3)
❤️ AWSis there a simple trick for using the same code when catching a few different exception types? I need to differentiate my exception handling and I want to eg. handle IllegalMonitorStateException the same way I handle IllegalStateException
(but not the same way I handle other Exception classes)
I can do this via a macro of course
bfabry: i use an absolutely minimal gen-class with :impl-ns referring to a delegate ns that does all the work. works just great, at least for servlets. what could be simpler?
@mobileink for the person coming across your code? not having to read the documentation for gen-class
I kind of wish I could just use (try ... (catch [IllegalStateException IllegalMonitorStateException] ...))
oh nice, I figured someone should have written that already
https://dev.clojure.org/jira/browse/CLJ-2124 referenced on this ticket
I've been using clojure professionally for 4 years and I would have to look up what :impl-ns is, and then I would get mad at how confusing the gen-class documentation is again, and then I would admire how it doesn't actually lead to much less code than a java class that delegates to clojure code
@bfabry agreed, every time I need gen-class I end up writing a java shim instead
Is it possible to extend MySpecialException
to all those and then a catch MySpecialException
?
@bfabry and when you want to do that I highly recommend this : https://github.com/ztellman/virgil
@bfabry fair enough. but docs is a separate matter. my gen-class code is so simple nobody should ever need to read it. :impl-ns, that's about it. in fact my build tool generates it and then discards it.
auto-reloads java code when it changes. Works well enough for writing shims
@john I think that gets the inheritance tree upside down - you need to catch a superclass and I can’t add new supers
@bfabry: e.g.
(ns servletsgen2293)
(gen-class :name greetings.hello
:extends javax.servlet.http.HttpServlet
:impl-ns greetings.hello)
dunno if that works for all cases, but it works great when you have a container looking for bytecode to bootstrap.
shrug I get that it works, but a java class with Clojure.var(...).invoke works too, and requires less new knowledge and has no downsides imo
another fun thing is that clojure try/catch lets you write two catch clauses for the same class - one of which won’t ever get called?
try is a special form iirc
but feel free to just log a bug on that
:thumbsup:
@alexmiller surely the compiler could be extended to test specs against special forms
that does not seem worth the time to do :)
seems like the compiler should just catch this
like, is it a useful design maxim, to eliminate as much of the difference between special forms and macros as possible, I don't know, but being a lisp crank is fun
#noob question, on http://www.braveclojure.com/core-async/ there is a pipeline example that follows the pattern: (defn upper-caser [in] (let [out (chan)] (go (while true (>! out (clojure.string/upper-case (<! in))))) out)) Now, if I chain a bunch of these my understanding is that they'll stay running forever, are they ever collected? My usecase is to receive a request, fan-out with core async, do some concurrent work, then wait 200ms and return whatever I manage to accumulate in those 200ms, I guess I can't use this as I'd leak these sort-of go-routine at each request, what is the recommended approach?
usually it’s better to write your loop to check whether you received nil from the in channel (indicating it’s closed), and then either end and/or close the out channel
that way external callers can close the in channel to clean things up
(defn upper-caser [in]
(let [out (chan)]
(go-loop [val (<! in)]
(if val
(do
(>! out (clojure.string/upper-case val))
(recur))
(close! out)))))
alexmiller: you probably meant
(defn upper-caser [in]
(let [out (chan)]
(go-loop []
(if-let [val (<! in)]
(do
(>! out (clojure.string/upper-case val))
(recur))
(close! out)))))
yes, thanks! :)
I just typed that in without an editor, so excuse typos or unbalanced parens
@alexmiller thanks, awesome I didn't know about go-loop, ty!
it can also be handy to check for a nil on your channel write (and exit)
If I have a large, deeply-nested data structure (maps, vectors, etc) and I want to get an approximate count of how big it is for instrumentation, what would be a fast/efficient way to do it? My initial thought was just (-> data flatten count)
but I wonder if that might be too naive from a memory standpoint, and just now I realized it doesn't do nested maps
I'm generally interested in just a count of the elements or something similar
I guess there's always just using (walk)
with an atom as in here: https://clojuredocs.org/clojure.walk/postwalk#example-542692d1c026201cdc326f2b
=> (->> {:a [1 2 3] :b {:c 33}} (tree-seq coll? seq) count)
13
the number will be larger than a flatten would give you, it includes an item for each branch, not just leaves
but that’s extra structure / space usage…
Ah, that looks great, haven't seen treeseq before
it walks ever node of nested data, returning each node
it’s lazy too (which doesn’t help here, but is handy)
you may want to look at something like http://openjdk.java.net/projects/code-tools/jol/
oh, and the tree-seq as shown won’t count how big eg. a string or java array is (but by replacing the coll? and seq args it could)
Well, my data is finite and I just want to have a number of some kind that I can watch for changes between implementations
So I don't need anything super-precise
Yeah, exactly. I'm just looking for a heuristic aid
there is always (count (pr-str x)), but I guess I would be surprised if that is faster
Yeah, wanted to avoid that
@bcbradley maybe adding a layer over http://www.mapdb.org/ ?