Fork me on GitHub
#clojure
<
2017-06-07
>
fabrao00:06:46

Hello all, if I use doall with pmap to threading some functions, if any of thread fails, fails the result for doall?

hiredman00:06:00

it is complicated, basically yes, but really no

hiredman00:06:09

you will get an exception from doall, but pmap may have processed some items "beyond" that point

hiredman00:06:24

pmap is not great

fabrao00:06:36

so, how can I run concurrent threads without affecting the result?

fabrao03:06:27

Is it that bad to do something like this?

(defn function-long-run [ip conc message]
	(try 
		(swap! message assoc (keyword ip) (icmp ip))
	(catch Exception e (swap! message assoc (keyword ip) :fail))
	(finally (swap! conc inc))))

(let [  conc (atom 0)
		output (atom {})
		ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]
		limit (count ips)]
		(pmap #(function-long-run % conc output) ips)
		(while (< @conc limit)
			(do (Thread/sleep 2000)))
		(println @output))

noisesmith03:06:38

I'd expect a lot of retries on that atom

noisesmith03:06:03

and pmap only uses a limited number of threads, it's not optimized for io bound tasks

noisesmith03:06:55

you eliminate the retries if you return [ip result] from each function call, and conj them all into a map

noisesmith03:06:26

there's nothing here that requires an atom

fabrao03:06:27

even if I need println only when wait to all finishing?

noisesmith03:06:23

pmap returns a specific number of results, when they all come back, you are done

noisesmith03:06:33

any other alternative will offer something similar easily

fabrao03:06:00

I tried with (doall (pmap ...)) but sometimes it fails

fabrao03:06:10

when icmp fail

noisesmith03:06:32

also, pmap is lazy, so not consuming results guarantees that you lock up

noisesmith03:06:01

isn't the try catch sufficient for the icmp failure issue?

fabrao03:06:05

that´s because I´m using doall

noisesmith03:06:17

but in that code you are not using doall, it will lock up

noisesmith03:06:46

unless the number of items handed to pmap is small enough that there's no backlog

fabrao03:06:51

in that code I´ll wait for conc catch limit

noisesmith03:06:10

but the pmap results don't happen because you aren't using pmap's return value

fabrao03:06:59

it doesn´t matter if the result is in output atom

noisesmith03:06:11

the result won't happen because pmap is lazy

noisesmith03:06:18

you don't even understand what this code is doing

fabrao03:06:24

I tried and it´s working, but I´m asking if is it smell

noisesmith03:06:54

it's not even a smell, it's an error

noisesmith03:06:09

because nobody looked at pmap's return value

noisesmith03:06:25

+user=> (do (pmap (fn [_] (swap! done inc)) (range 100000)) nil)
nil
+user=> @done
32

noisesmith03:06:32

so 32 items out of 100k get done

fabrao03:06:12

haha - I tried with doall, freezed my emacs

noisesmith03:06:38

illustrating what I was saying about retries:

+user=> (do (pmap (fn [_] (swap! done (fn [x] (print (str \space x \space)) (inc x)))) (range 100000)) (println))
 0  0  0  0  1  1  1  2  2  3  3  4  4  4  4  4  5  5  5  5  6  6  6  7  7  7  8  8  8  9  9  10  11  12  13  13  13  14  14  14  15  14  15  13  16  16  16  17  17  17  18  18  18  19  19  20  20  21  21  22  22  22  22  22  23  23  23  23  24  24  24  0  25  25  25  25  26  26  26  27  27  27  28  28  28  29  29  30  31 
nil

noisesmith03:06:50

notice how many times it needs to run to get 32 results

noisesmith03:06:38

the problem goes away, and the code becomes simpler when you stop using an atom, and just put the return values from pmap into a hash map directly

noisesmith03:06:11

if try/catch isn't fixing the errors you were seeing before, you weren't using try/catch properly

fabrao04:06:47

so this

(defn function-long-run [ip conc message]
	(try 
		{(keyword ip) (icmp ip)}
	(catch Exception e {(keyword ip) :fail})))

(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
    (println (pmap #(function-long-run %) ips)))
will be nice?

fabrao04:06:16

sorry, remove conc and message from function

seancorfield04:06:21

#(function-long-run %) is just a long way of writing function-long-run then

seancorfield04:06:16

I assume your list of IP addresses would be large? Much bigger than 32 elements?

john04:06:00

(map #(future (function-long-run %)) ips)

seancorfield04:06:42

Except you need to (map deref ...) over that... and it's chunked, so beware.

john04:06:24

what's chunked?

fabrao04:06:51

no more than 32

john04:06:56

hmm, maybe a (into [] (comp (map #(future (function-long-run %))) (map #(deref %))) ips)

fabrao04:06:34

john, your propose runs one ip by time

fabrao04:06:50

seems that the future is not runing in "parallel"

john04:06:15

will it really? hmm. Did you try it? Maybe the eagerness in the pipeline is running the first deref before the second future...

fabrao04:06:39

Yes, I tried now

fabrao04:06:33

pmap seems to do the work

john04:06:57

Well, transducers nicely translate into thrushes. Does this work? (into [] (->> (map #(future (function-long-run %)) (map #(deref %))) ips)

john04:06:24

pmap will throttle on a max number of threads. Which may be what you want.

fabrao04:06:20

clojure.lang.LazySeq cannot be cast to clojure.lang.IFn 🙂

john04:06:13

silly me

john04:06:32

(into [] (->> ips (map #(future (function-long-run %)) (map #(deref %))))

fabrao04:06:16

this worked

john04:06:44

So it depends on how many ips you have and how many threads you want to spawn at once.

fabrao04:06:46

so, this will not limit to pmap threads?

john04:06:08

Yeah, that'll spawn thousands of futures, I believe. Futures use clojure's agent send-off pool. Not sure if there's any rate limiting or maximums on that pool.

john04:06:35

I don't think so though

fabrao04:06:50

to understand what is the use of deref?

john04:06:37

As @seancorfield kindly pointed out, a future sorta wraps threads in an atom-like derefable interface

fabrao04:06:26

my brain messed up lol

john04:06:27

I can either return a pending status (I think) or the final value of the thread's return value. And thereafter always returns that value on deref.

fabrao04:06:59

hum, I understand

john04:06:06

So, you can do (let [f (future (inc 1))] (= 2 @f)) => true

john04:06:22

well, after you wait long enough

john04:06:28

otherwise, false

seancorfield04:06:57

@f will block until the future is completed.

john04:06:05

ah, nice

seancorfield04:06:32

pmap is a very blunt instrument and is nearly always the wrong solution -- unless you really want something quick'n'dirty.

fabrao04:06:04

so, if some of futures never come back, is there a deadlock?

seancorfield04:06:12

Using map like this for side-effecting functions has some... subtleties... that you really ought to avoid.

seancorfield04:06:42

If a future expression takes "forever" to execute, then a deref on it will block "forever".

fabrao04:06:18

so, is there any way to timed out the future?

seancorfield04:06:38

@fabrao I missed the beginning of this thread (yes, I know I could scroll back) but are you just trying to do something throwaway or is this meant to be "production" quality?

john04:06:44

It probably isn't a good idea to fire off 1000 parallel ping requests under any circumstances.

seancorfield04:06:47

If you want timeouts, use core.async.

seancorfield04:06:10

In fact, if you want any real level of control over this, use core.async.

seancorfield04:06:31

Then don't use pmap and don't use map future.

fabrao04:06:21

I have in production with pmap, and I´m having problems with this

fabrao04:06:42

so, ur advice is use what

fabrao04:06:34

thanks a lot for the information

john04:06:27

You should be rate limiting your pings. Create batches of a certain size, send them out in parallel. It's not good etiquette to DOS networks with ICMP packets.

fabrao04:06:48

@john I´ll not do 1000 parallel ping, but some 20 yes

fabrao04:06:38

I can´t figure out doing this with core.async

fabrao04:06:24

who is the last to close channel?

seancorfield04:06:29

There's a #core-async channel if that helps.

seancorfield04:06:37

But, yeah, it's definitely got a learning curve.

fabrao04:06:19

yes, I´m newbe about core-async

john04:06:20

right, so you might be okay with rate limiting a stream of futures... but yeah, once you get into fine grained concurrency management, there's core.async. If you're pinging thousands of hosts with super long response times and other long term / session oriented connections for monitoring, etc, managing those using core.async can save you from a lot of wasted thread management.

fabrao04:06:42

sometimes it´s hard shifting from imperative programming

john04:06:40

It's hard, but necessary. You know what you must do, fabrao. Go forth and code 🙂

seancorfield04:06:40

Clojure makes concurrency a lot easier than with many other languages but this sort of stuff is never trivial.

fabrao04:06:41

thanks for all folks help

fabrao04:06:09

never trivial as you said

fabrao04:06:05

but it´s flexible as I´ve never seen before

seancorfield04:06:48

FWIW, when we were starting out with Clojure, I wrote a process that scanned our database and ran searches against a custom search engine and then generated and sent HTML emails. When we first powered it up, it could do about 250K emails a day. A pmap later, it brought down our search engine... ahem

seancorfield05:06:10

So I tweaked things and expanded our search engine cluster and got it up to 750K a day.

seancorfield05:06:36

Eventually I got it up to 3M a day but pmap was not the right solution at any level.

john05:06:03

It's mostly a shiny object 🙂

seancorfield05:06:24

Moral: be very careful with pmap and just doing everything in a future...

fabrao05:06:16

I´ve met clojure by chance, and now I can´t figure out my life without () lol

fabrao05:06:25

I´m learning now that core.logic to replace somethig like http://www.clara-rules.org/

fabrao05:06:25

imperative -> functional -> logic 😱

danielsu05:06:09

I want to perform a operation on a series of values, each operation feeding the return value as a argument to the next. Kind of like The threading macros, but i must work on a sequence. Is there a core function like this?

seancorfield05:06:21

@fabrao Clojure changes the way you work -- we started using it for "small" stuff back in 2011 and it became our primary language in 2015 and we've been rolling out all sorts of pure Clojure apps in the last year.

seancorfield05:06:50

We're at close to 60K lines of Clojure these days...

danielsu05:06:11

@john oh that's right, i must have overlooked The usage of reduce. Thanks!

fabrao05:06:17

think 60K Clojure in Java lol

fabrao05:06:56

Here in Brazil the company NuBank (credit card) is using Clojure in almost projects

viesti05:06:41

hum, having interesting classpath problem:

dev=> (require ‘[compojure.core :as c])

CompilerException java.lang.RuntimeException: No such var: response/send, compiling:(compojure/core.clj:160:6)

viesti05:06:56

ps auxf | grep java shows that compojure 1.6.0 is on the classpath, as requested in :dependencies by

[ring “1.6.1”]
                 [compojure “1.6.0" :exclusions [commons-codec
                                                 medley]]

viesti05:06:35

should try to narrow what other dependency causes this

seancorfield05:06:45

viesti: Use lein deps :true to see if unexpected older versions of libraries are coming in.

viesti05:06:08

ah, didn’t know about :true

seancorfield05:06:28

sorry, typing with one hand -- beer in the other

viesti05:06:43

too early morning for beer now 🙂

seancorfield05:06:17

10:30pm for me 🙂

fabrao05:06:22

maybe what some :refer ?

seancorfield05:06:08

fabrao: More likely to be a classpath / version issue -- nothing to do with :refer

Yehonathan Sharvit06:06:54

I need to read a file that contains psql queries in clojure with and I am facing issues with \COPY

Yehonathan Sharvit06:06:25

\COPY is a postgres command but it confuses the reader

Yehonathan Sharvit06:06:38

I probably need to escape the backslash. But how?

pesterhazy10:06:38

@viebel what are the contents of the file?

pesterhazy10:06:29

pretty should that io/reader is agnostic vis-a-vis file contents

pesterhazy10:06:47

my guess is the issue lies elsewhere?

Yehonathan Sharvit10:06:49

\COPY commodity_details_tmp FROM 'path/to/aa.csv' DELIMITER ',' CSV;

pesterhazy10:06:40

you should be able to slurp that file (or the reader). You could try that and print the output

razum2um11:06:36

Reading https://github.com/juxt/bidi#wrapmiddleware and don’t fully understand what it suggest instead of ->WrapMiddleware. Do they mean writing a postwalk function which will replace handlers to fns with applyed middleware to the handler? or something else?

pesterhazy12:06:51

@razum2um can't you just use replace your handler with (wrap-params handler)?

dominicm12:06:43

@razum2um the postwalk refers to a situation like:

[["/"
    ["admin/" [["index" handler-A] ["logout" handler-B]]
    ["" homepage]]]
If you wanted to add middleware to all of the admin/* routes, you should use a postwalk. Does this answer your question?

pesterhazy12:06:23

so that would be something like this

["admin/" (postwalk #(if (fn? %) (wrap-params %) %) [["index" handler-A] ["logout" handler-B]])]
?

dominicm12:06:30

@pesterhazy yeah, exactly something like that.

pesterhazy12:06:01

that's not too bad

pesterhazy12:06:18

the fact that bidi uses simple data structures enables so many hacks

dominicm12:06:37

hacks? psshht, features!

pesterhazy12:06:58

yeah when all you have is data, the line between hack and feature is blurred

dominicm12:06:24

data enables you to make decisions for yourself. You're not dependent on a library author to say "Okay, I've thought about it. Having middleware apply to sections of routes is okay, you can do that now".

pesterhazy12:06:41

what's a feature if not a hack that you're stuck with?

dominicm12:06:45

A really unique feature is route-seq, you could build a site-map from that for example.

pesterhazy12:06:52

data also enables you to pretty much immediately estimate how hard implementing something would be: "oh, that's just a postwalk away"

dominicm12:06:25

postwalk is something we use a lot. Yada + Bidi is a pretty powerful combo in this way.

pesterhazy12:06:11

those four letter libs are growing on me, bit by bit

dominicm12:06:44

JUXT, bidi, skip, yada, aero, mach, roll, baby, edge, tick, pull. Everything is 4.

pesterhazy12:06:04

true quip, mate

dominicm12:06:04

Our training is called grok. I've run out of things we've named now.

graemenelson13:06:57

what is the idiomatic way for mapping [{:id 1, :value 10}, {:id 2 :value 20}, {:id 1 :value 15}] => {:id 1, :values [10, 15], id: 2 :values [20]}. I can envision doing this with a recursive function starting with an empty map, but I would assume there is a much more elegant way of doing this in clojure. Any tips would be much appreciated.

bronsa13:06:55

@graemenelson that's not a valid map

jeff.terrell13:06:52

@graemenelson - usually reduce would be preferred to a recursive function. Although it's possible there's something more elegant you could do here.

graemenelson13:06:20

You are right @bronsa — sorry haven’t had my coffee yet. I meant output to [{:id 1 :values [10 15]}, {id: 2 :values [20]}]

dominicm13:06:47

Some kind of reduce I'd guess.

graemenelson13:06:02

I was looking into to reduce and into, maybe I am just not groking it yet. I will keep looking

jeff.terrell13:06:15

I might try something like this:

(letfn [(join-values [maps]
           {:id (-> maps first :id)
            :values (map :value maps)})]
  (->> seq-of-id-val-maps
    (group-by :id)
    (join-values)))
Not sure that that would actually work, but something like that would.

dominicm13:06:33

@graemenelson You might want to start by doing a (group-by :id), that should make it easier to merge values of the same kind together. Then you can reduce over each value in the map using a function similar to the one I linked above 🙂

alricu13:06:39

hi all, is it possible to create my own webapp with ssl support (https) without a signed certificate by any entity such as Godaddy? I just need to show a proof that it is possible to use ssl with clojure!!!

dominicm14:06:07

@alricu it's possible to use ssl with clojure. https://github.com/SevereOverfl0w/aleph-yada-ssl I made this the other day

dominicm14:06:35

it has self-signed certs in already, so you can use those, or generate your own using the instructions I documented

graemenelson14:06:28

Thanks @dominicm I will take a look

graemenelson14:06:44

Thanks group-by might work

graemenelson14:06:30

with reduce-kv

dominicm14:06:19

graemenelson:

(def data [{:id 1, :value 10}, {:id 2 :value 20}, {:id 1 :value 15}])

(map (fn [[_ v]]
         (reduce (fn [m [k v]]
                 (case k
                   :id (assoc m k v)
                   :value (update m k conj v)))
               {}
               (apply concat v)))
     (group-by :id data))

graemenelson14:06:20

Thanks for the help!

alricu14:06:28

Thanks @dominicm !!! Is it possible without using yada and aleph?? problem is that I only need to use http-kit and jetty

alricu14:06:52

😥 I know that is bad; but those are my restrictions !!!

dominicm14:06:04

@alricu I imagine it is. But I haven't tried.

alricu14:06:58

ah ok thanks @dominicm , I will take a look to your code!!!

dominicm14:06:24

Hopefully the self-signed certs will be useful at least

johnj14:06:37

@alricu look up ring-ssl

not-raspberry14:06:04

@alricu It is possible, screw SSL on the clojure side and upload the cert to your cloud-hosted loadbalancer or if you don't use LBs, make your nginx use it.

not-raspberry14:06:43

No one I know actually bothers with SSL on the app side.

alricu14:06:58

yep, I have seen those options, my problem is that I have a lot of restrictions with regards to the libraries I can use

dominicm14:06:58

https://github.com/http-kit/http-kit/issues/32 It doesn't look like http-kit supports https in the server.

fabrao14:06:14

Hello @john I observe that the (into [] (->> ips (map #(future (function-long-run %)) (map #(deref %)))) serialize when function-long-run is doing network communication, how can it be?

dominicm14:06:33

@not-raspberry sometimes not possible, sometimes all data in transmission must be encrypted for example

dominicm14:06:04

@alricu I don't think http-kit supports ssl for server. So use jetty.

john14:06:48

@fabrao I'd suggest putting up an example of function-long-run so some folks can test out what you're working with.

fabrao14:06:53

@john

(defn- disable-certificate []
  (doto
      (java.util.Properties.)
    (.put "StrictHostKeyChecking" "no")))

(defn- session [host port user password]
  (doto
      (.getSession
       (com.jcraft.jsch.JSch.) user host port)
    (.setPassword password)
    (.setConfig (disable-certificate))
    (.connect)))

(defn available? [ip port username password & {:keys [timeout] :or {timeout 60000}}]
  (try
    (let [sess (session ip port username password)]
      (try
        (catch Exception e (throw e))
        (finally
          (.disconnect sess))))
    (catch Exception e (throw e))))

(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
	(into [] (->> ips
    	(map #(future (available? % 22 "someuser" "somepassword"))) (map #(deref %)))))

fabrao15:06:36

session is blocking future?

ghadi15:06:45

You're launching the future immediately before dereffing it: (map (deref (future blah...)) collection)

ghadi15:06:02

it's de facto serializing the operations

ghadi15:06:48

if you want truer concurrency you could launch all the futures first & eagerly. Then in a separate step dereference the future

fabrao15:06:27

sorry, what´s eagerly means in this context?

ghadi15:06:19

mapv not map

ghadi15:06:31

opposite of lazy-sequence

fabrao15:06:17

(let [ips ["192.168.0.10" "192.168.0.11" "192.168.0.12"]]
	(into [] (->> ips
    	(mapv #(future (available? % 22 "someuser" "somepassword"))) (mapv #(deref %)))))
?

john15:06:40

Give it a shot

fabrao15:06:49

I tried and it worked

fabrao15:06:00

well, I have to learn a lot before consider functional programmer 🙂

fabrao15:06:25

so, for my dumb knowleadge, can you explain what happened?

dominicm15:06:03

as an aside, you may want to look at pmap

fabrao15:06:30

future is in past with map but in future with mapv?

fabrao15:06:03

@dominicm someone told me that not use pmap in this context

john15:06:45

You could still use pmap for this. Just know that once your last available thread is waiting on your icmp response, and all others are hanging, all other resources that leverage that pool will also wait for those icmp responses to return. So all those extra cores are going to waste waiting on IO.

john15:06:51

So, in general, bad idea to do IO on resources designed for CPU bound processes.

john15:06:08

But, if you weren't using thread pools anywhere else, you could use pmap's queue to throttle your thread creation.

genmeblog15:06:26

@fabrao you don't have to use (into [] ...) mapv does that job

gowder15:06:55

Hey wise people... offhand, does anyone know offhand which types one has to extend to have a protocol work on all maps? Poking around, I seem to be able to turn up at least clojure.lang.PersistentTreeMap, clojure.lang.PersistentArrayMap, clojure.lang.PersistentHashMap --- I'm thinking I might have to extend them all...?

john15:06:35

As to your question, why mapv worked instead of map, I guess because map is lazy, the (map #(deref... isn't pulling the next invocation of the upstream map function until its own deref operation completes.

genmeblog15:06:03

@gowder all extend APersistentMap

genmeblog15:06:33

but I'm not sure if you can extend abstract class

john15:06:03

@tsulej @fabrao yeah, you no longer need the (into [] as that was only necessary in the previous transducer example.

dpsutton15:06:21

not saying to use it but he lists all the interfaces you need to implement and then he does as much as he can and tells you which ones are left to you

gowder15:06:32

oooh. oooh thank you @dpsutton and @tsulej !

dpsutton15:06:06

also mark engleberg wrote a priority map https://github.com/clojure/data.priority-map and you can see all that he did to make it "mappy"

dpsutton15:06:11

if you wanted a real world example

dpsutton15:06:22

helpful comments about some of the methods as well

john15:06:44

Can't you also use defrecords to make a custom, extensible map?

dpsutton15:06:25

my understanding is that defrecord makes a class but if you dissoc from it you lose the custom ness and it reverts back to a standard map

dpsutton15:06:45

and you wouldn't be able to add special behavior, which would be the reason to make a special map in the first place

dpsutton15:06:47

but i cut off @ghadi who probably had much more correct and interesting things to say

danielgrosse16:06:13

I have some problems with lazy evaluation, maybe one could give me some hints on this Question: https://stackoverflow.com/questions/44417831/swap-value-doesnt-resolve

kwladyka16:06:32

hi, i think you should better precise your question, what exactly you call, what is returned and what you expect

kwladyka16:06:35

it could be helpful

kwladyka16:06:41

BTW I am working now on some code to transform XML into vector of maps like convert XML orders into Clojure data

danielgrosse16:06:30

Thank you in advance

lxsameer16:06:39

@tbaldridge hey man, I just saw your talk about core.async on clojure west. first of all kudos. I'm really interested in that design topic and patterns which you were talking about, can you please introduce me to a resource to learn more on this topic ?

kwladyka16:06:54

@kyle.burton i found your library and it looks very useful https://github.com/kyleburton/clj-xpath . I was thinking about write something like that 🙂 Is it parse XML always lazy? How it look with GB XML files compare to https://github.com/clojure/data.xml . You did great job.

tbaldridge16:06:06

@lxsameer Most of those patterns were ones I discovered in the wild. And they're pretty broad. Books on distributed systems, Functional Reactive Programming or Dataflow are good places to start

tbaldridge16:06:39

I've heard great things about this book (even if all the examples are in Scala 🙂 ) https://www.manning.com/books/reactive-design-patterns

john17:06:40

ghadi: That's a great tip. I'm going to start using that method.

plexus17:06:00

how do people use :gen-class in a repl-based workflow?

mobileink18:06:01

plexus: depends on what you need to do. you need gen-class if you have a container (e.g. servlet container) that needs to find bytecode on disk. use :impl-ns and think of it as delegation. then you get sth close to repl-ability for the delegate ns, which is good enough in my experience.

plexus15:06:23

hey @mobileink sorry for the late reply, somehow I missed your response. I really appreciate you giving me a real answer, instead of all the non-answers that I got earlier.

mobileink16:06:25

np. dunno why :gen-class gets so much grief. the docs aren’t so great, I admit, but it works just fine, used correctly. oh well.

plexus17:06:22

is there any way to recompile and reload a class?

kwladyka17:06:52

java -cp foo-0.1.1-standalone.jar clojure.main -m foo.core you don’t need it totally, you can run your app like this

hiredman17:06:06

gen-class is a specific tool for a specific task, which you almost always don't need

hiredman17:06:02

sometimes you do need it, and in that case, beyond some work arounds here and there, you are stuck with its limitations

hiredman17:06:26

(which I think are largely there to avoid classloader visibility problems)

bfabry17:06:16

@plexus I do not use gen-class ever. I've found every situation where we could use it ended up simpler/more understandable with just a java shim class

mobileink18:06:51

bfabry: i use an absolutely minimal gen-class with :impl-ns referring to a delegate ns that does all the work. works just great, at least for servlets. what could be simpler?

hiredman17:06:44

gen-class also brings with it aot compilation which has its own set of issues

alricu17:06:01

Hi all, I have been trying to add ssl support to a webapp in Clojure; but my server never starts, I am using jetty I create my server certificate files and this is the code: (ns restserver.core (:require [ring.adapter.jetty :as jetty] [ring.util.response :refer [response]) (defn -main "A very simple web server using Ring & Jetty" [& args] (jetty/run-jetty (-> app wrap-json-response) {:port (Integer. "8123") :join? false :ssl? true :ssl-port 9999 :keystore "sitepkcs12.keystore" :key-password "qwerty"})) (defroutes app .... Do I need to do something more??? I cannot figure out what is wrong!!! I run boot run It tries to start but then nothing

hiredman17:06:05

I would try an absolute path

hiredman17:06:31

"then nothing" isn't really useful

hiredman17:06:42

do you get errors when you hit the ssl port?

hiredman17:06:52

by nothing do you mean nothing is printed out?

hiredman17:06:57

is there an error printed out?

hiredman17:06:02

does it just exit?

hiredman17:06:51

I know ':keystore' can take either a string path or a Keystore object, but I think I have only ever used it with a Keystore object

alricu17:06:53

@hiredman this last one snipped is the only output

alricu17:06:26

but then the console starts again

hiredman17:06:47

oh, because you have join? false

hiredman17:06:58

the main thread exits, so boot exits

alricu17:06:59

Ohhhh that is right!!!!

alricu17:06:34

thanks!! @hiredman !!!! I have been around with that problem for all the morning!!!

lvh17:06:13

is ther ea standard way for using docker to a) build uberjar b) put that uberjar in a docker image that just has a JRE?

lvh17:06:36

(the standard clojure docker base image is several times larger than openjdk:8-jre-alpine)

esanmiguelc21:06:09

lvh: don’t know if you got a response, maybe this article will be helpful: http://blog.alexellis.io/mutli-stage-docker-builds/

noisesmith17:06:38

so a docker that builds a jar and makes a new docker that just runs that jar?

noisesmith17:06:56

(I have no idea but that sounds useful if so)

noisesmith17:06:15

IOP - image oriented programming

lvh18:06:53

java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:(aws.clj:176:3)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:(aws.clj:176:3)
❤️ AWS

noisesmith18:06:19

is there a simple trick for using the same code when catching a few different exception types? I need to differentiate my exception handling and I want to eg. handle IllegalMonitorStateException the same way I handle IllegalStateException

noisesmith18:06:35

(but not the same way I handle other Exception classes)

noisesmith18:06:29

I can do this via a macro of course

mobileink18:06:51

bfabry: i use an absolutely minimal gen-class with :impl-ns referring to a delegate ns that does all the work. works just great, at least for servlets. what could be simpler?

bfabry18:06:27

@mobileink for the person coming across your code? not having to read the documentation for gen-class

noisesmith18:06:58

I kind of wish I could just use (try ... (catch [IllegalStateException IllegalMonitorStateException] ...))

noisesmith18:06:39

oh nice, I figured someone should have written that already

bfabry18:06:52

I've been using clojure professionally for 4 years and I would have to look up what :impl-ns is, and then I would get mad at how confusing the gen-class documentation is again, and then I would admire how it doesn't actually lead to much less code than a java class that delegates to clojure code

dpsutton18:06:53

asking for exactly what you are looking for

tbaldridge18:06:54

@bfabry agreed, every time I need gen-class I end up writing a java shim instead

john18:06:07

Is it possible to extend MySpecialException to all those and then a catch MySpecialException?

tbaldridge18:06:36

@bfabry and when you want to do that I highly recommend this : https://github.com/ztellman/virgil

mobileink18:06:51

@bfabry fair enough. but docs is a separate matter. my gen-class code is so simple nobody should ever need to read it. :impl-ns, that's about it. in fact my build tool generates it and then discards it.

tbaldridge18:06:52

auto-reloads java code when it changes. Works well enough for writing shims

noisesmith18:06:04

@john I think that gets the inheritance tree upside down - you need to catch a superclass and I can’t add new supers

mobileink18:06:46

@bfabry: e.g.

(ns servletsgen2293)

(gen-class :name greetings.hello
           :extends javax.servlet.http.HttpServlet
           :impl-ns greetings.hello)

mobileink18:06:56

dunno if that works for all cases, but it works great when you have a container looking for bytecode to bootstrap.

bfabry18:06:19

shrug I get that it works, but a java class with Clojure.var(...).invoke works too, and requires less new knowledge and has no downsides imo

mobileink19:06:21

matter of taste. i try to avoid mixed language coding at all costs. 😉

noisesmith19:06:25

another fun thing is that clojure try/catch lets you write two catch clauses for the same class - one of which won’t ever get called?

bfabry19:06:20

ha, that's interesting. you could add a spec to the try macro to validate that

alexmiller19:06:33

try is a special form iirc

alexmiller19:06:39

but feel free to just log a bug on that

bfabry19:06:43

ah, well not so much. oh well

hiredman19:06:33

@alexmiller surely the compiler could be extended to test specs against special forms

alexmiller19:06:04

that does not seem worth the time to do :)

alexmiller19:06:18

seems like the compiler should just catch this

hiredman19:06:39

yeah, but the compiler wouldn't catch it in the same way a spec would

mobileink19:06:15

nice catch! heh

hiredman19:06:20

like, is it a useful design maxim, to eliminate as much of the difference between special forms and macros as possible, I don't know, but being a lisp crank is fun

andrea19:06:50

#noob question, on http://www.braveclojure.com/core-async/ there is a pipeline example that follows the pattern: (defn upper-caser [in] (let [out (chan)] (go (while true (>! out (clojure.string/upper-case (<! in))))) out)) Now, if I chain a bunch of these my understanding is that they'll stay running forever, are they ever collected? My usecase is to receive a request, fan-out with core async, do some concurrent work, then wait 200ms and return whatever I manage to accumulate in those 200ms, I guess I can't use this as I'd leak these sort-of go-routine at each request, what is the recommended approach?

alexmiller19:06:52

usually it’s better to write your loop to check whether you received nil from the in channel (indicating it’s closed), and then either end and/or close the out channel

alexmiller19:06:04

that way external callers can close the in channel to clean things up

alexmiller19:06:01

(defn upper-caser [in]
  (let [out (chan)]
    (go-loop [val (<! in)]
      (if val
        (do
          (>! out (clojure.string/upper-case val))
          (recur))
        (close! out)))))

mpenet06:06:22

alexmiller: you probably meant

(defn upper-caser [in]
  (let [out (chan)]
    (go-loop []
      (if-let [val (<! in)]
        (do
          (>! out (clojure.string/upper-case val))
          (recur))
        (close! out)))))

mpenet06:06:31

otherwise the (recur) call will fail

mpenet06:06:49

doesn't really matter for the point you're making but for correctness sake 🙂

alexmiller14:06:08

yes, thanks! :)

alexmiller19:06:22

I just typed that in without an editor, so excuse typos or unbalanced parens

andrea19:06:24

@alexmiller thanks, awesome I didn't know about go-loop, ty!

noisesmith20:06:50

it can also be handy to check for a nil on your channel write (and exit)

timgilbert20:06:47

If I have a large, deeply-nested data structure (maps, vectors, etc) and I want to get an approximate count of how big it is for instrumentation, what would be a fast/efficient way to do it? My initial thought was just (-> data flatten count) but I wonder if that might be too naive from a memory standpoint, and just now I realized it doesn't do nested maps

timgilbert20:06:23

I'm generally interested in just a count of the elements or something similar

timgilbert20:06:32

I guess there's always just using (walk) with an atom as in here: https://clojuredocs.org/clojure.walk/postwalk#example-542692d1c026201cdc326f2b

noisesmith20:06:07

=> (->> {:a [1 2 3] :b {:c 33}} (tree-seq coll? seq) count)
13

noisesmith20:06:50

the number will be larger than a flatten would give you, it includes an item for each branch, not just leaves

noisesmith20:06:59

but that’s extra structure / space usage…

timgilbert20:06:19

Ah, that looks great, haven't seen treeseq before

noisesmith20:06:38

it walks ever node of nested data, returning each node

noisesmith20:06:52

it’s lazy too (which doesn’t help here, but is handy)

hiredman20:06:56

estimating size in a pointer machine is non-trivial

hiredman20:06:08

the jvm, even more so

noisesmith20:06:47

oh, and the tree-seq as shown won’t count how big eg. a string or java array is (but by replacing the coll? and seq args it could)

hiredman20:06:48

(which may be tricky to use with clojure)

timgilbert20:06:16

Well, my data is finite and I just want to have a number of some kind that I can watch for changes between implementations

timgilbert20:06:30

So I don't need anything super-precise

hiredman20:06:49

domain specific size I guess

timgilbert20:06:10

Yeah, exactly. I'm just looking for a heuristic aid

hiredman21:06:11

there is always (count (pr-str x)), but I guess I would be surprised if that is faster

timgilbert21:06:40

Yeah, wanted to avoid that