Fork me on GitHub
#clojure
<
2018-06-12
>
datran03:06:15

anyone using re-frame? Is there a way to turn off the re-frame.core/debug interceptor in production?

Nick Cabral08:06:01

jdbc question: if I use jdbc/query to execute a postgres SELECT… FOR UPDATE query within a jdbc/with-db-transaction, will I have locks on the selected rows for the remainder of the transaction? Or do I need to use something other than jdbc/query? The expected behavior isn’t clear to me from the docs.

pesterhazy09:06:17

@nick652, afaik the behavior should be the same as in a psql session wrapped in "BEGIN TRANSACTION ... COMMIT"

Nick Cabral09:06:35

Okay thanks. So you wouldn’t expect that I need to use jdbc/execute! rather than jdbc/query for the SELECT...FOR UPDATE query within that transaction?

pesterhazy09:06:39

according to the docstring, execute! is for INSERT, UPDATE etc., not SELECT: >>> perform a general (non-select) SQL operation

pesterhazy09:06:36

but it's a good idea to verify this from the repl to familarize yourself with how postgres transactions and jdbc work

Nick Cabral10:06:31

Yeah that part is clear. My uncertainty is because the postgres docs suggest that from a concurrency standpoint, SELECT FOR UPDATE acts more like an UPDATE than it does a SELECT. Although that’s an implementation detail of postgres, it’s not clear to me if jdbc/execute! is performing some sort of setup required to enable the UPDATE-like concurrency aspects of the query.

Nick Cabral10:06:29

In any case, you’re right that writing tests is necessary on my end to confirm. Just helps to have some idea of what to expect going in.

lmergen10:06:53

From the perspective of Clojure's JDBC, the important thing is whether a result set is returned or not. Execute just returns the number of affected rows, not the actual result set. Even a "UPDATE .. WHERE x = y RETURNING *" would be more appropriate to do using jdbc/query

👍 4
Nick Cabral10:06:50

@U0M8Y3G6N that clears it up completely, thanks!

pesterhazy11:06:20

That might be a useful addition to the docstring actuallly

👍 4
kah0ona10:06:02

Hi! In my system I shell out to libreoffice executable to generate a PDF from a word file. Occassionally it (libreoffice) hangs or chokes on the docx passed in. In such a case, my threads fill up, and at somepoint my uberjar process halts. Is there a way to timeout the shell call, and when it does stop the thread? What would be idiomatic?

kah0ona10:06:17

Are futures the way to go?

Hendrik Poernama10:06:26

You could try deref ing the future with timeout. If it times out, future-cancel the future

kah0ona10:06:34

Ah I’ve got a nice core async solution as well

kah0ona10:06:42

ah thanks Hendrik, yeah was thinking about that

kah0ona10:06:19

I wonder if that clogs up the thread pool in the long run though?

kah0ona10:06:34

ie. if the passed in function hangs forever… are resources being freed up? Haven’t got enough knowledge on runtime internals, i’m afraid…

Hendrik Poernama10:06:06

Hmm.. Not sure if core async will clean up the parked thread in jvm

Hendrik Poernama10:06:56

My feeling is you may need to isolate each shell call on its own thread instead of a pool, and kill the thread on timeout.

Hendrik Poernama11:06:25

I'm not sure if this can be done using a thread pool. Actually, now I'm not sure if future-cancel can terminate the shell.. Would have to test

kah0ona11:06:43

how do you actually test such a thing?

kah0ona11:06:57

can you somehow list everything that is parked?

kah0ona11:06:20

or do you need jvm command line calls or whatever

Hendrik Poernama11:06:26

Make a thread or future shell out a long sleep command, then future cancel it?

Hendrik Poernama11:06:59

Then ps and see if it kills the sleep

kah0ona11:06:05

aaah like that

kah0ona11:06:07

yeah of course

Hendrik Poernama11:06:17

Otherwise your LibreOffice process won't get killed and you risk running out of ram?

kah0ona11:06:26

yeah that ’s what happened ..

kah0ona11:06:46

alright; thanks!

Hendrik Poernama11:06:34

A hackier workaround outside the box is to have a cron job regularly look for stuck LibreOffice process and kill them.. Haha

kah0ona11:06:50

i was even thinking about moving that call to a different server, and make some sort of http-call in between them

kah0ona11:06:10

in that case i can use http’s normal timeout mechanism 🙂

kah0ona11:06:27

but that’s so much work; i hoped for a simple solution / wrapper function 🙂

kah0ona11:06:01

I am going to wrap the libreoffice call in a .sh script and call that instead from clojure. In that shell script, i use GNU Coreutils timeout: http://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html

Andreasp199414:06:40

Hey there. Can anyone point out some resources on how to integrate compojure with ring swagger?

valtteri14:06:19

https://github.com/metosin/compojure-api does that. Or did you mean integrating ring-swagger into existing compojure codebase? There’s #ring-swagger where you might find help.

gfredericks14:06:50

Does anybody know where the conj is this year?

Andreasp199414:06:01

Yeah I am trying to figure out how to do it without compojure api cause I have got some legacy routes... but would you suggest wrapping everything with compojure-api?

Alex Miller (Clojure team)14:06:58

@gfredericks we are finalizing contracts now and hope to announce soon

4
valtteri14:06:02

@andreasp1994 AFAIK compojure-api supports vanilla compojure out of box so starting a new project with c-api and migrating the legacy routes there could work.

Andreasp199414:06:47

So if I just remove compojure dependencies and add compojure api everything well be supported?

ikitommi14:06:10

@andreasp1994 mostly, yes. Compojure-api just wraps Compojure, so underlaying routing engine is the same.

Andreasp199417:06:27

Hey I am trying out the compojure-api but I am getting this error when doing a post request.`"title": "(not (instance? java.lang.String a-java.lang.Class))"` any ideas what this means?

ikitommi18:06:14

there is #ring-swagger for c-api related things.

Andreasp199418:06:34

Oh alright sorry

cjsauer20:06:40

Anyone know how I can easily round numbers in a cljc file?

cjsauer20:06:23

Or do I have to do some compiler-dependent importing...

xiongtx22:06:42

Looking at pmap more closely, it seems that n is not the degree of parallelism: https://github.com/clojure/clojure/blob/clojure-1.8.0/src/clj/clojure/core.clj#L6735 Instead, pmap launches as many threads (`future`s) as possible to process the elements of coll. What is the purpose of n, then?

noisesmith22:06:48

the laziness there means n is actually controlling the launch rate

noisesmith22:06:53

if I read it correctly

xiongtx22:06:42

How is n controlling the launch rate?

xiongtx22:06:04

IIUC n doesn't come into play until we're at the last n elements of coll

noisesmith22:06:54

no, notice that step uses n, and that n items are dropped from the input when passed to step the first time

noisesmith22:06:12

so rets is lazy, as each item is forced a future starts for that item

noisesmith22:06:23

the control of parallelism is hidden in the control of realization of elements of rets

noisesmith22:06:38

the thing with lazy-seq and cons inside step is kind of odd and I'm not sure why it's constructed like that

xiongtx23:06:53

For n to be the degree of parallelism, there'd have to be some point where new threads (`future`s) are not launched until n existing ones have been derefed. I don't see that anywhere.

noisesmith23:06:30

it's hidden in the laziness. (do (map #(future ....) coll) nil) starts no futures

noisesmith23:06:06

since there's a map call that creates futures, the forcing of items coming from that call (called rets here), controls the parallelism

hiredman23:06:15

it is complicated, and interacts with things like chunked-seqs, but n is sort of the amount of eager parallelism

xiongtx23:06:05

Let's say n = 3 and coll = (range 9). I'll use <i> to indicate (future (f i)). Then we end up w/ (lazy-seq (cons @<0> (lazy-seq (cons @<1> ... (map deref '(<6> <7> <8>)), correct?

xiongtx23:06:52

So the n only applies to the last n derefs, no?

noisesmith23:06:57

one thing missing here is that the destructure in the args list forces at least one item

noisesmith23:06:05

the seq coll could be doing a force as well

noisesmith23:06:44

it's not just derefs that matter here, it's the first realization of an item from ret (even if you are just checking if it's there)

noisesmith23:06:31

if you have (let [[x & ys] ret] ...) x has been forced even if deref isn't called yet

hiredman23:06:20

range is chunked, so if you pmap over range you will get chunk size number of futures

hiredman23:06:33

the derefs don't matter at all, the derefs don't cause futures to run or not, they just retrieve the value, so in the analysis of when a futures run, they just don't matter

xiongtx23:06:39

I don't see that n or chunk size has anything to do w/ how many futures run at a time. Chunk size may affect how many are launched at a time, but not how many are in flight. Is that a correct understanding?

noisesmith23:06:29

why would number in flight be in any way independent of how many are launched?

hiredman23:06:30

pmap is terrible

noisesmith23:06:19

the deref call blocks, which is a backpressure on the number in flight

noisesmith23:06:06

and agreed, this is weird code that probably isn't doing what anyone actually wants from it (and it's hard to even know what it's doing precisely due to the structure)

xiongtx23:06:55

What I'm saying is that I was expecting n to be the degree of parallelism, i.e. how many threads are actually being used to process the collection. But pmap uses future, which uses Agent/soloExecutor, which is a CachedThreadPool (no limit to # of threads running simultaneously). I wonder if the intent w/ n is to use the Agent/pooledExecutor instead, as that is a FixedThreadPool w/ has 2 + Runtime.getRuntime().availableProcessors() threads.

noisesmith23:06:44

what I'm saying is that laziness is controlling the number of threads in flight in pmap, it's not eagerly starting futures across your full collection

noisesmith23:06:38

and n is effecting the laziness due to it's usage inside step, but due to the complex code the relationship is hard to determine

xiongtx23:06:04

If I (doall (pmap f coll)), how many threads are running at the same time?

seancorfield23:06:45

This may be slightly easier to follow

user=> (def n (atom 0))
#'user/n
user=> (defn slow-f [_] (swap! n inc) (Thread/sleep 1000) (swap! n dec))
#'user/slow-f
user=> (doall (pmap slow-f (range 50)))
(30 29 31 23 28 27 26 25 22 24 21 16 13 18 17 20 14 15 11 10 9 4 8 12 19 7 6 5 2 0 1 3 16 17 15 14 13 11 12 10 9 8 6 7 5 0 4 1 3 2)
user=> (doall (pmap slow-f (range 100)))
(29 31 30 28 25 27 26 24 20 23 22 21 19 18 16 17 15 14 11 13 12 10 9 8 7 6 4 3 5 2 31 30 31 27 30 29 28 24 26 25 23 22 21 20 19 16 18 17 15 13 12 14 11 10 9 6 8 7 5 4 3 2 0 1 28 27 29 26 30 31 25 23 22 24 21 19 20 18 17 14 16 15 13 10 12 11 9 8 7 6 3 5 4 2 0 1 0 1 2 3)
user=> (doall (pmap slow-f (into [] (range 50))))
(31 27 29 28 30 26 25 24 22 23 21 19 20 18 17 16 14 15 13 11 12 9 10 8 6 7 4 5 3 2 0 1 15 14 17 13 16 11 12 10 9 8 5 6 4 3 7 1 2 0)
user=> (doall (pmap slow-f (into [] (range 100))))
(30 31 28 27 24 29 25 26 23 22 21 20 19 17 18 16 15 14 13 12 11 9 10 8 7 6 5 0 4 2 3 1 28 26 30 31 25 29 27 23 22 24 20 21 19 18 17 16 15 14 11 13 12 10 8 9 6 7 4 5 0 3 2 1 25 27 16 31 29 26 30 28 24 21 23 22 20 19 15 18 17 13 14 12 11 9 10 8 7 5 6 3 4 5 4 6 3 2 1 0)

👍 4
seancorfield23:06:32

The maximum number in-flight is 32.

seancorfield23:06:39

(which is the chunk size) <--- this is wrong

xiongtx23:06:55

OK, I think that makes sense

seancorfield23:06:02

(the result is the sequence of one less than the in-flight number at each point)

xiongtx23:06:23

B/c w/ each chunk we block until all the futures in the chunk have been derefed, correct?

noisesmith23:06:54

notice what's happening with (rest s) on line 6740, it seems like that's being done to keep some number of items ahead of the derefs

seancorfield23:06:18

If I make slow-f faster, 100ms instead of 1s, then I get more functions in flight.

noisesmith23:06:31

at first I was worried something was lost from s, but s only exists for forcing purposes

the2bears23:06:31

Great discussion by the way, thanks to those who take the time to explain things!

noisesmith23:06:56

since everything in s is already in the first arg to step

seancorfield23:06:13

user=> (defn slow-f [_] (swap! n inc) (Thread/sleep 100) (swap! n dec))
#'user/slow-f
user=> (doall (pmap slow-f (into [] (range 100))))
(26 28 29 30 24 31 27 25 21 23 18 22 20 19 16 15 17 13 14 11 12 10 8 9 28 33 36 34 33 35 32 34 22 30 31 29 28 27 25 26 23 24 20 21 19 17 18 16 15 14 13 12 11 10 9 8 7 37 36 26 35 34 32 33 30 29 8 28 31 27 25 26 24 23 22 21 19 18 20 15 17 16 13 14 11 12 10 9 11 10 5 9 6 7 8 4 1 2 0 3)
Notice that I get up to 38 in-flight here.

noisesmith23:06:29

given that s starts as (drop n coll), n is deciding how far ahead of your derefs of results the launches of new futures by forcing them out of laziness are

noisesmith23:06:32

in a weird way

seancorfield23:06:27

It's a very weird and unpredictable function.

8
seancorfield23:06:41

Depending on how slow the function is and how large the collection is, I get somewhere between 32 and 38 in-flight (on a machine that reports 8 processors).

noisesmith23:06:03

that into doesn't prevent chunking, but replacing range with iterate does

user=> (doall (pmap slow-f (take 100 (iterate inc 0))))
(5 3 2 1 0 6 4 6 5 4 3 2 1 0 6 5 4 3 1 2 0 5 1 6 4 3 2 0 6 5 4 3 2 1 0 4 5 0 1 2 6 3 5 4 6 3 2 1 0 6 5 4 3 2 1 0 5 6 2 4 3 2 1 6 6 4 6 5 3 2 6 6 6 4 5 2 3 6 6 3 2 4 5 6 6 6 3 5 6 4 2 6 6 5 6 4 3 2 1 0)

noisesmith23:06:21

user=> (.. Runtime getRuntime availableProcessors)
4