Fork me on GitHub
#core-async
<
2017-03-23
>
tbaldridge01:03:20

To some extent, async for SQL is a tad pointless, since the remote end isn't highly concurrent. Threadpooling will probably work just as well,

noisesmith01:03:33

@tbaldridge my motivation here is that a bottleneck on my app is the number of running threads (and the resources each thread uses), so if I can use proper async for IO bound tasks I can make my response times more reliable and avoid load induced outages. Given this I would like to do as much as I can using natively async io, and one of my biggest io usages is db communication.

noisesmith01:03:17

perhaps I'm not thinking this through properly though (and yeah, I'll still have a thread pool for things that are eg. CPU bound, but the more that can do its own thing and not get claimed by db usage the better)

tbaldridge01:03:30

Right, but by going async you don't really gain anything since the SQL server is blocking and locking tables

tbaldridge01:03:23

So if only one thread can touch a table at a time, might as well just have a SQL thread pool on the clients that's limited to x threads

tbaldridge01:03:51

Then have your go blocks funnel their requests through that pool.

noisesmith01:03:54

ahh, so a fixed dedicated pool for the db, that makes sense, thanks

donaldball02:03:03

Perhaps neither here nor there, but postgresql now uses SSI for serializable transactions

elenacanovi14:03:43

Hi all! I have a question about transducers in core async channels. is it idiomatic to define a transducer which depends (without side-effects) on a globally defined atom? I posted a longer version of this question with an example here: https://groups.google.com/forum/#!topic/clojure/JCQmEXkIYms

tbaldridge15:03:12

Nothing really wrong with that approach, and it really shows of the power of Clojure's concurrency models.

tbaldridge15:03:53

Although to be careful how much work you do inside those transducers. The transducers on a channel run inside a lock on the channel itself. So while the transducer is executing, no one many read or write from the channel

tbaldridge15:03:16

Not bad if you're doing a hash-map lookup, but get into a O(n*n) situation and "you're gonna have a bad time"

elenacanovi15:03:16

@tbaldridge I see your point. Maybe for a tougher task (not just a lookup as in the example) it would be better to change the approach

dergutemoritz20:03:24

Is there any technical reason why PromiseBuffer uses a custom NO-VAL sentinel rather than just nil?

noisesmith20:03:04

nil usually indicates closed

noisesmith20:03:15

(not sure if that works in that context though…)

dergutemoritz20:03:32

Well yeah but you can't put a nil into a channel, so that wouldn't be ambiguous

noisesmith20:03:22

yeah - that is just my best guess, someone else might have a better idea - core.async generally tends to be fastidious about overloading nil though, so it could just be part of that tendency

noisesmith20:03:54

@dergutemoritz look at the methods - it sets it to nil if closed

noisesmith20:03:00

which is what I said, right?

noisesmith20:03:09

so you can differentiate closed vs. not yet having a value

dergutemoritz20:03:41

There is no closedness check that depends on val though

noisesmith20:03:44

otherwise it would be ambiguous (or you’d need a separate state field)

dergutemoritz20:03:54

My best guess is that it was used to make the code clearer maybe

dergutemoritz20:03:39

If what you said was the case, NO-VAL could escape from the buffer just as well, which is certainly not intended 🙂

dergutemoritz20:03:31

I verified it just in case: works fine with nil instead, too!

noisesmith20:03:22

but if I read the code and know how nil is used in channels, I would find that code harder to understand if it used nil

noisesmith20:03:16

for example, I could end up asking “does line 90 imply that I can deliver a new value to my promise after closing it?"

noisesmith20:03:38

oh wait - that’s probably a bug that using nil for no-val would introduce 😄

noisesmith20:03:52

dergutemoritz check it out - can you deliver again to a closed promise chan when you do that?

dergutemoritz20:03:19

Yeah that's what I was referring to by making the code clearer / look less ambiguous... I think closedness is maintained in the chan so it shouldn't be possible but I'll try!

noisesmith20:03:38

where would it be maintained?

noisesmith20:03:06

(other than in exactly the code we’ve been looking at, that is)

noisesmith20:03:26

but that code is the chan

dergutemoritz20:03:34

No it's the buffer

noisesmith20:03:40

oh, now I get what you are saying

dergutemoritz20:03:54

Works just fine, too! Can't put anything after closing with nil as sentinel

dergutemoritz21:03:47

From the description of http://dev.clojure.org/jira/browse/ASYNC-103 it looks like the author assumed that the buffer is responsible for producing the nil value when the chan is closed

noisesmith21:03:16

I still like that it has a special sentinel other than nil - that’s how I would write it

noisesmith21:03:28

(though I might use a namespaced keyword rather than Object)

dergutemoritz21:03:28

Hm weird, close-buf! was even introduced with that patch just for this purpose!

noisesmith21:03:46

oh, interesting

dergutemoritz21:03:49

But ... close-buf! only sets val to nil if it's still undelivered o_O

noisesmith21:03:12

I bet this represents a coordination pattern

noisesmith21:03:28

where you want to be clear whether it was delivered first or closed first for later steps

noisesmith21:03:43

(even if closing and delivering are always both attempted)

dergutemoritz21:03:04

Wait a second ...

dergutemoritz21:03:45

That's the effect - it still allows to take from the chan even when closed

dergutemoritz21:03:01

take the value if one was delivered, that is

dergutemoritz21:03:27

That behavior is not apparent from the docstring

dergutemoritz21:03:53

Or am I not reading it right?

noisesmith21:03:17

so that’s what I had in mind for the “coordination pattern” - you launch two async logics, one closes it, the other delivers, but you can easily see which one got to it first even though both run

noisesmith21:03:33

there’s useful things you can do with that behavior

dergutemoritz21:03:06

If only the docstring would indicate this 😄

dergutemoritz21:03:11

OK the comments section in the JIRA issue above discusses this. The behavior was indeed intentionally changed to ignore closes on delivered promise-chans

dergutemoritz21:03:28

So I guess the docstring was just forgotten about at that point, seems like a pretty last minute change

dergutemoritz21:03:57

Hmm one could argue that this behavior is to be expected because for other buffers it's also possible to take all remaining values after closing. A delivered promise-chan just happens to contain infinite number of remaining values 😉

mpenet21:03:32

Fits the definition of a promise

dergutemoritz21:03:53

Yep, the behavior is definitely desirable

dergutemoritz21:03:19

And in line with the semantics of closed channels anyhow

dergutemoritz21:03:31

I just didn't connect all the dots 🙂

dergutemoritz21:03:07

Thanks for thinking along @noisesmith and @mpenet