This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-05-25
Channels
- # aws (2)
- # beginners (57)
- # boot (31)
- # carry (15)
- # cider (9)
- # cljs-dev (9)
- # cljs-experience (32)
- # cljsrn (94)
- # clojure (129)
- # clojure-dusseldorf (3)
- # clojure-greece (4)
- # clojure-italy (8)
- # clojure-norway (3)
- # clojure-russia (344)
- # clojure-sg (39)
- # clojure-spec (2)
- # clojure-uk (39)
- # clojurescript (84)
- # core-async (99)
- # cursive (10)
- # data-science (1)
- # datascript (4)
- # datomic (66)
- # emacs (10)
- # graphql (4)
- # hoplon (28)
- # jobs (15)
- # luminus (3)
- # lumo (5)
- # off-topic (23)
- # om (4)
- # onyx (32)
- # pedestal (24)
- # re-frame (2)
- # reagent (7)
- # ring-swagger (32)
- # spacemacs (4)
- # untangled (57)
- # utah-clojurians (1)
you would need a new channel type if you wanted to get the contents of the chan (unless you want to dump the contents into a collection and send that I guess) - what is the purpose of sending a chan?
I definitely have abstractions where each item coming from another host is decoded by transit and put onto a chan, but I can’t think of why I would want to send a chan, or if I did how I would want that chan to behave
@noisesmith our use-case is we’re getting a chan of db rows from cassandra, and we want to use that as a response body in pedestal
but we need to encode the chan as transit, and have it read back as a collection on the client
don’t send the chan, send the things that come off the chan
a chan is not a container
it’s an object that does io
the response can be large and it’d be great if we could use the chan as a backpressure medium
@jfntn sure, you can pipe from the channel into a stream that encodes each element as it comes off for example
a transit writer that loops on chan reads can do this
There’s a hack that works, where we wrap the chan and return a chan of transit strings, we wrap the contents in brackets, and interpose commas, the client reads that back as a vector
you don’t need this with transit - just keep writing more objects to the writer
and make sure you loop on reading on the other end
the output stream coming from the writer will have the bytes ready as they come if you tie it together properly
none of this is core.async or chan specific though
hmm need to look into that, think this would have to bridge to java.nio to work with pedestal though
the transit shouldn’t ever need to become a string in this scenario
@jfntn make the chan’s writer out of your nio output stream direclty
You can pass a custom chan to execute-chan-buffered that has a transducer that does transit encoding of each chunk
it’s simpler to just wrap your network io object in the transit writer htough
and write each item as you get it
I am not familiar with transit, and the particular use case, but there are options for surr :)
@mpenet I happened to stumble upon the issue you opened for pedestal regarding blocking calls in the go-loop, do you think this use case is worth the trouble given the current limitations?
if this is about doing io inside go, definitely always put io writes and reads inside a thread call if you are in a go block
or is this something where pedestal is already doing it wrong and you don’t really get to decide?
that’s too bad
Details here for anyone intersted https://github.com/pedestal/pedestal/issues/497
It was acknoledged to be a real issue in privmsg on slack, and would be fixed asap, but apparently it s still here
It might be better to turn the results chan into a ReadableByteChannel
like @noisesmith suggested, this would be more performant than my core.async hack, and would sidestep the core.async issue in pedestal :thinking_face:
:thumbsup:
in fact, code that writes transit directly to your destination will actually be simpler than the code that used strings in many cases - I think it’s intentional that transit doesn’t provide any string conveniences
@noisesmith I’m not familiar with nio, would you happen to know whether I should be looking into encoding to a ReadableByteChannel
or a ByteBuffer
? These are the two streamed and async containers that pedestal supports.
use the one that can most conveniently be the target of an OutputStream - iirc that’s easy with a ByteBuffer
@jfntn actually, taking a quick browse, this might be your ticket https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Channels.html
you can make a readablebytechannel from an inputstream using newchannel, the inputstream can be tied to the outputstream you write to from transit
(I haven’t done this with nio - hope it’s not too tricky but that api looks like it will do it)
@jfntn I would move the a/thread to directly wrap the io call, and park on the result in a go block
But yeah, that's petty close to it
(go-loop [] …. (<! (a/thread (transit/write ...))) …)
that way you get the advantage of using core.async (go blocks that can park) without starving the thread pool
Oh ok, you wait on the values from the chan in a go block, but you start a thread for the write?
right, that is what a/thread is for
so you can park on something that would otherwise block
So this would have the advantage of not using up a thread unless doing actual i/o right?
right - spending more time parking rather than using a thread
(it lets you use <! instead of <!! in the channel take above, which likely takes longer than the individual transit/write calls do)
yup, and the go could be a go-loop, but that’s just syntax it’s the same behavior
what? - you still try/catch
but inside go-loop
oh - wait, I think I get it now
right
I still find it really difficult to get a sense for when work should be offloaded to a thread vs right in the go block, esp if things are cpu bound. Say I had a transform fn in there, as in transform before writing, I think in the latest version i’d just apply it before the transit/write
inside the thread, but I wouldn’t be confident that’s the best place to do it
yeah- there’s some ambiguity - but in my experience things that use enough CPU to be a real issue will be pretty rare and I’ll know what they are - and they definitely go in a thread call
like if I’m calculating graph metrics on an adjacency list and I know it’s going to take a few minutes
even the transit write - until you get a network hiccup it’s actually not going to block long enough tobe a problem - it’s just the unpredictable network lag that makes the thread call neccessary
I’d say unless it aproaches a decent fraction of a second - but it’s worth measuring - the symptoms of go being overloaded are noticible
right so if you benchmarked it it’d be faster in the go block, but when things get congested in prod you’re better off with defering to a thread
and it will be a different consideration depending on how many parts of your app are using go blocks
@jfntn yeah - or in my app’s case I profile and I look for long green rectangles in the CPU usage overview inside threads that belong to core.async
I take each one and move them to thread calls, and then I just need to profile at least ~once per major release of the code
which is a good thing to be doing anyway
One thing I’m not clear on is I used to only have one transit/read
invocation per payload on the ClojureScript side. But with the function we looked at it seems I’ll need some out of band information to decide when to use a different read fn that will wrap multiple invocations of transit/read
in a lazy-seq, is that how you’d do it @noisesmith ?
@jfntn the most elegant thing would be to always return a sequence of results, and the old code would just get a single item
but that means changing the old code’s consumers of course
because no matter how many times a single transit was written to, you can always loop until it has no more and put them all in a coll
but really I’d use my best judgment based on how deeply the concept of only one result from transit is baked into the app logic
yeah, but I don’t like using hacks I don’t need, and transit already handles the case the brackets / commas thing was meant to cover
but it’s your code, of course 😄
I think the best solution for our re-frame app will be to set a stream?
flag on our remote effect and use that in the interpreter as a signal to use the lazy reader
It will be a while before I can run some full-stack benchmarks to compare the solutions, but I feel like this has been a productive yak shaving session, thanks for your help @noisesmith & mpenet !
heh - no problem, I hope it proves productive
@noisesmith starting to look like the ClojureScript reader for transit doesn’t actually support parsing that, have you tried that approach yourself?
I’ve done multiple writes followed by multiple reads… but not between cljs and clj
I'm writing a simple patch for core.async, but I don't know how to run the clojurescript tests. lein cljsbuild test
builds a test.js file, but doesn't seem to execute it. I haven't touched ClojureScript in a couple of years, not sure how to proceed.
I've installed the current version of v8, but still fails:
16:08:30 ~/workspaces/github/core.async > V8_HOME="/usr/local/bin" script/test
Compiling ClojureScript...
Compiling "tests.js" from ["src/test/cljs" "src/main/clojure/cljs"]...
WARNING: Wrong number of args (1) passed to cljs.core.async.runner-tests/pause at line 71 src/test/cljs/cljs/core/async/runner_tests.cljs
WARNING: Wrong number of args (1) passed to cljs.core.async.runner-tests/pause at line 83 src/test/cljs/cljs/core/async/runner_tests.cljs
Successfully compiled "tests.js" in 9.292 seconds.
Testing with V8
tests.js:2: ReferenceError: document is not defined
if(typeof goog == "undefined") document.write('<script src="../out/goog/base.js"></script>');
^
ReferenceError: document is not defined
at tests.js:2:32
SPIDERMONKEY_HOME not set, skipping SpiderMonkey tests
JSC_HOME not set, skipping JavaScriptCore tests
16:09:08 ~/workspaces/github/core.async >
I do not think that script is actually used now (or at least I haven’t used it recently, maybe David uses it)
I usually:
1. lein do clean, cljsbuild once
2. open script/runtests.html
3. Open JavaScript console and look for test output like:
“Ran 43 tests containing 134 assertions. 0 failures, 0 errors.”
@alexmiller I'm updating CONTRIBUTING.md as part of this. Will it break anything if I change "runtests.html" to simply say "Check console for results."?