This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # aws (2)
- # beginners (57)
- # boot (31)
- # carry (15)
- # cider (9)
- # cljs-dev (9)
- # cljs-experience (32)
- # cljsrn (94)
- # clojure (129)
- # clojure-dusseldorf (3)
- # clojure-greece (4)
- # clojure-italy (8)
- # clojure-norway (3)
- # clojure-russia (344)
- # clojure-sg (39)
- # clojure-spec (2)
- # clojure-uk (39)
- # clojurescript (84)
- # core-async (99)
- # cursive (10)
- # data-science (1)
- # datascript (4)
- # datomic (66)
- # emacs (10)
- # graphql (4)
- # hoplon (28)
- # jobs (15)
- # luminus (3)
- # lumo (5)
- # off-topic (23)
- # om (4)
- # onyx (32)
- # pedestal (24)
- # re-frame (2)
- # reagent (7)
- # ring-swagger (32)
- # spacemacs (4)
- # untangled (57)
- # utah-clojurians (1)
you would need a new channel type if you wanted to get the contents of the chan (unless you want to dump the contents into a collection and send that I guess) - what is the purpose of sending a chan?
I definitely have abstractions where each item coming from another host is decoded by transit and put onto a chan, but I can’t think of why I would want to send a chan, or if I did how I would want that chan to behave
@noisesmith our use-case is we’re getting a chan of db rows from cassandra, and we want to use that as a response body in pedestal
but we need to encode the chan as transit, and have it read back as a collection on the client
the response can be large and it’d be great if we could use the chan as a backpressure medium
@jfntn sure, you can pipe from the channel into a stream that encodes each element as it comes off for example
There’s a hack that works, where we wrap the chan and return a chan of transit strings, we wrap the contents in brackets, and interpose commas, the client reads that back as a vector
the output stream coming from the writer will have the bytes ready as they come if you tie it together properly
hmm need to look into that, think this would have to bridge to java.nio to work with pedestal though
You can pass a custom chan to execute-chan-buffered that has a transducer that does transit encoding of each chunk
I am not familiar with transit, and the particular use case, but there are options for surr :)
@mpenet I happened to stumble upon the issue you opened for pedestal regarding blocking calls in the go-loop, do you think this use case is worth the trouble given the current limitations?
if this is about doing io inside go, definitely always put io writes and reads inside a thread call if you are in a go block
or is this something where pedestal is already doing it wrong and you don’t really get to decide?
It was acknoledged to be a real issue in privmsg on slack, and would be fixed asap, but apparently it s still here
It might be better to turn the results chan into a
ReadableByteChannel like @noisesmith suggested, this would be more performant than my core.async hack, and would sidestep the core.async issue in pedestal :thinking_face:
in fact, code that writes transit directly to your destination will actually be simpler than the code that used strings in many cases - I think it’s intentional that transit doesn’t provide any string conveniences
@noisesmith I’m not familiar with nio, would you happen to know whether I should be looking into encoding to a
ReadableByteChannel or a
ByteBuffer? These are the two streamed and async containers that pedestal supports.
use the one that can most conveniently be the target of an OutputStream - iirc that’s easy with a ByteBuffer
@jfntn actually, taking a quick browse, this might be your ticket https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Channels.html
you can make a readablebytechannel from an inputstream using newchannel, the inputstream can be tied to the outputstream you write to from transit
(I haven’t done this with nio - hope it’s not too tricky but that api looks like it will do it)
@jfntn I would move the a/thread to directly wrap the io call, and park on the result in a go block
that way you get the advantage of using core.async (go blocks that can park) without starving the thread pool
Oh ok, you wait on the values from the chan in a go block, but you start a thread for the write?
So this would have the advantage of not using up a thread unless doing actual i/o right?
(it lets you use <! instead of <!! in the channel take above, which likely takes longer than the individual transit/write calls do)
I still find it really difficult to get a sense for when work should be offloaded to a thread vs right in the go block, esp if things are cpu bound. Say I had a transform fn in there, as in transform before writing, I think in the latest version i’d just apply it before the
transit/write inside the thread, but I wouldn’t be confident that’s the best place to do it
yeah- there’s some ambiguity - but in my experience things that use enough CPU to be a real issue will be pretty rare and I’ll know what they are - and they definitely go in a thread call
like if I’m calculating graph metrics on an adjacency list and I know it’s going to take a few minutes
even the transit write - until you get a network hiccup it’s actually not going to block long enough tobe a problem - it’s just the unpredictable network lag that makes the thread call neccessary
I’d say unless it aproaches a decent fraction of a second - but it’s worth measuring - the symptoms of go being overloaded are noticible
right so if you benchmarked it it’d be faster in the go block, but when things get congested in prod you’re better off with defering to a thread
and it will be a different consideration depending on how many parts of your app are using go blocks
@jfntn yeah - or in my app’s case I profile and I look for long green rectangles in the CPU usage overview inside threads that belong to core.async
I take each one and move them to thread calls, and then I just need to profile at least ~once per major release of the code
One thing I’m not clear on is I used to only have one
transit/read invocation per payload on the ClojureScript side. But with the function we looked at it seems I’ll need some out of band information to decide when to use a different read fn that will wrap multiple invocations of
transit/read in a lazy-seq, is that how you’d do it @noisesmith ?
@jfntn the most elegant thing would be to always return a sequence of results, and the old code would just get a single item
because no matter how many times a single transit was written to, you can always loop until it has no more and put them all in a coll
but really I’d use my best judgment based on how deeply the concept of only one result from transit is baked into the app logic
yeah, but I don’t like using hacks I don’t need, and transit already handles the case the brackets / commas thing was meant to cover
I think the best solution for our re-frame app will be to set a
stream? flag on our remote effect and use that in the interpreter as a signal to use the lazy reader
It will be a while before I can run some full-stack benchmarks to compare the solutions, but I feel like this has been a productive yak shaving session, thanks for your help @noisesmith & mpenet !
@noisesmith starting to look like the ClojureScript reader for transit doesn’t actually support parsing that, have you tried that approach yourself?
I’ve done multiple writes followed by multiple reads… but not between cljs and clj
I'm writing a simple patch for core.async, but I don't know how to run the clojurescript tests.
lein cljsbuild test builds a test.js file, but doesn't seem to execute it. I haven't touched ClojureScript in a couple of years, not sure how to proceed.
I've installed the current version of v8, but still fails:
I do not think that script is actually used now (or at least I haven’t used it recently, maybe David uses it)
lein do clean, cljsbuild once
“Ran 43 tests containing 134 assertions. 0 failures, 0 errors.”
@alexmiller I'm updating CONTRIBUTING.md as part of this. Will it break anything if I change "runtests.html" to simply say "Check console for results."?