This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-04-09
Channels
- # announcements (26)
- # asami (1)
- # babashka (7)
- # beginners (51)
- # calva (1)
- # cider (18)
- # clj-kondo (6)
- # clj-together (1)
- # cljsrn (4)
- # clojure (78)
- # clojure-australia (3)
- # clojure-europe (25)
- # clojure-losangeles (1)
- # clojure-nl (2)
- # clojure-serbia (4)
- # clojure-spec (23)
- # clojure-uk (19)
- # clojurescript (23)
- # community-development (13)
- # deps-new (1)
- # emacs (24)
- # figwheel-main (5)
- # fulcro (61)
- # graalvm (1)
- # honeysql (11)
- # jobs (1)
- # lsp (7)
- # malli (4)
- # meander (6)
- # membrane (3)
- # off-topic (10)
- # polylith (13)
- # quil (7)
- # re-frame (54)
- # reagent (18)
- # remote-jobs (2)
- # shadow-cljs (81)
- # sql (16)
- # tools-deps (12)
- # xtdb (10)
I just asked a question in #clojure that probably should have been here, but there's too much content to paste this time.
It's an issue I'm having with next-jdbc and streaming.
I'm dipping my toes in the 'next-jdbc' waters, and ran into a small issue. It's probably a gap in my knowledge of streaming, but here goes.... [9:01 AM] I have a query using 'plan' that then can be reduced 'into []' with no issue.. it's a huge result set, which is what I expect. [9:02 AM] But when I try to reduce that into something streamable, I'm getting an empty result.. [9:02 AM] So...
[9:03 AM] (ring-io/piped-input-stream
(fn [ostream]
(let [^java.io.Writer w ( ostream {})]
(get-big-data-streaming w))
(.close ostream)))
[9:03 AM] This is in the handler ☝️:skin-tone-2:
[9:03 AM] And the function of interest:
[9:04 AM] (reduce (fn [^java.io.Writer w record]
(let [m (datafiable-row record *db* {})]
(.write w m)
))
writer
(jdbc/plan *db* sv))
[9:05 AM] We can safely assume that `sv` is a sqlvec that works and generates a large dataset outside of this streaming code
[9:05 AM] So the 'w' passed in is the Writer and locally is named `writer`
[9:07 AM] Hitting the endpoint takes about 4 seconds, the same as the non-streaming version, but returns no data.Well, the copy/paste was easier than I thought it would be.
@jmckitrick the function f
supplied to reduce
has to return a value to be passed on, but I have a feeling this bit here
(fn [^java.io.Writer w record]
(let [m (datafiable-row record *db* {})]
(.write w m)
))
doesn’t actually return anything after the first call?You're right, I missed that. But once I fixed it, I'm still getting the empty result....
(fn [^java.io.Writer w record]
(let [m (datafiable-row record *db* {})]
(.write w m))
w)
I think you already know this, but if reduce is working fine in the non-stream version, then if you manually write something to the stream, does it work as intended?
Well, when I output a debugging line after the .write
it is apparently only reached once.
@jmckitrick your original call to .close
is redundant fwiw, that's done automatically.
@jmckitrick What does datafiable-row return? The fn name makes it sound like a map or something, but it should be a string/char/int
I think that's my issue. I wrapped it in 'str' in the call to '.write' and that seems to have done the trick....
@jmckitrick consider setting up https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions as it will help reveal these errors
Ah, great! Thanks@
Actually, in this case, I'm lying. ring-io uses future
for it's threads, that means they will hold onto their exception until deref'd. A little annoying.
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html#get() docs this behaviour. You'll have to do your own try/catch to make sure you have visibility into these fns, or rewrite the piped-input-stream to just use a threadpool instead.