Fork me on GitHub
#sql
<
2018-04-27
>
joefromct21:04:59

hi, I'm trying to query a large resultset onto an async channel so my :row-fn looks something like #(>!! db-chan) , what would be a good :result-set-fn ? I am using j/query and wondering if i have to get lower level to (patiently) stream out a lot of results.

joefromct21:04:15

obviously do-all would be greedy, and i think even vec also

joefromct21:04:36

and i'm afraid anything else might close the cursor if the chan is blocking (temporarily)

seancorfield22:04:33

@joefromct Take a look at reducible-query. I think that might be a better option.

joefromct22:04:21

@seancorfield sure thing, thank you

seancorfield22:04:46

@joefromct FYI, you do not need with-db-connection there -- you can pass db-spec directly to query.

seancorfield22:04:18

Did you look at reducible-query? It's intended for lazily read large streams of data from the DB.

joefromct22:04:45

yeah i thinki have somehting that replaces most of ^ thing

joefromct22:04:08

I'm still playing with it, i think i'm going to toy with my partitioning in the channels and the insert-multi!

joefromct22:04:12

it's pretty cool though

joefromct22:04:41

maybe async is overkill, i just like t he idea of being able to inject other functions into the xf in that pipeline i guess

joefromct22:04:55

and this might have kafka involved at some point

joefromct22:04:15

i basically am looking for the best way for me to move data from one db to another sorta "streaming" with multiple threads on the insert side

joefromct22:04:28

i am probably going to stick with one reader though not sure