Fork me on GitHub
#sql
<
2020-04-17
>
salokristian06:04:25

@vachichng I think go channels are best used with non-blocking libraries (which next.jdbc is not, and I don't know a Clojure SQL client that is) or for CPU-bound tasks, which this is not. https://eli.thegreenplace.net/2017/clojure-concurrency-and-blocking-with-coreasync/ is a great read on the subject.

Vachi07:04:20

then maybe Pulsar? it's a Fiber library. http://docs.paralleluniverse.co/pulsar/

salokristian09:04:09

Yeah, I think I'll look into it. It looks promising.

nikolavojicic18:04:07

Use async/thread instead async/go for blocking operations. Channels work the same for both blocking and non-blocking operations (i.e. ! and !!). Core.async works great with next.jdbc.

seancorfield16:04:16

If you treat next.jdbc/plan as a producer for a channel of results, that's reasonable. Channel consumption will automatically create back pressure on reduce-over-`plan` as it streams the result set onto the channel. It's trickier if you want a way to terminate early (and return reduced) since you need a way to tell the reduce to stop but it will be (deliberately) blocked trying to put data on the channel.

jonpither16:04:13

Howdy @seancorfield. Do you ever find yourself wanting a clojure keyword mapping from/to java.sql.Types? I.e. to map to/fro java.sql.Types/VARCHAR to :varchar

jonpither16:04:37

Had a scan around next.jdbc and clojure.java.jdbc and couldn't see such a mapping

dcj16:04:54

I implemented https://www.bevuta.com/en/blog/using-postgresql-enums-in-clojure/ in next.jdbc , now trying to extract the defined enum types out of postgres so they are not hard-coded into the coercion code....

seancorfield16:04:26

@jonpither I do run into situations where I have a keyword and want to store it as a string (via name usually) but not often enough that I'd want an automatic conversion: I'd rather have it fail in the cases where I didn't expect to get a keyword (and that has, indeed, uncovered several bugs for me in the past).

seancorfield16:04:23

In general, I tend to find keywords in Clojure may get mapped to ENUM in SQL (MySQL) so having an explicit conversion is safer. Overall, I prefer explicit conversions to/from SQL than global implicit ones.

seancorfield16:04:59

(I don't even leverage the next.jdbc.date-time auto-conversions from Java Time to SQL date/timestamp)

jonpither16:04:14

thanks @seancorfield

emccue20:04:26

@salo.kristian Have you/are you using connection pooling yet?

emccue20:04:34

Kinda obvious but I did some testing locally

emccue20:04:54

(time
  (doseq [_ (range 100)]
    (with-open [conn (jdbc/get-connection (jdbc/get-datasource db-spec))]
      (jdbc/execute! conn ["SELECT * FROM USERS"]))))
"Elapsed time: 7557.7078 msecs"
=> nil
(time
  (doseq [_ (range 100)]
    (with-open [conn (jdbc/get-connection datasource)]
      (jdbc/execute! conn ["SELECT * FROM USERS"]))))
"Elapsed time: 282.6531 msecs"
=> nil

emccue20:04:37

(time
  (doseq [_ (range 10000)]
    (with-open [conn (jdbc/get-connection datasource)]
      (jdbc/execute! conn ["SELECT * FROM USERS"]))))
"Elapsed time: 2021.8485 msecs"

emccue20:04:56

you might come pretty close to your perf requirements with just that

salokristian09:04:52

I haven't yet done any tests, since I've mainly been familiarizing myself with the alternatives. However, I have a HikariCP connection pool already setup.

salokristian09:04:25

How large a connection pool did you use for your tests? They look very promising.

emccue17:04:11

just the default one, 10 active connections

emccue17:04:59

(in my case at least)

seancorfield20:04:24

@emccue It never occurred to me to even ask that question -- good point! I just sort of assume that anyone who cares about performance is already making sure they use connection pooling and don't try to stand up a new connection for every query. 🙂