Fork me on GitHub
#sql
<
2022-04-06
>
kwladyka21:04:48

next.jdbc question: Am I right how this fns work? (connection/->pool ...) - the same as (jdbc/get-connection ds) but keep many connections (jdbc/get-connection ds) - open 1 connection and keep it open, if closed then re-open (get-datasource ...) - open and close 1 connection for each query

seancorfield21:04:33

Does this section of the docs not explain it well enough? https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.2.780/doc/getting-started#connection-pooling Do you understand what connection pooling is?

kwladyka21:04:17

well yes, but didn’t have opportunity to code pooling myself before

seancorfield21:04:34

connection/->pool produces a datasource that manages connections for you. You can call get-connection on it (as long as you also close the connection -- with-open is your friend there) but you can also just pass the (pooled) datasource directly to execute! etc (and next.jdbc will get a connection from the pool, use it, and return the connection to the pool).

seancorfield21:04:21

The pool keeps some connections open/warm so that you can get/use/close (return) connections quickly without the overhead of standing up a new connection to the database each time.

kwladyka21:04:02

(jdbc/execute-one!
    (connection/->pool com.zaxxer.hikari.HikariConfig
                       {:jdbcUrl d})
    ["SHOW timezone"])
Execution error (IllegalArgumentException) at next.jdbc.protocols/eval3855$fn$G (protocols.clj:14).
No implementation of method: :get-datasource of protocol: #'next.jdbc.protocols/Sourceable found for class: com.zaxxer.hikari.HikariConfig
> you can also just pass the (pooled) datasource directly to execute! etc What I miss?

seancorfield21:04:30

First off, there's no point in creating a pooled datasource and just using it once -- that's going to be slower. You need to keep that pooled datasource around and pass it into each next.jdbc call.

kwladyka21:04:02

> First off, there’s no point in creating a pooled datasource and just using it once it was for testing purpose

seancorfield21:04:36

Second, I'm pretty sure that should be com.zaxxer.hikari.HikariDataSource -- I'll have to see how that incorrect reference to HikariConfig got into the docs (the examples that follow that all use HikariDataSource)

kwladyka21:04:30

(def pool (connection/->pool com.zaxxer.hikari.HikariConfig {:jdbcUrl d}))
=> #'crypto-trading.db.integrant/pool
(jdbc/execute-one! pool ["SHOW timezone"])
Execution error (IllegalArgumentException) at next.jdbc.protocols/eval3855$fn$G (protocols.clj:14).
No implementation of method: :get-datasource of protocol: #'next.jdbc.protocols/Sourceable found for class: com.zaxxer.hikari.HikariConfig
`

kwladyka21:04:15

thank again

kwladyka21:04:40

it is midnight for me, sorry

kwladyka21:04:16

That is why I couldn’t make it work

seancorfield21:04:15

Yup, it was a typo back in 1.2.737 and no one has spotted it until now. Fixing...

kwladyka21:04:09

(let [log-ds (next.jdbc/with-logging d println println)]
    (jdbc/execute! log-ds ["SHOW timezone"]))
Syntax error compiling at (src/crypto_trading/db/integrant.clj:1:195).
No such var: next.jdbc/with-logging
https://cljdoc.org/d/com.github.seancorfield/next.jdbc/1.2.780/doc/getting-started#logging Why? In fact I don’t see with-logging in next.jdbc.

seancorfield21:04:15

Make sure you're not pulling in an older next.jdbc as a transitive dependency.

kwladyka21:04:15

1.2.659 I am curious why clj -M:outdated --upgrade doesn’t upgrade it higher

kwladyka21:04:34

:outdated {:extra-deps {com.github.liquidz/antq {:mvn/version "1.6.0"}}
                      :main-opts ["-m" "antq.core"]}}

seancorfield21:04:54

No idea. I don't let tools modify my deps.edn. I get the report and do it manually.

seancorfield21:04:04

What's in your deps.edn?

kwladyka21:04:55

seancorfield/next.jdbc {:mvn/version "1.2.659"}
in project
:outdated {:extra-deps {com.github.liquidz/antq {:mvn/version "1.6.0"}}
                      :main-opts ["-m" "antq.core"]}}
in global deps

kwladyka21:04:19

> Could not find artifact seancorfield:next.jdbc:jar:1.2.780 in central (https://repo1.maven.org/maven2/)

seancorfield21:04:29

com.github.seancorfield/next.jdbc

kwladyka21:04:56

ah…. thank you

kwladyka21:04:36

I am curious how many other deps I use like that…

seancorfield21:04:37

I'm surprised antq didn't spot the group ID change -- it's been pretty good about reporting those for me. Clojars changed their group name policy about a year ago and several libraries changed their group IDs to reflect that. I double-published for a while and then announced I was going to stop double-publishing.

seancorfield21:04:06

The README explicitly calls that out BTW:

The documentation on GitHub is for develop since the 1.2.780 release -- see the CHANGELOG and then read the corresponding updated documentation on GitHub if you want. Older versions of next.jdbc were published under the seancorfield group ID and you can find older seancorfield/next.jdbc documentation on cljdoc.org.

kwladyka21:04:26

yes I remember this policy change, but some libraries changed namespace, some not…. not easy to follow

seancorfield21:04:37

Read the README? 🙂

kwladyka21:04:47

each of library? 🙂

seancorfield21:04:02

(you should always read the readme and changelog whenever you are considering updating a library)

kwladyka21:04:39

yes, but I use clj -M:outdated --upgrade to detect if there is something to update

seancorfield21:04:38

Which is a bad idea. Because it will change things without giving you any idea of what changed in a library.

seancorfield21:04:58

It could easily upgrade a library to some incompatible version.

kwladyka21:04:10

I mean I use it to detect and then I can read vs read all deps readme and then update.

kwladyka21:04:56

the most important is: with-logs works now! 🙂

seancorfield21:04:18

The change to the coordinates was documented in 1.1.643, BTW:

1.1.643 -- 2021-03-06

Change coordinates to com.github.seancorfield/next.jdbc (although new versions will continue to be deployed to seancorfield/next.jdbc for a while -- see the Clojars Verified Group Names policy).

seancorfield21:04:56

Anyways, glad you have it working now. Also, please stop using depstar and switch to tools.build (per the depstar README) 🙂

👍 1
kwladyka21:04:01

I know, I should notice it, but I didn’t.

seancorfield21:04:39

(just in case you're relying on antq to update depstar and hadn't noticed that it is deprecated/archived now 🙂 )

kwladyka21:04:21

this one I noticed

1
kwladyka21:04:44

the last question: Can I somehow preview for debug purpose the final raw SQL for

(jdbc/execute! log-ds ["SELECT ?::timestamptz" "2022-01-01 00:00:00"])

kwladyka22:04:58

not very important right now but today I wanted to know what is the final form of Instant in SQL query. Finally I found it in Postgres logs. So I could see if the query contain 2022-04-04 02:00:00+02 vs 2022-04-04 00:00:00+00.

kwladyka22:04:31

Any good practice to make it UTC always and always? I see there is :serverTimezone for jdbc-url but I am not sure about the effect and don’t know if it will work with pool.

isak22:04:10

Does it need to be a parameter? Can you just do

now() at time zone 'utc'
so the db will resolve it?

seancorfield22:04:47

Re: raw SQL actually sent to the DB -- no, JDBC doesn't make that available.

seancorfield22:04:33

Re: timezones -- my recommendation is always to have your database and the server it runs on and your JVM and the server that runs on all set to UTC timezone.

kwladyka22:04:47

I can but then I will have to write at time zone 'utc' in each query.

kwladyka22:04:34

Re: timezones -- my recommendation is always to have your database and the server it runs on and your JVM and the server that runs on all set to UTC timezone.
Sure. I am looking what I can do more, but maybe this is enough.

isak22:04:03

That depends on your setup - some people run the dbs on developer laptops

kwladyka22:04:36

I use docker-compose for third party software deps, so not problem here

seancorfield22:04:50

FWIW, we always have &characterEncoding=UTF-8&serverTimezone=UTC in our JDBC URLs at work (for QA/production -- for dev/CI we omit serverTimezone and rely on local time since we don't control the TZ for those "servers").

👍 1
kwladyka22:04:36

But I am writing crypto currency bot which will play this game with my own money and if somehow something will break about timezone everything can end very bad

kwladyka22:04:36

Is it possible to set {:builder-fn jdbc-rs/as-unqualified-maps} on pool or by default?

kwladyka22:04:29

Thank you for help. I am going sleep. Goodnight.

seancorfield22:04:50

@kwladyka Look at with-options in the docs -- but also the caveats around calling get-connection and with-transaction on that (since those unwrap the options and you need to rewrap them). Although, I strongly recommend trying to get used to and work with the qualified keys in what next.jdbc returns -- and only switching to unqualified keys if you need to expose that data to some external system.

👍 1
kwladyka22:04:54

I interpret it as (jdbc/with-options pool {:builder-fn rs/as-unqualified-lower-maps}) work also with pool. Once again thank you. Goodnight.

kwladyka22:04:22

Great everything works perfect.