Fork me on GitHub
#sql
<
2024-02-07
>
Mark Wardle07:02:01

Hi all. Using SQLite with next.jdbc, and I'm returning namespace qualified keys based on the table name. However, I'm doing a calculation - and using either a join or a view to get it, but get one key without a namespace.

"select * from uk_composite_imd_2020_mysoc a left join (select lsoa,ntile(4) over(order by UK_IMD_E_rank) as UK_IMD_E_pop_quartile from uk_composite_imd_2020_mysoc) b on a.lsoa = b.lsoa where a.lsoa=?"
=>
{:uk_composite_imd_2020_mysoc/lsoa "95ZZ06W1",
 :uk_composite_imd_2020_mysoc/UK_IMD_E_pop_decile 1,
 :uk_composite_imd_2020_mysoc/UK_IMD_E_pop_quintile 1,
 :uk_composite_imd_2020_mysoc/UK_IMD_E_rank 1,
 :uk_composite_imd_2020_mysoc/UK_IMD_E_score 123.00849692191194,
 :UK_IMD_E_pop_quartile 1}
This same result occurs if I create a view and SELECT on that instead. I presume SQLite is not providing enough information behind the scenes to link a column to a table. Is there a way of providing a default, or am I better just using plan, or a non-namespaced key result set builder, and then change the namespaces afterwards?

igrishaev08:02:46

The simplest solution would be to skip namespaces by passing this argument:

{:as next.jdbc.result-set/as-unqualified-maps}
Pay attention that the namespaces are fetched using an extra query. It means, each time you select something, you perform two queries under the hood.

👍 1
rage4 1
Mark Wardle09:02:26

I didn't know about the double fetch thing - I assumed it came in the ResultSet internally but didn't dig into this.

seancorfield18:02:36

> Pay attention that the namespaces are fetched using an extra query That is false.

seancorfield18:02:59

Either the ResultSetMetaData provides that information or it doesn't. e.g., Oracle simply doesn't implement that in its drivers. MS SQL Server only provides it if certain settings are present.

seancorfield18:02:31

Computed columns never provide that information, which is why that column in the OP's query doesn't have a qualifier.

seancorfield18:02:36

For performance reasons, you are better using plan in all cases since that avoids constructing Clojure hash maps from ResultSet objects.

👍 1
Mark Wardle18:02:23

Thanks @U04V70XH6 - I appreciate your comments and advice. I've kept it as simple as possible and delegated giving the namespaced keys into my application rather than next.jdbc which is working well. Thank you.

seancorfield18:02:32

As you can probably tell, I get rather annoyed when people just throw out the recommendation to use the unqualified builders... That's not what people should turn to as a solution.

👍 1
seancorfield18:02:11

If you use plan for performance, the column names are unqualified, because no builder is used -- which is where the performance boost comes from.

Mark Wardle18:02:39

Yes I understand that. I think in my case using plan makes sense given what I'm doing and it makes it explicit. Thank you again.

igrishaev18:02:53

> That is false I haven't checked this case with other databases, but for Postgres, it's true. Fetching column names from the RSMetadata triggers this query:

igrishaev18:02:03

Which returns the following result:

igrishaev18:02:59

One can ensure when running Postgres with -E flag (log everything).

seancorfield18:02:56

Is that specifically for the .getTableName() call or for ResultSetMetaData overall? Does it run it every time the main query is run or cache it as part of the plan or other data about a query?

igrishaev18:02:35

I believe it gets executed once when getTableName is called, and then cached

igrishaev18:02:23

But in fact, every time we select something with f.q. keys, it doubles the number of queries

seancorfield18:02:54

Given the focus on performance from the metosin folks when they were helping with the initial implementation of next.jdbc, I'm surprised they didn't call that out as a problem (but we were all probably more focused on plan which is the recommended approach anyway).

igrishaev18:02:08

Maybe because this query is done under the hood by the driver. To spot it, one should enable logging all the expressions with the -E flag. I always set it to true and observe in the Docker console all the expressions executed. Often, there are some really weird ones produced by the drivers.

seancorfield18:02:58

Looking at the MySQL driver, it seems that all that information is pulled directly as part of what is needed to support even calling .getValue() on a row in a ResultSet -- it uses the field definition which has the table name and column type information all baked in. So I'm guessing different drivers handle this very differently...

seancorfield18:02:38

(so if MySQL ends up running multiple queries, it isn't .getTableName() that triggers it as far as I can tell from the source code)

igrishaev18:02:40

It looks like MySQL passes the names of the columns that participate in the result. But Postgres doens't. They only pass OIDs: oid of the type, of the table, and the column. To resolve the name of the tables, one should query the pg_catalog table

igrishaev18:02:21

It's out of the scope of the origin question as it's about SQLite, but just in case

seancorfield18:02:44

Interesting... I'll create an issue to add notes to the Tips & Tricks docs and maybe some of the mentions of qualified columns names for PG's behavior. Definitely not what I would have expected, given how PG prides itself on being so advanced 😉 where other DBs / drivers can handle this without the extra query 🙂 TIL.

igrishaev18:02:11

yeah... another problem with Postgres is enum types. All the standard types have hardcoded OIDs so they're defined in the code. But when someone defines their own enum, it gets a random oid, and on the driver's level, it's unclear what is it.

igrishaev18:02:43

you only have type_oid = 8883, no name.

seancorfield18:02:20

Ouch. So user-defined enums have a performance overhead there?

seancorfield18:02:55

PG weirdness is the bane of my life, both with next.jdbc and with HoneySQL! 🙂

igrishaev19:02:15

No I just meant it's unclear how to parse these unknown types. Having the name of the type would be great, and the client could provide a mapping like <type_name = parsing function>.

igrishaev19:02:51

but we have integer oids which might differ depending on the server

k3nj1g08:02:37

I need to process a large set of results. I use jdbc/plan for this. Like this:

(require '[next.jdbc :as jdbc] '[next.jdbc.result-set :as rs])

(with-open [db-conn (raw-connection ctx)]
  (run! (fn [row]
          (let [result (-> (select-keys row [:jsonb-field])
                           (update :jsonb-field #(-> % (str) (json/decode keyword))))]
            (some-side-effect! result)))
        (jdbc/plan db-conn query-sql {:builder-fn rs/as-unqualified-lower-maps})))
As you can see, one column is json. So I parse it with cheshire. Is there a more convenient way to get the result set in the form of clojure structures for this kind of scenario?

igrishaev08:02:02

Is it Postgres? If yes, you need to extend some protocols and types for automatic conversion. See this link: https://github.com/seancorfield/next-jdbc/blob/44b3cc206fe4635702331814da0860e63261e9ec/doc/tips-and-tricks.md?plain=1#L364

k3nj1g10:02:05

It's Porstgres, right. Thanks for the tip, I'll take a look.

James Amberger17:02:27

So—if I’m stuck on Oracle, the nice things next.jdbc does with Datafy/Nav basically don’t work or don’t work out of the box because of the unqualified keywords?

seancorfield18:02:55

Oracle's JDBC drivers do not provide table name information (as noted in the Oracle-specific Tips & Tricks page). Nothing next.jdbc can do about that, unfortunately.

James Amberger18:02:15

and, just reading up on datafy/nav basically for the first time, but it seems to me your implementations of nav would rely on that, wouldn’t they?

seancorfield18:02:37

It depends on your FK naming conventions, not the qualifiers.

seancorfield18:02:10

If table foo has bar_id, then datafy/`nav` assumes that will join to table bar column id.

seancorfield18:02:22

If you need to map table/column pairs in order to define a custom :schema mapping, then no it won't work without qualifiers. But if you have been consistent in your column naming conventions, it should all work.

James Amberger18:02:24

(too bad they are not my conventions 😞 )

seancorfield18:02:30

What are your conventions? The latest next.jdbc allows you to specify more patterns.