Got a fairly interesting error with taoensso.nippy last night

when I use it on data returned by jdbc/execute! it tries to serialize a non-serializable class

(def m (chat/messages-since
         system
         {:time (.minus (Instant/now) (Duration/ofDays 1))
          :n 1
          :participants [1 1]}))
2020-11-09 11:26:51,238 [nRepl-session-b5d597e4-c019-4378-829a-5e1bfdb10589] INFO  space.egge.db - Name:Query Logger, Connection:13, Time:10, Success:True, Type:Prepared, Batch:False, QuerySize:1, BatchSize:0, Query:["SELECT * FROM page_message
         WHERE created_at > ? AND
             ((page_id_from = ? AND page_id_to = ?) OR
              (page_id_from = ? AND page_id_to = ?))
         ORDER BY created_at ASC
         LIMIT ?"], Params:[(2020-11-08 11:26:51.2118528,1,1,1,1,1)] 
=> #'space.egge.main/m
m
=>
[#:page_message{:id 2,
                :page_id_from 1,
                :page_id_to 1,
                :contents {:hello "world"},
                :reactions [],
                :created_at #object[java.time.Instant 0x33e82c30 "2020-11-09T02:51:04.526352Z"],
                :updated_at #object[java.time.Instant 0x5aab54c4 "2020-11-09T02:51:04.526352Z"]}]
(type m)
=> clojure.lang.PersistentVector
(require '[taoensso.nippy :as nippy])
(nippy/freeze-to-string m)
=> nil
Execution error (ExceptionInfo) at taoensso.nippy/throw-unfreezable (nippy.clj:1001).
Unfreezable type: class next.jdbc.result_set$navize_row$fn__4663

so something somewhere in the results is hanging on to a reference to that function

using transit everything works fine and I am moving on with the project

but its still an interesting error and I'm curious how it comes about

its the metadata for datafy i think. seems nippy serializes metadata and the metadata includes functions for datafy

oh that makes sense

https://github.com/ptaoussanis/nippy/commit/db71943a5bf3bf958d27776a0f170832002a967e#diff-04c23ef13a320fe16a238924ddb8382fe642b3bce6d765f53bed9c206f70cbafR283 looks like there's a way to prevent that

(nippy/freeze-to-string (mapv #(with-meta % nil) m))
=>
"TlBZAG4BcAdqD3BhZ2VfbWVzc2FnZS9pZCoAAAACahlwYWdlX21lc3NhZ2UvcGFnZV9pZF9mcm9tKgAAAAFqF3BhZ2VfbWVzc2FnZS9wYWdlX2lkX3RvKgAAAAFqFXBhZ2VfbWVzc2FnZS9jb250ZW50c3ABagVoZWxsb2kFd29ybGRqFnBhZ2VfbWVzc2FnZS9yZWFjdGlvbnMRahdwYWdlX21lc3NhZ2UvY3JlYXRlZF9hdE8AAAAAX6iumB9ffoBqF3BhZ2VfbWVzc2FnZS91cGRhdGVkX2F0TwAAAABfqK6YH19+gA=="

you'd probably need to walk it but there is a dynamic var to do this for you so you don't walk twice

setting the binding seems better

(binding [nippy/*incl-metadata?* false]
  (nippy/freeze-to-string m))
=>
"TlBZAG4BcAdqD3BhZ2VfbWVzc2FnZS9pZCoAAAACahlwYWdlX21lc3NhZ2UvcGFnZV9pZF9mcm9tKgAAAAFqF3BhZ2VfbWVzc2FnZS9wYWdlX2lkX3RvKgAAAAFqFXBhZ2VfbWVzc2FnZS9jb250ZW50c3ABagVoZWxsb2kFd29ybGRqFnBhZ2VfbWVzc2FnZS9yZWFjdGlvbnMRahdwYWdlX21lc3NhZ2UvY3JlYXRlZF9hdE8AAAAAX6iumB9ffoBqF3BhZ2VfbWVzc2FnZS91cGRhdGVkX2F0TwAAAABfqK6YH19+gA=="

the more you know i guess

Only the rows will have the metadata so running mapv over it should be sufficient, in case you want other metadata to be preserved.

I found a problem when using next.jdbc . Basically when using (with-transact [tx ds] ...) if I have options on ds that I set with with-options then I lose them. So, for example, here I get a syntax error because I'm not transforming dashes into underscores in the :label/store-id attribute:

(let [ds (->> {:dbtype "postgres" ...}
              (jdbc/get-datasource)
              (jdbc/with-options jdbc/snake-kebab-opts))]
  (jdbc/with-transaction [tx ds]
    ...
    (sql/insert! tx :label
      {:label/name "Beverages"
       :label/store-id #uuid "610d75fa-7aeb-43d3-bcea-a26d1d2042ae"})))

(let [ds (jdbc/get-datasource {:dbtype "postgres" ...})]
  (jdbc/with-transaction [tx ds]
    ...
    (sql/insert! tx :label
      {:label/name "Beverages"
       :label/store-id #uuid "610d75fa-7aeb-43d3-bcea-a26d1d2042ae"}
      jdbc/snake-kebab-opts)))

@franquito That is specified called out and explained in the documentation.

> Note: See the next.jdbc/with-option examples in the Datasources, Connections & Transactions below for some caveats around using this function.

where it says: > Note: Because get-datasource and get-connection return plain JDBC objects (javax.sql.DataSource and java.sql.Connection respectively), next.jdbc/with-options cannot flow options across those calls, so if you are explicitly managing connections or transactions as above, you would need to have local bindings for the wrapped versions: And has examples to show how to handle that.

(with-open [con (jdbc/get-connection ds)]
  (let [con-opts (jdbc/with-options con some-options)]
    (jdbc/execute! con-opts ...) ; committed
    (jdbc/with-transaction [tx con-opts] ; will commit or rollback this group:
      (let [tx-opts (jdbc/with-options tx (:options con-opts)] ; <=== rewrapping tx with the same options from the connection
        (jdbc/execute! tx-opts ...)
        (jdbc/execute! tx-opts ...)
        (into [] (map :column) (jdbc/plan tx-opts ...))))
    (jdbc/execute! con-opts ...))) ; committed

It uses with-options again on the tx.

Ahh, that's better than my approach. Gotta read the docs then 😅

There's a lot of stuff in the docs -- even in just the Getting Started page. Definitely worth reading it all. Several times, probably.

I try to update the docs with each release, adding more and more information, as folks ask questions about usage etc. If you have any suggestions for improving the docs -- or the docstrings -- feel free to discuss here or create issues on GitHub.