Clojurians Log v2

btw you can opt out of the clerk cache by setting the clerk.disable_cache system prop to a value that isnt false

Thanks for the tip. My current notebook contains "training of a model", which is slow. So using Clerk at all only makes sense, if caching is enabled. For this concrete issue, "cleaning the cache ones" is good enough as work around. Nevertheless I think we need a in-memory cache, even maybe as default. It seems to me that the nippy frezze / unfreeze has many issues and should not be the default.

that’s a strong statement 🙃

it does work incredibly well with regular Clojure data

there is an in-memory cache, I just need to fix an issue where it’s not used for things that aren’t nippy freezable which I’m doing right now

@mkvlr an idea occurs to me about this issue, which I have also faced: could metadata be used to allow Clerk users to annotate values with their own caching functions (analogous to custom viewers)? certainly ordinary Clojure data is easy to persist using Nippy, but if Clerk's caching mechanism were extensible to other disk-backed formats (e.g. CSV/Arrow/etc) then you could potentially cache to a file format that makes sense for bigger things like TMD datasets (or images, etc).

@afoltzm yes! Something @jackrusher has also mentioned and definitely on the roadmap.

that's a strong statement :upside_down_face:

on my issue with TDM

class [D cannot be cast to class [Ljava.lang.Object; ([D and
   [Ljava.lang.Object; are in module java.base of loader 'bootstrap')

Does this means that "something" (Nippy ?) converts object arrays into double arrays or the other way arround? Or are we still see a "rendering" issue in Clerk ? It goes away when cleaning caches (but comes back on next JVM restart)

do you have a small repro of the above error?

https://github.com/behrica/kaggleHP

doing: 1. (clerk/show! "src/kaggle.clj") 2. JVM restart 3. (clerk/show! "src/kaggle.clj")

should trigger it

thanks, I’ll try…

btw, you’re still mostly working with the file watcher, right?

can highly recommend trying with a hotkey for clerk/show! https://github.com/nextjournal/clerk#editor-workflow

Regarding "configurable caching": I have my TMD datasets often nested inside a map. This seems to make specific caching via annotations pretty hard. Viewers are simpler, as we control well what to view, while data structures are often just big maps. An other way to say this: The caching should be IMHO completely invisible even if this means that persistent caching (across restarts) is not possible. To wonder about the viewers is quite some work already, to wonder about "how to store objects" should be avoided. (specially as I have not yet a use case for the persistence)

No, I went to hot key in emacs. Very good, indeed ! "C-c c" does all of it:+1:

> The caching should be IMHO completely invisible even if this means that persistent caching (across restarts) is not possible. that’s how it works now or am I misunderstanding?

(well besides the bug you’re encountering 😹)

do you know which cell it is that causes the failure?

seems to be this one (def test-data\n (load-hp-data \"test.csv.gz\"))

this seems to be a more minimal repro

(ns kaggle-min
  (:require [nextjournal.clerk :as clerk]
            [tablecloth.api :as tc]))



(defn load-hp-data [file]
  (println "load a file : " file)
  (-> (tc/dataset file {:key-fn keyword})

      (tc/convert-types (zipmap [:BedroomAbvGr
                                 :BsmtFullBath
                                 :BsmtHalfBath
                                 :Fireplaces
                                 :FullBath
                                 :GarageCars
                                 :HalfBath
                                 :KitchenAbvGr
                                 :OverallCond
                                 :OverallQual
                                 :MoSold
                                 :TotRmsAbvGrd
                                 :MSSubClass]
                                (repeat :string)))))

(def df (load-hp-data "train.csv.gz"))

(defn ->table [df]
  (clerk/table {:head (tc/column-names df)
                :rows (tc/rows df :as-seqs)}))


;;  # The data
^{::clerk/width :full}
(->table df)

In my personal experience as a data science practitioner, I'd say if you're working with models and datasets that are expensive to retrain/recompute, you probably are going to need to think explicitly about how serializing to disk fits into a workflow at some point. I don't see why Clerk asking its users to be explicit about custom caching is really that different than a lot of work that's already a big part of the data science lifecycle. I think Clerk is a very useful library, but expecting it to work well with heavyweight computation like that without user-supplied configuration may be asking too much of it. My personal preference is for flexible configuration via programming that may require more upfront effort than configuration via settings that is more brittle.

That is for sure true. There is a moment where exploration becomes engineering, and then Clerk is the wrong tool. But this line can be dynamic, and I think Clerk should support the use case where single form computations maybe takes 1 minute. And that minute I only want to wait if really needed (after changes of relevant code). And I think (and it seems so), that Clerk can bring that via caching.

The issue above is clearly related to caching. When I disable nippy cache, it goes away.

@mkvlr I found the root cause of thge exception

Unhandled java.lang.ClassCastException
   class [D cannot be cast to class [Ljava.lang.Object; ([D and
   [Ljava.lang.Object; are in module java.base of loader 'bootstrap')

it happens when Clerk calls on a dataset:

(#{:nextjournal/missing} df)

Inside the loop of all predicates here: https://github.com/nextjournal/clerk/blob/120b1223cfe6a37a12ce6e13bdbe11fabc6161dc/src/nextjournal/clerk/viewer.cljc#L302

The same exception hppens doing this:

(clerk/eval-string "df")

So does this mean that the nippy caching changes somehow the dataset object and makes it "inconsistent" or something like this ?

It is not a problem of Clerk. https://github.com/techascent/tech.ml.dataset/issues/287