Clojurians Log v2

yes, hiding code is fine. It goes then into writing "tutorials" for others, with hiding parts of the code. In notespace we can do this individually for expressions, but as well globally

I was just trying to load a large file, 287 MB csv. In teh repl, this just work in 2 seconds. In notespace as well, as long a I don't visualize it in the browser. In Clerk I cannot prevent it's visulaisation (except with a workarround),

so the browser crashes.

The work arround is this:

(def _
  (def large-ds
    (->
     (tc/dataset "/home/carsten/Dropbox/sources/scicloj-data-science-handbook/Reviews.csv"))))

Therefore I am still wondering, if the default behaviour of Clerk to "render" a "def ..." can scale up well.

None of the other notebooks does this. Jupyter, notespace, gorilla and others don't render the variable defined in a "def"

^:clerk/no-render as metadata might make sense, similarly to the no-cache option, for hiding code

yes, sounds good.

Indeed only a few expressions would need this.

(Nevertheless It would be different then in other notebooks, but that would be ok for me)

I just see the reasoning behind the decision....

Caching...

so I should not use a def for this case, right ?

I should make a method with defn, which loads the data and rely on caching

(defn get-large-ds []
  (tc/dataset "/home/carsten/Dropbox/sources/scicloj-data-science-handbook/Reviews.csv"))
(-> (get-large-ds)
    ds/row-count)


(-> (get-large-ds)
    ds/shape)


(-> (get-large-ds)
    ds/column-names)

@carsten.behring the Clerk viewers should only render a subset of the data, so what your seeing is most definitely a bug I want to fix. Note that this currently doesn't apply to the table viewer which currently loads all data. Fixing that is also very high priority.

the reasoning for showing the contents of a var is seeing the contents, which I do believe is helpful. Obviously only once the issues you're seeing with large data are fixed. Still considering showing the var alongside with the contents though.

I also think I can put some safeguards in place to prevent too much data to be transmitted to the browser.

We have worked quite a lot wit large data structures in notebooks. "preventing" and "safeguards" seem to be a tricky business to me, and has a risk of failure in special cases.

I have a hard time with Cider repl as well, that sometimes it tries to print my large data structures, which makes Emacs crash.

Yep, I understand. The safeguard I have in mind is not printing the value of the var if it exceeds a certain size. @thheller does this successfully in shadow inspect https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/remote/runtime/LimitWriter.java

@thheller are you aware of any cases where the limit writer approach fails ie shadow inspect crashes the browser?

The limiting writer looks good to me. The idea of limit by byte seems (more) robust then "counting of vector lengths" or similar. The limit should be configurable, to account for large clj files with lots of sexp vs small file with a few sex. In Cider I saw the "limit" always as a measurement to "avoid user mistake". Because in practice I don "want" to see several pages of output on the screen. It should only happen (in Cider), if I do a mistake and evaluate something by mistake.

In Clerk it's a bit different, as we cannot limit as a user , so the system limit need to work in "all cases". But "byte counting" seems to be the correct approach.

no known issues with limit writer. well except that it very dumb and usually cuts stuff off mid structure. a smarter one that doesn't truncate keywords or so would be nice but its usually not an issue

Did the viewer for svg disappear ? It's used in the demo, but I cant find it ?

@carsten.behring it’s not there yet, still on the list. It was in my talk, but only as a picture. 🤫

I'd like to report something minor about clerk. I wanted to try from Windows. It launched ok until this point:

user=> (clerk/show! "notebooks/hello.clj")
Execution error (AssertionError) at rewrite-clj.node.whitespace/newline-node (whitespace.clj:89).
Assert failed: (string-of? s r/linebreak?) 

Hiya @ales.najmann, this was a problem in rewrite-clj v0, but should work in rewrite-clj v1. Are you using rewrite-clj v1?

@ales.najmann oh but it’s working for you on windows? I also got reports that it doesn’t work at all with this stacktrace https://gist.github.com/mmwaikar/007cb6f76798527c76d78cba75b3b9f2

btw I'm fan of Nextjournal. I'm not a data scientist but I like playing around with it. Check my dashboard here if you like https://nextjournal.com/littleli/homomorphic-encryption-in-clojure

Makes sense. Do you think being able to hide just the code is enough and the result should always stay visible? I'm leaning towards yes then we can use that as an indicator so folks and show the code cell again. Becomes harder when we hide both.