Clojurians Log v2

Hey 👋 Sorry for the long wait. Clerk is finally ready for testing! See https://nextjournal.com/mk/clerk-preview for how to get started.

I still would like to polish it a bit and open source it then. In the meantime https://clojars.org/io.github.nextjournal/clerk. The quickest way to try it out is to clone the http://github.com/nextjournal/clerk-demo.

this is so incredibly simple and cool, can’t wait to try it out! one obvious use-case would be to use this for documentation, examples and tutorials.

There are a bunch of developers around here who have uploaded example/starter repositories on github and are interested in teaching and guidance to some degree or another. I think this might be a very valuable tool for these use-cases.

yeah, I look forward to using this for docs

not yet documented but Clerk also supports generating a static html bundle, here’s an example https://storage.googleapis.com/nextjournal-cas-eu/data/8VwPbq6xWQadigCghK7KH5Q3WhUdCNU1d3LCgBnDNUJPyNH1Qw1JfDGFcBmCodHTY8rQDZDTS6UobY3dTLw4suxpGK

https://storage.googleapis.com/nextjournal-cas-eu/data/8VwPbq6xWQadigCghK7KH5Q3WhUdCNU1d3LCgBnDNUJPyNH1Qw1JfDGFcBmCodHTY8rQDZDTS6UobY3dTLw4suxpGK#/notebooks%2Frule_30.clj being my favorite demo of custom viewers

@mkvlr Github doesn't allow you to embed this kind of stuff directly in a README right, so you would have to use Github pages for docs I guess?

Perhaps cljdoc can play a role here

@borkdude yep, Clerk should obviously generate its own homepage 😼

never used github pages, but I was under the impression that it doesn’t do a build-step and you are supposed to check in the compiled (html) target?

yeah, I think that would be the way to go

I tried it quickly and first impression was good.

I put it into docker now, and was wondering which port is needed for websocket to work ? I get the 7777 forwarded, but I hnk I need to open as well for websocket

@carsten.behring websocket is using the same port and address. You should see the connection code in the page source const ws = new WebSocket(document.location.origin.replace(/^http/, 'ws') + '/_ws')

would appreciate a RT https://twitter.com/mkvlr/status/1446425912321392640

I saw that

(def x (make-very-large-map)) 

@carsten.behring are you sure? it should only render the first 20 elements. If not, it's a bug. Not at the computer atm so I can't check.

Ok , I have not tried with a normal "map". I have used with a "dataset", which is a custom deftype, representing a columnar table. It implements a lot of standart Clojure interfaces, so in this sense it is a map. https://github.com/techascent/tech.ml.dataset/blob/a730e652117bb538e9ba5b4513a3a0cb29384765/src/tech/v3/dataset/impl/dataset.clj#L125 It is "big".... I would never think about "printing" it in a browser. So doing "(def x .....) of one of it (reading it from a csv file, for example) will try to show it in the browser. It behaves ok in a repl, because it implements some stuff to be repl friendly

ok, in that case it's not a map in the map? sense? This should be a good example for a custom viewer, will look into this when I'm back at my computer later

It is a "map?"

It implements java.util.Map

I tired a custom viewer.

But what should be the predicate ?

We have a clojure fn "dataset?", but that does not run in sci.

In "reality" I want to have this result in browser, for example:

(->
 (tc/dataset "data/AOP non dogmatic. KE4dev5 210928.xlsx" {:key-fn csk/->kebab-case-keyword})
 (tc/column-names data))

so just the column names.

But usualy, I define the data instance first via def

(def data
  (tc/dataset "data/AOP non dogmatic. KE4dev5 210928.xlsx" {:key-fn csk/->kebab-case-keyword}))

(->
 data 
 (tc/column-names data))

I will try some more things. It is for sure an edge case (large data structure, but implementing most clojure standard interfaces)

Currently it "hangs"

I'll take a look in a bit

Thanks.

@carsten.behring do you have a minimal repro for me maybe? deps.edn + a code snippet?

deps.edn: org.clojure/clojure {:mvn/version "1.10.3"} io.github.nextjournal/clerk {:mvn/version "0.1.164"} scicloj/tablecloth {:mvn/version "6.012"}

notebook:

(require '[tablecloth.api :as tc] '[tech.v3.dataset.impl.dataset :refer [dataset?]] '[tech.v3.libs.fastexcel] '[nextjournal.clerk :as clerk] '[camel-snake-kebab.core :as csk]) (def data (tc/dataset "data/AOP non dogmatic. KE4dev5 210928.xlsx" {:key-fn csk/->kebab-case-keyword})) (-> data (tc/column-names data))

the file to load can be any csv or xlsx file

(inlining 'data' will make it work)

as then the data is never printed in browser

@carsten.behring thanks, seems to work ok here, notice that the values are only partially loaded

So had a quick play. All the notebooks render nicely for me except how-clerk-works which had no access to next.jdbc and weavejester dependency. Added some likely looking lines to deps.edn- this leads to SQL error or missing database (no such table: tracks)

Commenting out 59-63 fixes the issue at the cost of losing the query-results function

@adwelly thanks! Just pushed https://github.com/nextjournal/clerk-demo/commit/c5358a6a372df6f9e93f97e7274ce2d06f81b754

the rendering error with the dependency graph is something I still need to look into, now need to enjoy a bit of the ☀️

Hi, I'm enjoying Clerk very much! 🙏 Also, the code is nice and enlightening. One question, looking at the code: I see that nextjournal.clerk.webserver/broadcast! can be used to update all clients with all the values to be viewed. Is it possible to broadcast a partial update, without passing all the values? This can affect the experience when some values are big data structures. (In Notespace we handle it by broadcasting not the actual values but rather some corresponding ids. Then, the client can check which ids are new, and request the server for the corresponding actual values.) (Another approach is representing everything as Datascript transactions. The Literate tool by @pedrorgirardi does that, and it is amazingly simple and elegant. https://github.com/pedrorgirardi/literate .)

@daslu great to hear, thanks! Regarding the broadcasting of large values: for results Clerk doesn’t send the whole to the client, but only ids (hash of the content) and a description of the values (the shape, viewers and counts). Then the client fetches the data it wants to display and only if the data changed.

Oh, thanks!

note that not all viewers do partial loading, the table viewer still needs work in this regard. And plotly and vega always load the complete result (but only if it changed) because you wouldn’t want to plot a subset. 🙃

@mkvlr Seems to depend on dataset and browser. My dataset is larger then iris, and it works on Firefox, not on Chrome

Has long strings as well.

@carsten.behring strings should also be shortened.

Will find a tricky one which is online and send you link.

Om Chromium screen stays blank.

strange, I’m using Brave for development so should be the same. Any errors in the console?

Chromium behaves far worse, for sure. No errors in concole, just blank screen

On Linux.

Version 94.0.4606.61 (Official Build) Arch Linux (64-bit)

Clearly a chromium problem, on Firefox it works very well !

😞 browsers, I’ll try to reproduce it tomorrow

any extensions that could be causing issues? Can you try without them or in a private window in chromium?

For my larger dataset it makes sense to wrap it in the "result" viewer:

(clerk/with-viewer :clerk/result
  (def data (tc/dataset "/workdir/data/AOP non dogmatic. KE4dev5 210928.xlsx" {:key-fn csk/->kebab-case-keyword
                                                                                      :file-type :xlsx})))

This outputs "nil", and makes the whole rendering faster

Alterantively:

(def _
  (def data (-> (tc/dataset "/workdir/data/AOP non dogmatic. KE4dev5 210928.xlsx" {:key-fn csk/->kebab-case-keyword
                                                                                   :file-type :xlsx})
                (tc/head 1000))))