Clojurians Log v2

Right now I'm mostly just exploring. I'm thinking that I'd likely want helper functions in a different namespace.

Nice! Thank you. Works great 🙂

yes, I would expect clerk to only show the first 20 elements and allow you to lazy load more on demand.

see https://github.com/nextjournal/clerk-demo/blob/main/notebooks/pagination.clj and https://github.com/nextjournal/clerk-demo/blob/main/notebooks/viewer_api.clj#L14

can you run those and tell me if they work? Curious if you also see the faulty behavior on eg (range)

Clerk should only send a tiny preview of the 300mb object to the browser, never the whole thing

(range) works fine

How does it make the "tiny preview" on an arbitrary object ?

I start to see the special think about my data.

text columns

so 500 000 times a "medium size text".

the medium size text is too short to get truncated, but we have lots of those.

still should never serve all of them to the browser, just the first 20

This starts to make issues:

(repeat 10000 (apply str (repeat 1000 "a")))

oh probably haven't implemented a protocol for some of the specific types

https://github.com/borkdude/finitize/blob/62840885886354a5a95b7404f04fe4596c58e75e/src/finitize/core.cljc#L12-L56

@carsten.behring ok great, will dig into this

@borkdude I’m afraid it’s entirely my fault and it’s all in plain clojure causing troubles

I meant, it might be such an issue, not sure actually how you implemented it :)

@carsten.behring not running into an issue here on the first try though

I remember this case from Cider, as I tried to configure some "logic", when to cut of the output. The seq of long strings is a difficult case, and it does happen in real life. Text columns of datasets.

as you can see Clerk only gets the first 20 elements (and a preview of 80 chars for each)

for you it crashes the browser?

I see there’s an issue with lazy loading in this case though

gtg now, will look more into this later

Whats about that:

(repeat 1000
        (zipmap
         (range)
         (apply vector (repeat 100 (apply str (repeat 100 "a"))))))

so a 2d table of 10000 strings of length 100

for me it does not render

on firefox.

just tried opera and it works....

This file and TMD fails always: https://www.dropbox.com/s/qearl4fh0c9eirr/Reviews.csv?dl=0

with this code:

(ns prepare-data

  (:require
   [tablecloth.api :as tc]
   [tech.v3.dataset.impl.dataset :refer [dataset?]]
   [tech.v3.libs.fastexcel]
   [nextjournal.clerk :as clerk]
   [nextjournal.clerk.viewer :as v]
   [camel-snake-kebab.core :as csk]))

(def raw-data-2
  (tc/dataset "/home/carsten/Dropbox/sources/scicloj-data-science-handbook/Reviews.csv"
                {:key-fn csk/->kebab-case-keyword}))
(tc/shape raw-data-2)

(tc/column-names raw-data-2)

clojure is fast on reading it: Clerk evaluated 'notebooks/prepare_data.clj' in 5914.30252ms.

But the browser window stays blank forever.

I think TMD has as well some other characteristics and maybe @chris441 can talk about it in detail. While working in the repl, a TMD dataset hardly ever gets "fully realized" So maybe our impression that we can "easely work with it in the repl", is mainly thanks to that. And the "rendering" break that by "realizing it" in some form, and teh teh "cutting" off does not really help to keep performance acceptable. Not sure, if this applies as well to the "Pure loading" of a dataset as I do above.

The large dataset above prints in the repl instantently and even nicely (my pasting here hides this ...) prepare-data> raw-data-2 ;; => /home/carsten/Dropbox/sources/scicloj-data-science-handbook/Reviews.csv [568454 10]: | :id | :product-id | :user-id | :profile-name | :helpfulness-numerator | :helpfulness-denominator | :score | :time | :summary | :text | |----:|-------------|----------------|---------------------------------|-----------------------:|-------------------------:|-------:|-----------:|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 | B001E4KFG0 | A3SGXH7AUHU8GW | delmartian | 1 | 1 | 5 | 1303862400 | Good Quality Dog Food | I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than most. | | 2 | B00813GRG4 | A1D87F6ZCVE5NK | dll pa | 0 | 0 | 1 | 1346976000 | Not as Advertised | Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as "Jumbo". | | 3 | B000LQOCH0 | ABXLMWJIXXAIN | Nat;; => alia Corres "Natalia Corres" | 1 | 1 | 4 | 1219017600 | "Delight" says it all | This is a confection that has been around a few centuries. It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar. And it is a tiny mouthful of heaven. Not too chewy, and very flavorful. I highly recommend this yummy treat. If you are familiar with the story of C.S. Lewis' "The Lion, The Witch, and The Wardrobe" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch. | | 4 | B000UA0QIQ | A395BORC6FGVXV | Karl | 3 | 3 | 2 | 1307923200 | Cough Medicine | If you are looking for the secret ingredient in Robitussin I believe I have found it. I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda. The flavor is very medicinal. | | 5 | B006K2ZZ7K | A1UQRSCLF8GW1T | Michael D. Bigham "M. Wassir" | 0 | 0 | 5 | 1350777600 | Great taffy | Great taffy at a great price. There was a wide assortment of yummy taffy. Delivery was very quick. If your a taffy lover, this is a deal. | | 6 | B006K2ZZ7K | ADT0SRK1MGOEU | Twoapennything | 0 | 0 | 4 | 1342051200 | Nice Taffy | I got a wild hair for taffy and ordered this five pound bag. The taffy was all very enjoyable with many flavors: watermelon, root beer, melon, peppermint, grape, etc. My only complaint is there was a bit too much red/black licorice-flavored pieces (just not my particular favorites). Between me, my kids, and my husband, this lasted only two weeks! I would recommend this brand of taffy -- it was a delightful treat. | | 7 | B006K2ZZ7K | A1SP2KVKFXXRU1 | David C. Sullivan | 0 | 0 | 5 | 1340150400 | Great! ;; => Just as good as the expensive brands! | This saltwater taffy had great flavors and was very soft and chewy. Each candy was individually wrapped well. None of the candies were stuck together, which did happen in the expensive version, Fralinger's. Would highly recommend this candy! I served it at a beach-themed party and everyone loved it! | | 8 | B006K2ZZ7K | A3JRGQVEQN31IQ | Pamela G. Williams | 0 | 0 | 5 | 1336003200 | Wonderful, tasty taffy | This taffy is so good. It is very soft and chewy. The flavors are amazing. I would definitely recommend you buying it. Very satisfyin ... output truncated to 9.8kB ...

in Emacs, Cider cuts it of as well, it seems. The ... output truncated to 9.8kB ... comes from Cider

ahha, we are getting somewhere:

playing with different sizes of the columns and try to print them, I see finally error messages:

So I suppose that with a larger text, we can a lot of errors and that makes something crash.

at 10000 the browser stops being responsive:

(-> raw-data-2 (tc/head 10000) :text)

will try this later, thanks! 🙏

@chris441 At a certain moment, I see as well the internal column metadata, instead of the column values ....

The only thing I have to add at this point is that I hope that everyone is aware of https://github.com/cnuernber/tmdjs if they are in fact attempting to ship some subset of their data to the client and manipulate it there. Not sure why you would see metadata at this point.

@chris441 no manipulation, just display a (tiny) subset, i.e. the first 20 rows

@carsten.behring can reproduce the issue now, thanks!

@carsten.behring think I have a fix but will need some more testing. Also you obviously want show this dataset not as nested maps but as a table. We’re working on an overhaul of our table viewer which is just a prototype currently.

Very nice. Regarding the way to view this: It is fine like it is now, as a ddfault, for a few reasons: 1. Dataset is a map?

2. I never want to view it in "full". It is just a side effect of Clerk, that is shows all defs. So this should be fast and not crash.

3. Maybe a custom viewer would be nice.