Fork me on GitHub
#cljdoc
<
2018-09-03
>
martinklepsch18:09:58

URLs now have much better meta tags, which should be nice on Twitter, Facebook, Slack and whatnot: https://cljdoc.xyz/d/clojure.java-time/clojure.java-time/0.3.2/doc/readme

martinklepsch18:09:06

Would be awesome to generate a dynamic image with the library name on the fly but not sure how much work that would be (if anyone has experience with that I'd be happy to hear it)

mkvlr18:09:58

I’ve recently looked into that and it worked very well using chrome headless

martinklepsch18:09:11

ah, interesting. So basically: HTML page with contents, chrome headless to make a "screenshot", store/return that

mkvlr18:09:12

could add that as part of the build process I guess

martinklepsch18:09:28

ah, that's also an interesting option

mkvlr18:09:16

(def chrome-command
  (if (-> "/usr/bin/google-chrome" io/file .exists)
    "/usr/bin/google-chrome"
    "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"))

(defn make-screenshot!
  ([url]
   (make-screenshot! url "/tmp/screenshot.png"))
  ([url filename]
   (log/debug :capture! url)
   (let [{:keys [exit] :as r}
         (shell/sh chrome-command
                   "--headless"
                   "--hide-scrollbars"
                   "--no-gpu"
                   "--window-size=600,600"
                   "--virtual-time-budget=25000"
                   (str "--screenshot=" filename)
                   "--no-sandbox"
                   url)]
     (log/info :capture! {:url url :exit-code exit})
     r)))

mkvlr18:09:46

that’s from what we’re using

mkvlr18:09:17

guess you could also have some mapping of library name and version to file name and just generate it on demand if it doesn’t exist

mkvlr18:09:46

what would you like to generate? is it about having a preview for facebook / twitter?

mkvlr18:09:08

not sure I fully understood what you meant by image with the library name

martinklepsch18:09:37

yeah pretty much. If you share a cljdoc link now the image is just the cljdoc logo. it would be nice if it would include at least the project name and version

mkvlr18:09:48

most of the work for our thing was figuring out when to take the screenshot (make sure react was fully loaded) but that shouldn’t be an issue with cljdoc

mkvlr18:09:03

ok, wouldn’t a preview of the actual page be even better?

martinklepsch18:09:40

maybe but at the current layout I doubt it's going to be readable

mkvlr18:09:19

it doesn’t have to be I think

mkvlr18:09:18

maybe just screenshot the main div

mkvlr18:09:40

or something like this would just be fine?

martinklepsch18:09:54

you're right, that actually doesn't look all that bad

mkvlr18:09:07

let me try to sketch something

mkvlr18:09:19

been meaning to contribute for a while anyway

martinklepsch18:09:03

oh awesome! yeah, let's do it

martinklepsch18:09:08

I also just discovered this "microservice" type thing but not sure if that's necessary https://github.com/pressly/screenshot

mkvlr18:09:14

guess the main issue is naming the namespace and where to put it 😼

mkvlr18:09:33

the puppeteer thing is nice if you need more control

mkvlr18:09:56

but for cljdoc I think it’s overkill

martinklepsch18:09:39

guess the only issue we could run into is chrome startup/runs using to much resources but we can just work that out as we go I guess

mkvlr18:09:12

I think circleci should be able to handle it

mkvlr18:09:18

the site is pretty lean after all

mkvlr18:09:44

could also run as it’s own task

martinklepsch18:09:06

ah, do you want to create & store the image during the CI run?

mkvlr18:09:01

yeah, I think doing it on build makes the most sense?

mkvlr18:09:15

or triggering it’s own build for the image?

mkvlr18:09:21

but running it on circle ci either way

martinklepsch18:09:58

I was thinking it might be better to generate it on demand because: - we need the CI run to finish and (web-)hooked back into cljdoc before we can render any docs - we'd need to re-run for all already imported projects

mkvlr18:09:18

regarding the first point: yes you need the docs but could also open them locally, right?

mkvlr18:09:08

where’s cljdoc hosted?

martinklepsch18:09:11

not in CircleCI... it's a very stripped down environment where you can really just analyze some Clojure code

mkvlr18:09:18

could we trigger a circle ci build on demand?

mkvlr18:09:52

I think using some service where the sceenshotting happens is probably better then doing it in process

mkvlr18:09:18

or I guess as plain as the cljdoc site is maybe javafx is just fine and we do it in-process?

martinklepsch18:09:24

Let me describe the rough build architecture: 1. A build is requested via the website/API 2. This triggers a build on CircleCI where a very small Clojure program runs and stores information about namespaces in an .edn file. That file is exposed as a build artifact. 3. Once notified via webhook we get that file and import it into an SQLite database. 4. If the .pom for the artifact specifies a Git remote we clone that and import various things into SQLite (Readme, Changelog, other files, ...) 5. The build is done Now when a user comes and requests some documentation it is fetched from SQLite and returned as some HTML like with any other dynamic website.

mkvlr18:09:57

ok, the scrollbars are a bit ugly using JavaFx

martinklepsch18:09:42

heh, I just did exactly that 😄 nextjournal is pretty cool 🙂

mkvlr18:09:51

I see, so if we want to screenshot as part of the build process it would be some extra work to serve just that edn file

martinklepsch18:09:22

Linux font rendering also sucks unfortunately 😞

martinklepsch18:09:39

> I see, so if we want to screenshot as part of the build process it would be some extra work to serve just that edn file not sure I understand

mkvlr18:09:07

I mean serve the docs on circle ci within that build…

mkvlr18:09:16

so we can screenshot locally

martinklepsch18:09:24

We can't do that 😅 None of the rendering code is loaded/available in CI

mkvlr18:09:01

this is the screenshot in chrome on linux for comparison

mkvlr18:09:16

coudl still increase the window size I guess

mkvlr18:09:27

the scrollbars also aren’t too pretty

mkvlr18:09:52

font rendering is ok for sharing I guess

mkvlr18:09:18

ah, there’s also --hide-scrollbars

mkvlr18:09:35

this looks pretty ok, what do you think?

martinklepsch19:09:31

What I originally had in mind was something more like this. I agree that an actual preview is nicer though.

mkvlr19:09:16

could be nice as well

mkvlr19:09:44

if we serve the markup for that at a url that should be easily doable as well

martinklepsch19:09:36

right, or java that generates a png but not sure how hard/easy that is with layouting and that kind of stuff

mkvlr19:09:53

but I guess the way twitter and facebook sharing works the actual content would be better suited

martinklepsch19:09:35

--enable-font-antialiasing chrome has this option

mkvlr19:09:59

doesn’t seem to make a difference, get the same image back

mkvlr19:09:16

(we’re using content-addressing for data)

martinklepsch19:09:54

interesting that the result is pixel perfect (found a GH issue where some people mentioned it's not consistently pixel-perfect)

mkvlr19:09:58

not sure if that depends on the gpu flag and if you’re using js

mkvlr19:09:50

but yes, was also surprised

mkvlr19:09:38

trying to see if we can remove scrollbars with jBrowserDriver but can’t find an easy way

martinklepsch19:09:32

I'd love to have this but I have a few concerns still: - using chrome we'll likely want to move this onto another machine - cljdoc's dev/ops setup isn't exactly pretty and this will likely introduce more moving parts, making things worse unless more time is spent on it - we'll (probably?) need caching (currently nothing is cached)

mkvlr19:09:38

> currently nothing is cached

mkvlr19:09:51

does that mean that there’s no filesystem state except the sqlite db?

martinklepsch19:09:27

there is some ephemeral state when cloning a git repo as well

mkvlr19:09:36

ok yeah, certainly don’t want to generate a screenshot every time but only once

martinklepsch19:09:02

we could also just store the files on disk, right 🙂

mkvlr19:09:10

that should be fine

mkvlr19:09:05

another thing to maybe take into account is renaming the files if the global layout and css changes…

mkvlr19:09:28

is there just one server running at this point?

mkvlr19:09:25

re moving onto another machine: yes, it’s the right thing to do long-term, not sure it’s needed right away though.

martinklepsch19:09:15

This would be an utter hack (once again) but we could do the same what we're doing with the analysis. Just run a CircleCI task when the image is requested for the first time. Store it as build artifact, import it to disk once webhook tells us it's done.

martinklepsch19:09:55

I assume they have Chrome images that wouldn't require any further bootstrapping so job could be done in ~10s

mkvlr19:09:28

they must have chrome somewhere or allow for arbitrary docker images

mkvlr19:09:00

too bad the nextjournal articles as api endpoints feature isn’t ready yet, that would be perfect for this

mkvlr19:09:38

but as you already do stuff on circle ci if you hit a URL I guess taking a screenshot isn’t so different

martinklepsch19:09:43

We'd need to track that we started a build so we're not starting 15 when the user is clicking around (not sure if browsers request these images though)

mkvlr19:09:08

is there currently any duplicate tracking / locking for doc builds?

martinklepsch19:09:54

there wasn't until very recently, now you'll be redirected to a running build if there is one

martinklepsch19:09:31

for regular users this isn't really necessary but there's some stuff upcoming where it's been the right thing to do

mkvlr19:09:04

ok, and that goes through the db or how is it done?

martinklepsch19:09:31

yeah, builds are stored in DB, it just checks if there's a running one for the project/version

martinklepsch19:09:05

(in reality it also limits the query to the last 10min because some builds may end up in limbo if the webhook gets lost)

mkvlr19:09:14

I guess keeping track of running screenshots in the db (or in memory as long as there’s only one process) is fine for now?

martinklepsch19:09:35

yeah, in this case an atom is probably just fine

mkvlr19:09:49

btw, should you ever want client-side routing, reitit supports pedestal and clojurescript and has been working nicely for us

martinklepsch19:09:36

ah, cool to hear of people using reitit, I've been looking at it but pedeistal is ok for now. I like where it's headed generally though

martinklepsch19:09:08

Although I gotta say that this still drives me mad about pedestal: https://github.com/pedestal/pedestal/issues/572#issue-330157184

martinklepsch19:09:43

(url-for :thing :path-params {})
;;=> "/thing/:type/:id" ;        <- it just returns the parameter key if you don't supply a value for it

mkvlr19:09:53

I’m with you on that one

martinklepsch19:09:41

alright. I haven't eaten dinner yet so we're gonna take a stroll and hunt down some food 😄

martinklepsch19:09:55

The previews would definitely be a cool addition, so if that's interesting/fun to you lets do it 🙂

martinklepsch19:09:56

We can also set up a call some time (e.g. 5pm tmrw) and I'd be happy to walk you through the code etc. Or we pair. Or ...

martinklepsch19:09:25

That said, if the screenshot thing sounds too annoying there's also a ton of other interesting stuff to work on 🙂

mkvlr19:09:14

alright, I’ll take a look, dev setup seems to work like a charm

mkvlr19:09:47

I’m currently in the US but back next wednesday

mkvlr19:09:06

but I think I should be able to figure it out, seems like it’s very well documented

mkvlr19:09:28

have a good night!