This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-09-03
Channels
- # aleph (3)
- # announcements (1)
- # beginners (116)
- # boot (8)
- # braveandtrue (4)
- # cider (40)
- # cljdoc (120)
- # cljsrn (10)
- # clojure (29)
- # clojure-austin (4)
- # clojure-dev (43)
- # clojure-germany (1)
- # clojure-italy (4)
- # clojure-nl (17)
- # clojure-russia (19)
- # clojure-uk (76)
- # clojurescript (118)
- # cursive (13)
- # datascript (11)
- # datomic (73)
- # emacs (24)
- # figwheel-main (176)
- # fulcro (40)
- # hyperfiddle (4)
- # leiningen (3)
- # off-topic (1)
- # pedestal (4)
- # re-frame (6)
- # reagent (22)
- # reitit (1)
- # ring (3)
- # rum (8)
- # shadow-cljs (41)
- # spacemacs (11)
- # specter (19)
- # unrepl (1)
URLs now have much better meta tags, which should be nice on Twitter, Facebook, Slack and whatnot: https://cljdoc.xyz/d/clojure.java-time/clojure.java-time/0.3.2/doc/readme
Would be awesome to generate a dynamic image with the library name on the fly but not sure how much work that would be (if anyone has experience with that I'd be happy to hear it)
ah, interesting. So basically: HTML page with contents, chrome headless to make a "screenshot", store/return that
ah, that's also an interesting option
(def chrome-command
(if (-> "/usr/bin/google-chrome" io/file .exists)
"/usr/bin/google-chrome"
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"))
(defn make-screenshot!
([url]
(make-screenshot! url "/tmp/screenshot.png"))
([url filename]
(log/debug :capture! url)
(let [{:keys [exit] :as r}
(shell/sh chrome-command
"--headless"
"--hide-scrollbars"
"--no-gpu"
"--window-size=600,600"
"--virtual-time-budget=25000"
(str "--screenshot=" filename)
"--no-sandbox"
url)]
(log/info :capture! {:url url :exit-code exit})
r)))
guess you could also have some mapping of library name and version to file name and just generate it on demand if it doesn’t exist
yeah pretty much. If you share a cljdoc link now the image is just the cljdoc logo. it would be nice if it would include at least the project name and version
most of the work for our thing was figuring out when to take the screenshot (make sure react was fully loaded) but that shouldn’t be an issue with cljdoc
maybe but at the current layout I doubt it's going to be readable
you're right, that actually doesn't look all that bad
oh awesome! yeah, let's do it
I also just discovered this "microservice" type thing but not sure if that's necessary https://github.com/pressly/screenshot
guess the only issue we could run into is chrome startup/runs using to much resources but we can just work that out as we go I guess
ah, do you want to create & store the image during the CI run?
I was thinking it might be better to generate it on demand because: - we need the CI run to finish and (web-)hooked back into cljdoc before we can render any docs - we'd need to re-run for all already imported projects
regarding the first point: yes you need the docs but could also open them locally, right?
not in CircleCI... it's a very stripped down environment where you can really just analyze some Clojure code
I think using some service where the sceenshotting happens is probably better then doing it in process
or I guess as plain as the cljdoc site is maybe javafx is just fine and we do it in-process?
maybe just like this https://nextjournal.com/mk/screenshotting-with-sparkledriver
https://nextjournal.com/mk/screenshotting-with-chrome is the chrome alternative
Let me describe the rough build architecture:
1. A build is requested via the website/API
2. This triggers a build on CircleCI where a very small Clojure program runs and stores information about namespaces in an .edn
file. That file is exposed as a build artifact.
3. Once notified via webhook we get that file and import it into an SQLite database.
4. If the .pom
for the artifact specifies a Git remote we clone that and import various things into SQLite (Readme, Changelog, other files, ...)
5. The build is done
Now when a user comes and requests some documentation it is fetched from SQLite and returned as some HTML like with any other dynamic website.
https://nextjournal.com/data/QmUS4x61ouzvb1UoHF61BVMDCxBifj2Yhc8wPzSxg8ArcS?content-type=image%2Fpng
heh, I just did exactly that 😄 nextjournal is pretty cool 🙂
I see, so if we want to screenshot as part of the build process it would be some extra work to serve just that edn file
Linux font rendering also sucks unfortunately 😞
> I see, so if we want to screenshot as part of the build process it would be some extra work to serve just that edn file not sure I understand
We can't do that 😅 None of the rendering code is loaded/available in CI
https://nextjournal.com/data/QmS7MZFm6zcYZhFjMw7jdqqoQyh1VBDxMR6Wcmrm3m1WaD?content-type=image%2Fpng
https://nextjournal.com/data/QmZ6xFpJRJuxr7fR1zKvkm3gApR237XJUZUK7NiAJLbA5C?content-type=image%2Fpng
What I originally had in mind was something more like this. I agree that an actual preview is nicer though.
right, or java that generates a png but not sure how hard/easy that is with layouting and that kind of stuff
but I guess the way twitter and facebook sharing works the actual content would be better suited
--enable-font-antialiasing
chrome has this option
https://nextjournal.com/data/QmZ6xFpJRJuxr7fR1zKvkm3gApR237XJUZUK7NiAJLbA5C?content-type=image%2Fpng
interesting that the result is pixel perfect (found a GH issue where some people mentioned it's not consistently pixel-perfect)
trying to see if we can remove scrollbars with jBrowserDriver but can’t find an easy way
might be doable with http://machinepublishers.github.io/jBrowserDriver/com/machinepublishers/jbrowserdriver/Settings.Builder.html#headScript-java.lang.String-
I'd love to have this but I have a few concerns still: - using chrome we'll likely want to move this onto another machine - cljdoc's dev/ops setup isn't exactly pretty and this will likely introduce more moving parts, making things worse unless more time is spent on it - we'll (probably?) need caching (currently nothing is cached)
there is some ephemeral state when cloning a git repo as well
we could also just store the files on disk, right 🙂
another thing to maybe take into account is renaming the files if the global layout and css changes…
re moving onto another machine: yes, it’s the right thing to do long-term, not sure it’s needed right away though.
This would be an utter hack (once again) but we could do the same what we're doing with the analysis. Just run a CircleCI task when the image is requested for the first time. Store it as build artifact, import it to disk once webhook tells us it's done.
I assume they have Chrome images that wouldn't require any further bootstrapping so job could be done in ~10s
too bad the nextjournal articles as api endpoints feature isn’t ready yet, that would be perfect for this
but as you already do stuff on circle ci if you hit a URL I guess taking a screenshot isn’t so different
We'd need to track that we started a build so we're not starting 15 when the user is clicking around (not sure if browsers request these images though)
there wasn't until very recently, now you'll be redirected to a running build if there is one
for regular users this isn't really necessary but there's some stuff upcoming where it's been the right thing to do
yeah, builds are stored in DB, it just checks if there's a running one for the project/version
(in reality it also limits the query to the last 10min because some builds may end up in limbo if the webhook gets lost)
I guess keeping track of running screenshots in the db (or in memory as long as there’s only one process) is fine for now?
https://github.com/cljdoc/cljdoc/commit/4a2f0f4effe2afd566f06bdc88f725dd3aaa8885
yeah, in this case an atom is probably just fine
btw, should you ever want client-side routing, reitit supports pedestal and clojurescript and has been working nicely for us
ah, cool to hear of people using reitit, I've been looking at it but pedeistal is ok for now. I like where it's headed generally though
Although I gotta say that this still drives me mad about pedestal: https://github.com/pedestal/pedestal/issues/572#issue-330157184
(url-for :thing :path-params {})
;;=> "/thing/:type/:id" ; <- it just returns the parameter key if you don't supply a value for it
alright. I haven't eaten dinner yet so we're gonna take a stroll and hunt down some food 😄
The previews would definitely be a cool addition, so if that's interesting/fun to you lets do it 🙂
We can also set up a call some time (e.g. 5pm tmrw) and I'd be happy to walk you through the code etc. Or we pair. Or ...
That said, if the screenshot thing sounds too annoying there's also a ton of other interesting stuff to work on 🙂