Say more!

getting there...

if I use vim - no luck xed - no luck nano - works ok...

with logs enabled, I see exceptions of the form 2023-02-01 14:02:07 DEBUG DirectoryWatcher:372 - DirectoryWatcher got an exception while watching! java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /home/xxx/clerk-demo/notebooks/data_science.clj/? at java.base/sun.nio.fs.UnixPath.encode(UnixPath.java:121) ...

Ah, it’s not ignoring the temp files I guess? Works with emacs too…

Sorry, you said that!

The exception is raised by the watcher itself - so quite low level. maybe due to files the are quickly removed.

I looked closer at the DirectoryWatcher - the exception is thrown here https://github.com/gmethvin/directory-watcher/blob/main/core/src/main/java/io/methvin/watcher/DirectoryWatcher.java#L372 according to the default listener this is just logged so should not cause errors. In the logs following this the watcher skips a ENTRY_CREATE event and then creates an ENTRY_MODIFY Event - I am not sure why this is skipped by clerk for update. log details and stack traces FYI 2023-02-01 15:58:48 DEBUG DirectoryWatcher:301 - ENTRY_DELETE [/home/.../clerk-demo/notebooks/data_science.clj] 2023-02-01 15:58:48 DEBUG DirectoryWatcher:372 - DirectoryWatcher got an exception while watching! java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /home/.../clerk-demo/notebooks/data_science.clj/? at java.base/sun.nio.fs.UnixPath.encode(UnixPath.java:121) at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:68) at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:278) at java.base/java.nio.file.Path.of(Path.java:147) at java.base/java.nio.file.Paths.get(Paths.java:69) at io.methvin.watcher.PathUtils.subMap(PathUtils.java:43) at io.methvin.watcher.DirectoryWatcher.runEventLoop(DirectoryWatcher.java:362) at io.methvin.watcher.DirectoryWatcher.lambda$watchAsync$1(DirectoryWatcher.java:232) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) 2023-02-01 15:58:48 DEBUG DirectoryWatcher:301 - ENTRY_CREATE [/home/.../clerk-demo/notebooks/data_science.clj] 2023-02-01 15:58:48 DEBUG DirectoryWatcher:474 - Skipping create event for path [/home/.../clerk-demo/notebooks/data_science.clj]. Path already hashed. 2023-02-01 15:58:48 DEBUG DirectoryWatcher:301 - ENTRY_MODIFY [/home/.../clerk-demo/notebooks/data_science.clj]

I tracked this down - the problem is in the detail of the way that an editor writes files. in short - if using neovim then: :set backupcopy=yes see here for more details....https://github.com/gmethvin/directory-watcher/issues/88

hmm, that sounds like a bug, right @U0303ET1R2T?

@U0FT7SRLP Did you build the newer version by manually going to https://github.clerk.garden/jeroenvandijk/hello-garden/commit/0d5d5548abcadbbcbaa5b9d45cee10a4fdfb734e/ or through the form on https://github.clerk.garden/? The form goes to https://github.clerk.garden/jeroenvandijk/hello-garden and appends a query parameter ?update=1 , which makes that link update to point to the latest build. If you simply build a separate commit by directly visiting the stable url with the commit sha, the mutable url won't get updated.

@U0303ET1R2T Yeah I changed the commit manually the second time. And then I tried from the normal url and that pointed to the old one

Ok, try https://github.clerk.garden/jeroenvandijk/hello-garden/?update=1

Ah great! Thank you both

The original reasoning behind this was, that you could use the mutable url as a way to share your notebook with the world and you want updates to that be controlled. But the current UX is not great and we'll probably rework it (and document it better).

Yeah makes sense

The only “bug” would be that I tried again through the form. But I’m guessing Garden saw the cache and thought it didn’t have to update anything?

Oh, if you try again through the form it should update. That's indeed a bug. I'll look into it.

Thank you 🙏

Btw, really cool I can publish a db like this. Didn’t think about this before

@U0303ET1R2T I was just trying the update=1 and it seems that it only works with and not with => the extra slash

Oh, thanks for the catch!

The project is here: https://github.com/jaydeesimon/nivoline-clerk

I set up this barebones project with only io.github.nextjournal/clerk and org.mentat/clerk-utils in the deps.edn

I created a Clojure namespace, jaydeesimon.notebook, to act as a notebook and a CLJS namespace, jaydeesimon.viewer, that will eventually house the nivo/line component for the viewer but for now, just has a dummy function that I’m hoping I can see will show up in a viewer via SCI

The first time I ran (mentat-clerk/serve! {:cljs-namespaces ['jaydeesimon.viewer]}), it generated a package.json, added some dependencies, invoked shadow-cljs and tried to build but it failed with an error

I was thinking about this, I think there’s an extra dependency I need to add, let me help out in a few when I’m back at my computer!!

We’ll get this dialed and I’ll use it to write the guide today

> I was thinking about this, I think there’s an extra dependency I need to add, let me help out in a few when I’m back at my computer!! Sounds good. Take your time. I've got some info I'll put down on this thread. Just ignore it til you have time

This was the first error

The required JS dependency "react-dom/client" is not available, it was required by "nextjournal/clerk/static_app.cljs".

I added react-dom to the package.json, tried again and it failed looking for another JS dependency. I realized it was looking for all of the dependencies in Clerk’s package.json so I ended up adding all of the https://github.com/nextjournal/clerk/blob/main/package.json#L3-L33 to my package.json

I was able to get past it looking for all of the JS dependencies by adding it to the package.json but now the build is failing with this message

The required namespace "reagent.ratom" is not available, it was required by "nextjournal/clerk/render/hooks.cljs".

Seems like it can’t find reagent. I tried adding reagent to the deps.edn and that didn’t work. I also created a shadow-cljs.edn and added it as a dependency there but also no luck

It seems like I’ll have a ways to go after the build part gets resolved but at the moment, I’m stuck

Here’s a https://gist.github.com/jaydeesimon/ab5f88fad276d62d7cb1d1faade1b344 to start it up and the package.json that was initialized by clerk-utils

Try again after adding this dependency too https://github.com/mentat-collective/clerk-utils/blob/1681208f9af7e644897008e79dc9f5af64f53b69/deps.edn#L17

Making sure its SHA matches the Clerk dependency

You should be able to delete the shadow-cljs.edn, it’ll be ignored

I THINK that should do it, but I’ll roll my sleeves up and make sure once I get home… I’m so glad you’re trying this, it’s important that this custom JS thing end up feeling as easy as possible @U098UL4QP

Ok yup that did it. I https://github.com/jaydeesimon/nivoline-clerk/commit/771532882d52cb3d8707db0fe55556f9ebe086df io.github.nextjournal/clerk.render as a dependency and it built and now works!

I'm not 100% sure how this kind of dependency works but I'll come back to that

io.github.nextjournal/clerk.render
  {:git/url ""
   :git/sha "d387037aa45303c32e311b62580b6f85e0b72919"
   :deps/root "render"}

Yesss

It’s a dependency that Clerk stuck in a subfolder of the repo

So the “render” folder in the clerk repo holds a deps.edn that specifies extra stuff , that’s what this is pointing at

Ok gotcha I see it now

Next challenge is the SCI customization

Right. I want to get my jaydeesimon.viewer/add var available in SCI so I can invoke it from the viewer

Ok I'll work on that. You sent an example of how to make the namespace available in SCI yesterday. Gonna look for that. Will let you know how it goes

https://github.com/mentat-collective/JSXGraph.cljs/blob/main/dev/jsxgraph/clerk_ui.cljs

Basically do this but swap your namespace in for jsxgraph.core

And if this namespace in your project requires the namespace with “add”, you only need to pass this sci customizing namespace to serve!

And it will pull in anything else you use via the require

With live reloading of the JS . You probably have to restart the page if you add new functions but then on reload they will be available for the viewers

How/where does or should the SCI customization code happen? In the JSXGraph project, I think the jsxgraph.clerk-ui namespace is being https://github.com/mentat-collective/JSXGraph.cljs/blob/main/shadow-cljs.edn#L18 and it's building it. Should I make my own shadow-cljs.edn and follow this example?

With the clerk-utils code all you need to do is make a cljs file analogous to the one above and stick it in src or dev,

Gotcha 👍

And then pass it’s namespace symbol via :cljs-namespaces

So close!!

@U098UL4QP okay, back at the computer, so I can type properly formatted code 🙂

Works! Amazing!

Here's the https://github.com/jaydeesimon/nivoline-clerk/commit/435552a4549d293fbda27cc30cdf208c1429fbc3

@sritchie09 I gotta go afk for a bit but this is great. Next step for me is to add the nivo/line dependencies, get it to bundle it up, add it to sci and then see if I can make a viewer from that

@U098UL4QP nice! and you don’t even need to have the viewer namespace in there; it will get pulled in automatically, since you’ve required it in jaydeesimon/sci.cljs

Sounds good. I'll be back. Thanks for your help! Let me know what I can do to help with the guide

@U098UL4QP all you should have to do is npm install @nivo/line , then you’re back on the happy path of your repro the other day

@U04V15CAJ do you have thoughts on the cleanest thing to add to this SCI customization namespace, if @U098UL4QP wants to use a js library? I’m assuming create-ns won’t work when passed a js namespace…

I think we solved something like this, I can’t track down which project it was in

you should use the :classes option for JS libraries.

Similar to what is done here: https://github.com/babashka/sci/blob/master/doc/async.md

so (:require ["@nivo/line" :as nline]), then :classes {'nivo nline} , something like that?

It depends on what clerk does in its :load-fn

see the example I linked

got it

I guess this must be extensible: https://github.com/nextjournal/clerk/blob/d387037aa45303c32e311b62580b6f85e0b72919/src/nextjournal/clerk/sci_env.cljs#L88-L108

I guess if you (set! clerk.sci/libname->class (assoc clerk.sci/libname->class "@whatever/your-lib" the-lib) then it would work

ah, I did this once and it worked:

(:require ["three" :as three])

(def three-ns
  (-> (into {}
            (map (fn [[k v]] [(symbol k) v]))
            (.entries js/Object three))))

(sci.ctx-store/swap-ctx!
 sci/merge-opts
 {:namespaces {'three three-ns}})

it works, but it's not recommended

it does have the advantage of not requiring that I control the load-fn

Sure. But this would solve it already for you: https://clojurians.slack.com/archives/C035GRLJEP8/p1675276217535389?thread_ts=1675271575.764119&amp;cid=C035GRLJEP8

This would be the only change

awesome, I’ll try that out and note it for the docs

in @U098UL4QP’s case if he wants to convert the components like

(def line-ns
  (-> (into {}
            (map (fn [[k v]]
                   [(symbol k) (r/adapt-react-class v)]))
            (.entries js/Object line))))

I just told you that I don't recommend turning JS libs into "cljs namespaces" ;)

haha no but read the example

this is taking everything inside the namespace and calling r/adapt-react-class on it

so I end up with Reagent components for every component inside some JS namespace

(in my head this is different, and something I would manually do if I were wrapping a React component library)

Do what you need to do in user space

I don't have enough context to say if this is a good idea in general

could be

(def line-ns
  {'component1 (r/adapt-react-class line/component1)
   'component2 (r/adapt-react-class line/component2)})

this would be the alternative

why do you need to make a namespace out of this?

so folks can use those components in their clerk viewers

like [line/component1 {:k "v"} …]

I would just expose the JS lib and let people do what they normally do in reagent, but that's just my preference

It would be the same except with a :> as the first vector entry …

yes, which seems good enough to me

As long as I can document how to do it super easily for someone, that’s my top concern writing these Clerk guides

And you solved that above

I suppose you can defer those folks to the reagent docs, you don't have to protect them too much by wrapping stuff, that often leads to more surprises

but of course it all depends :)

That's just my take, there is no single best answer I think

Sometimes I wrap if I’m mixing in other reagent components that I’ve added, so I don’t want the confusion of :>

I did it for the mathbox primitives so I could add docstrings

Im a pathological component wrapper I guess, full of justifications

As long as you also expose the raw stuff, so users have the flexibility, I think it's good. Sometimes a wrapper hides stuff by mistake, which is mostly what I'm concerned about

Agree

Looking forward to your final solution :)

@U04V15CAJ this is weird,

------ WARNING #1 - :undeclared-var --------------------------------------------
 File: /Users/sritchie/code/clj/clerk-utils/dev/clerk_utils/sci_extensions.cljs:14:7
--------------------------------------------------------------------------------
  11 |   (sci/copy-ns clerk-utils.custom
  12 |                (sci/create-ns 'clerk-utils.custom)))
  13 | 
  14 | (set! cs/libname->class
-------------^------------------------------------------------------------------
 Use of undeclared Var nextjournal.clerk.sci-env/libname->class
--------------------------------------------------------------------------------
  15 |       (assoc cs/libname->class
  16 |              "react" react))
  17 | 
  18 | ;; This next form mutates SCI's default environment, merging in your custom code
--------------------------------------------------------------------------------

but it’s definitely there…

maybe try the fully qualified name?

I can't see the full context just from this snippet

sorry, yes, I’m trying to figure out if I have some wonky version of clerk, will report shortly

Here is the context… when I call the following ina cljs namespace:

(prn (ns-publics 'nextjournal.clerk.sci-env))

{mount #'nextjournal.clerk.sci-env/mount,
 code-namespace #'nextjournal.clerk.sci-env/code-namespace,
 hooks-namespace #'nextjournal.clerk.sci-env/hooks-namespace,
 parser-namespace #'nextjournal.clerk.sci-env/parser-namespace,
 onmessage #'nextjournal.clerk.sci-env/onmessage,
 doc-url #'nextjournal.clerk.sci-env/doc-url,
 !edamame-opts #'nextjournal.clerk.sci-env/!edamame-opts,
 render-namespace #'nextjournal.clerk.sci-env/render-namespace,
 initial-sci-opts #'nextjournal.clerk.sci-env/initial-sci-opts,
 viewer-namespace #'nextjournal.clerk.sci-env/viewer-namespace,
 ->viewer-eval-with-error #'nextjournal.clerk.sci-env/->viewer-eval-with-error,
 set-state #'nextjournal.clerk.sci-env/set-state,
 ->viewer-fn-with-error #'nextjournal.clerk.sci-env/->viewer-fn-with-error,
 read-string #'nextjournal.clerk.sci-env/read-string,
 eval-form #'nextjournal.clerk.sci-env/eval-form}

when I search the Clerk codebase for parser-namespace , for example, I don’t see anything at all?? what a mystery

maybe try clojure -X:deps tree :aliases '[...]' (and fill in your aliases there too) and see what version of clerk you got?

weird, it doesn’t seem to show git dependencies: EDIT sorry still working…

io.github.nextjournal/clerk 4180ed3

that’s what I had expected…

@U5H74UNSF do these vars look familiar? https://clojurians.slack.com/archives/C035GRLJEP8/p1675359147121999?thread_ts=1675271575.764119&amp;cid=C035GRLJEP8

I’m looking in the .gitlibs directory at the clerk dep, and I see the code I expect, but not the publics from that link

oh… I must have updated deps.edn after starting cider long ago…

let’s see… sorry folks

🎉 @nivo/line is successfully rendering! Here's my https://github.com/jaydeesimon/nivoline-clerk/commit/6cc8e329752ce11f83f70d89dd5afbfed5c9d93d It was as easy as adding the @nivo/line npm deps and then adding ResponsiveLine to the :classes SCI context

Woohoo, that’s excellent news!!

That looks like a great library, looks good in the notebook too

I haven't tried building the static site yet but will try that out later. Thanks again for the help!

When you do you won’t be able to open index.html directly, you’ll need to run a web server so the site doesn’t complain about the relative JS link… copy the command from the clerk-utils package.json

Npm run serve I think? I will add this to the guide

@sritchie09 If your guide already uses babashka, you can also use: https://github.com/babashka/http-server/

or you can add it to deps.edn and run with clojure -X

Thank you! This is getting better and better

If I can replace the gh-pages package with babashka I can ditch the package.json completely

What's the gh-pages package?

@U04V15CAJ sorry, back at the comp. I imagine this is easy to recreate; it’s an npm package https://github.com/tschaub/gh-pages that I’ve been using to accomplish the task of “move public/build (or whatever output-dir you choose) to the gh-pages branch and commit”

which has the side effect of triggering a GitHub Pages Publish

@U04V15CAJ I was GOING to say: “for https://github.com/babashka/http-server/blob/main/src/babashka/http_server.clj, I think one thing we need is the ability to set the Cache-Control header to max-age=0, since the shadow-cljs build produces main.js and the browser caches it and won’t pick up changes….” But I forgot I’d gone the CAS route, so the path changes every time

boom, works great

@sritchie09 I'm using a github actions for that purpose. https://github.com/oxalorg/4ever-clojure/blob/cc66e5d218fb89c3a8c0e5e447a16685b480b961/.github/workflows/deploy.yml#L41-L52

me too, but sometimes when I’m working on a notebook I like to publish locally too without merging a PR (since I only have it run on changes to main)

I don't merge any PRs with that setup

but you do have to push to main right

true

which is definitely the right way to do it, that’s what I have set up too

but that can be controlled here: https://github.com/oxalorg/4ever-clojure/blob/cc66e5d218fb89c3a8c0e5e447a16685b480b961/.github/workflows/deploy.yml#L5

but when I am first working on these libraries and trying to share default stuff sometimes I run bb release-gh-pages to get my local branch etc pushed up

anyway, committing to gh-pages is scriptable