Clojurians Log v2

I'm still not seeing any source maps in lib with bb dev

I'm on your branch and the only difference is that I've added a nbb.edn

on the newest commit

This is the command generated:

"-M:test -m shadow.cljs.devtools.cli --force-spawn watch modules --config-merge shadow-tests.edn --config-merge '{:compiler-options {:source-map true}}'"

even when removing the first shadow-tests.edn, no luck. darn

thheller told me that source maps are the default behavior

but I'm not getting any

ok, it works, the maps are in lib/cljs-runtime

anyway, in the profiler output I just see trees of evaluation + analysis, kind of what is expected. if you load 50 namespaces it may take 1 second, just like in JVM Clojure I guess

it all has to be parsed, analyzed and evaluated on the fly

if there's any quick wins I'd want to to know but it's not apparent to me where the gains are

Mornin. Good to hear sourcemaps worked. Thanks for taking a look and bummer there’s no quick win. Fwiw one downside of the lazy approach is that nbb bundle can’t be used since the lazy namespaces aren’t included. For now I’ll just distribute the CLI with nbb-logseq as a dep and https://github.com/logseq/logseq/pull/12000/commits/77f5dcb6152ec456009f73e3b48cb3c22f30bf50

> Good to hear sourcemaps worked. Well, they did appear on disk but I didn't see them work in the chrome tab

if you want even faster startup times, move stuff to pre-compiled CLJS would still be my advice for now

I decided to use nbb’s bundle approach to see if it would speed up the CLI. It still takes about 1.5s+ to start. Most of the time is in loadString. I’ve attached the bundled cli which has 55 loadString calls. I tried https://github.com/logseq/nbb/blob/2888eb678b92631eb78b1e2a6434c7e296b998ec/src/nbb/impl/bundler.cljs#L203-L204 to batch all the loadString calls into one loadString but reading load-string it looks like it’s only meant for one ns. Any thoughts on how I could speed up the initial load? Perhaps there’s a way to batch load sci namespaces that’s not in the public api

Is there a specific nbb version where it became slow? or just happened over time?

I'm getting this:

$ node out-cli.mjs
node:internal/modules/esm/resolve:264
    throw new ERR_MODULE_NOT_FOUND(
          ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/private/tmp/dude/node_modules/@logseq/nbb-logseq/lib/nbb_cljs_bean_transit.js' imported from /private/tmp/dude/out-cli.mjs

This is a https://github.com/logseq/logseq/pull/12000 I’m writing and probably has more namespace deps than previous CLIs. This is my first time profiling this CLI so I’ve only ever used it based off of nbb 1.3.204

And if you use nbb in the unbundled way, is it still doable? Perhaps this is due to loading all the things while you could just lazy load things?

I'm also getting:

Could not resolve symbol: tc/to-date

> I’m getting this: I don’t know how you’re building it but it seems like it’s not picking up nbb-logseq’s deps. nbb-logseq is nbb bundled with extra deps. With nbb-logseq checked out, I build it with rm -rf .shadow-cljs && NBB_CLI_NAME=nbb-logseq NBB_NPM_LIB_NAME='@logseq/nbb-logseq' bb release . There’s one fix for nbb-features I should push if you want to repro it locally

I just installed nbb-logseq from npm

I can try your branch and building it locally but some other time this week

> And if you use nbb in the unbundled way, is it still doable? Perhaps this is due to loading all the things while you could just lazy load things? Unbundled are the times I reported at the top of the thread so it’s not doable as far as what I can have users using. This would be logseq’s main CLI which possibly thousands of users would use. I’m open to lazy-loading namespaces but unsure which ones to do and if that would shave off significant time. All these namespaces are also used by the web, desktop and mobile apps so it adds loading complexity there as well

The nbb-logseq on npm is fairly out of date. I’ll push up a complete repro on that branch with specific instructions so hopefully a repro is pretty quick

You mentioned that you need source maps. Since you build your own fork off of nbb, is it possible to enable them?

I tried enabling them but I couldn’t get them to work as expected with node’s inspector. I’m happy to push those changes on to this branch to if you want to try them

it would be helpful to find out what part of nbb.core is slow. There is some stuff that could be optimized, I'm thinking of sci/binding especially since that mimicks JVM clojure semantics, but it shouldn't have to in CLJS. But until we find out what is causing slowness using source maps and profilers, it's just a guess

Perhaps functions like seq? etc are also slow since it uses satisfies? in CLJS

stuff like that would be good to optimize

but I'm just stabbing in the dark here

zooming out a bit, is there a specific reason you're using nbb - you need others to use eval for example?

you can move more stuff to compiled CLJS for example to make the part that's interpreted smaller

Not sure if you’re asking why nbb in general or for this specific CLI. In general, because it would allow us to provide an ecosystem for users to write and share scripts that quickly do useful things with logseq. For this specific CLI, I could move to compiled cljs although I would be sad to leave behind nbb’s more productive dev env

for sure. we'll take a look where we can optimize stuff

so the easier it is for me to run the repro somewhere this week, the better

Can you summarize for me how I can build your project locally and run it, step by step?

Mornin. I thought more on what you said about lazy loading and realized that lazy loading in the CLI is enough to fix my issue - https://github.com/logseq/logseq/pull/12000/commits/456f853174c93f4597ca03ec48eddf77b296361d . Instead of requiring 55 namespaces at 1.7s, the cli now requires 6 namespaces at 0.25s. That works great for me! Thanks for the suggestion

Down the road it may be nice to improve load times if nbb scripts start to average requiring 10+ namespaces. One other load time data point - https://github.com/logseq/logseq/blob/master/deps/db/script/query.cljs loads 20 namespaces (5 coming from malli) and loads in about 0.7s

Ok if you have some ready to go profiling stuff going on happy to take a look to improve things in the future but happy that you managed with lazy loading for now :)

I don’t have a specific profiling thing setup. If you’re curious, here’s my wip sourcemap attempt with profiling that I attempted. I added optional sourcemaps to nbb with https://github.com/logseq/nbb/commit/4e3a6ebaa65e73e697faea3d59646b065830671d. I then enabled sourcemaps and invoked the inspector on the node nbb script with node --enable-source-maps --inspect-brk cli.mjs . The .js files showed up in chrome’s devtools Sources tab but then disappeared after I ran profile

if you hit profile, how do you actually execute something without completely restarting?

ok I figured it out

now how do I load those 50 namespaces?

I did remotes/origin/add-optional-sourcemap) $ node --enable-source-maps --inspect-brk cli.js --help

but it seems to be vanilla nbb right

I think even without source maps I'd get a pretty good idea where to look

this should give me some useful info maybe:

node --enable-source-maps --inspect-brk cli.js  -e "(require '[honey.sql])"

{:deps {com.github.seancorfield/honeysql {:mvn/version "2.2.868"}}}

Yep. Adding deps and getting to about 50 load-file calls should repro the load time I was seeing. medley/medley, metsonin/malli, io.github.nextjournal/nbb-test-runner are other nbb deps that you could add

hmm, yeah now I can see how source maps are going to be useful

I don't see shadow spitting out source maps though

perhaps that should be configured

Putting a print statement in nbb.core/load-file was an easy way I found to monitor the number of calls

$ ls lib/
cljs-runtime/              nbb_cljs_bean.js           nbb_deps.js                nbb_impl_main.js           nbb_math.js                nbb_promesa.js             nbb_repl.js                nbb_tests.js               nbb_zip.js
nbb_api.js                 nbb_core.js                nbb_goog_crypt.js          nbb_js_interop.js          nbb_nrepl_server.js        nbb_reagent_dom_server.js  nbb_spec.js                nbb_tools_cli.js
nbb_bundler.js             nbb_data.js                nbb_goog_string.js         nbb_main.js                nbb_pprint.js              nbb_reagent.js             nbb_test.js                nbb_transit.js

there should be source maps here right?

Yes. With my branch source maps are optional so build/release should be called with {:include-source-map? true}

I'm using bb dev but not seeing source maps

can you do ls lib/ and show that you have source maps?

I only ever used bb release. I could try with bb dev in a min

bb release is painful if you want to iterate

After pushing https://github.com/logseq/nbb/commit/52c3cdea432fa3785ca48565dff65468f71dfe6b, I’m seeing sourcemaps in lib/ for dev and release tasks

oh thanks!

I'll take a look tomorrow

It depends how you're invoking nbb I guess

$ time node_modules/.bin/nbb -e '(+ 1 2 30)'
33
node_modules/.bin/nbb -e '(+ 1 2 30)'   0,07s  user 0,01s system 97% cpu 0,091 total

I’m seeing similar times when I comment out loadFile in my https://github.com/logseq/logseq/pull/12000/files#diff-5a2292e8c38c8cb7586476ce5a505e0956b6800c00a57c4e3dee02824505de8e:

# When I comment out loadFile and remaining lines in cli.mjs
$ time node cli.mjs
node cli.mjs  0.07s user 0.01s system 99% cpu 0.083 total
# When I just uncomment line 10 containing loadFile call in cli.mjs
$ time node cli.mjs
node cli.mjs  1.77s user 0.08s system 200% cpu 0.925 total

perhaps try running with a dev build and insert some printlns?

I can try that. I’m also going to see if I can enable source maps and see the specific nbb.core fns in the chrome profiler. I’m using nbb-logseq which is using the latest nbb - 1.3.204