This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-09-18
Channels
- # announcements (1)
- # asami (2)
- # babashka (21)
- # beginners (23)
- # cider (5)
- # clj-kondo (10)
- # clojure (31)
- # clojure-europe (3)
- # clojure-nl (1)
- # clojurescript (47)
- # deps-new (1)
- # figwheel-main (7)
- # fulcro (7)
- # gratitude (1)
- # jobs-discuss (2)
- # lein-figwheel (1)
- # lsp (5)
- # off-topic (11)
- # pathom (5)
- # re-frame (1)
- # react (5)
- # reagent (4)
- # releases (1)
- # shadow-cljs (63)
- # tools-deps (16)
- # xtdb (26)
I've been working on some stuff to do with parallelism with SAB's atomics and it's really hard to get a good demo site going on github pages, with the latest security header requirements for SABs. So I've been wanting to migrate my testing over to nodejs and flesh it over there, until browsers settle on a security model. But I'm not seeing a lot of prior art out there for using CLJS with worker_threads. If I end up working on building something out on nodejs, I'd probably want to bust the thing that wraps the worker_threads api out into its own lib. Things is, it's inherently a code deploy situation, where you're pushing some set of code into the worker. How you go about doing that depends on your build environment. When I build my tau.alpha framework for the browser, I kinda hardcoded into the code where to find the sources to inject into the worker based on figwheel's semantics, iirc. I'm wondering if I can come up with some strategy to inject code into the worker in a way that will be compatible with vanilla clojure, figwheel and shadow - a build-tool agnostic method by which I can easily push my build into nodejs webworkers
In the browser, I would just basically load the entire app in the worker. Then I would use conditional checks on whether I was in a worker or not to trigger certain behaviors on wokers vs main. Going that route, it's a little easier to perhaps engineer compatibility with vanilla, figwheel and shadow. However, thinking about selectively loading modules in particular workers, for instance, is a little less certain, wrt cross-build-tool compatibility
So for this nodejs impl, I'm thinking about getting a little more nuanced and allowing for selective loading of deps in workers, but I'm not sure what the lay of the land is, in terms of what strategies would be compatible with the most build tools
I mean, I know there's a webworker story already out there for the default cljs, figwheel and shadow, and things could maybe be patched up to work with worker_threads, in each case. But until that time comes, in a lib, I could just do the job of grabbing the sources and putting things where they belong myself. I'm just not sure if "grabbing the sources" is really a coherent story across all three build strategies
@john not sure what you are trying to do but given that node doesn't care much about build size I would go with two separate builds. two :target :node-script
basically. one :main
one :main your.worker/main
. no clue what you mean by "selective loading of deps". if that refers to loading CLJS namespaces on demand then your only option is going self hosted
worker_threads itself is pretty much the same setup as regular workers in a browser. you load a file designed for it and start listining for or posting messages back and forth
I consider this isMainThread
checking all over the place bad practice (just like regular web-workers) so I always go with writing your own entry namespace for the worker that is only ever loaded in the worker and as such never has to check what it is supposed to be
I’m having trouble getting --config-merge
to work:
clojure -m shadow.cljs.devtools.cli --config-merge '{:js-options {:js-package-dirs ["external/frontend_npm/node_modules"]}}' release frontend
returns:
Searched for npm packages in:
/Users/arohner/Programming/griffin/banksy/node_modules
i.e. without updating the actual list of package-dirs. Is there a way to debug what the config looks like post merge?
@arohner I guess that is a bug. the config merge happens later, after :js-package-dirs
is used
you can set :release {:js-options {:js-package-dirs ["external/frontend_npm/node_modules"]}}
in your build config to make it only apply to release builds but not dev?
although thinking about it that may have the same issue. how come you only want to set this conditionally?
I’m in the middle of transitioning our build to bazel. Some users are still on lein, and official builds will be done via bazel soon
feel free to open an issue about this though. not intentional that this isn't config-merge-able
It’s more just to avoid repeating myself. Bazel is very opinionated in general, and has a strict separation of input and output files. In Bazel’s opinion, no new files should be created in the git repo when you do a build, which includes ./node_modules. So bazel runs everything in a separate dir, which changes the path of node modules. Bazel defines its own node_modules directory, so it’d be nice if I could pass that in directly, because it’s a small DRY violation to list it in shadow. But yeah, not the end of the world.
I get that you put npm packages in a different dir. I do it myself and even recommend doing it. I just don't get why you don't do it always. Why are your "users are still on lein" not doing that also? independent of bazel? I mean this is not related to bazel in any way AFAICT?
Hello, i am using shadow-cljs to make a node module my code works when `:target :node-script` and it was working also before somedays with `:target :npm-module` (i dont know what changed and now doesn't work) now i get `cljs$core$ExceptionInfo [Error]: Promise error` `cause: MongoServerSelectionError: cljs is not defined` when i try to use `:target :npm-module` anyone knows what might caused this `cljs is not defined` ?
borkdude i used :node-library
and it is working fine, thank you for you help 🙂 i was stuck like 4 hours or more , we didnt find why it didn't work, but now all seem fine
@takis_ nice! you can also use :target :esm
for both browser and node now btw, but it's somewhat undocumented
more info on that here: https://clojureverse.org/t/generating-es-modules-browser-deno/6116
npm-module is maintained just fine. it just has some fundamental issues that make it a bit hacky. cljs is not defined
is not one of them but I can't say more without seeing actual code and config and how you use it. Don't know what the MongoServerSelectionError
is about?
i am so new in both cljs/node/shadow-cljs, but i will send you my code, dont know if it can help alot
(defn f1 []
(go (let [mongodb (js/require "mongodb")
MongoClient (.-MongoClient mongodb)
_ (prn "Until here is ok")
_ (prn MongoClient)
client (<p! (.connect (MongoClient. "" (clj->js {"useUnifiedTopology" true}))))
_ (prn "Will never be printed if :npm-library")
])))
(f1)
Uncaught:
cljs$core$ExceptionInfo [Error]: Promise error
at new cljs$core$ExceptionInfo (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.js:37699:10)
at Function.cljs$core$IFn$_invoke$arity$3 (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.js:37760:9)
at /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.async.interop.js:32:84 {
data: {
meta: null,
cnt: 1,
arr: [ [Object], [Object] ],
__hash: null,
'cljs$lang$protocol_mask$partition0$': 16647951,
'cljs$lang$protocol_mask$partition1$': 139268
},
cause: MongoServerSelectionError: cljs is not defined
at Timeout._onTimeout (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/mongodb/lib/sdam/topology.js:310:38)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
reason: TopologyDescription {
type: 'Unknown',
servers: [Map],
stale: false,
compatible: true,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
logicalSessionTimeoutMinutes: undefined
}
},
description: undefined,
number: undefined,
fileName: undefined,
lineNumber: undefined,
columnNumber: undefined
}
before some days it worked with :npm-library
now only works with :node-library
and :node-script
probably its my code or the way i did the compilation/used it from node, and its not related to cljs/shadow/mongo or the nodejs mongo driver
~/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp$ node
Welcome to Node.js v12.22.6.
Type ".help" for more information.
> var c=require("./queries/clojurescriptapp.core");
undefined
> "Until here is ok"
#object[MongoClient]
Uncaught:.....the above error........
why is this mongodb (js/require "mongodb")
in a go
block? none of this should be a in a go block
but what is your intention with :npm-module
in the first place? why do you want to use it?
:builds {:library {:target :npm-module
:output-dir "./queries"
:compiler-options {:infer-externs true}
:entries [clojurescriptapp.core]}
:library2 {:target :node-library
:output-to "dist/index.js"
:exports {:f1 clojurescriptapp.core/f1}}
:app {:target :node-script
:output-to "target/main.js"
:source-map true
:main clojurescriptapp.core/main
:compiler-options {:infer-externs true}
:devtools {:repl-init-ns clojurescriptapp.core
:repl-pprint true}
}}
ok good, it works also, thank you for you time and shadow, i am so new but helped me already alot
which shadow-cljs version do you use? which clojurescript and closure-library versions? do you use deps.edn or project.clj or just plain shadow-cljs.edn with no extra tools?
{
"name": "clojurescriptapp",
"version": "0.1.0",
"description": "",
"main": "index.js",
"scripts": {
"watch": "shadow-cljs watch app",
"compile": "shadow-cljs compile app",
"build": "shadow-cljs release app"
},
"keywords": [],
"author": "takis",
"license": "MIT",
"devDependencies": {
"shadow-cljs": "^2.10.12",
"source-map-support": "^0.5.19",
"ws": "^7.3.0",
"mongodb": "4.0.0"
}
}
that doesn't tell me which version you use. I mean its a version range but 2.10
is ancient history by now
{:source-paths ["src"]
:dependencies [[cmql "0.1.0-SNAPSHOT"]
[cmql-js "0.1.0-SNAPSHOT"]]
:builds {:library {:target :npm-module
:output-dir "./queries"
:compiler-options {:infer-externs true}
:entries [clojurescriptapp.core]}
:library2 {:target :node-library
:output-to "dist/index.js"
:exports {:f1 clojurescriptapp.core/f1}}
:app {:target :node-script
:output-to "target/main.js"
:source-map true
:main clojurescriptapp.core/main
:compiler-options {:infer-externs true}
:devtools {:repl-init-ns clojurescriptapp.core
:repl-pprint true}
}}}
shadow-cljs info
shadow-cljs - config: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/shadow-cljs.edn
=== Version
jar: 2.15.9
cli: 2.15.9
deps: 1.3.2
config-version: 2.15.9
=== Paths
cli: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/shadow-cljs/cli/dist.js
config: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/shadow-cljs.edn
project: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp
cache: .shadow-cljs
thank you for your time, i will update read more of the user-guide and i will retry it, probably i did something wrong, but node-library works and you said its the right way , so all works
FWIW the code without go would be something like
(ns whatever
(:require ["mongodb" :as mdb]))
(defn f1 []
(-> (mdb/MongoClient. "" #js {:useUnifiedTopology true})
(.connect)
(.then (fn []
(prn [:mongodb-connected])
))))
~/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp$ node
Welcome to Node.js v12.22.6.
Type ".help" for more information.
> var c=require("./queries/clojurescriptapp.core");
undefined
> c.f1();
Promise { <pending> }
> (node:8986) UnhandledPromiseRejectionWarning: MongoServerSelectionError: cljs is not defined
at Timeout._onTimeout (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/mongodb/lib/sdam/topology.js:310:38)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7)
(node:8986) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 1)
(node:8986) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.