Fork me on GitHub
#shadow-cljs
<
2021-09-18
>
john01:09:48

I've been working on some stuff to do with parallelism with SAB's atomics and it's really hard to get a good demo site going on github pages, with the latest security header requirements for SABs. So I've been wanting to migrate my testing over to nodejs and flesh it over there, until browsers settle on a security model. But I'm not seeing a lot of prior art out there for using CLJS with worker_threads. If I end up working on building something out on nodejs, I'd probably want to bust the thing that wraps the worker_threads api out into its own lib. Things is, it's inherently a code deploy situation, where you're pushing some set of code into the worker. How you go about doing that depends on your build environment. When I build my tau.alpha framework for the browser, I kinda hardcoded into the code where to find the sources to inject into the worker based on figwheel's semantics, iirc. I'm wondering if I can come up with some strategy to inject code into the worker in a way that will be compatible with vanilla clojure, figwheel and shadow - a build-tool agnostic method by which I can easily push my build into nodejs webworkers

john01:09:54

In the browser, I would just basically load the entire app in the worker. Then I would use conditional checks on whether I was in a worker or not to trigger certain behaviors on wokers vs main. Going that route, it's a little easier to perhaps engineer compatibility with vanilla, figwheel and shadow. However, thinking about selectively loading modules in particular workers, for instance, is a little less certain, wrt cross-build-tool compatibility

john01:09:14

So for this nodejs impl, I'm thinking about getting a little more nuanced and allowing for selective loading of deps in workers, but I'm not sure what the lay of the land is, in terms of what strategies would be compatible with the most build tools

john01:09:34

@thheller wondering if you have any opinions on this yet

john01:09:27

I mean, I know there's a webworker story already out there for the default cljs, figwheel and shadow, and things could maybe be patched up to work with worker_threads, in each case. But until that time comes, in a lib, I could just do the job of grabbing the sources and putting things where they belong myself. I'm just not sure if "grabbing the sources" is really a coherent story across all three build strategies

thheller06:09:35

@john not sure what you are trying to do but given that node doesn't care much about build size I would go with two separate builds. two :target :node-script basically. one :main one :main your.worker/main. no clue what you mean by "selective loading of deps". if that refers to loading CLJS namespaces on demand then your only option is going self hosted

thheller06:09:43

worker_threads itself is pretty much the same setup as regular workers in a browser. you load a file designed for it and start listining for or posting messages back and forth

thheller06:09:10

I consider this isMainThread checking all over the place bad practice (just like regular web-workers) so I always go with writing your own entry namespace for the worker that is only ever loaded in the worker and as such never has to check what it is supposed to be

arohner13:09:26

I’m having trouble getting --config-merge to work: clojure -m shadow.cljs.devtools.cli --config-merge '{:js-options {:js-package-dirs ["external/frontend_npm/node_modules"]}}' release frontend returns: Searched for npm packages in: /Users/arohner/Programming/griffin/banksy/node_modules i.e. without updating the actual list of package-dirs. Is there a way to debug what the config looks like post merge?

thheller15:09:19

@arohner I guess that is a bug. the config merge happens later, after :js-package-dirs is used

thheller15:09:10

you can set :release {:js-options {:js-package-dirs ["external/frontend_npm/node_modules"]}} in your build config to make it only apply to release builds but not dev?

thheller15:09:46

although thinking about it that may have the same issue. how come you only want to set this conditionally?

arohner18:09:53

I’m in the middle of transitioning our build to bazel. Some users are still on lein, and official builds will be done via bazel soon

thheller21:09:03

but why does that change which npm modules you include?

thheller21:09:53

feel free to open an issue about this though. not intentional that this isn't config-merge-able

arohner11:09:36

It’s more just to avoid repeating myself. Bazel is very opinionated in general, and has a strict separation of input and output files. In Bazel’s opinion, no new files should be created in the git repo when you do a build, which includes ./node_modules. So bazel runs everything in a separate dir, which changes the path of node modules. Bazel defines its own node_modules directory, so it’d be nice if I could pass that in directly, because it’s a small DRY violation to list it in shadow. But yeah, not the end of the world.

thheller14:09:12

I get that you put npm packages in a different dir. I do it myself and even recommend doing it. I just don't get why you don't do it always. Why are your "users are still on lein" not doing that also? independent of bazel? I mean this is not related to bazel in any way AFAICT?

thheller14:09:30

but I'll fix that bug regardless

thheller16:09:39

2.15.10 should allow you to config-merge js-package-dirs

Takis_19:09:01

Hello, i am using shadow-cljs to make a node module my code works when  `:target :node-script` and it was working also before somedays with `:target :npm-module`  (i dont know what changed and now doesn't work) now i get `cljs$core$ExceptionInfo [Error]: Promise error`  `cause: MongoServerSelectionError: cljs is not defined`  when i try to use  `:target :npm-module`  anyone knows what might caused this `cljs is not defined` ?

borkdude20:09:20

I think I've heard thheller say that npm-module isn't very well maintained

Takis_20:09:54

it was working before some days, i dont know what i might did

Takis_20:09:23

borkdude i used :node-library and it is working fine, thank you for you help 🙂 i was stuck like 4 hours or more , we didnt find why it didn't work, but now all seem fine

borkdude20:09:01

@takis_ nice! you can also use :target :esm for both browser and node now btw, but it's somewhat undocumented

Takis_20:09:26

thank you i will read it : )

thheller21:09:36

npm-module is maintained just fine. it just has some fundamental issues that make it a bit hacky. cljs is not defined is not one of them but I can't say more without seeing actual code and config and how you use it. Don't know what the MongoServerSelectionError is about?

Takis_22:09:47

i am so new in both cljs/node/shadow-cljs, but i will send you my code, dont know if it can help alot

Takis_22:09:56

(defn f1 []
  (go (let [mongodb (js/require "mongodb")
            MongoClient (.-MongoClient mongodb)
            _ (prn "Until here is ok")
            _ (prn MongoClient)
            client (<p! (.connect (MongoClient. "" (clj->js {"useUnifiedTopology" true}))))
            _ (prn "Will never be printed if :npm-library")

            ])))
(f1)

Takis_22:09:14

Uncaught:
cljs$core$ExceptionInfo [Error]: Promise error
    at new cljs$core$ExceptionInfo (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.js:37699:10)
    at Function.cljs$core$IFn$_invoke$arity$3 (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.js:37760:9)
    at /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/queries/cljs.core.async.interop.js:32:84 {
  data: {
    meta: null,
    cnt: 1,
    arr: [ [Object], [Object] ],
    __hash: null,
    'cljs$lang$protocol_mask$partition0$': 16647951,
    'cljs$lang$protocol_mask$partition1$': 139268
  },
  cause: MongoServerSelectionError: cljs is not defined
      at Timeout._onTimeout (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/mongodb/lib/sdam/topology.js:310:38)
      at listOnTimeout (internal/timers.js:554:17)
      at processTimers (internal/timers.js:497:7) {
    reason: TopologyDescription {
      type: 'Unknown',
      servers: [Map],
      stale: false,
      compatible: true,
      heartbeatFrequencyMS: 10000,
      localThresholdMS: 15,
      logicalSessionTimeoutMinutes: undefined
    }
  },
  description: undefined,
  number: undefined,
  fileName: undefined,
  lineNumber: undefined,
  columnNumber: undefined
}

Takis_22:09:56

i sended here because it said as cause cljs is not defined

Takis_22:09:05

before some days it worked with :npm-library now only works with :node-library and :node-script

Takis_22:09:02

probably its my code or the way i did the compilation/used it from node, and its not related to cljs/shadow/mongo or the nodejs mongo driver

Takis_22:09:19

~/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp$ node
Welcome to Node.js v12.22.6.
Type ".help" for more information.
> var c=require("./queries/clojurescriptapp.core");
undefined
> "Until here is ok"
#object[MongoClient]
Uncaught:.....the above error........

thheller22:09:22

why is this mongodb (js/require "mongodb") in a go block? none of this should be a in a go block

Takis_22:09:58

i didnt had them there, i just moved them there after the problem

thheller22:09:34

go in general makes everything 1000x harder to debug, so avoid it if you can

thheller22:09:04

but what is your intention with :npm-module in the first place? why do you want to use it?

Takis_22:09:23

i just want a node-app to call my clojurescript code

Takis_22:09:23

:builds {:library {:target :npm-module
                    :output-dir "./queries"
                    :compiler-options {:infer-externs true}
                    :entries [clojurescriptapp.core]}
                    
          :library2 {:target :node-library
                     :output-to "dist/index.js"
                     :exports {:f1 clojurescriptapp.core/f1}}          
 
          :app {:target :node-script
                :output-to "target/main.js"
                :source-map true
                :main clojurescriptapp.core/main
                :compiler-options {:infer-externs true}
                
                :devtools {:repl-init-ns clojurescriptapp.core
                           :repl-pprint true}
                }}

thheller22:09:31

for that :node-library is recommended

Takis_22:09:17

ok good, it works also, thank you for you time and shadow, i am so new but helped me already alot

Takis_22:09:31

if i read more i will re-try it in the future

thheller22:09:04

which shadow-cljs version do you use? which clojurescript and closure-library versions? do you use deps.edn or project.clj or just plain shadow-cljs.edn with no extra tools?

Takis_22:09:55

{
  "name": "clojurescriptapp",
  "version": "0.1.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "watch": "shadow-cljs watch app",
    "compile": "shadow-cljs compile app",
    "build": "shadow-cljs release app"
  },
  "keywords": [],
  "author": "takis",
  "license": "MIT",
  "devDependencies": {
    "shadow-cljs": "^2.10.12",
    "source-map-support": "^0.5.19",
    "ws": "^7.3.0",
    "mongodb": "4.0.0"
  }
}

Takis_22:09:05

thats my package.json

thheller22:09:23

that doesn't tell me which version you use. I mean its a version range but 2.10 is ancient history by now

thheller22:09:05

do you have :deps or :lein in shadow-cljs.edn?

Takis_22:09:16

shadow-cljs.edn only

thheller22:09:24

shadow-cljs info will tell you the version

Takis_22:09:38

{:source-paths ["src"]
 :dependencies [[cmql "0.1.0-SNAPSHOT"]
                [cmql-js "0.1.0-SNAPSHOT"]]
 :builds {:library {:target :npm-module
                    :output-dir "./queries"
                    :compiler-options {:infer-externs true}
                    :entries [clojurescriptapp.core]}
                    
          :library2 {:target :node-library
                     :output-to "dist/index.js"
                     :exports {:f1 clojurescriptapp.core/f1}}          
 
          :app {:target :node-script
                :output-to "target/main.js"
                :source-map true
                :main clojurescriptapp.core/main
                :compiler-options {:infer-externs true}
                
                :devtools {:repl-init-ns clojurescriptapp.core
                           :repl-pprint true}
                }}}

Takis_22:09:51

this is all my shadow-cljs.edn

Takis_22:09:26

shadow-cljs info
shadow-cljs - config: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/shadow-cljs.edn
=== Version
jar:            2.15.9
cli:            2.15.9
deps:           1.3.2
config-version: 2.15.9

=== Paths
cli:     /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/shadow-cljs/cli/dist.js
config:  /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/shadow-cljs.edn
project: /home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp
cache:   .shadow-cljs

thheller22:09:39

ok thats all fine

Takis_22:09:06

thank you for your time, i will update read more of the user-guide and i will retry it, probably i did something wrong, but node-library works and you said its the right way , so all works

thheller22:09:34

FWIW the code without go would be something like

(ns whatever
  (:require ["mongodb" :as mdb]))

(defn f1 []
  (-> (mdb/MongoClient. "" #js {:useUnifiedTopology true})
      (.connect)
      (.then (fn []
               (prn [:mongodb-connected])
               ))))

thheller22:09:41

I don't know about the error otherwise

Takis_22:09:45

~/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp$ node
Welcome to Node.js v12.22.6.
Type ".help" for more information.
> var c=require("./queries/clojurescriptapp.core");
undefined
> c.f1();
Promise { <pending> }
> (node:8986) UnhandledPromiseRejectionWarning: MongoServerSelectionError: cljs is not defined
    at Timeout._onTimeout (/home/white/IdeaProjects/cmql-projects/nodeapp/clojurescriptapp/node_modules/mongodb/lib/sdam/topology.js:310:38)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7)
(node:8986) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 1)
(node:8986) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

thheller22:09:17

yeah no clue

Takis_22:09:22

thank you for you time, because its possible that i done many things wrong, if i know more in the future i will re-try it