Clojurians Log v2
shadow-cljs
I'm working on adding (finally!) a CSP in a production CLJS app, using shadow-cljs with code splitting etc. I have managed to get everything working, as long as there's only one module (the base one) loaded at a time. However, the moment another module is lazy loaded, then the whole process stalls, and eventually craps out with a Error loading <module>: Consecutive load failures (?) (seems to be coming from goog.module.ModuleManager) - there's nothing else in the console to indicate a CSP failure, the actual network call to load <module>.js succeeds.
Oh, this is with shadow-cljs 2.28.23, we haven't upgraded to 3.x yet
Adding a permissive CSP with script-src: 'unsave-eval' restores functionality but defeats the purpose (well, a major part of it). Trying now with 3.2.1 - seems like there's now apparent issues when upgrading, which is nice
Nope, shadow-cljs 3.2.1 still has the same issue.
Just to clarify, in the stacktrace, I see our own nosco.lazy namespace which in turn uses shadow.lazy/load
yes, the regular shadow.lazy loader uses the goog.module.ModuleManager stuff to load things and requires unsafe-eval
you can instead use :target :esm and the new method of lazy loading stuff, which uses the native browser import() and requires no eval
Oof, thanks for chiming in, I can stop trying various combinations now. So the old module loader just passes in a string in eval? I thought it would also try to append a script tag to the DOM
see https://github.com/thheller/shadow-cljs/issues/1238#issuecomment-3424826578 for an example of the new stuff
you can make it use script tags yeah. forgot what the closure define for that is though
I'd strongly recommend using :esm instead though. much better overall and all browser built-in stuff
I'll look into :esm - the scare, as always, is the amount of time it would take to migrate the codebase/build tools. But it seems config-wise, that it's roughly similar
yeah pretty much the same
Hm, need to figure out how to migrate
(defn lazy-component* [loadable]
(React/lazy
(fn []
;; React lazy expects a promise, that returns an ES6 module with a React Component as default export
(-> (shadow.lazy/load loadable)
(.then
(fn [root-el]
(if-not js/goog.DEBUG
;; in production mode, just do that
#js {:default root-el}
;; in dev mode, we need wrap the loaded component one
;; extra level so live-reload actually works since React
;; will keep a reference to the initially loaded fn and
;; won't update it
#js {:default (React/forwardRef
(fn lazy-loaded [props maybe-ref]
(if maybe-ref
(let [props' (goog.object/clone props)]
(goog.object/set props' "ref" maybe-ref)
(React/createElement @loadable props' (.-children props)))
(React/createElement @loadable props (.-children props)))))})))))))Plus
(defmacro lazy-component [the-sym]
`(nosco.lazy/lazy-component* (shadow.lazy/loadable ~the-sym)))a lot simpler
doesn't need a macro at all anymore
Well, I still need a wrapper to add the React.lazy machinery, but I guess that doesn't need to be a macro, indeed.
Looks to be working... but I'm getting issues with the upgrade to modules I guess > shadow-cljs - failed to load module$node_modules$sse_DOT_js$lib$sse (loading sse.js)
(defmacro lazy-component [the-sym]
`(nosco.lazy/lazy-component* (subs (str ~the-sym) 1)))(defn lazy-component* [loadable-name]
(React/lazy
(fn []
(js/console.log "Lazy Loading Loadable " loadable-name)
;; React lazy expects a promise, that returns an ES6 module with a React Component as default export
(-> (esm/load-by-name loadable-name)
(.then
(fn [root-el-f]
(js/console.log "ROOT EL", root-el-f)
(if-not js/goog.DEBUG
;; in production mode, just do that
#js {:default (root-el-f)}
;; in dev mode, we need wrap the loaded component one
;; extra level so live-reload actually works since React
;; will keep a reference to the initially loaded fn and
;; won't update it
#js {:default (React/forwardRef
(fn lazy-loaded [props maybe-ref]
(if maybe-ref
(let [props' (goog.object/clone props)]
(goog.object/set props' "ref" maybe-ref)
(React/createElement (root-el-f) props' (.-children props)))
(React/createElement (root-el-f) props (.-children props)))))})))))))(The macro is not needed, but for now I'm looking into just migrating)
and why did it fail loading? there is another error that tells you why, the one you pasted is just what failed to load
js.js:90 TypeError: Cannot set property SSE of #https://www.npmjs.com/package/sse.js?activeTab=code The offending code might be
/** @type {number} */
SSE.INITIALIZING = -1;
/** @type {number} */
SSE.CONNECTING = 0;
/** @type {number} */
SSE.OPEN = 1;
** @type {number} */
SSE.CLOSED = 2;It seems like the processed file looks like so:
shadow$provide["module$node_modules$sse_DOT_js$lib$sse"] = function(require,module,exports) {
Object.defineProperties(exports, {__esModule:{enumerable:true, value:true}, SSE:{enumerable:true, get:function() {
return SSE;
}}});
<...module code follows...>event.responseCode = e.currentTarget.status; is the line it complains about
Hm, but would that be triggered by just requiring the module?
I don't have the slightest clue what any of that does 😛
(sse.js:198:1) is line 198 in that sse.js file. that is all I know.
Well, the line you mentioned is supposed to run inside an event handler. I have disabled all access to the module - it fails just by requiring it.
I can't really comment on any of this. a repro would help. don't see why it would be behaving differently in :target :esm than before
other than the browser using modules being a little more strict on stuff
dunno where you got the snippet above, but it seems incomplete?
Sorry, I meant to type that the module code follows.
I'm getting this through the browser debugger when I click through the stack trace
I'm trying to edit the file directly in node-modules/... but it seems like it's not getting picked up, probably there's a cache somewhere
yes, touch the node_modules/sse/package.json file to invalidate the cache
maybe best way to go about debugging is doing a compile build once, then edit the module$node_modules$sse_DOT_js$lib$sse.js file manually to add some console.log or whatever
This file lives in the .shadow-cljs cache directory, right?
I've got it. The issue is with
if (typeof exports !== "undefined") {
exports.SSE = SSE;
}var SSE3 = function(url, options) {
console.log("look at me");
}
// Export our SSE module for npm.js
if (typeof exports !== "undefined") {
exports.SSE3 = SSE3;
}
// Export as an ECMAScript module
export { SSE3 };exports.SSE3 = SSE3 doesn't.Same if I revert the entirety of the file and remove the offending lines.
And it all makes sense since shadow injects
Object.defineProperties(exports, {__esModule:{enumerable:true, value:true}, SSE:{enumerable:true, get:function() {
return SSE;
}}});SSE which has only a getter, but the code later tries to set the SSE property (which is already set). I guess the library is at fault here, since this is an ESM module and shouldn't try to to add to exports, right?no clue why it would be trying to do both
do not mess with source file
debug with the file shadow-cljs has generated. thats the only file I can maybe comment on.
it lives in the output dir
don't edit it while a watch is running, since that'll just overwrite whatever you edited
Sorry, the module.exports = SSE is coming from the library file.
Looking at the generated file, line 180 (that flashes before my eyes before the sourcemap takes over and breaks things) is:
typeof exports !== "undefined" && (exports.SSE = SSE);import "./cljs_env.js";
import "./shadow.js.js";
shadow$provide.module$node_modules$sse_DOT_js$lib$sse = function(require, module, exports) {
Object.defineProperties(exports, {__esModule:{enumerable:!0, value:!0}, SSE:{enumerable:!0, get:function() {
return SSE;
}}});
var SSE = function(url, options) {
if (!(this instanceof SSE)) {
return new SSE(url, options);
}
this.url = url;
options = options || {};
this.headers = options.headers || {};
this.payload = options.payload !== void 0 ? options.payload : "";
this.method = options.method || this.payload && "POST" || "GET";
this.withCredentials = !!options.withCredentials;
this.debug = !!options.debug;
this.autoReconnect = options.autoReconnect !== void 0 ? options.autoReconnect : !1;
this.reconnectDelay = options.reconnectDelay !== void 0 ? options.reconnectDelay : 3000;
this.maxRetries = options.maxRetries !== void 0 ? options.maxRetries : null;
this.retryCount = 0;
this.reconnectTimer = null;
this.useLastEventId = options.useLastEventId !== void 0 ? options.useLastEventId : !0;
this.FIELD_SEPARATOR = ":";
this.listeners = {};
this.xhr = null;
this.readyState = SSE.INITIALIZING;
this.progress = 0;
this.lastEventId = this.chunk = "";
this.addEventListener = function(type, listener) {
this.listeners[type] === void 0 && (this.listeners[type] = []);
this.listeners[type].indexOf(listener) === -1 && this.listeners[type].push(listener);
};
this.removeEventListener = function(type, listener) {
if (this.listeners[type] !== void 0) {
var filtered = [];
this.listeners[type].forEach(function(element) {
element !== listener && filtered.push(element);
});
filtered.length === 0 ? delete this.listeners[type] : this.listeners[type] = filtered;
}
};
this.dispatchEvent = function(e) {
if (!e) {
return !0;
}
this.debug && console.debug(e);
e.source = this;
const onHandler = "on" + e.type;
return this.hasOwnProperty(onHandler) && (this[onHandler].call(this, e), e.defaultPrevented) ? !1 : this.listeners[e.type] ? this.listeners[e.type].every(function(callback) {
callback(e);
return !e.defaultPrevented;
}) : !0;
};
this._markClosed = function() {
this.xhr = null;
this.progress = 0;
this.chunk = "";
this._setReadyState(SSE.CLOSED);
this.autoReconnect && (this.maxRetries !== null && this.retryCount >= this.maxRetries ? (this.debug && console.debug(`SSE max retries (${this.maxRetries}) reached, stopping reconnection attempts`), this.autoReconnect = !1, this.close()) : (this.reconnectTimer && clearTimeout(this.reconnectTimer), this.debug && console.debug(`SSE will attempt to reconnect in ${this.reconnectDelay}ms (attempt ${this.retryCount + 1}${this.maxRetries ? "/" + this.maxRetries : ""})`), this.reconnectTimer = setTimeout(() =>
{
this.reconnectTimer = null;
this.retryCount++;
this.stream();
}, this.reconnectDelay)));
};
this._setReadyState = function(state) {
const event = new CustomEvent("readystatechange");
this.readyState = event.readyState = state;
this.dispatchEvent(event);
};
this._onStreamFailure = function(e) {
const event = new CustomEvent("error");
event.responseCode = e.currentTarget.status;
event.data = e.currentTarget.response;
this.dispatchEvent(event);
this._markClosed();
};
this._onStreamAbort = function() {
this.dispatchEvent(new CustomEvent("abort"));
this._markClosed();
};
this._onStreamProgress = function(e) {
if (this.xhr) {
if (this.xhr.status < 200 || this.xhr.status >= 300) {
this._onStreamFailure(e);
} else {
this.retryCount = 0;
e = this.xhr.responseText.substring(this.progress);
this.progress += e.length;
e = (this.chunk + e).split(/(\r\n\r\n|\r\r|\n\n)/g);
var lastPart = e.pop();
e.forEach(function(part) {
part.trim().length > 0 && this.dispatchEvent(this._parseEventChunk(part));
}.bind(this));
this.chunk = lastPart;
}
}
};
this._onStreamLoaded = function(e) {
this._onStreamProgress(e);
this.dispatchEvent(this._parseEventChunk(this.chunk));
this.chunk = "";
this._markClosed();
};
this._parseEventChunk = function(chunk) {
if (!chunk || chunk.length === 0) {
return null;
}
this.debug && console.debug(chunk);
const e = {id:null, retry:null, data:null, event:null};
chunk.split(/\n|\r\n|\r/).forEach(function(line) {
const index = line.indexOf(this.FIELD_SEPARATOR);
let field;
if (index > 0) {
const skip = line[index + 1] === " " ? 2 : 1;
field = line.substring(0, index);
line = line.substring(index + skip);
} else if (index < 0) {
field = line, line = "";
} else {
return;
}
field in e && (field === "data" && e[field] !== null ? e.data += "\n" + line : e[field] = line);
}.bind(this));
if (e.data === null) {
return null;
}
e.id !== null && (this.lastEventId = e.id);
chunk = new CustomEvent(e.event || "message");
chunk.id = e.id;
chunk.data = e.data || "";
chunk.lastEventId = this.lastEventId;
return chunk;
};
this._onReadyStateChange = function() {
if (this.xhr && this.xhr.readyState === XMLHttpRequest.HEADERS_RECEIVED) {
const headers = {};
var headerPairs = this.xhr.getAllResponseHeaders().trim().split("\r\n");
for (var headerPair of headerPairs) {
const [key, ...valueParts] = headerPair.split(":");
headerPairs = valueParts.join(":").trim();
headers[key.trim().toLowerCase()] = headers[key.trim().toLowerCase()] || [];
headers[key.trim().toLowerCase()].push(headerPairs);
}
headerPair = new CustomEvent("open");
headerPair.responseCode = this.xhr.status;
headerPair.headers = headers;
this.dispatchEvent(headerPair);
this._setReadyState(SSE.OPEN);
}
};
this.stream = function() {
if (!this.xhr) {
this._setReadyState(SSE.CONNECTING);
this.xhr = new XMLHttpRequest();
this.xhr.addEventListener("progress", this._onStreamProgress.bind(this));
this.xhr.addEventListener("load", this._onStreamLoaded.bind(this));
this.xhr.addEventListener("readystatechange", this._onReadyStateChange.bind(this));
this.xhr.addEventListener("error", this._onStreamFailure.bind(this));
this.xhr.addEventListener("abort", this._onStreamAbort.bind(this));
this.xhr.open(this.method, this.url);
for (let header in this.headers) {
this.xhr.setRequestHeader(header, this.headers[header]);
}
this.useLastEventId && this.lastEventId.length > 0 && this.xhr.setRequestHeader("Last-Event-ID", this.lastEventId);
this.xhr.withCredentials = this.withCredentials;
this.xhr.send(this.payload);
}
};
this.close = function() {
this.readyState !== SSE.CLOSED && (this.reconnectTimer && (clearTimeout(this.reconnectTimer), this.reconnectTimer = null), this.autoReconnect = !1, this.xhr.abort());
};
(options.start === void 0 || options.start) && this.stream();
};
SSE.INITIALIZING = -1;
SSE.CONNECTING = 0;
SSE.OPEN = 1;
SSE.CLOSED = 2;
typeof exports !== "undefined" && (exports.SSE = SSE);
};
//# sourceMappingURL=module$node_modules$sse_DOT_js$lib$sse.js.mapHere's the entirety of the generated output file
I do not understand why a file would have exports and export. that seems idiotic to me. so I fully expect this to break in many ways. just don't see how its related to switching to :esm in any way?
for context. this bit
Object.defineProperties(exports, {__esModule:{enumerable:!0, value:!0}, SSE:{enumerable:!0, get:function() {
return SSE;
}}});export { SSE };Perhaps it's missing when the target is :browser?
so this is all entirely working as intended. it just doesn't know what to do with the exports.SSE bit since I have never seen a file that tries to be esm and commonjs in one
no, browser rewrites this in the exact same way. the code that rewrites npm code isn't target specific, there is only one implementation
only guess I have is that ESM is treated more strictly by the browser, but I'm guessing entirely blind. make a repro and I can take a look, otherwise I'm out of answers
I will file an issue with the library. It doesn't make sense, I agree - they clearly state in package.json that it's a type: "module" so why are they mucking about with exports.
The only workaround I can think, is renaming the function signature to be function(require, module, _exports) so that the code checking that exports is not defined will skip over.
well its called exports so actual commonjs code works, which is still most of npm
Ah, yes. I forgot it's the same handler for modules and commonjs.
Hi guys!
I have the following shadow-cljs.edn:
{:deps
{:aliases [:dev :test :shadow]}
:builds
{:app
{:target :node-script
:main app.core/main
:output-to "projects/app/target/core.js"}}}watch command like this:
yarn shadow-cljs watch appwatch with cider but when I want load namespace I have No available JS runtime
So I execute node core.js and I got this error:
SHADOW import error /Users/jnbdt/Documents/beop/backend/.shadow-cljs/builds/page-extractor/dev/out/cljs-runtime/shadow.js.shim.module$ws.js
/Users/jnbdt/Documents/beop/backend/projects/page-extractor/target/main.js:67
throw e;
^
Error: Cannot find module 'ws'
Require stack:
- /Users/jnbdt/Documents/beop/backend/projects/page-extractor/target/main.js
at Function._resolveFilename (node:internal/modules/cjs/loader:1383:15)
at defaultResolveImpl (node:internal/modules/cjs/loader:1025:19)
at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1030:22)
at Function._load (node:internal/modules/cjs/loader:1192:37)
at TracingChannel.traceSync (node:diagnostics_channel:328:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:237:24)
at Module.require (node:internal/modules/cjs/loader:1463:12)
at require (node:internal/modules/helpers:147:16)
at /Users/jnbdt/Documents/beop/backend/.shadow-cljs/builds/page-extractor/dev/out/cljs-runtime/shadow.js.shim.module$ws.js:3:61
at global.SHADOW_IMPORT (/Users/jnbdt/Documents/beop/backend/projects/page-extractor/target/main.js:64:44) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/Users/jnbdt/Documents/beop/backend/projects/page-extractor/target/main.js'
]
}shadow-cljs is not installed globally?
https://clojurians.slack.com/archives/C6N245JGG/p1715785801951599?thread_ts=1715784816.056719&cid=C6N245JGG looks like this was discussed a bit ago. perhaps this can help you out?
https://clojurians.slack.com/archives/C6N245JGG/p1665850329119049?thread_ts=1665834443.697369&cid=C6N245JGG here a bit more explicit
I have already see this post. It work now when I use node_modules instead of .yarn/cache.
does that mean you are good?
Yes
Thank you
wasn’t sure if it was easy to swap to using node_modules or if that was an imperfect solution. have you tried to yarn install it so you can have it in yarn’s preferred way of working?
I just run yarn config set node-modules && yarn install with the ws package inside de devDependencies