Fork me on GitHub
#shadow-cljs
<
2024-01-22
>
pmooser14:01:35

I'm still trying to see if there's a way to get the newest automerge working in the context of a cljs application built by shadow-cljs. I'm hoping that I can load the wasm myself, and avoid any kind of weird external bundling, because I was running into race conditions having to do with using webpack as an external bundler. I'm sort of trying to start from scratch a little bit, and depending on @automerge/[email protected], when I try to do a shadow-cljs build, I see the following:

[:app-simple] Configuring build.
[:app-simple] Compiling ...
[:app-simple] Build failure:
Failed to inspect file
  /Users/me/project-wasm/node_modules/@automerge/automerge-wasm/bundler/automerge_wasm_bg.wasm

Errors encountered while trying to parse file
  /Users/me/project-wasm/node_modules/@automerge/automerge-wasm/bundler/automerge_wasm_bg.wasm
  {:line 1, :column 0, :message "Character '' (U+0000) is not a valid identifier start char"}
Does that kind of error message give you any clue about what might be happening ? I don't know if this could be because of how this library is being referenced in my imports ?

thheller14:01:35

didnt we go over this before? shadow-cljs does not support bundling .wasm files. so it any sources do import ... from "./some.wasm" that will fail, as shadow-cljs expects to find a file with JS syntax, which a .wasm file is not

thheller14:01:15

as such you get the above parser error

pmooser14:01:36

No, in fact, this is in line with the last thing we discussed - where we left this was, with the race condition introduced by webpack, this may be impossible, but in fact if I can load the wasm asynchronously and pass it to the library itself that needs it to be initialized, it might work, without needing any kind of special bundling from shadow-cljs or webpack or anything.

pmooser14:01:09

I can try to dig up that thread but I'm reasonably certain that's where we left it, so that's why I'm trying to go back and remove the webpack bundling.

thheller14:01:48

thats what led you to webpack in the first place. shadow-cljs cannot process this. webpack can, but implies that the loading mechanism becomes async

pmooser14:01:02

No, that's not the conversation we had.

pmooser14:01:06

I'll look for the thread.

thheller14:01:50

well the facts haven't changed so I doubt I said anything different

thheller14:01:10

automerge moved from something where you can supply the wasm yourself

thheller14:01:18

to something were it relies on the bundler supplying it

pmooser14:01:27

It appears quite clear.

thheller14:01:45

yes, exactly. that would be an option, but it was a hypothetical for something that doesn't exist in automerge

pmooser14:01:56

Ok, what aren't you understanding?

pmooser14:01:01

It does exist in automerge in fact -

pmooser14:01:07

that's entirely what I'm talking about.

pmooser14:01:36

That's why I said "if I can load the wasm asynchronously and pass it to the library itself" ...

thheller14:01:25

ok. there is a misunderstanding here I guess.

thheller14:01:34

the error you just posted that started this thread

thheller14:01:47

I assume you get when you (:require ["@automerge/automerge" :as whatever])

pmooser14:01:15

Yes, I'm just including:

["@automerge/automerge/next" :as am]

thheller14:01:30

ok, and doing ONLY that yields the above error?

pmooser14:01:32

I'm trying to figure out how I would remove all references to automerge in the code, since this is an existing codebase that goes back years (but of course used a different version).

pmooser14:01:53

I don't see any of my debugging logs get emitted that would occur when I try to do something with the module ...

thheller14:01:25

ok, let me put it another way. some of the transitive dependencies brought in by your automerge require ends up doing import ... from "./automerge_wasm_bg.wasm" which shadow-cljs is trying to process but cannot

thheller14:01:43

and since its in those files you cannot configure or remove it, well unless you are willing to rewrite automerge

pmooser14:01:10

Ok, so if I understand correctly then, since this just happens as a byproduct of that require, there's actually no workaround, unless I have a local version of automerge that doesn't perform such an import automatically.

pmooser14:01:47

Thank you for your help.

pmooser14:01:59

I think that's enough for me to go back and talk to the automerge folks again.

thheller14:01:48

the screenshot of the JS example you posted isn't maybe clear enough about what structure you'd need for CLJS, so let me give a second snippet to complement

thheller14:01:17

import * as Automerge from "@automerge/automerge";
let wasm = await fetch_util("/some/location/automerge.wasm");
let doc = Automerge.init(wasm);

thheller14:01:41

the goal being that wasm can be supplied by other methods than just import

thheller14:01:17

this would enable YOU to just take that wasm file out of node_modules, put it on your server, load it via fetch or whatever in CLJS and start automerge that way

pmooser14:01:32

Yes, so like I mentioned, they do have such a facility in automerge, and it is in use by something else already (not a clojure project) ...

thheller14:01:38

without shadow-cljs ever knowing or caring that you use wasm files

pmooser14:01:58

And I think the fact that the main project imports stuff from the automerge-wasm project results in this breaking for us from a cljs perspective.

pmooser14:01:37

Anyway, I do understand what you're saying. Basically the automerge project itself can't actually have project references to the wasm project, or they'll get loaded transitively.

thheller14:01:49

yes, as this is a non-standard feature not all bundlers support. and no browsers at all.

thheller15:01:19

so basically expecting you to use webpack, but for webpack to do all this has to make the entire loading process async

thheller15:01:25

thus leading to your race conditions

thheller15:01:30

the issue is that you could configure shadow-cljs to just ignore this wasm require

thheller15:01:57

but the code using that require is expecting it to be available, so it will fail regardless

pmooser15:01:54

Ok, thank you thheller.

thheller15:01:25

> Yes, so like I mentioned, they do have such a facility in automerge

thheller15:01:52

as far as I understood it they *used* to have this, but removed it in favor of the package dealing with it internally

thheller15:01:14

at least as far as I can tell from looking at the code

thheller15:01:29

since there is a non-optional import * as wasm from "@automerge/automerge-wasm"; in line 55 here https://unpkg.com/browse/@automerge/[email protected]/dist/mjs/stable.js and the line after that immediately using it. no chance to do anything async inbetween.