This error occur with normal compilation?

Not in dev no, this is strictly with advanced compilation, local as well as CI. It’ll randomly work sometimes. I saw the same problem when initially switching to Electric with incremental compilation. I went around and (it felt like) randomly moved things around in the codebase, and the problem disappeared. The background is that I’m working on a commit where I’ve thrown out our old (non-Electric) code, and this problem has now reoccurred in this branch. It includes no changes to the Electric part of the app, but is a deletion of 200 files and 30k lines of code of non-Electric stuff. The fact that I saw it the first time when switching to IC hints that it might be Electric related, but the fact that I haven’t touched Electric code in this branch indicates that it might be something else. I’m a bit lost as to how to pin down what the source of the problem is.

This is very strange indeed. The CI has cache or something like that?

Yeah, normal Docker cache stuff for deps, but not for code. Also, since I’m seeing the same thing locally, where I can control caching entirely, indicates that something else is at foot.

Do you use Rama? I have similar issue with IC and Rama together.

Aha, interesting to hear. No, not in this branch.

The randomness could indicate that it has something to do with load ordering. It happens in the CLJ phase of Shadow-cljs compilation anyway.

Some more examples. What fails exactly differs from time to time, but it all kind of seems to bottom out in libraries rather than our own code.

On closer inspection, all errors seem to be complaining about CLJC files of this nature:

(ns some-ns
  #?(:cljs (:require-macros some-ns))

please confirm - can you ever repro locally (not CI) with deleted shadow caches?

Yes, it’s reproducible locally with a brutally clean repo.

But only advanced compile, it doesn’t occur in dev.

I’ve managed to proceed by removing :require-macros and doing some-ns/macro-fn instead. This eliminates that particular NS from the pile of possible error candidates.

I suspect that if I refactor the rest, it will go through predictably. I’ll confirm and get back to you.

please post shadow version

2.27.2

Random fact: our app weighs in at ~13mb uncompressed (2.9mb compressed)

An early theory was that it was some OOM and failing to load nses during compilation, but it seems that is not it.

i don’t recall seeing any issues that match this previously but IC has not been out very long, i’ll get back to you

I encountered the same when initially switching to IC.

I've seen similar error beforehttps://github.com/hyperfiddle/electric/blob/ae1341bc8c5566b45179c9407bc9d902e91c0847/src/hyperfiddle/electric/impl/expand.clj#L32

I also encountered compilation errors when updating to IC with this sort of cljc code

(ns some-ns
  #?(:cljs (:require-macros some-ns))

I’ve now removed these in favour of direct refers, and it looks like I can now compile it consistently. The seeming “soft limit” on this is confusing, it seems like the kind of thing that should fail consistently or not at all. Not trigger after some level of saturation.

Please can you copy paste an example of exactly what you had at first and what you changed it to?

Sure. So for example (where text-area-primary is a macro):

[app.electric.component.text-area :refer [text-area-primary]]

…

(text-area-primary …)

[app.electric.component.text-area :as text-area]

…

(text-area/text-area-primary …)

(ns app.electric.component.text-area
  #?(:cljs (:require-macros app.electric.component.text-area))
  (:require …))

(ns app.electric.component.text-area
  (:require …))

Bundle size seems to have shrunk as well. Hang on, I’ll confirm. False alarm, I was comparing a compile with debug symbols to one without.

So in principle we should be able to reproduce this in starter app, the problem is nondeterminism – only shows up in large electric codebases?

Please say more about the "soft limit" idea, I didn't follow that

I’m not saying there’s an actual “soft limit”, it’s more of an expression of my confusion. This pattern of requiring macros was repeated pretty often in the codebase, ~15 times. So: 1. Why did I not see this problem since the initial IC migration, and only now after removing unrelated code? 2. Removing one instance of this pattern of macro requirement excluded that instance from showing up (but there were still problems). 3. Removing a second one dramatically increased the chances of getting a compiling build. 4. Removing all of them now seems to produce predictably successful builds. Did I happen to refactor the two instances that actually had some kind of problem with them? Or could it have been any of them, just that I’m decreasing some kind of pressure somewhere by omitting them?

The app size is pretty substantial, so it’s not impossible that there’s some path of graph expansion would lead to a giant tree which subsequently blew some fuse, making the compiler blame the nearest thing at hand (The term “graph expansion” used loosely; I don’t understand the particulars well enough to be precise in my terminology).

Because come on shadow; blaming internals of XTDB to be missing while compiling CLJS? Really?

Ok, and just to confirm, the reproduction always involves :require-macros in the module ns and :refer [my-macro] in the consuming ns?

Yes, that seems to be the culprit. Backing to a commit where this hasn’t been refactored promptly reintroduced the problems.

And the intervening commits contained only refactoring of this.

Btw @UHZPYLPU1, depending on how you require your Rama stuff (which is a giant pile of macros), refactoring this might eliminate your problems.

Not to mention it would be interesting to have the pattern confirmed if it is indeed the same problem that you saw.

So did you get the 30k LOC deleted?

It landed at 20k in the end, but still a relief.

(e/fn [] x) will type check as a missionary continuous flow, which is to say that it is a flow which has an initial value and is defined for all time (always has a most recent value). Strictly speaking, "continuous flow" and "signal" are not quite the same thing. In missionary the word "signal" means: a continuous flow that has been memoized. m/signal is the missionary operator which allocates and supervises the memo buffer. Strictly speaking, (e/fn [] x) is not a signal, it is just a continuous flow. Electric let is the electric operator which compiles down to m/signal (i.e., all electric let bindings are memoized).

My statements to beginners regarding missionary signals may be blurring the distinction between signal and continuous flow, which is hard to avoid when matching people's priors from other libraries which may not separate or cleanly distinguish between the two

I don't think we have clear docs currently about the interaction; we used to have some examples but they are "in transition" and need to be checked and republished

The electric lang test suite is a great place to look

big picture: as you know, e/fn returns a value that typechecks as missionary continuous flow. Similarly, new—in addition to "calling"/booting an electric lambda—will accept a missionary continuous flow and join it into the electric program

i.e., you might think of this aspect of new as await on missionary flows

Thank you, that helps a lot

I don't think we've ever tried to do that

RCF has a clojure.test integration. You can configure it to generate clojure.test deftest and then use clojure.test/run-all-tests #"ns-regex.*"

Do you mean: I could pass the JVM opt for the :dev alias and this will generate deftests?

Yup, that works. Thanks!

Any possible footguns?

it’s meant for CI, due to mutable global namespace state i think you can dangle tests if you rename or delete

To unpack that for future reference: • deftest has this 'dangling' problem. rcf makes it (more?) unmanageable by not having stable test names. • Enabling rcf to emit deftest at dev time hence introduces this footgun, which doesn't exist in the usual rcf workflow. I think this may be enough for me not to enable the integration at dev time. I guess another option is to manually trigger a re-evaluation of nses.

you can also do it from the CLI (which gives you clj and cljs in the same run). you could shell out from the repl?

Yep, yep. The motivation is actually not having to run through CLI when the cljs setup is already running (during dev).

Evidently haha https://github.com/hyperfiddle/electric/blob/ae1341bc8c5566b45179c9407bc9d902e91c0847/src/hyperfiddle/incseq.cljc#L128