Fork me on GitHub
#shadow-cljs
<
2023-03-06
>
Eugen09:03:52

hello, so a shadow-cljs build is failing on my CI. the same code builds locally. on CI I get an error that says it cannot find a specific ns:

ExceptionInfo: no source by provide: dre.core
I checked and the file is there on CI. I run the same command I always ran, and the one I use to build locally and same issue. I have been looking at this for the past ~3h and I am stuck. Any ideas on what I should check?
clojure -Spath -M:app/build

Eugen09:03:07

I have same version of java on Local and CI:

openjdk version "17.0.5" 2022-10-18
I have on CI Clojure CLI version 1.11.1.1200 On local: Clojure CLI version 1.11.1.1237 Using shadow-cljs 2.18.0

thheller09:03:06

what is the full stacktrace?

thheller09:03:38

also maybe update shadow-cljs. I vaguely remember fixing an issue like this a while ago

Eugen10:03:05

3 | (ns 
-------^------------------------------------------------------------------------
An error occurred while generating code for the form.
ExceptionInfo: no source by provide: dre.core
	shadow.build.data/get-source-id-by-provide (data.clj:187)
	shadow.build.data/get-source-id-by-provide (data.clj:184)
	shadow.build.data/get-source-by-provide (data.clj:190)
	shadow.build.data/get-source-by-provide (data.clj:189)
	shadow.build.compiler/eval14043/fn--14046/fn--14049 (compiler.clj:361)
	clojure.core/map/fn--5935 (core.clj:2770)
	clojure.lang.LazySeq.sval (LazySeq.java:42)
	clojure.lang.LazySeq.seq (LazySeq.java:51)
	clojure.lang.RT.seq (RT.java:535)
	clojure.core/seq--5467 (core.clj:139)
	clojure.core/filter/fn--5962 (core.clj:2826)
	clojure.lang.LazySeq.sval (LazySeq.java:42)
	clojure.lang.LazySeq.seq (LazySeq.java:51)
	clojure.lang.RT.seq (RT.java:535)
	clojure.core/seq--5467 (core.clj:139)
	clojure.core/seq--5467 (core.clj:139)
	shadow.build.compiler/eval14043/fn--14046 (compiler.clj:366)
	clojure.lang.MultiFn.invoke (MultiFn.java:234)
	shadow.build.compiler/do-compile-cljs-resource/fn--14137/fn--14146/fn--14151 (compiler.clj:670)
	shadow.build.compiler/do-compile-cljs-resource/fn--14137/fn--14146 (compiler.clj:669)
	shadow.build.compiler/do-compile-cljs-resource/fn--14137 (compiler.clj:664)
	shadow.build.compiler/do-compile-cljs-resource (compiler.clj:607)
	shadow.build.compiler/do-compile-cljs-resource (compiler.clj:565)
	shadow.build.compiler/maybe-compile-cljs/fn--14240 (compiler.clj:958)
	shadow.build.compiler/maybe-compile-cljs (compiler.clj:957)
	shadow.build.compiler/maybe-compile-cljs (compiler.clj:933)
	shadow.build.compiler/par-compile-one (compiler.clj:1066)
	shadow.build.compiler/par-compile-one (compiler.clj:1021)
	shadow.build.compiler/par-compile-cljs-sources/fn--14276/iter--14298--14302/fn--14303/fn--14304/fn--14305 (compiler.clj:1139)
	clojure.core/apply (core.clj:667)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/with-bindings* (core.clj:1990)
	clojure.core/apply (core.clj:671)
	clojure.core/bound-fn*/fn--5818 (core.clj:2020)
	java.util.concurrent.FutureTask.run (FutureTask.java:264)
	java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
	java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
	java.lang.Thread.run (Thread.java:833)

--------------------------------------------------------------------------------
   4 |   {:authors ["Joël Kuiper" "Jozef Wagner" "Michiel Borkent"]}
   5 |   (:require [dre.core :as datahub]
   6 |             [dre.sentry :as sentry]))
   7 | 

Eugen10:03:22

I think it can't load dependencies or something ?!

Eugen10:03:43

I tried it in another branch and I get a similar error

thheller10:03:00

hmm no immediate guess. try upgrading, see if that fixes it

thheller10:03:47

it happens while trying to generate code, so I'm not sure how anything lazy gets there?

Eugen10:03:35

moved to another branch t osee if it builds and find diffs

thheller10:03:44

some CI systems aggressively kill stuff sometimes. don't know how that would manifest that way but maybe try setting a fixed memory limit?

thheller10:03:01

-Xmx1G or so via jvm opts?

Eugen10:03:06

on another branch it builds - so something changed with this branch

Eugen10:03:26

I will wait for the devs to check it out and will post back once we find the issue

Eugen10:03:37

I bellieved I tracked it down to a npm dependency that required g++ to build and that is not available on CI

Eugen10:03:43

once installed seems to work

Eugen10:03:56

dependency is microtime

Eugen10:03:24

and it builds a native gyp module

thheller11:03:21

not sure how it ends up failing in that way though. do you do any crazy macro stuff in this repo?

thheller11:03:50

I mean npm is not involved in any of the code running during emit?

thheller11:03:06

but best not ask question if its working now 😉

Eugen13:03:04

found the issue

Eugen13:03:02

the branch name is enhancement/decouple-frontend-3161 and contains / which ends up on the path on the CI. Jenkins Pipeline plugin creates a directory for each branch. The directory name is encoded on the filesystem though. But somehow it breakes the build.

drwxr-xr-x  2 jenkins jenkins 4096 Mar  6 12:18 enhancement%2Fdecouple-frontend-3161

Eugen13:03:45

not sure exactly if this is an issue with shadow-cljs or with Jenkins and how we use shadow-cljs in our build

thheller13:03:32

shadow-cljs doesn't know or care about git or branches, so not sure why this matters?

thheller13:03:44

drwxr-xr-x 2 jenkins jenkins 4096 Mar 6 12:18 enhancement%2Fdecouple-frontend-3161 what does this mean? I mean is it a directory in the project or "where" is this exactly?

Eugen13:03:10

I think it might be caused by shadow-cljs. Reproducer would be:

mv my-project enhancement%2Fdecouple-frontend-3161
cd enhancement%2Fdecouple-frontend-3161

clojure -J-Dclojure.main.report=stderr -M:app/build

thheller13:03:48

I don't see any shadow-cljs command yet 😉

Eugen13:03:08

:app/build
  {:main-opts   ["-m" "shadow.cljs.devtools.cli" "release" ":app"]
   :extra-paths ["src-prod/"]
   :extra-deps  {org.slf4j/slf4j-simple {:mvn/version "1.7.36"}}}

thheller13:03:35

that looks fine

Eugen13:03:18

if you have a shadow-cljs project using the jar way of calling shadow-cljs, you could rename it with that name - so it has the path with encoded / . the build should fail

thheller13:03:45

compiles fine for me

thheller13:03:04

/mnt/c/Users/thheller/code/tmp/enhancement%2Fdecouple-frontend-3161

Eugen13:03:07

hmm, I wonder what could be the issue then

Eugen13:03:41

I tried that locally for this project and it failed

thheller13:03:44

well did you update?

thheller13:03:58

I tested with current version of course

Eugen13:03:06

not yet (I am devops, not frond-end)

thheller13:03:37

shouldn't be more than changing the version in deps.edn from 2.18.0 to 2.21.0

thheller13:03:54

unless you are still running jdk8, that would be a problem

thheller13:03:12

ah nvm 17 is fine

Eugen13:03:34

I did update, same results

Eugen13:03:50

I'll ask the frontedn devs to check if they can reproduce this

Eugen13:03:02

but in the meantime it works

thheller15:03:24

do you have an updated stacktrace? maybe there is a clue in there somewhere

Eugen15:03:30

no, but someone from my team saw the same issue using the reproducer above.

Eugen15:03:44

perhaps it is related to a plugin or something?

Eugen15:03:34

this is the app config we are using

{:target :browser

   :output-dir "resources/js/"

   :modules
   {:app {:entries []
          #_:init-fn #_TODO}}

   :devtools
   {:after-load dre.core/reload!
    :preloads [dre.preload
               devtools.preload
               day8.re-frame-10x.preload]}

   :dev
   {:compiler-options
    {:closure-defines
     {re-frame.trace.trace-enabled?        true
      day8.re-frame.tracing.trace-enabled? true}}}

   :release
   {:build-options
    {:ns-aliases
     {day8.re-frame.tracing day8.re-frame.tracing-stubs}}}}

thheller15:03:18

nope. all fine.

thheller15:03:34

well, technically a macro could be doing absolute crazy stuff. can't control that

Eugen15:03:36

then I don't know unfortunatelly

thheller15:03:53

but an updated stacktrace would still be useful. the last one didn't map to current code.

Eugen15:03:05

ok, I am not familiar with the code. I do know there are some macros there from times gone by

Eugen15:03:19

but not sure what they do / if they are used in this context

Eugen15:03:44

updated stacktrace => stacktrace from 2.21.0 ?

thheller15:03:19

yeah, from 2.21.0

Eugen16:03:04

3 | (ns 
-------^------------------------------------------------------------------------
An error occurred while generating code for the form.
ExceptionInfo: no source by provide: dre.core
        shadow.build.data/get-source-id-by-provide (data.clj:192)
        shadow.build.data/get-source-id-by-provide (data.clj:187)
        shadow.build.data/get-source-by-provide (data.clj:195)
        shadow.build.data/get-source-by-provide (data.clj:194)
        shadow.build.compiler/eval15144/fn--15147/fn--15150 (compiler.clj:374)
        clojure.core/map/fn--5935 (core.clj:2770)
        clojure.lang.LazySeq.sval (LazySeq.java:42)
        clojure.lang.LazySeq.seq (LazySeq.java:51)
        clojure.lang.RT.seq (RT.java:535)
        clojure.core/seq--5467 (core.clj:139)
        clojure.core/filter/fn--5962 (core.clj:2826)
        clojure.lang.LazySeq.sval (LazySeq.java:42)
        clojure.lang.LazySeq.seq (LazySeq.java:51)
        clojure.lang.RT.seq (RT.java:535)
        clojure.core/seq--5467 (core.clj:139)
        clojure.core/seq--5467 (core.clj:139)
        shadow.build.compiler/eval15144/fn--15147 (compiler.clj:379)
        clojure.lang.MultiFn.invoke (MultiFn.java:234)
        shadow.build.compiler/do-compile-cljs-resource/fn--15230/fn--15239/fn--15244 (compiler.clj:684)
        shadow.build.compiler/do-compile-cljs-resource/fn--15230/fn--15239 (compiler.clj:683)
        shadow.build.compiler/do-compile-cljs-resource/fn--15230 (compiler.clj:678)
        shadow.build.compiler/do-compile-cljs-resource (compiler.clj:621)
        shadow.build.compiler/do-compile-cljs-resource (compiler.clj:575)
        shadow.build.compiler/maybe-compile-cljs/fn--15334 (compiler.clj:972)
        shadow.build.compiler/maybe-compile-cljs (compiler.clj:971)
        shadow.build.compiler/maybe-compile-cljs (compiler.clj:947)
        shadow.build.compiler/par-compile-one (compiler.clj:1092)
        shadow.build.compiler/par-compile-one (compiler.clj:1047)
        shadow.build.compiler/par-compile-cljs-sources/fn--15374/iter--15396--15400/fn--15401/fn--15402/fn--15403 (compiler.clj:1165)
        clojure.core/apply (core.clj:667)
        clojure.core/with-bindings* (core.clj:1990)
        clojure.core/with-bindings* (core.clj:1990)
        clojure.core/apply (core.clj:671)
        clojure.core/bound-fn*/fn--5818 (core.clj:2020)
        java.util.concurrent.FutureTask.run (FutureTask.java:264)
        java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
        java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
        java.lang.Thread.run (Thread.java:833)

--------------------------------------------------------------------------------
   4 |   {:authors REDACTED}
   5 |   (:require [dre.core :as datahub]
   6 |             [dre.sentry :as sentry]))
   7 | 
--------------------------------------------------------------------------------

thheller16:03:03

hmm, I don't know how it would get there like that. I guess :compiler-options {:parallel-build false} might fix it?

Eugen16:03:42

I added that to:

:release
   {:build-options
    {:ns-aliases
     {day8.re-frame.tracing day8.re-frame.tracing-stubs}}
    :compiler-options {:parallel-build false}}}
no change

Jakub Šťastný16:03:02

How do I set an instance property (`this.errors = errors`)? I tried (set! specError (clj->js spec-error)))), but it's telling me set! target must be a field or a symbol naming a var. I can't find set! documentation in the ShadowCLJS manual.

Jakub Šťastný16:03:24

For context, this is the full code:

(defclass InvalidDataError
  (extends js/Error)
  (constructor [this message spec-error]
               (super message)
               (set! specError (clj->js spec-error))))

thheller16:03:58

either you specify (field specError) after the (extends ...)

thheller16:03:04

then the set! should be ok

thheller16:03:12

otherwise (set! this -specError ...)

👍 2
thheller16:03:22

do you do this for interop with JS? in CLJS you generally (throw (ex-info "the message" {:spec-error data}))

Jakub Šťastný16:03:58

Exactly, the output is a JS lib that's meant to be used by JS devs.

👍 2
Jakub Šťastný17:03:04

By the way I keep getting Could not set npm package requirements. Error getting response at for package "shadow-cljs": An npm specifier not found in cache: "shadow-cljs", --cached-only is specified. all the time while in deno repl. Anyone knows how to fix it? It's super annoying.

thheller17:03:14

did you try my suggestion from last time?

thheller17:03:39

I doubt many people use deno, so you are in uncharted territory for the most part

Jakub Šťastný18:03:55

@thheller the npm cache clean --force? Yes I tried that but it didn't work unfortunately.

thheller18:03:57

is shadow-cljs in devDependencies? I don't know know deno, I don't know why it looks at shadow-cljs at all. it is not a runtime dependency of the output

Jakub Šťastný18:03:54

{
  "name": "bizmentor-core",
  "version": "0.0.1",
  "private": true,
  "devDependencies": {
    "shadow-cljs": "2.21.0"
  }
}

thheller18:03:25

then I don't know. this is not an error shadow-cljs would emit. it is entirely deno, so consult their docs

thheller18:03:27

one option may be putting the output into its own folder, with its own package.json and then running deno in that dir

thheller18:03:37

then it shouldn't see shadow-cljs at all

thheller18:03:59

I don't even know what it means. shadow-cljs is just a regular npm package, not sure what would be wrong with it

Jakub Šťastný18:03:41

deno cache npm:shadow-cljs

Jakub Šťastný18:03:53

This solves it turns out!

Jakub Šťastný18:03:58

In case someone else needs it 🙂

👍 2