hmm, was :external-config at some point explained in the docs? I recall reading about it at some point, and it saved my bacon yesterday, but I can’t find a good explanation anywhere

I’ve posted a gist of one way I have been handling injection of environment variables using a shadow-cljs build hook and juxt/aero: https://gist.github.com/mhuebert/a87795a74bf3f3452e6b032f1c7ee25d

It seems like a bit more ceremony than I would like but is the cleanest way I have found (so far) to satisfy all my requirements.. I would be very happy to discover ways to simplify it

I guess it is time I finally write down how to do runtime configuration for CLJS 🙂

:external-config and the macro approaches break the assumptions made by the caching stuff so it very likely breaks caching in some way

yes, whenever that config changes, the whole app recompiles

yeah thats the nuclear option I added for external config

since there is no other way to tell what it may affect 😛

I tried another approach where I compared the prev-config to current-config, and conditionally added just the env namespace to :cache-blockers if things had changed, but that never worked for the first compile

so changes to the config while the server wasn’t running, or re-running with a different release flag, wouldn’t get picked up

that actually looked like it might be a bug or nuance with how :cache-blockers works

you can skip the hook though and just

(defn release
  "Build :browser release, with advanced compilation"
  ([] (release "local"))
  ([release-flag]
   (-> (shadow-config/get-build! :browser)
       ;; note, we add ::release-flag to our build-config, we need this later.
       (assoc-in [:compiler-options :external-config ::env] (read-env :release))
       (shadow/release* {}))))

for release that would work, but not for watch

for watch we need something to happen to on every recompile, no?

well the problem here is that you are treating runtime configuration like you would in clojure

but what you are actually turning it into is compile time constants

which isn't actually what you want

it is both runtime and compile time configuration

the stuff is also exposed to macros, for DCE etc

eek 😛

the idea is to have a single config map which gives you the same benefits as goog-define-ing things

read via aero so you can pull in system environment variables or keep things in the edn directly

you only get goog-define via actual goog-define

(defmacro goog-define*
       "Like goog-define, but reads `k` from `env/config` at compile time "
       [name k default]
       `(~'goog-define ~name ~(get config k default)))

^works with the above approach

hmm ok so you use them as actual goog-defines just with different defaults

hmm I think your hook solution is the cleanest way

so the single config map is exposed as a plain map in cljs-land, and then i only use that goog-define* bit when I want DCE. We also have some clj scripts which rely on the config being available in clojure-land (generating static html from hiccup, some other deploy-related tasks)

you just can skip the entire custom build function

ah nope .. you have three profiles

re: the custom build function, i just couldn’t figure out how to pass in my own command line argument

meh ... I hate env variables so much for compiling stuff

alright 2second example: in a browser app I recommend passing runtime config via the HTML

so assuming you generate it on the server

you call <script>your.app.init(some-json-config-data);</script>

or you load it via xhr (depends on the app you have)

things like the url you use in your example should probably come from the html directly since the server probably knows these things

or if just static stuff you just have a index-dev.html index-staging.html index.html for prod

yeah, server or compile environment. this is all just deployed to s3 but we generate the index.html from hiccup in a post-compile build hook

so it would just be index.html but generated specifically for the current env

yes

I tried describing my approach a long time ago https://code.thheller.com/blog/web/2014/12/20/better-javascript-html-integration.html

this is what I do

and currently that build hook can seamlessly read from the same app.env/config var that is exposed to cljs

and we can also put things like :devtools false in the same config map, which needs to be a compile-time flag to keep stuff out of prod builds

do you have an example of something that should actually be a compile time constant?

ie. something that affects DCE?

well, our devtools flag

we have an additional tab that is only visible in dev and sometimes staging, and includes a bunch of extra stuff that we don’t want bundled always

would that not be solved by using :preloads [your.app.dev-stuff]?

hmm. not in the way it currently works, it is used as a conditional in a few places

or do you constantly check (when devtools (do-more-stuff))?

hmm ok

eg. in routing (cond-> routes env/devtools? (merge {.. dev routes ..}))

hmm that for example probably won't be properly DCE'd by closure

did you verify that is actually removed and not just disabled?

or is that a pure closure-define?

that is a closure-define

using that goog-define* macro which reads from the env map at compile time

so, a “pure” closure-define but derived from our canonical config

sure. you don't actually gain anything by making it a goog-define since you are never redefining it

just something that ends up as (def devtools? false) would have the same effect as far as closure is concerned

eg. (def devtools? (get-config :devtools false))

ah. that is good to know

if the get-config is a macro and just emits literals you don't need extra goog-define gimmicks

hmm although ...

hmm I'm not sure I can think of a clean way to do this

do you ever use watch with a non-dev release-flag?

yeah, sometimes I use a prod release flag if i want my local build to talk to the prod api server

you could always skip all this entirely

and just generating a static src/app/env.cljs before starting the compile

no macro trickery and compiler magic required 😛

hm. if that is done in a build hook, would shadow notice changes to that file “in time” to invalidate its cache if it has changed?

if config variables are read from a static/unchanging map, can DCE still happen? eg. (when (:devtools static-config) ...)

hmm good point

no that won't work

but the code you generate can just emit a static variable

generating something like

(ns app.env)

(def devtools? false)

(def config {:url "foo.bar"})

yeah.. that code-generation looks to me more complicated than what we are doing, where there is a straightforward config map

with lots of macro trickery 🙂

yeah. although, the macro trickery has a straightforward API once it’s set up, and there is no uncertainty about where a given environment variable is coming from. if you know the release flag, there is just one edn file where you can always look things up. and changes are picked up during recompile

hmm can't really think of anything other than what you already do

I could add an option that lets you modify the config before it is used

so maybe it is the most minimal thing that solves for the constraints/requirements we’ve accepted

is :cache-blockers supposed to work on first compile?

but that still wouldn't solve the extra "release-flag" argument

:cache-blockers complete blocks all cache for the namespaces mentioned. yes it works on the first compile

npm run lets you add command line parameters after a --, which are then passed to the underlying script, eg. npm run watch -- staging. that’s actually what we have set up. i could imagine something similar for shadow, allowing parameters at the end to be passed in as command-line-args or something, but I haven’t thought a lot about that.

sure that would be easy but where would you read them?

ok. I think there might be a bug there, as my previous attempt was dynamically setting :cache-blockers in :compile-prepare stage, and on the first compile it wasn’t recompiling everything that I added to it

I would read the *command-line-args* from build hooks

you can't really modify the config on :compile-prepare

ok. in :compile-prepare is where I am setting :external-config

that’s where I re-read the environment and update app.env/config

hmm yeah but the watch logic will not trigger recompiles if you modify the config there

hm. i guess that could explain why it wasn’t working on the 1st compile, but seemed to work later

you can just set (ns ^:dev/always app.env) to never cache it

that slows everything down a lot

cache-blockers does the same no?

yeah, but i was only adding it to :cache-blockers when the config changed

(-> build-state
        (assoc ::prev-config public-config)
        (update-in [:shadow.build/config :cache-blockers] (if env-changed?
                                                            #(conj % 'web3.env)
                                                            #(remove #{'web3.env} %))))

that has no effect

that seemed to work as intended except for the 1st compile

but maybe it was always only affecting the next compile?

so assuming the file is not compiled and cached already

in watch the file will be compiled once and after that kept in the compiler state

if the file is touched on the filesystem the version in the compiler state is invalidated

if a file is in the compiler state and not otherwise affected it won't be recompiled

so if you add cache-blockers after it is already compiled nothing happens?

ah no it works but since the first compile was cached and written to disk

the next time you compile if can load that

meh this makes my head hurt even thinking about it 😛

hehe

i think i need to have a look at the source

hmm no I can't see how changing cache-blockers affects anything once a file is compiled and in memory

config changes typically trigger a fill reconfigure and recompile

source file changes only trigger a partial recompile

you mean this part

(let [output (get-in state [:output resource-id])]
    ;; skip compilation if output is already present from previous compile
    ;; always recompile files with warnings
    (if (and output (not (seq (:warnings output))))
      output
      (maybe-compile-cljs state src)
      ))

yes

well, now i am wondering how it was working before

so seq-compile-cljs-sources uses that code, par-compile-cljs-sources calls maybe-compile-cljs directly without that output check but is not used for partial incremental compiles

seriously?

unless i am misreading

hehe indeed

ah nvm. it does the same logic though

https://github.com/thheller/shadow-cljs/blob/cbd91dfad8b972773b50a8b26dee032af90185da/src/main/shadow/build/compiler.clj#L890

ah, yes

well, this isn’t so important anyway, for our purposes :external-config is fine, and we can live with recompiling everything when config changes, as that is not so often. I guess in a hyper-optimised world we would figure out how to add a particular namespace to the list of things-to-recompile from within a :compile-prepare hook.

(update state :output dissoc resource-id)?

would something like shadow-cljs release app --config-hook "(your.ns/some-fn 1 2 3)" help?

so (your.ns/some-fn config-from-file 1 2 3) would be called whenever the config is loaded?

yes if you remove the :output it will recompile

(shadow.build.data/get-source-id-by-provide state 'app.env) will you you the proper resource-id

probably should use (shadow.build.data/remove-source-by-id state resource-id) instead of just removing the output

if we remove ’app.env, would namespaces that depend on app.env also recompile?

depends on if it was cached before

i am thinking about the --config-hook thing, i don’t fully follow how that fits in

since the logic will load a cached file from disk when not in memory

if the cache logic decides that the disk cache was ok then no recompiles happen at all

(not even app.env)

hmm. so this wouldn’t really trigger a recompile then, because the file isn’t changing

yes

we would need to remove it from the output and also add it to cache-blockers

do you actually need app.env/config on the clojure side?

yes - not just the DCE stuff (ie. in macros) but also scripts that generate HTML, interop with Sentry

which are all triggered in build hooks

right now it sounds like you are abusing builds hooks a little bit too much 🙂

like

:build-hooks [(app.build/load-env)
                                  (app.build/compile-static-assets-hook)
                                  (app.sentry/push-source-maps!)]

🙂

they are reliable workhorses

why not

(defn release
  "Build :browser release, with advanced compilation"
  ([] (release "local"))
  ([release-flag]
   (-> (shadow/get-build-config! :browser)
       ;; note, we add ::release-flag to our build-config, we need this later.
       (assoc ::release-flag release-flag)
       (shadow/release* {})
       (compile-static-assets)
       (send-source-maps-to-sentry)
       )))

seems overkill to run all the hooks all the time in watch? granted I don't know what you do in them but probably the first thing you do is check if they should actually run?

yeah, the sentry fn starts with (when (= :release (:shadow.build/mode build-state)) ...)

the static assets are affected by changes to config, so i want those to recompile, and all the heavier bits are memoized so it adds max handful-of-milliseconds

i previously had these things in the watch/release functions like you have there

what made you move them into hooks?

I like how far you are pushing things with hooks and stuff though

interesting to see what they are used for 🙂

seems all ok ... just probably want something cleaner for the command line stuff

before, there were multiple commands that had to be run / coordinated at the right time, in order to get a correct build

now, running the shadow build ensures that everything is created declaratively / at the right time / using a consistent environment

it is not really possible to compile the app without simultaneously creating all the appropriate assets

it has made it easier for onboarding / booting up in new environments, there is little to remember

I think at one point I was hoping to do away with the custom release/watch scripts altogether,

the only reason they are there is for the release flag

overall the build hooks let me substantially simplify our whole build/deploy process

the scripts are hid behind npm scripts, so in practice we run npm run watch -- staging and then it delegates to shadow’s clj-run

I have been thinking about ways to make configs adjustable from the command line

ie. load the config from the file but also merge in extra values

that has been helpful as i have been able to evolve how this stuff works without breaking anyone’s habits

I could add shadow-cljs release browser --config-merge '{:some "data"}' for example

cljs.main lets you specify multiple --compile-opts and merges them together

that is a good idea imho

hmm that could be useful. would that be added to shadow’s config and validated according to that schema, or could we put arbitrary keys there

if you want to see some real abuse of build hooks, https://github.com/mhuebert/js-interop-example/blob/master/shadow-cljs.edn#L15

you could add anything. config spec validation is open.

hehe wow

I would probably have outsourced the html generation to an external config somewhere but if you like it there that is fine 🙂

for any real project i would do that

I've been thinking about this whole static site generation business a lot the past few weeks

but for just a quick sketch i didn’t want to have to create an extra file

and 90% of the time for these cljs projects, all i need in terms of HTML is a title, and a couple style tags and script elements

what kind of use-cases are you focused on?

nothing in particular. just looking at what the rest of the world is doing/using

it all doesn't really fit what I'm doing for work stuff

just interested to see what others are doing really

I have probably never written anything that didn't have a massive backend that was at least as big as the frontend

that function in the build hook, write-assets!, makes use of the :output-dir and :asset-path in the shadow build config, and in :release mode it appends content hashes to paths

but the "get something simple out" part for CLJ(S) still sucks compared to something like gatsby or so

yeah

I always use the manifest.edn for that https://shadow-cljs.github.io/docs/UsersGuide.html#BrowserManifest

(granted that I always have a server that consumes that file at runtime)

this project has a big backend for the api server, but it doesn’t generate any html, the web app is all on s3

so deployment happens via aws CodeBuild, a remote box performs the shadow build and spits all the html / js to disk

all triggered by pushing to specific branches on GitHub

I have used manifest.edn as well. i ended up writing something more general because we needed the same cache-busting stuff for generated html and css also

our hosting is configured to cache everything except index.html et al very aggressively

yeah makes sense. Just surprised that shadow-cljs is used to drive it all

My JVM returns

(System/getProperty "java.version")
=> "11.0.1"

{"java.specification.version" "11"
 "java.vm.specification.version" "11"}