Hello 👋 I'm trying to cache shadow-cljs runs on CircleCI. In it's YAML DSL I'm using:
1. restore_cache
2. shadow-cljs release ...
3. save_cache with .shadow-cljs folder.
The cache is properly stored, and loaded on the next job. The problem is it seems shadow misses some caches. Running with --verbose shows entries like this
-> Cache read: common/components.cljs
<- Cache read: common/components.cljs (30 ms)
-> Compile CLJS: common/components.cljs
<- Compile CLJS: common/components.cljs (1449 ms)
-> Cache write: common/components.cljs
<- Cache write: common/components.cljs (83 ms)
which are confusing. like it's picking up the cache, but then it writes it again?Is there any way to see why a cache missed?
Some CI systems write back the cache if anything changed in the cached folders/files. What is your cache based on, e.g., file hashes?
If your .shadow-cljs folder contents change on a normal CI run, I would expect it to get written back at the end I think?
What makes you think there's a cache miss?
I'm interpreting this ouput as a cache miss
-> Cache read: common/components.cljs
<- Cache read: common/components.cljs (30 ms)
-> Compile CLJS: common/components.cljs
<- Compile CLJS: common/components.cljs (1449 ms)
-> Cache write: common/components.cljs
<- Cache write: common/components.cljs (83 ms)
Thanks for the observations, I'll see what CircleCI logs when restoring the cache, see if I can link it to a previous runYep, I'm using file hashes. And the CircleCI cache seems to be working
Found a cache from job 11693 at v9-frontend-BYQz89GzZtdi1b78eqoEFsCE2MxtrAmO16uxkCiqCsY=-R4aA88N6NtSRW4jYI59Nv9USaSP46fA6BXiLssHe1go=-RsQdSDJ2+_hAM_WMZ3NEHhbke5asjgOEHg7pPj5p5aE=-PvQyMxQuUdfiKoSY8vXDsmbIaxshSPfi3qr4L9fSH_Q=
Size: 83 MiB
Cached paths:
* /root/.m2
* /root/project/node_modules
* /root/project/shadow/node_modules
* /root/project/shadow/.shadow-cljs
Downloading cache archive... Downloading cache file in parallel
Unarchiving cache...
Extracting using zstd
Extraction duration for v9-frontend-{{ checksum "deps.edn" }}-{{ checksum "package-lock.json" }}-{{ checksum "shadow/yarn.lock" }}-{{ checksum "shadow/shadow-cljs.edn" }}: 2.230509079s
Double checked job ID 11693 was the previous job. Also just in case SSHd into the machine to list the files in .shadow-cljsSo... it's doing the right thing, and you just thought it wasn't?
Well, yes and no. The first time I run the job, shadow-cljs compiles all, the CI stores the cache and I get this log line:
> Build completed. (476 files, 326 compiled, 0 warnings, 49.95s)
The next run, it loads the cache, but shadow-clljs has to recompile files again, when it should show something like:
> Build completed. (476 files, 0 compiled, 0 warnings, 40.08s)
(which is what I get when I run the shadow-cljs release command locally)
So the Q for @thheller is probably "how does shadow-cljs determine if it needs to recompile a file?" and I suspect the answer is modified timestamp, and CircleCI's way of restoring the cache doesn't maintain the timestamps? To debug that, you could add a line to your CI job that lists the cached files with their modification dates perhaps?
Mmm thank you, I'll check that 🙏