Fork me on GitHub
#shadow-cljs
<
2022-04-25
>
Piotr Roterski09:04:01

Hello everyone! 👋 shadow-cljs outputs source-maps in “sections” format but Sentry (error tracking SaaS) fails to parse it. The “sections” format looks like:

{
    version : 3,
    file: "app.js",
    sections: [
     { offset: {line:0, column:0}, url: "url_for_part1.map" }
     { offset: {line:100, column:10}, map:
       {
         version : 3,
         file: "section.js",
         sources: ["foo.js", "bar.js"],
         names: ["src", "maps", "are", "fun"],
         mappings: "AAAA,E;;ABCDE;"
       }
     }
    ],
}
While a more traditional “section-less” format, that works with Sentry, looks like:
{
"version" : 3,
"file": "out.js",
"sourceRoot": "",
"sources": ["foo.js", "bar.js"],
"sourcesContent": [null, null],
"names": ["src", "maps", "are", "fun"],
"mappings": "A,AAAB;;ABCDE;"
}
(some info on source-maps formats I found https://sourcemaps.info/spec.html) I went through the shadow-cljs code (https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/build/output.clj#L355-L375) but I didn’t find any option to output source-maps in “section-less” format so I tried to convert it in {:shadow.build/stage :flush} build hook. However, the naive merging sections back to the top-level map didn’t work - it went past initial sentry’s parsing error "Sourcemap was invalid or not parseable" but then it hit errors like "Invalid location in sourcemap: (37101, 117)". my naive (not working) attempt:
(defn flatten-source-map [path]
  (let [source-map (-> (io/file path)
                       slurp
                       ch/decode)
        sections (get source-map "sections")]
    (->> sections
         (reduce (fn [acc section]
                   (-> acc
                       (update-in ["names"] (fnil into []) (get-in section ["map" "names"]))
                       (update-in ["mappings"] #(if (not-empty %)
                                                  (str % ";" (get-in section ["map" "mappings"]))
                                                  (get-in section ["map" "mappings"])))
                       (update-in ["sources"] (fnil into []) (get-in section ["map" "sources"]))
                       (update-in ["sourcesContent"] (fnil into []) (get-in section ["map" "sourcesContent"]))))
                 (dissoc source-map "sections"))
         ch/encode
         (spit path))))

(defn shadow-cljs-after-build-hook
  {:shadow.build/stage :flush}
  [build-state]
  (let [{:shadow.build/keys [config mode]
         :shadow.build.modules/keys [module-order]} build-state
        {:keys [output-dir]} config
        filename (-> module-order
                     first
                     name
                     (str ".js.map"))
        path (str output-dir "/" filename)]
    (when (= :release mode)
      (println "shadow-cljs-after-build-hook running (flatten-source-map " path ")")
      (flatten-source-map path)))
  build-state)
I was wondering whether I should try to facilitate some functions from https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/source_map.clj or try to tweak shadow-cljs config but that appears to be a deep hole to dig in, and I feel kinda out of my depth. I would much appreciate any help/hints on what direction to pursue so shadow-cljs’s source-maps can work on Sentry. Thank you! clojure-spin

😮 1
👀 1
sebastibe06:11:00

Hi @pt.roterski, I was wondering if you ended up finding a solution for this? We have I believe the same issue trying to integrate with sentry-cli new tooling around source maps with a the following error:

error: Invalid sourcemap at ./resources/public/js/app.js.map
  caused by: encountered incompatible sourcemap format
after running npx sentry-cli sourcemaps ./resources/public/js

Piotr Roterski06:11:27

sorry but AFAIR I didn't find a good solution back then and I don't really remember anything about this issue now and I don't have access to that codebase anymore 😅

sebastibe06:11:55

Alrighty 😄 I remember I managed to get it working for a time at some point on some other projects but the flow is a bit different now and I don't pass the local sentry-cli steps on the latest sentry browser version it seems anymore. I will document here if I find anything useful.

thheller10:04:09

@pt.roterski the source map mappings encode in them line number information. so basically each section acts as an offset for that line information. as such without the offset and just concatenating the mappings does not work. you could rewrite source maps by adjusting those mappings accordingly in theory but I have never attempted to do so

thheller10:04:08

what issue exactly does sentry have with this? I mean sections aren't exactly new and as far as I can tell all browsers support them? even common JS libs like source-map do?

Piotr Roterski10:04:54

Thanks for the swift response @thheller! I understand that sections are widely supported (and they work for this code in my browser) but not by Sentry apparently and their error output is really poor (literally just those two strings "Sourcemap was invalid or not parseable" and then "Invalid location in sourcemap: (37101, 117)") which makes debugging it difficult. I’ve also tried their validation tool https://docs.sentry.io/platforms/javascript/sourcemaps/validating/ but it also just fails with no descriptive stacktrace so my guess is that “sections” format is just not supported by them and I need to convert the source-maps to “section-less” format (because the same codebase’s source-maps used to work there before the transition from lein+figwheel to shadow-cljs build). Thank you for the hint, I’ll try to improve my flatten-source-map function based on that.

thheller10:04:52

do yourself a favor and don't do it as a hook though

thheller10:04:13

just write a function that takes an already generated .map file and spits out a new one

Piotr Roterski11:04:21

I guess that's the thing I already try to do in this hook, and the :flush hook seemed just like a convenient place to do it but I guess it could be called from different place as well

thheller11:04:31

I'd recommend calling it from the REPL while developing

thheller10:04:29

this doesn't need to happen during a shadow-cljs build

thheller10:04:05

if you get to a working solution and wish to share it we can talk about having shadow-cljs spit this out directly

thheller10:04:27

a build hook just isn't the place to do this

thheller10:04:15

hmm yeah that website for validating is useless. just threw the shadow-cljs UI map at it and just gave me

Errors 1
InvalidSourceMapFormatError
Invalid SourceMap format: "." is not in the SourceMap.

thheller10:04:20

no clue where it gets "." from. no complaints about sections though?

thheller10:04:05

pretty sure this is just the source-map npm package wrapped in a bad website

thheller10:04:30

what leads you do believe this is about sections?

Piotr Roterski11:04:38

The previous lein+figwheel build had "section-less" source-map format and it worked, the shadow-cljs's source-maps use sections format so I assumed that's the cause but of course I might be wrong about it and it could be something else :thinking_face:

thheller11:04:13

the lein+figwheel likely also had much smaller source map due to all cljsjs packages being unmapped

thheller11:04:06

I mean you can just trim down the source map by removing all except the last sections entry

thheller11:04:43

the early sections are all for npm packages so maybe the error also goes away if you remove them?

Piotr Roterski11:04:57

I can try doing that

thheller11:04:50

and it is indeed just using the source-map npm package for the validation. source-map definitely supports sections just fine, so must be something else

👀 1
thheller11:04:44

this entire website seems completely useless, just check one of their own examples

thheller11:04:52

who is supposed to make sense of these errors 😉

Piotr Roterski11:04:30

yeah, it’s a bit of a hit-or-miss debugging game 🐛 🔨 more than anything

pinkfrog13:04:51

Hi. Is it possible to specify the autobuild part in shadowcljs.edn file

(shadow.cljs.devtools.api/watch :app {:autobuild false})

thheller18:04:01

actually I forgot. you can set :devtools {:autobuild false} in your build config and that would be same as above

pinkfrog22:04:36

i’d believe that’s not documented in the manual.

thheller14:04:32

you mean disable it? no.

cldwalker16:04:59

Hi. Is it possible to run shadow-cljs compilation using a shadow-cljs.edn from another directory? I tried passing it to --config-merge but the builds aren't recognized and I get an error like

{:build-id :modules, :known-builds #{}}
ExceptionInfo: no build with id: :modules

thheller17:04:36

I guess the answer is no? can give a better answer if you provide more details about what you are trying to do specifically

cldwalker17:04:43

Sure. I'm building a bb lib which will allow users to build custom versions of https://github.com/babashka/nbb. The users of this lib will be running shadow-cljs from their own project but will want to pull in https://github.com/babashka/nbb/blob/main/shadow-cljs.edn as their initial config

cldwalker17:04:37

I can workaround this by copying the shadow-cljs.edn before compilation and deleting after but it would be nice if there was a way to just point shadow to use the config from the lib directory

thheller17:04:02

you cannot do with any default build commands

thheller17:04:40

but you can manually call all the API build functions with a build config map you created yourself by any means

👍 1
thheller17:04:20

not sure what problem this is supposed to solve though. what is the point of a custom nbb version?

cldwalker17:04:36

Ok. No worries. I'll probably rely on my suggested workaround for now

cldwalker17:04:06

Ship versions of nbb with X number of features/libraries enabled on the classpath. There are a large number of cljs libraries that aren't sci compatible yet

cldwalker17:04:18

Will be pretty neat when nbb has this ability as users will be able to put preconfigured features/libraries on the classpath like https://github.com/babashka/nbb/tree/main/features and build a version of nbb that just works with those libraries

thheller17:04:10

yeah, sorry I still don't get why anyone wouldn't just do a regular build (eg. with shadow-cljs) and then use the generated output

thheller17:04:31

which will no longer depend on the build tool used, or incur any cost regarding to that on startup and such

thheller17:04:17

I get the ad-hoc scripting part being nice and fast feedback

thheller17:04:32

I don't get it at all for any kind of deployment or published library

cldwalker17:04:24

> yeah, sorry I still don't get why anyone wouldn't just do a regular build (eg. with shadow-cljs) and then use the generated output This probably comes down to the type of users you're thinking of. In my case I'm largely focused on supporting cljs beginners who will only write a couple small ad-hoc scripts and nothing else

cldwalker17:04:16

A lot of their scripts will use datascript which has a much better cljs api than js one so just giving them a batteries-included approach to write their scripts saves them time

thheller17:04:16

wouldn't a REPL make more sense for teaching purposes?

thheller17:04:56

I mean you have a plan so go build it. I'm by no means the target audience so I don't have to get it 😉

cldwalker17:04:11

Hehe. Happy to explain, especially to someone like you who's been so key in making cljs and nodejs work so smoothly together. We're a cljs editor, https://github.com/logseq/logseq and almost all our users are only familiar with nodejs. Distributing a nbb-like lib will allow them to not only run adhoc scripts with a larger part of the datascript API but also let them do useful things like run CI jobs, without having to think of compilation