Fork me on GitHub
#shadow-cljs
<
2020-07-31
>
timrichardt07:07:26

https://github.com/thheller/shadow-cljs/issues/127 @thheller I am really interested in th is feature. It would make the life for my coworker much easier. I would like to work on it, if you don't mind? Could you point me to the right starting point? 🙂

thheller08:07:35

@timrichardt how so? this really isn't that relevant anymore? it only affects ES6 files in node_modules? why does that affect you?

timrichardt08:07:14

There are people working on jsx Files. I babel them and then bundle with shadow. Errors inside the babel-output are not source mapped to the original jsx Files. Do you have any idea how to tell the browser the mapping from jsx->js?

thheller08:07:39

how do you include them in the project?

timrichardt08:07:33

They live in a separate folder /src/js along with the .js.map -Files.

thheller08:07:24

and the files do have the sourceMappingURL comment?

thheller08:07:41

then source maps should already be working

thheller08:07:21

I can take a look if you make a reproducible example. I haven't tested this in a long time so maybe its broken.

👍 3
thheller08:07:43

but it isn't what #127 is about so it would be a separate issue

timrichardt08:07:44

I'll do a minimal example.

thheller08:07:57

I basically only need a .jsx file and its converted form + map

timrichardt08:07:34

I have included the js and map

thheller09:07:35

@timrichardt I think the problem is that the map file needs to contain the sourcesContent. closure will not find the file otherwise.

thheller09:07:51

ah nvm its already in there

thheller09:07:06

is it possible to make babel inline the source map into the file?

timrichardt09:07:22

I also saw this warning now.

------ WARNING #1 -  -----------------------------------------------------------
 Resource:
 Failed to resolve sourcemap at Component.js.map: Component.js.map

timrichardt09:07:33

Babel has this option, let me try.

timrichardt09:07:38

Inlining in principle works, but Chrome and Firefox seem to not recognize the source map comment.

thheller09:07:35

I'll check it out later. can you add the inline option to the repo?

timrichardt09:07:30

OK, i'll do. Thanks anyway.

ghaskins12:07:46

@thheller Do you know if there is a way to use both :handler and :proxy-url with the :dev-http feature? After a quick look at the code, I think the answer is "no" but I wanted to check in case I missed something

ghaskins12:07:21

ultimately what I am after is the ability to define what should be wildcarded to my SPA and what should be forwarded to the proxy

ghaskins12:07:05

The follow-up question is: if there is not, would you be interested in a patch that does?

thheller12:07:10

@ghaskins "define what should be wildcarded to my SPA" what does that mean? we are talking about :dev-http, what does the SPA have to do with that?

ghaskins12:07:05

what I mean is: i have a backend with different endpoints...lets just say there is an identity provider on /oauth, an api backend on /graphql, and an admin ui at /admin....and /admin is a shadow-cljs compiled SPA

thheller12:07:32

if you have a backend anyway why not use that directly? I mean why do you need the proxy?

ghaskins12:07:42

what id like is to define something like a push handler that says /admin/* should be index.html, anything else should be forwarded

ghaskins12:07:00

its just during development of the SPA

thheller12:07:26

why do you need the proxy? does it proxy to some remote production server?

thheller12:07:56

or is your backend running locally anyways?

ghaskins12:07:57

right, though its a sandbox deployment, but same difference

ghaskins12:07:02

no, its not local

ghaskins12:07:12

its running in k8s in aws, except for my ui

ghaskins12:07:46

the dev-proxy works great, except for the inability to specify what is a route handled in the spa

thheller12:07:02

you are confusing me with your wording

thheller12:07:19

"route handled in the SPA" to me means something handled by the client that never even reaches the server

ghaskins12:07:57

we have a kubernetes architecture with a bunch of microservices...one type of microservice is a shadow-cljs compiled ui that has an SPA in it

ghaskins12:07:37

during develoment, its nice to work on the UI code locally where I want any of the paths that would normally hit the UI backend to be served by dev-proxy, and all others to forward on

ghaskins12:07:28

the SPA has kee-frame/reitit routes, so things like /admin/foo/bar are actually served from /admin/app.js

ghaskins12:07:51

but everything else needs to forward to the server

thheller12:07:27

ah you are using push state routes and don't want to forward those to the proxy?

thheller12:07:56

like the server always give you the same html regardless of what path you ask for?

ghaskins12:07:58

i think that is correct, though I am unsure of the terminology

ghaskins12:07:52

right, or really once the /admin/[index.html, app.js] are running, it knows that foo/bar is a route inside the browser

ghaskins12:07:15

which works great

ghaskins12:07:47

but the problem is if someone tries to navigate to /admin/foo/bar externally, the proxy forwards that to the backend

ghaskins12:07:18

i.e. it only works if you come in through the front index.html

thheller12:07:35

yes but you handle that in production too right?

ghaskins12:07:15

so what I was thinking is something like a push-handler that can be configued in conjunction with :proxy-url, and the push-handler can decide if it should be handled locally or forwarded

ghaskins12:07:19

yes, prod is fine

ghaskins12:07:30

once its in the k8s backend, the routing all works

ghaskins12:07:40

this is just when the :dev-proxy is in front during dev

ghaskins12:07:16

so the first question is: do you already allow this and I missed it? and if you don't, would you be willing to consider a patch for the feature?

ghaskins12:07:08

looking at the :dev-proxy code, I suspect we can put an undertow handler in front of the LoadBalancingProxyClient in a fairly straight forward way

ghaskins12:07:51

sorry, :dev-http code, i meant

ghaskins12:07:06

what im thinking is something that would basically allow this and :proxy-url to both be configured, details TBD

thheller12:07:42

hmm I'm undecided. usually I recommend running your own server if you need more complicated setups

ghaskins12:07:19

thats an acceptable response, though I think what i need/suggest is not too far off from what you already offer

thheller12:07:20

like how would you want to configure this?

thheller12:07:02

one person may say that you want to exclude certain paths, another may want to include certain paths

ghaskins12:07:28

im thinking it would look something like

{...
 :dev-http
 {8000
  {:root "public"
   :handler 
   :proxy-url ""}}}

thheller12:07:01

yes but that doesn't work since the proxy always answers

ghaskins12:07:29

> one person may say that you want to exclude certain paths, another may want to include certain paths Yeah, so to that point, the protocol for the :handler would need to allow it to express whether to serve locally or move on to the proxy

ghaskins12:07:55

im thinking its either a path to a local file (e.g. "index.html" or nil, in which case the proxy is queried

thheller12:07:12

the issue is that at the point where the handler is called the http request is in blocking mode and you can't proxy anymore

ghaskins12:07:08

its been a while since I wrote an undertow backend, but I think one thing that may work is if we add an undertow handler in front of the reverse-proxy

ghaskins12:07:27

since undertow already has handler chaining, IIRC

thheller12:07:40

yes you can add a handler to decide which other handler to take next

ghaskins12:07:52

thats what i was thinking was the integration point

thheller12:07:57

but how would you configure that?

ghaskins12:07:39

im thinking its the same :handler spec as above, and if it returns a path, we serve that, and if it doesnt, we move to the next handler

ghaskins12:07:20

so in my example, I could map "/admin/*" to "index.html", and everything else is nil

ghaskins12:07:44

or I could code it for the opposite, e.g. "/oauth/* -> nil

thheller12:07:47

just trying to figure out how to express this in config without getting too complex

ghaskins12:07:50

whatever the use case is

thheller12:07:32

some kind of :proxy-skip

thheller12:07:46

pointing to a (defn should-this-proxy [req] true|false)

thheller13:07:25

or :proxy-skip [#"^/something" #"^/else"]

ghaskins13:07:00

right..if we can assume that the non-proxy behavior of shadow.http.push-state/handle is what we want, then the new config really just needs to be a predicate

ghaskins13:07:55

thats why I was thinking I wouldnt propose a new configuration knob, just allow the :handler to decide

ghaskins13:07:20

but id be happy with the above, too

thheller13:07:37

the handler can't decide because of how undertow works

thheller13:07:03

as soon as it enters "ring" mode (which the handler is) I switch it to "blocking" mode

thheller13:07:19

but proxy only works when its in the async mode

ghaskins13:07:39

let me check how I did this in protojure

thheller13:07:48

and as far as I know there is no way to go back to async mode

thheller13:07:35

granted that worrying about this at all for a dev server is overkill

thheller13:07:45

but I still want to do it correctly

ghaskins13:07:59

i understand completely

ghaskins13:07:55

so, details TBD, obviously, but this is what I was thinking

ghaskins13:07:16

specifically "a handler can pick the next handler to invoke based on the current request."

thheller13:07:26

yes I understand that

ghaskins13:07:48

in protojure, im currently only installing one handler, just like shadow

ghaskins13:07:23

but im wondering if I hook it at the Undertow/builder level, if we can get both the :handler and :proxy-url components working together

ghaskins13:07:35

i get what you are saying about blocking vs async mode

ghaskins13:07:40

might have to play around with that

thheller13:07:16

yeah the way this is built is quite strange but it made sense to me at the time I built it

ghaskins13:07:26

i also understand that completely, heh

ghaskins13:07:59

ok, let me understand how you did that part

ghaskins13:07:22

i also forget the nuances of the undertow api...i remember there are things that you do that implicitly set the mode

thheller13:07:33

I think I have an idea how to add what you want without getting too complicated

ghaskins13:07:41

oh? that would be awesome

thheller13:07:45

I sort of created a DSL for describing the handler setup

thheller13:07:52

looks something like

[:shadow.cljs.devtools.server.dev-http/file-recorder
 {:on-request
  #object[shadow.cljs.devtools.server.dev_http$start_build_server$file_request_fn__19970 0x6fce3691 "shadow.cljs.devtools.server.dev_http$start_build_server$file_request_fn__19970@6fce3691"]}
 [:shadow.undertow/soft-cache
  [:shadow.undertow/ws-upgrade
   [:shadow.undertow/ws-ring
    {:handler-fn
     #object[shadow.cljs.devtools.server.dev_http$start_build_server$http_handler_fn__19965 0x6109c518 "shadow.cljs.devtools.server.dev_http$start_build_server$http_handler_fn__19965@6109c518"]}]
   [:shadow.undertow/compress
    {}
    [:shadow.undertow/file
     {:root-dir
      #object[java.io.File 0x7ee0cc9 "out\\demo-test-browser"]}
     [:shadow.undertow/classpath
      {:root "shadow/cljs/devtools/server/dev_http"}
      [:shadow.undertow/blocking
       [:shadow.undertow/ring
        {:handler-fn
         #object[shadow.cljs.devtools.server.dev_http$start_build_server$http_handler_fn__19965 0x6109c518 "shadow.cljs.devtools.server.dev_http$start_build_server$http_handler_fn__19965@6109c518"]}]]]]]]]]

ghaskins13:07:22

ah, that probably explains the [::undertow/xx] stuff I saw

ghaskins13:07:28

i didnt dig into that, but I was curious

thheller13:07:59

yeah constructing the handlers by hand was quite annoying otherwise

thheller13:07:49

it probably overkill but it solves the issues I wanted to solve 😛

ghaskins13:07:33

very clever

ghaskins13:07:52

its not quite clear to me what I need to do to express this in the DSL though...guidance appreciated

thheller13:07:30

you can't express it in the DSL

ghaskins13:07:38

ah, sorry...misunderstood

thheller13:07:54

problem is that the "next" logic of the proxy handler requires this returning null

thheller13:07:21

but it never does

thheller13:07:46

but I guess I can subclass that proxy client and have that return null whenever the clojure fn returns false

ghaskins13:07:58

that sounds reasonable

thheller13:07:06

or just leave all that alone and add a regular handler instead

thheller13:07:16

probably simpler .. especially since subclassing in clojure is annoying

ghaskins13:07:34

the handlers are a simple reify, so probably easier than a full gen-class

ghaskins13:07:56

though the 'next' concept seems to be only in the deprecated constructors

ghaskins13:07:31

maybe they've changed the way the chaining works

Old account13:07:13

please help me debug my build configuration:

:builds       {:app {:target     :npm-module
                      :output-dir "node_modules/shadow-cljs"}}}
gives me such error:
[:app] Configuring build.
[:app] Build failure:
Invalid configuration
-- Spec failed --------------------

  {:target :npm-module,
   :output-dir "node_modules/shadow-cljs",
   :build-id :app}

should contain key: :entries

| key      | spec            |
|==========+=================|
| :entries | (coll-of        |
|          |  simple-symbol? |
|          |  :distinct      |
|          |  true           |
|          |  :min-count     |
|          |  1              |
|          |  :kind          |
|          |  vector?)       |

-- Relevant specs -------

:shadow.build.targets.npm-module/target:
  (clojure.spec.alpha/keys
   :req-un
   [:shadow.build.targets.shared/output-dir
    :shadow.build.targets.npm-module/entries]
   :opt-un
   [:shadow.build.targets.npm-module/runtime
    :shadow.build.targets.shared/devtools])
:shadow.build.config/build+target:
  (clojure.spec.alpha/and
   :shadow.build.config/build
   (clojure.spec.alpha/multi-spec
    shadow.build.config/target-spec
    :target))

-------------------------
Detected 1 error

Old account13:07:42

altho

{:frontend
                 {:target :browser
                  :modules {:main {:init-fn }}
                  }}
does not contain :entries but does not throw the same error.

thheller13:07:54

:target :browser has different config that :target :npm-module

thheller13:07:18

for :target :npm-module you need to specify :entries [at-least-one.of-your.namespaces]

thheller13:07:35

basically you need to configure which namespaces you want to have available from JS

Old account13:07:48

@thheller how do I choose which ns to add?

thheller13:07:02

whichever you want to have available from JS?

thheller13:07:34

why are you using :npm-module in the first place?

Old account13:07:35

this is from react native app

thheller13:07:28

react-native should be using :target :react-native and not :npm-module

ghaskins13:07:34

I think I get the gist of your DSL now

ghaskins14:07:33

@thheller im struggling to see where you serve the local-files in front of the proxy from

ghaskins14:07:25

oh, i think i see it now

thheller14:07:54

see :shadow.undertow/file in the DSL example from above

ghaskins14:07:05

yep, i see it now

thheller14:07:20

it checks if the file exists and calls the next handler if it does not

thheller14:07:19

will make a release in a bit, just want to look at something else first

ghaskins14:07:48

there are some cases where it may be easier to whitelist or blacklist what should be proxied, so I wonder if a predicate-fn is more flexible

ghaskins14:07:04

for example, I have two UIs, one at /foo and another at the default / page, so the exclude works well for the former but not the latter...but a simple predicate-fn would work for both

ghaskins14:07:58

so what im thinking is, maybe :proxy-pushstate-predicate returns true means invoke the :handler, false means invoke the proxy

ghaskins14:07:29

and there could be a default pushstate-predicate that handles exclude/include options, if we wanted

Saikyun15:07:10

is it possible to get the absolute path of a file (alternatively relative path + project directory)? in vanilla cljs cljs.analyzer/*cljs-file* does the trick, but with shadow-cljs I only seem to get the part of the path after the source directory

Saikyun15:07:22

e.g. "hello_world/other.cljs" rather than "/Users/.../src/hello_world/other.cljs"

thheller17:07:48

@saikyun what for? and what would it report for files from libraries in .jar files?

adamrenklint19:07:51

I’m trying to use the :target :bootstrap feature in Shadow and am running into trouble with macros. The :entries vector of my bootstrap build contains instaparse.core, and the build fails with:

The required namespace "instaparse.macros" is not available, it was required by "instaparse/core$macros.cljc".
"instaparse/macros.clj" was found on the classpath. Maybe this library only supports CLJ?
Tried patching the problem by copying https://github.com/Engelberg/instaparse/blob/master/src/instaparse/macros.clj into my src folder, but named macros.cljc (thanks for the ability to quickly add custom implementations of arbitrary namespaces, btw). This solved that particular problem, but another similar error pops up:
The required namespace "" is not available, it was required by "cljs/repl$macros.cljc".
"clojure/java/io.clj" was found on the classpath. Maybe this library only supports CLJ?
I feel like I must be holding this tool wrong, as the same code and namespaces can be used to generate an analyzer cache with my own old solution, which generated a node script with Boot and simply executed it. Any tips where I should look next?

thheller19:07:45

@adamrenklint the most likely explanation is that the library just doesn't support self-hosted use. especially if it imports namespaces like http://clojure.java.io

adamrenklint19:07:07

That’s the weird thing, I have created this analyzer cache before, and it definitely included Instaparse, but it was all done with Boot and is horribly slow to develop with, so I want to use Shadow instead.

adamrenklint19:07:38

But maybe you can explain the first error. Why is importing macros implemented using CLJ instead of CLJC throwing an error? The namespace is obviously there, the error message even states it.

thheller20:07:20

I can't explain it sorry. It has been several years since I looked at and implemented this stuff and I cannot remember the fine details

thheller20:07:29

there is a secondary pass that does the macro compilation

thheller20:07:37

maybe you just need to exclude some

thheller20:07:39

I honestly cannot remember any of this 😞 self-hosted confuses me when it comes to macros

thheller22:07:05

@ghaskins 2.10.19 lets you specify :proxy-predicate some.ns/some-fn. should be (defn some-fn [ex config] true|false) where ex is the undertow HttpServerExchange (eg. (.getRequestURI ex) to get request path) and config is the :dev-http config map

thheller22:07:28

should return true if the request should go to the proxy and false if the handler should be taken instead

👍 3
thheller22:07:55

@timrichardt I looked over the source map issue but I can't figure out how to make it work

Björn Ebbinghaus23:07:15

Does anyone have a clue why I am getting DOMException: Failed to construct 'WebSocket': The URL 'ws://:8237/chsk?... Exceptions in my console, when I install a service worker? Everything works fine as far as I can see. Live reloading is working..

thheller23:07:57

no clue. looks like you are trying to connection to ws without a host?

Björn Ebbinghaus08:08:20

This only happens when a service worker is installed. I am playing around with them for the first time.

thheller20:08:35

is the websocket in the service worker the one trying to create the connection? maybe you are using document.location or something which isn't present in service workers?