Fork me on GitHub
#shadow-cljs
<
2022-08-26
>
folcon00:08:18

Is there an alternative way to recursively convert js objects to cljs ones? I've been spotting some odd behaviour where js->clj doesn't actually convert anything and I'm unsure why. This is post advanced compilation if that makes any difference? Not super confident if this is due to shadow specifically, I'll try and make a minimal reproducible case later. Hmm, it's an :type #object[Array], but it's failing (array? x) Ok, no idea why, but these predicates fails:

(identical? (type x) js/Object)
(identical? (type x) js/Array)
whereas these succeed:
(= (str (type x)) (str js/Object))
(= (str (type x)) (str js/Array))
No idea what's causing it...

thheller06:08:24

dunno what you are doing so can't say much. there are some known issues with js->clj and :advanced. mostly related to very short property names clashing with protocols, this can be solved by externs.

thheller06:08:06

you can use https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/json.cljs#L4 which doesn't have that particular issue. don't know if its in any way related to what you are doing though.

folcon13:08:31

Thanks, to be honest I'm super surprised this doesn't work? Surely this is a bug? Also any idea if this a clojurescript issue or a shadow one? (I'm assuming clojurescript at the moment)

thheller14:08:48

I don't know what you are working with so hard to say. I'm tempted to say that its neither a bug in shadow-cljs nor cljs but instead something weird you are doing in your code

folcon14:08:43

I mean I'm calling js->clj on a js object and then just doing a console log into chrome.

folcon14:08:00

It's very literally a stack of console.log's which I'm checking... I pulled the cond predicates from js->clj and printed them as a sanity check to see which branch it goes down... The answer is none of them, which is why it doesn't convert...

(println :satisfies? (satisfies? IEncodeClojure x))
             (println :seq? (seq? x))
             (println :map-entry? (map-entry? x))
             (println :coll? (coll? x))
             (println :array? (array? x))
             (println :object (object? x))
             (println :identical? (identical? (type x) js/Object))
             (println :type (type x))
             (println :type js/Object (identical? (type x) js/Object) (= (str (type x)) (str js/Object)) js/Array (identical? (type x) js/Array) (= (str (type x)) (str js/Array)))

thheller14:08:07

and what does the js object look like?

thheller14:08:42

does the problem go away if you run with shadow-cljs release app --pseudo-names?

folcon14:08:00

Elided example:

[
    {
        "limit": 30,
        "label": "Export ...",
        "fn_name": "get_...",
        "search_query": "...",
        "task-id": "...",
        "container_id": "214319799975"
    },
    {
        "limit": 20,
        "label": "Run ...",
        "fn_name": "run_...",
        "search_query": "...",
        "task-id": "...",
        "container_id": "799975628099"
    }
]

thheller14:08:43

that should be fine

folcon14:08:55

Yea, it's really bugging me...

thheller14:08:27

> does the problem go away if you run with shadow-cljs release app --pseudo-names?

thheller14:08:44

just trying to rule out externs issues

folcon14:08:09

Just tried that, the only true is on the string check ones, ie: (println :type js/Object (identical? (type x) js/Object) (= (str (type x)) (str js/Object)) js/Array (identical? (type x) js/Array) (= (str (type x)) (str js/Array))):

:satisfies? false
:seq? false
:map-entry? false
:coll? false
:array? false
:object false
:identical? false
:type #object[Array]
:type #object[Object] false false #object[Array] false true
:satisfies? false
:seq? false
:map-entry? false
:coll? false
:array? false
:object false
:identical? false
:type #object[Object]
:type #object[Object] false true #object[Array] false false

thheller14:08:39

where do you get this object from? that looks like it might be coming from a different context? eg. iframe?

folcon14:08:55

I mean I have a work around using a custom version of js->clj which uses string checks, though I can probably also use your to-clj, but it would be good to figure out how to fix it

folcon14:08:08

So it's in an iframe within the browser.

thheller14:08:25

ok thats the issue then

folcon14:08:34

Not sure why being in an iframe should break it though?

thheller14:08:00

that is an isolated context. you don't actually have the real objects then, just proxies to it

folcon14:08:03

The entire app resides and runs within the iframe

folcon14:08:43

Ah, so this is a case of don't expect core stuff to work properly? So this is expected behaviour and thus not a bug?

thheller14:08:42

yes, this is not expected to work. you must properly "transfer" the object to the correct context before working on them

folcon14:08:36

Hmm, how does one do that? Or is that not possible in this case as the app sits in an iframe? Happy to read it up if you can point to a resource or tell me what to google 😃...

thheller14:08:59

usually postMessage to send the message between the contexts

folcon14:08:22

Hmm, ok so I think postmessage is being called on my behalf, but as the app is residing in the iframe I suppose I'll still have to deal with proxies for the moment... Ok. thanks @thheller 😃... Appreciate the help!

thheller14:08:20

a slightly worse version can just JSON.stringify and JSON.parse the object 😉

folcon14:08:49

Hmm, to be honest the hit is probably not that high as the objects are small, I'll give that a go, if it works, then great!

folcon14:08:18

Ok, that works 😃... Not the fastest, but good enough:tm:...

👍 1
pez06:08:30

@kbosompem did you get it to work? Looking at the user guide link you provided, it is talking about the option to use shadow-cljs from npm and letting it start Leiningen. With that setup; in Calva, you should use the shadow-cljs project type, not Leiningen + shadow-cljs when jacking in. You also should have :lein {:profile "+cljs"} in your shadow-cljs.edn. (Or simplify things for yourself and skip the :cljs alias, like @thheller mentioned. If you do that, then use :lein true, in the shadow config.)

vlad_poh19:08:47

@pez I am able to work by keeping them separate. i.e do shadow-cljs watch frontend first then jack in with leiningen only. Only drawback is calva treats the cljs files differently. I can’t jump to definitions etc Tried the :lein true option but the build kept failing with “The required namespaces is not available”

thheller20:08:29

> “The required namespaces is not available”

thheller20:08:59

likely means that your :source-paths from shadow-cljs.edn weren't in project.clj. they no longer apply from shadow-cljs.edn when using lein.

pez22:08:47

Yes, once you've moved dependencies and source-paths over to Leiningen, it should totally work. (I'm working on getting Calva to work using lein directly and start and hook up shadow via the nrepl middleware, but I'm stumbling a bit.)

pez11:08:46

@kbosompem The latest release of Calva should give you some more options for jack-in/connect. It fixes some missing pieces for the Leiningen + shadow-cljs project type. Note, however, that starting the REPL using the shadow-cljs npm executable is often still better with Calva, because it will keep shadow-cljs stdout and stderr output mostly in the jack-in terminal (or wherever you start the REPL). https://clojurians.slack.com/archives/CBE668G4R/p1661687545337289

👍 2
thheller06:08:31

@pez it should be fine to just start with leiningen? especially if the backend should also run in that process. it should not be required to run via shadow-cljs to get a working cljs setup?

thheller06:08:34

or does it start shadow-cljs AND leiningen?

pez06:08:05

I think Calva might do things wrong/in a way that prevents that from working. Or that something more is needed in project.clj or the lein command line. When using shadow to start things, it starts Leiningen, and then things work. If you have some time, I'd be happy to discuss this with you in some depth, because this is a very common Ux trap in Calva.

pez06:08:33

We have the same situation with deps.edn, btw. Starting with shadow-cljs is pretty easy to get working. Starting it with clojure is hella difficult

thheller06:08:43

you basically only need to do two things: (require 'shadow.cljs.devtools.server) and (shadow.cljs.devtools.server/start!)

thheller06:08:13

that will start the embedded server and get everything ready

thheller06:08:53

then starting a watch or repl is done via the shadow.cljs.devtools.api namespace

thheller06:08:18

so (shadow.cljs.devtools.api/watch :the-buid) and (shadow.cljs.devtools.api/repl :the-build) to get the REPL

thheller06:08:03

as for the clojure command thats just the usual clojure -A:whatever:aliases, nothing special in startup required there.

thheller06:08:34

I'm assuming you are starting via some kind of nrepl mechanism though? so you need to add the middleware for shadow-cljs there.

thheller06:08:11

I went over this with @kbosompem yesterday via DMs and the command calva issued was

lein update-in :dependencies conj '[nrepl,"1.0.0"]' -- update-in :dependencies conj '[cider/cider-nrepl,"0.28.5"]' -- update-in :plugins conj '[cider/cider-nrepl,"0.28.5"]' -- update-in '[:repl-options,:nrepl-middleware]' conj '["cider.nrepl/cider-middleware"]' -- with-profile +cljs repl :headless

thheller06:08:48

which would just need the shadow.cljs.devtools.server.nrepl/middleware added to the middleware injection, or hardcoded manually in the project.clj

thheller06:08:16

but this appeared to work fine when run via the command line

thheller06:08:28

but calva still wouldn't find the dependency when run via calva. not sure why

pez06:08:36

So, for Leiningen, what seems to be missing in Calva is the shadow.cljs.devtools.server steps and the middleware injection. I'll experiment a bit with it and see if I get it working and can fix Calva's support for this.

thheller06:08:58

if you point me to the places that actually run all this I can maybe help out

thheller06:08:02

if calva is able to handle 2 separate processes for the REPL it might also be an option to just run shadow-cljs separately and talk to that to handle CLJS things

pez06:08:28

Calva can't do this, though. But using shadow-cljs to start Leiningen things works.

pez06:08:02

The CLJS repls are created here in Calva: https://github.com/BetterThanTomorrow/calva/blob/dev/src/nrepl/connectSequence.ts#L212-L272 A thing to not is that the Figwheel config has startCode, but the shadow-cljs one hasn't.

pez06:08:04

The user can create custom start/connect sequences: https://calva.io/connect-sequences/

pez06:08:44

> but calva still wouldn't find the dependency when run via calva. not sure why This is strange and something I need to figure out...

pez06:08:36

> as for the clojure command thats just the usual clojure -A:whatever:aliases, nothing special in startup required there. Again, this is very tricky to get to actually work for Calva users. So there is something I need to fix, somewhere. Maybe it is the shadow.cljs.devtools.server stuff that's missing.

pez08:08:19

I created this issue and will spend some time today with trying to figure it out and maybe even fix it. https://github.com/BetterThanTomorrow/calva/issues/1842

pez13:08:36

I'm not having huge success here, but at least some. I get :missing-nrepl-middleware when selecting the build. I am injecting it, though. Starting the project like so:

lein update-in :dependencies conj '[nrepl,"1.0.0"]' -- update-in :dependencies conj '[cider/cider-nrepl,"0.28.5"]' -- update-in :plugins conj '[cider/cider-nrepl,"0.28.5"]' -- update-in '[:repl-options,:nrepl-middleware]' conj '[cider.nrepl/cider-middleware,shadow.cljs.devtools.server.nrepl/middleware]' -- repl :headless
This is what I try with:
clj꞉shadow-lein.server꞉> 
(do (require (quote shadow.cljs.devtools.server)) (shadow.cljs.devtools.server/start!) (require (quote shadow.cljs.devtools.api)) (shadow.cljs.devtools.api/watch :app))
shadow-cljs - server version: 2.19.9 running at 
shadow-cljs - nREPL server started on port 54224
:watching
; shadow-cljs - HTTP server available at 
; [:app] Configuring build.
; [:app] Compiling ...
; [:app] Build completed. (170 files, 0 compiled, 0 warnings, 1.24s)
clj꞉shadow-lein.server꞉> 
(do (require 'shadow.cljs.devtools.api) (shadow.cljs.devtools.api/nrepl-select :app))
:missing-nrepl-middleware

pez14:08:46

It's a bit unfortunate that shadow starts an nREPL server here, btw. Can I prevent that somehow?

thheller14:08:51

hmm dunno why the middleware would be missing there? injection looks correct

thheller14:08:03

you can just ignore the nrepl server started by shadow-cljs

thheller14:08:14

or set :nrepl false in shadow-cljs.edn

pez14:08:42

I can ignore it. It's that it could confuse the Calva user. I'll probably strip that message away.

pez14:08:20

Here's the project I am trying with: https://github.com/PEZ/shadow-w-backend

thheller14:08:25

looks fine. well kind of ancient style config but still ok

pez14:08:35

Repeating the start command used:

lein update-in :dependencies conj '[nrepl,"1.0.0"]' -- update-in :dependencies conj '[cider/cider-nrepl,"0.28.5"]' -- update-in :plugins conj '[cider/cider-nrepl,"0.28.5"]' -- update-in '[:repl-options,:nrepl-middleware]' conj '[cider.nrepl/cider-middleware,shadow.cljs.devtools.server.nrepl/middleware]' -- repl :headless
Removing Calva from the picture, here's a lein-only repro:
% lein repl :connect
Connecting to nREPL at 127.0.0.1:51722
REPL-y 0.5.1, nREPL 1.0.0
Clojure 1.11.1
OpenJDK 64-Bit Server VM 18.0.1+10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

shadow-lein.server=> (do (require (quote shadow.cljs.devtools.server)) (shadow.cljs.devtools.server/start!) (require (quote shadow.cljs.devtools.api)) (shadow.cljs.devtools.api/watch :app))
shadow-cljs - HTTP server available at 
shadow-cljs - server version: 2.19.9 running at 
shadow-cljs - nREPL server started on port 51913
[:app] Configuring build.
[:app] Compiling ...
[:app] Build completed. (170 files, 169 compiled, 0 warnings, 7.05s)
:watching
shadow-lein.server=> (do (require 'shadow.cljs.devtools.api) (shadow.cljs.devtools.api/nrepl-select :app))
:missing-nrepl-middleware
shadow-lein.server=> 
I'm pretty sure I've done something wrong in some config...

pez14:08:07

> kind of ancient style config I'm all ears! 😃

thheller14:08:24

{:lein true
 :dev-http {8700 "public"}
 :builds
 {:app {:target :browser
        :output-dir "public/js/compiled"
        :asset-path "/js/compiled"
        :modules {:main {:init-fn shadow-lein.core/init}}}}}

thheller14:08:52

in index.html remove <script>shadow_lein.core.init();</script>

pez14:08:45

Thanks. I considered something like that, but started wondering about the production case. Maybe that'll be a separate index.html?

thheller14:08:03

no? nothing changes for release

pez14:08:29

I’ll fix.

pez14:08:14

But my real problem is still to try support starting shadow projects with lein and deps. I think I'm close on the Calva side, but I’m not sure

pez19:08:15

I've updated the project config to be more modern now. 😃

😄 1
pez22:08:48

@kbosompem, that ^ tiny example project works with Calva jack-in if you choose the shadow-cljs project type.

pez07:08:12

If I use clojure to start the project, it works:

clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"}}}'  -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware,shadow.cljs.devtools.server.nrepl/middleware]"
I've updated the sample project repo with a deps.edn file.

pez07:08:01

I’m afk and can't check. But it struck me that maybe I have a bad lein version…!

pez08:08:24

% lein --version
Leiningen 2.9.8 on Java 18.0.1 OpenJDK 64-Bit Server V

pez09:08:04

I get the same :missing-middleware` result with Leiningen 2.9.10.

pez09:08:13

If I add this to project.clj it starts working.

:repl-options {:nrepl-middleware [shadow.cljs.devtools.server.nrepl/middleware]}
Seems command-line injection takes a backseat...

thheller09:08:45

injection works fine for me when working from the terminal though? and it if didn't work wouldn't the cider.nrepl middleware also be missing?

pez09:08:54

Do you get different results than I do with my repro above?

pez09:08:09

I get errors adding cider middleware in the project file, actually. It’s all very strange.

thheller09:08:23

I just tried running the command in the terminal directly and the middleware is added fine

thheller09:08:35

lein update-in :dependencies conj '[nrepl,"1.0.0"]' -- update-in :dependencies conj '[cider/cider-nrepl,"0.28.5"]' -- update-in :plugins conj '[cider/cider-nrepl,"0.28.5"]' -- update-in '[:repl-options,:nrepl-middleware]' conj '[cider.nrepl/cider-middleware,shadow.cljs.devtools.server.nrepl/middleware]' -- repl :headless

thheller09:08:53

so maybe has something to do with the way you invoke this? maybe some quoting issue or so? how do you actually start the process?

pez09:08:19

I run it from the terminal. Then in another terminal I use lein to connect to the nrepl server.

thheller10:08:35

your repro works fine for me?

thheller10:08:01

I mean manually in the terminal. I do not have vscode or calva installed

thheller10:08:55

d$ lein --version
Leiningen 2.9.4 on Java 17.0.3 OpenJDK 64-Bit Server VM

pez10:08:06

Calva is not involved in my repro. I’ll try with lein 2.9.4 later today.

thheller10:08:39

do you maybe have something on your machine that breaks it? like ~/.lein/profiles.clj or so?

pez10:08:30

I’ll check that as well. Afk now, but I just might have something there.

pez15:08:19

I don't have a ~/.lein/profiles.clj

pez16:08:00

I get the same :missing-middleware result using lein 2.9.4 and 2.9.1. And also the same on another machine I have.

pez16:08:30

I'm baffled this works on your machine, @thheller! Very strange issue.

thheller18:08:15

hmm now it doesn't work anymore

thheller18:08:01

I think I might have started the lein repl :connect in the wrong dir :P

thheller18:08:11

(and just connecting to my regular project)

pez18:08:56

😃 Well, then maybe I should now file an issue on Leiningen.

😅 1
pez08:08:23

When setting up a minimal shadow-cljs project with binaryage/devtools 1.0.6, I get tons of warnings such as this in the console:

DevTools failed to load source map: Could not load content for : Load canceled due to load timeout
Afaikt the source maps are created in the place the warning is talking about. Anyone recognize this situation?

thheller09:08:50

is this with a custom server or :dev-http?

thheller09:08:13

> Load canceled due to load timeout

thheller09:08:22

suggests the server is overloaded and doesn't reply properly?

pez09:08:16

Yes, overload could be it. I had a js linter running on a parent dir and it couldn't cope with the main.js produced by shadow and went into a spin.

pez09:08:36

(So it was my machine that was overloaded maybe)

pez11:08:46

@kbosompem The latest release of Calva should give you some more options for jack-in/connect. It fixes some missing pieces for the Leiningen + shadow-cljs project type. Note, however, that starting the REPL using the shadow-cljs npm executable is often still better with Calva, because it will keep shadow-cljs stdout and stderr output mostly in the jack-in terminal (or wherever you start the REPL). https://clojurians.slack.com/archives/CBE668G4R/p1661687545337289

👍 2