Fork me on GitHub
#cljs-dev
<
2022-02-04
>
thheller06:02:51

what makes this so terrible is that there are two variants of this

thheller06:02:13

one where the return value is unused and just causes webpack to include the file in the build somehow

thheller06:02:45

import "./some.css". this is fine and could be expressed via ns metadata or so without the need to do anything on the compiler side

thheller06:02:59

the other variant is where the return value is actually required and used

thheller06:02:48

import SomeImage from "./some.png" or const SomeImage = require("./some.png") where SomeImage will contain the URL to the image for base64 encoded string or whatever. or in case of react-native just an image id.

thheller06:02:44

this is much more problematic since the return value is actually required so the compiler needs to do something. at the very least collect all js/require calls so they can be transferred to whatever is doing the bundling

thheller06:02:55

the annoying part is that it uses (js/require ...) or import in the first place. so the same mechanism as referencing code

thheller06:02:59

my idea for this was creating a cljs.assets/reference or something macro that'll do the necessary work and then save some metadata in the analyzer data for that ns. after CLJS compilation the various targets could just add whatever code is necessary to wire things together

thheller06:02:47

dunno how this all would actually work since webpack actually allows dynamic requires for this mess, eg. require("./my-images/{foo}.png") but for the most part people seem to stick to static references

borkdude15:02:02

about iteration in CLJS, re-iterating what I said https://twitter.com/borkdude/status/1489615134297411591: The iteration function is specifically written for requesting services that have some kind of continuation token to fetch the next page. But in JS requesting such services is nowadays done using promises. Wouldn't it be better if CLJS had something directly based on promises (or callbacks), without having to pull in core.async? Copying iteration verbatim to CLJS hasn't much value imo, but I could be missing something.

quoll16:02:23

I don’t really have a say, but I do have an opinion! 🙂 While certain constructs don’t necessarily make sense in ClojureScript, there are occasions when people want to use code portably between Clojure and ClojureScript. After all, ClojureScript on Node can do many operations synchronously, and having promise-free alternatives can be useful in such an environment. When such moments arrive, discovering parity between platforms is very welcome.

1
lilactown16:02:25

it seems like the continue token passed to the step fn could in fact be a promise

lilactown16:02:38

IIRC how iteration works

borkdude16:02:46

While some things can be done sync on Node, that's not where JS is moving towards. And for browsers that's definitely not the option that's available.

borkdude16:02:15

@U4YGF4NGM it'd be interesting to see if verbatim iteration works with promises as is ...

lilactown16:02:22

yeah not really any better than a reduce

lilactown16:02:37

looking at the API again

lilactown16:02:38

for CLJS, it would be better if you passed a continuation callback that you called with the next token

lilactown16:02:22

that one change would make it useful for promises, callbacks, core.async, etc.

henryw37416:02:40

it'd be nice to have async opt on jvm also

borkdude16:02:58

There are plans to add a similar thing to core.async, but there are situations where in JS you just want to deal with promises directly

lilactown16:02:39

here's some pseudo-code of how an async iteration API might look

(defn fetch-summaries-async
  ([url]
   (fetch-summaries url 0))
  ([url page]
   (iteration-async
    (fn [cont page]
      (-> (js/fetch (str url "?page=" page))
          (.then #(.json %))
          ;; call `cont` callback with results to collect and continue fetching
          ;; the next page
          (.then cont)))
    :kf :next-page
    :vf :data
    :somef some?)))

borkdude16:02:58

+ error handling I guess

lilactown16:02:06

ah you probably want to pass in a done callback to be called when it has completed

lilactown16:02:10

right that too

borkdude16:02:10

and maybe the result value is an argument somewhere?

quoll16:02:31

> there are situations where in JS you just want to deal with promises directly This is true, but sometimes people don’t want to. They both need support

lilactown16:02:24

(defn fetch-summaries-async
  ([done err url]
   (fetch-summaries done err url 0))
  ([done err url page]
   (iteration-async
    (fn [cont fail page]
      (-> (js/fetch (str url "?page=" page))
          (.then #(.json %))
          ;; call `cont` callback with results to collect and continue fetching
          ;; the next page
          (.then cont fail)))
    :kf :next-page
    :vf :data
    :somef some?
    :done done
    :error err)))

lilactown16:02:36

you could then wrap this in a promise returning version

lilactown16:02:29

(defn iteration-promise
  [pf & {:keys [kf vf somef]}]
  (js/Promise.
   (fn [resolve reject]
     (iteration-async pf :kf kf :vf vf :somef somef :done resolve :error reject))))

lilactown16:02:37

I forgot initk but y'all get the picture

lilactown16:02:46

I should put this in a gist

borkdude16:02:03

:thumbsup: including iteration-async :)

lilactown17:02:02

I think that one major difference (other than async ofc) is that you can't rely on lazy seqs to consume each page in a lazy fashion

lilactown17:02:39

I think you'd want a much different API if you wanted to do this in a lazy way

dnolen15:02:54

@thheller that's actually a good summary thanks much!

dnolen15:02:08

@thheller what about ^{:js-assets [...] :js-named-assets {"./some.png" some-img}} ? don't care about the dynamic case at all

thheller15:02:42

metadata that creates a magical symbol some-img? that doesn't sound good

dnolen15:02:42

the named asset doesn't even really need compiler support other than recording for later that you need to generate the ns and assign that value

dnolen16:02:04

it would generate a var in that namespace

thheller16:02:44

I'd prefer a macro variant for that as mentioned

thheller16:02:08

nothing that causes Cursive and the others to show undeclared var warnings all over the place

dnolen16:02:25

I'm not really feeling the macro approach

thheller16:02:35

(cljs.assets/defasset some-img "./some.png") creates a deref'able some-img var usable via [:img {:src @some-img}] or so

thheller16:02:03

under the hood that derefable looks up the real url or whatever in some js object created by the compiler or so

dnolen16:02:53

ok, in the big picture that's not that different that what I'm thinking.

dnolen16:02:25

the thing I guess I'm liking is discouraging people from using js/require for this

dnolen16:02:53

I also wondering we can avoid the vars

dnolen16:02:08

(def some-img (cljs.asset/get-asset ::my-asset))

dnolen16:02:23

in the macro way - and just have a data api

dnolen16:02:32

the no magically symbol stuff in the metadata

thheller16:02:33

and where is ::my-asset coming from?

dnolen16:02:25

hrm well the ns metadata - but there's a reader problem then

dnolen16:02:56

not in any particular rush about this idea by the way - basically if there's some way we don't have do a compiler pass that would be ideal

dnolen16:02:32

macro is one way - but would prefer a design that

dnolen16:02:40

1. What does this ns need that's not a source file

dnolen16:02:48

2. API to get that thing if you need a reference

dnolen16:02:05

3. To your point - no var gen - let the user control that

dnolen16:02:41

4. Some way to prevent resource name clashes

thheller16:02:20

reason I prefer a var is hot-reload and easy collecting data from the analyzer data

thheller16:02:11

I mean whether you collect it from the ns or the ns :defs doesn't make much of a difference

thheller16:02:40

but I guess I don't care much whether its on the var or ns data either

thheller16:02:20

but the reader problem on ns metadata kinda means we can only use simple keywords which kinda sucks

thheller16:02:50

in webpack I kinda don't like that you have to tweak the build config if you want to inline vs image url

thheller16:02:17

so (js/require "./some-icon.png") may return a data url vs a regular url depending on build config

thheller16:02:34

would prefer for the code to control that but not a big deal I guess

thheller16:02:00

get-asset-url vs get-asset-data-url etc

thheller16:02:58

I think we are talking about the same thing really

dnolen16:02:35

yes, I would like it to be minimal - if we can get alignment

dnolen16:02:48

then maybe shadow could do purely additive stuff

thheller16:02:04

(defasset some-img "./some.png") results in (def ^{:cljs.assets/reference "class/path/some.png"} some-img (cljs.assets/get "class/path/some.png"))

dnolen16:02:19

for example get-asset-data-url etc. because since CLJS doesn't really care that you have to use Webpack

dnolen16:02:37

but it would be interesting to implement those as placeholders or something

thheller16:02:39

very lightweight macro really, just helping with resolving relative paths

dnolen16:02:10

hrm that's true

thheller16:02:10

we need this to be classpath aware or otherwise libraries can never make use of it

thheller16:02:55

and on build just copy off the classpath so webpack or so can find it

dnolen16:02:25

what I'm thinking is a design that's clearly flexible to interpretation or taste

dnolen16:02:36

i.e. if there well defined functions for accessing the assets

dnolen16:02:08

then CLJS can have placeholders that throw - but shadow could provide them since it manages this stuff

dnolen16:02:01

or cljs.asset could be fully portable, and shadow.asset provides extras

thheller16:02:05

thats why I'm bringing up the classpath. there are use cases where you want to reference stuff from npm packages

thheller16:02:12

and then there is stuff that will reference your local files

thheller16:02:41

which uses the same rules as npm resolve

dnolen16:02:50

but is the classpath really a problem?

dnolen16:02:12

my assumption would be this foo/bar must be classpath or node_modules

thheller16:02:23

path based things use the classpath. ie (ns foo.bar {:asset {:some-img "./some.png"})) resolves to foo/some.png on the classpath

dnolen16:02:28

if it is on the classpath it wins, and copy that asset

thheller16:02:47

whereas "module" requires such as react-dom/foo.css goes to node_modules

dnolen16:02:52

if it is a relative path, rewrite it (but this is an implementation detail, users don't care)

thheller16:02:30

classpath is important for library others that want to write say reagent components and include some css and/or images

dnolen16:02:54

but I think again we saying the same thing - I see three cases

dnolen16:02:04

1. non-relative, is it classpath? if so wins

dnolen16:02:17

2. not classpath, must be node.js - do something

dnolen16:02:37

3. relative - clearly do something

thheller16:02:39

there should be no if so ever

thheller16:02:00

./some.png and /some.png is classpath, some-lib/foo.css is always npm

thheller16:02:31

there should be no case where some-lib is tried on the classpath

dnolen16:02:37

but that doesn't make sense to me?

dnolen16:02:46

is ./some.png or /some.png a valid classpath thing?

thheller16:02:47

thats no the webpack/node world works

dnolen16:02:49

(I honestly don't know)

thheller16:02:26

we treat it as such. /some.png becomes ( "some.png)

thheller16:02:40

relative paths are resolved against the current ns

dnolen16:02:42

but what I mean is does this kind of path jive with how people use the classpath

dnolen16:02:04

I don't recall ever dealing w/ classpath resources w/ relative or fake absolute paths

dnolen16:02:33

if there is cognitive mismatch here I don't like it

thheller16:02:40

but there is no ambiguity here. clear rules for this behavior. I hate "look on the classpath first, if not found proceed to npm"

dnolen16:02:41

but as I said about maybe I don't know about this pattern

dnolen16:02:02

the ambiguity would be is nobody writes "/foo" for classpath stuff

dnolen16:02:09

that is the thing to not like - if that is true

thheller16:02:13

I just adopted the node/webpack resolve rules to fit for the classpath

thheller16:02:33

ie. in node (js/require "/foo.js") actually just looks up /foo.js on disk. which I replaced with classpath since that is how the jvm works

dnolen16:02:37

yeah I do not like it

dnolen16:02:56

if somebody says classpath, I know what that means - what you're talking about above is a strange combination

thheller16:02:21

but how do you differentiate then between "look this up on the classpath" vs "look this up in node_modules"

thheller16:02:29

I really really don't like to check both for everything

thheller16:02:52

since the classpath stuff needs to do more and node_modules basically nothing since webpack already knows how to find it

dnolen16:02:04

I just think those are implementation problems

thheller16:02:10

I didn't make the node/webpack resolve rules 😉

dnolen16:02:12

but you're making it sound like this is hard?

dnolen16:02:25

what is complicated about (some? (io/resource "foo.js"))

thheller16:02:46

ambiguous, not hard

dnolen16:02:06

but why does it matter?

thheller16:02:07

I look at the code and cannot tell if this is from npm or the classpath

thheller16:02:25

I look at ./some.png I know classpath

dnolen16:02:31

what I mean is that for the user - what could go wrong?

dnolen16:02:35

what are the actual problems?

thheller16:02:35

I look at some-lib/foo.css I know node_modules

dnolen16:02:13

but what is the value of knowing such a thing to the end user?

thheller16:02:26

well, classpath rules already establish good practice for where stuff goes and naming things

dnolen16:02:30

since they wrote it - they know what they are doing

thheller16:02:54

I want to prevent people include assets/logo.png in their stuff

dnolen16:02:15

stepping back

thheller16:02:18

and some library uses the same path and replaces the logo because it came first on the classpath

dnolen16:02:28

defasset & defresource

dnolen16:02:50

and skip all this strings meaning somehting - which is probably bad - both of our ideas 🙂

thheller16:02:02

I'm fine with differentiating between classpath and node_modules assets in other ways too

thheller16:02:14

I just want to avoid the look in two places thing

dnolen16:02:29

I like defasset and defresource

dnolen16:02:57

there's an interesting thing which is that defresource could be JVM friendly way for people to describe assets for the build

thheller16:02:14

I also would like something that is completely CLJ/S only and does not expect to only work with webpack

dnolen16:02:53

I think defresource would be that thing - decouple from webpack

dnolen16:02:23

but I don't think you need to follow JS conventions at all

dnolen16:02:34

just JVM classpath, and drop the Node-isms

thheller16:02:56

well the JVM classpath never has relative references so we lose that

dnolen16:02:05

this is a good thing

thheller16:02:42

they are kind of convenient but I don't care that much either

thheller16:02:53

(ns my.super.duper.library) (defresource some-img "./some.png") is kinda nice

thheller16:02:30

don't have to think about (defresource some-img "my/super/duper/some.png")

thheller16:02:55

but I don't feel strongly about either and they achieve the same result

thheller16:02:35

if it never tries to find node_modules/my/super/duper/some.png I'm happy

dnolen16:02:05

I see - the shortcut is interesting and natural - but could probably add the sugar later in the pondering process

👍 2
thheller17:02:40

btw there is prior art for all of this in https://clojureverse.org/t/using-none-code-resources-in-cljs-builds/3745. although I never went further than the inline macro. it does have the path resolve stuff though.

quoll18:02:53

I need to bring tests over from Clojure

quoll18:02:40

They need to vary a little, e.g. parsing the string form of Long/MAX_VALUE is beyond the safe integer limit in JS, and it gets rounded

quoll18:02:05

Oh! Sorry! Wrong patch! My mistake

quoll18:02:46

(I’m referring to CLJS-3348)

dnolen18:02:22

right, yeah this is the first one

mfikes20:02:53

Made a JIRA comment about 3 (likely internal helper symbols) that are currently public that don't appear in clojure.math

quoll21:02:38

Thank you. You’re right with 2 of them… I didn’t think about xor and get-little-endian. I don’t know about IEEE-fmod though. It’s an implementation of fmod (and actually returns the same answers as the fmod found in #include <math.h>). So even though it’s not exposed publicly in Java, it’s public in other circumstances. I gave this some thought when I wrote it, and actually removed the private flag from it. But I can put it back on.

mfikes13:02:01

@quoll Looks like the standard copyright notice needs to be added to the tops of the newly introduced files. (Just look at other files for what it looks like)

👍 1
mfikes15:02:55

@quoll (Perhaps minor thing): It appears that two-to-the-double-scale-up/down leaves stuff in :advanced output that appears to be unelidable and calculated at namespace load time. Maybe if they are actually constants they could be replaced with their values. (But maybe they are not VM-independent constants? Dunno.)

quoll15:02:01

That one will need a keyboard and not a phone. I’ll look into it later this morning 🙂

mfikes15:02:34

No worries. I think it is minor. Wondering if their values of 1.3407807929942597e+154 and 7.458340731200207e-155 can be placed literally in the code, or if those values depend on the VM you are running on... hrm.

quoll16:02:23

There is some rounding that can occur on the JVM vs JS, though the bit patterns are identical

quoll16:02:36

So it depends on the specific values

mfikes16:02:33

Yeah, it is perhaps dangerous to inline any particular constant value for those