Fork me on GitHub
#tools-deps
<
2019-02-13
>
stathissideris07:02:53

does the Clojure runtime have access to the aliases that were passed? Can my code access this information somehow?

dominicm07:02:09

There are some system properties set. I think only clojure.libfile

dominicm07:02:17

Why do you need to know?

stathissideris07:02:10

I’d like to know whether its a figwheel run or not

dominicm07:02:08

@stathissideris everyone is going to name their Figwheel alias differently. I'd check if the namespace is loaded/can be required, and depending on context, check if the state var is bound.

dominicm07:02:56

Your code might work with some lein setups too that way

stathissideris08:02:59

good point about about naming, but I need this for my specific setup, I’m not making a generic tool, so hardcoding it would be ok

stathissideris08:02:11

Is is possible to combine -e and -m to inject a var maybe?

dominicm08:02:52

I do this in edge using :jvm-opts

dominicm08:02:14

Jvm opts compose, main opts do not.

dominicm08:02:11

Using -D in jvm opts that is.

stathissideris08:02:08

oh thanks, I’ll check it out

dominicm13:02:22

I think there's a NPE in t.d.a:

Error building classpath. nil
java.lang.NullPointerException
        at clojure.tools.deps.alpha.extensions.maven$eval638$fn__641.invoke(maven.clj:41)
        at clojure.lang.MultiFn.invoke(MultiFn.java:239)
        at clojure.tools.deps.alpha$canonicalize_deps$fn__1027.invoke(alpha.clj:74)
        at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
        at clojure.core$reduce.invokeStatic(core.clj:6827)
        at clojure.core$reduce.invoke(core.clj:6810)
        at clojure.tools.deps.alpha$canonicalize_deps.invokeStatic(alpha.clj:73)
        at clojure.tools.deps.alpha$canonicalize_deps.invoke(alpha.clj:71)
        at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:189)
        at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:170)
        at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:236)
        at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:218)
        at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invokeStatic(make_classpath.clj:59)
        at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invoke(make_classpath.clj:52)
        at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:70)
        at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:64)
        at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:109)
        at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:84)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.main$main_opt.invokeStatic(main.clj:491)
        at clojure.main$main_opt.invoke(main.clj:487)
        at clojure.main$main.invokeStatic(main.clj:598)
        at clojure.main$main.doInvoke(main.clj:561)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.main.main(main.java:37)
this may be time sensitive as I'm using a just-deployed coordinate.

dominicm13:02:41

org.webjars.bower/react {:mvn/version "15.4.2"}
  org.webjars.bower/es6-promise {:mvn/version "4.0.5"}
  org.webjars.bower/fetch {:mvn/version "0.9.0"}
  org.webjars.bower/graphiql {:mvn/version "0.11.11"}
  org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"}
  org.webjars.npm/subscriptions-transport-ws {:mvn/version "0.8.3"}
these are the deps I just added

dominicm13:02:18

org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"} this appears to be the exact problem dep

Alex Miller (Clojure team)13:02:28

runtime does not have access to the aliases that were used @stathissideris

stathissideris13:02:12

@alexmiller thanks, I ended up using -D to indicate the alias via properties as @dominicm suggested

Alex Miller (Clojure team)13:02:59

I can repro the npe, will take a look. shouldn’t be timing-related

Alex Miller (Clojure team)13:02:22

there’s a version range in the transitive deps, wouldn’t surprise me if that’s involved

dominicm13:02:56

that seems more likely, yeah. I was worried that we'd caught maven in an inbetween state and it was causing an NPE.

Alex Miller (Clojure team)13:02:11

I think it’s finding no matches in the version range

Alex Miller (Clojure team)13:02:40

so code could definitely be tightened up around that to not npe but it will just not find the dep then

dominicm13:02:34

yeah. makes sense. webjars is a bit weird like this I guess 🙂

dominicm13:02:41

I can add explicit graphiql dep here.

Alex Miller (Clojure team)13:02:29

org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"} depends on org.webjars.npm/graphql {:mvn/version "[0.9.1,0.10)" but I don’t see anything older than 0.10.1 in maven central

dominicm13:02:05

webjars is a bit weird. You basically are deploying npm things. So if nobody has deployed graphiql 0.10, then it won't be there.

dominicm13:02:29

adding an explicit graphiql dep hasn't helped though, annoyingly. So I guess I will deploy 0.10 via http://webjars.org

Alex Miller (Clojure team)13:02:39

does npm actually have jars in it?

dominicm14:02:00

webjars takes stuff from npm, and puts it in a jar

Alex Miller (Clojure team)14:02:28

tdeps could have an extension to use webjars directly

dominicm14:02:54

do you mean npm directly? I don't know how you'd use webjars directly 🙂

dominicm14:02:55

but either way, that would be a huge +1 from me

Alex Miller (Clojure team)14:02:35

well, you would need to tell me more things for me to understand what’s possible, but anything that can be a source of named, versioned artifacts is a potential target

dominicm14:02:13

good question, admittedly I don't fully understand how webjars works.

Alex Miller (Clojure team)14:02:17

well, webjars seems to have a a cdn? whatever step you’re doing seems like some sort of mindless wrapping task

Alex Miller (Clojure team)14:02:21

programs are good at those

dominicm14:02:24

> All of the WebJar contents are available on the public jsDelivr CDN. Just prefix //cdn.jsdelivr.net/webjars/{groupId} in front of your static asset URLs. For instance, if using the org.webjars : jquery WebJar and your local URL to jquery.js is /webjars/jquery/2.1.0/jquery.js then the CDN URL would be: //cdn.jsdelivr.net/webjars/org.webjars/jquery/2.1.0/jquery.js So I guess the contents of the jars are available via the CDN, rather than the jars themselves.

dominicm14:02:41

the actual jars go to maven.

dominicm14:02:55

then you can request a new jar to go to maven via clicking buttons on http://webjars.org - that triggers an automated process

Alex Miller (Clojure team)14:02:11

“clicking buttons” :)

dominicm14:02:31

is your proposal that instead of deploying to maven, a webjars-like thing would build the jars and stick them in ~/.local/cache/webjars/foo.bar.1.jar ?

Alex Miller (Clojure team)14:02:47

why do you even need jars? they’re just files on the classpath right?

Alex Miller (Clojure team)14:02:39

it would be helpful to do something that actually helped people down the line as they build and deploy stuff

Alex Miller (Clojure team)14:02:08

I am too ignorant about cljs / npm / node / etc to not know what would be useful

dominicm14:02:51

you're right. jars not needed.

Alex Miller (Clojure team)14:02:58

npm is a central repository. it has stuff you want in your app. tools.deps has an extension point to plug in code to get stuff and put it on your classpath (and determine its dependencies)

dominicm14:02:59

fwiw, this is pretty unrelated to cljs.

Alex Miller (Clojure team)14:02:09

see how ignorant I am? :)

dominicm14:02:37

I think npm is a good target for t.d.a. So is bower.

dominicm14:02:48

maybe not bower, it's kinda old now.

Alex Miller (Clojure team)14:02:13

yeah, anything that didn’t come out this month in js is old :)

Alex Miller (Clojure team)14:02:14

I guess coming back to a problem statement - this is like js stuff that you want to serve as static resources from your app?

Alex Miller (Clojure team)14:02:10

does npm really not have any kind of namespacing?

Alex Miller (Clojure team)14:02:44

oh, it has scopes, I see

Alex Miller (Clojure team)14:02:40

well, seems like the info necessary exists if you want to file a ticket defining what’s useful

dominicm14:02:54

> I guess coming back to a problem statement - this is like js stuff that you want to serve as static resources from your app? Exactly!

dominicm14:02:05

Although, admittedly, I'd personally be quite happy with cdn integrations.

dominicm14:02:51

bower predates npm afaik. It was a big step forward at the time, but it's not really used. npm got more traction and was dominant by the time a majority of companies turned their attention to js.

dominicm14:02:16

bower doesn't recommend you use bower so ¯\(ツ)

dominicm14:02:42

I'll write down that I want to create this, and have a note for myself to think a little more about what I want 🙂

dominicm14:02:51

@alexmiller while I have you, is there a workaround for this graphiql issue? bintray appears to be having issues, and I can't get an older graphiql to deploy. :exclusions hasn't worked, nor has an explicit graphiql dependency on 0.11.11.

Alex Miller (Clojure team)14:02:37

not sure I have a good option for you

Alex Miller (Clojure team)14:02:34

if you made webjars deploy version 0.9.1 to maven central that would let it proceed

dominicm14:02:10

Unfortunately, that's failing 🙂 I thought you might say that though. It's okay

Alex Miller (Clojure team)14:02:09

kind of an interesting failure mode. would be better if it could defer long enough to be overridden

Alex Miller (Clojure team)14:02:56

in this particular case, the only reason it’s resolving early is due to the version range I think, trying to narrow that down to single version

dominicm16:02:23

Hmm, 0.9.1 now exists, but I'm still seeing a NPE. no it doesn't, i've been deploying the wrong package

dominicm16:02:57

graphiql vs graphql

dominicm17:02:20

You hit an NPE whenever you have a transitive dependency which doesn't exist. I'm seeing it with a version pinned to [1.0.3] (maybe that is a range though?)

dominicm17:02:35

Actually, no. I'm confused.

dominicm17:02:17

I've rm -rf .cpcache, and used -Sforce, just in case.

dominicm17:02:36

❯ clj -Sdeps '{:deps {org.webjars.npm/graphql {:mvn/version "0.9.1"}} :aliases {:v {:verbose true}}}' -A:v -Sforce
Initial deps to expand:
{org.clojure/clojure #:mvn{:version "1.10.0"},
 org.webjars.npm/graphql #:mvn{:version "0.9.1"}}
Expanding org.clojure/clojure #:mvn{:version 1.10.0}
        =>  include, pin top dep
Expanding org.webjars.npm/graphql #:mvn{:version 0.9.1}
Error building classpath. nil
java.lang.NullPointerException
        at clojure.tools.deps.alpha.extensions.maven$eval638$fn__641.invoke(maven.clj:41)

dominicm17:02:22

[ERROR] Failed to execute goal on project my-app: Could not resolve dependencies for project : Failed to collect dependencies at org.webjars.npm:graphql:jar:0.9.1 -> org.webjars.npm:iterall:jar:[1.0.3]: No versions available for org.webjars.npm:iterall:jar:[1.0.3] within specified range -> [Help 1] maven is struggling too

dominicm17:02:47

Oh. Interesting. It is fine with it as a top-level dependency, but not as a transitive dep.

Alex Miller (Clojure team)18:02:32

[1.0.3] is a version range. from Maven’s perspective just 1.0.3 by itself is a specific version

Alex Miller (Clojure team)18:02:55

I don’t think there’s any difference with top vs transitive - you probably just have it in your local Maven repo at ~/.m2/repository now

Alex Miller (Clojure team)18:02:50

it works for me as a transitive dep, and as a top-level dep with specific version, and as a top-level dep with version range [1.0.3]

Alex Miller (Clojure team)18:02:59

(clearing all caches before each one)

Alex Miller (Clojure team)18:02:53

$ rm -rf ~/.m2/repository/org/webjars
$ clj -Sdeps '{:deps {org.webjars.npm/graphql {:mvn/version "0.9.1"}}}' -Sforce
Downloading: org/webjars/npm/graphql/0.9.1/graphql-0.9.1.pom from 
Downloading: org/webjars/npm/iterall/maven-metadata.xml from 
Downloading: org/webjars/npm/iterall/1.0.3/iterall-1.0.3.pom from 
Downloading: org/webjars/npm/graphql/0.9.1/graphql-0.9.1.jar from 
Downloading: org/webjars/npm/iterall/1.0.3/iterall-1.0.3.jar from 
Clojure 1.10.0
user=>

Alex Miller (Clojure team)18:02:21

when you deploy to Maven central it can take 10 minutes to an hour to show up for download, so maybe it just wasn’t there yet

dominicm18:02:46

I'm a bit suspicious that perhaps version ranges are searched somewhere different than specific versions.

Alex Miller (Clojure team)18:02:24

but there is an extra piece of code to explicitly resolve them in tdeps

Alex Miller (Clojure team)18:02:09

if you haven’t, you should try selectively deleting part of your ~/.m2/repository/org/webjars/…

Alex Miller (Clojure team)18:02:32

there are metadata files about versions that are cached there and only updated once every 24 hrs

Alex Miller (Clojure team)18:02:55

(this is a setting we don’t currently surface - if you’re using mvn, you can pass -U to force an update)

dominicm18:02:52

Is it possible that metadata would store [1.0.3] separately from 1.0.3?

Alex Miller (Clojure team)19:02:19

it just stores known versions and then uses that to resolve

Alex Miller (Clojure team)19:02:41

but if you had already filled that cache for the day and the repository said that version didn’t exist, it’s going to keep telling you that

Alex Miller (Clojure team)19:02:51

it will not go re-check the repository

Alex Miller (Clojure team)19:02:37

this file in particular: ~/.m2/repository/org/webjars/npm/iterall/maven-metadata-central.xml

Alex Miller (Clojure team)19:02:46

last check timestamps are in ~/.m2/repository/org/webjars/npm/iterall/resolver-status.properties

stijn19:02:31

@alexmiller on the slow REPL startup of the latest java 8 (u202) on Linux, looking at the stack traces, it spends a lot of time compiling some java-time things (https://github.com/dm3/clojure.java-time). One CPU goes to 100% during this and after that stack trace is gone, the REPL starts.

hiredman20:02:23

java-time does lot of meta programming stuff, finding types, doing a graph traversal to find paths between types

hiredman20:02:55

so I could imagine types changing between jdk versions

dominicm20:02:19

blugh, after all this work, one of my transitive deps doesn't work because it has a custom gql license which webjars won't automatically repackage because it doesn't recognize. If we were a direct npm client this would be avoided. Very tempting proposition really.

dominicm20:02:46

@alexmiller if I wrote an npm client with an EPL license in clojure, would it be candidate for use in t.d.a? Or would you prefer it was under the clojure/core umbrella?

Alex Miller (Clojure team)20:02:38

could be an external lib (we use others already). I think there are some out there already. but maybe the amount of access needed by tda is so small to not even be worth that, not sure.

Alex Miller (Clojure team)20:02:40

I guess you probably want a cache ala tools.gitlibs though and that’s something we’d want outside tda

dominicm20:02:30

I haven't really looked at the npm client details yet. Might be worth seeing if we can share the yarn cache. Plus I'm a bit uncertain about how to handle http.

Alex Miller (Clojure team)20:02:07

well I think you have the same options as gitlibs - shell out to npm or make a direct implementation

Alex Miller (Clojure team)20:02:07

pros and cons. might be worth gathering some more information about the things you need to implement: 1) given a name/version, determine that lib’s deps (don’t need transitive, just shallow) 2) given a name/version, download the bits somewhere

dominicm20:02:12

Npm is far more amenable I think. Yarn is a reimplementation of the npm client. Plus, npm doesn't have a global cache, and because of the hierarchical nature, you probably want more control.

dominicm20:02:49

I suppose you have to figure out auth, which is the recurring issue.

Alex Miller (Clojure team)20:02:01

well public stuff is fine I assume

Alex Miller (Clojure team)20:02:22

for creds, we already have a means to do this via the maven settings.xml on a per-repo basis

Alex Miller (Clojure team)20:02:43

admittedly kind of weird to put npm auth there

stijn20:02:17

@hiredman, the weird thing is that it's only a minor version upgrade (from 8.u192 to 8.u202). With 192 the repl starts in seconds, with 202 it takes minutes

ghadi20:02:04

I know it's aside from your problem @stijn but I wouldn't use that library, to be perfectly honest.

ghadi20:02:27

java.time is perfectly approachable

dominicm20:02:03

https://juxt.pro/tick/docs/index.html makes java.time quite nice, and it works in cljs :face_with_rolling_eyes: (I work for JUXT)

seancorfield20:02:44

That's a matter of opinion @U050ECB92 🙂 I find clojure.java-time really does produce cleaner-looking code in a lot of situations.

ghadi21:02:40

clojure.java-time seems to have a steep cost in classloading and metaprogramming

seancorfield21:02:43

It's a one-off cost, the first time you use certain constructors. It's fast after that.

seancorfield21:02:06

The readme even shows how you can "warm up" that stuff.

seancorfield21:02:49

I was a bit surprised that it caused such a "blip" on 202(?) since it's fine on 192 (and on subsequent builds) according to the OP.

stijn08:02:30

I will investigate some more when I have the time. For now, went back to 192.

ghadi20:02:39

the actual JDK package java.time

hiredman20:02:08

maybe check to see if you have org.threeten.extra.Temporals and or org.joda.time.DateTime on your classpath, and if calling (Class/forName ...) with those class names changes behavior on the different jdk versions

stijn08:02:08

yes org.joda.time.DateTime is on the classpath. will check later what the impact is

Alex Miller (Clojure team)20:02:36

just a couple small enhancements, no biggie

dominicm20:02:55

Woah, very quick, thanks!

Alex Miller (Clojure team)20:02:55

hadn’t released in a while

seancorfield21:02:16

That Groups post hasn't gone out via email again, as far as I can see @alexmiller

seancorfield21:02:27

(I just replied to it on the web which should cause it to be seen via email 🙂 )

Alex Miller (Clojure team)22:02:52

I sent that one from a different account