Fork me on GitHub
#tools-deps
<
2021-06-30
>
greg15:06:48

How to pass vector as one of args for -X ? Example:

(ns xml)

(defn merge-files
  "Merges given XML files. Options:
   :files Vector of filenames, required
   :output Filename for merge output, optional"
  [{:keys [files output]}]
  ...)
Assuming there is no aliases specified in deps.edn, how to call merge-files? The attempt below seems to be invalid:
clj -X xml.merge-files :files '["file1.xml" "file2.xml"]' :output "file3.xml"

Alex Miller (Clojure team)15:06:03

why do you say it is invalid?

Alex Miller (Clojure team)15:06:32

you also likely need to single quote the last arg

Alex Miller (Clojure team)15:06:44

clj -X xml.merge-files :files '["file1.xml" "file2.xml"]' :output '"file3.xml"'

Alex Miller (Clojure team)15:06:23

alternately, you can now pass a trailing map in the latest version

clj -X xml.merge-files '{:files ["file1.xml" "file2.xml"] :output "file3.xml"}'

greg15:06:01

thanks, works now, kind of Actually previously I tried to pass the vector '[file1.xml file2.xml]' I'm having now another issue: `Unqualified function can't be resolved: xml.merge-files' The full call is like that

clojure -Sdeps '{:paths ["scripts"]}' -X xml.merge-files :files '["~/.m2/settings.xml" "/tmp/settings.xml"]' :output '"te.xml"'

greg15:06:16

scripts is just a directory holding xml.clj file xml.clj file defined internally (ns xml) and (defn merge-files) Is there anything else I need to do?

greg15:06:43

Alright, I should call clj -X xml/merge-files instead of clj -X xml.merge-files

greg15:06:54

no no, my fault, thanks for help 🙂

dpsutton22:06:02

i've got a strange bug that i just cannot figure out. We build our release with a clojure cli based program. It has local-roots for a project that builds the actual uberjar, and it includes the paths :paths ["src" "resources"]. And for whatever reason, this resources path is not showing up on the classpath for my coworker when running this release script, but it is for everyone else

dpsutton22:06:07

I got him to run clojure -e "(run! println (clojure.string/split (System/getProperty \"java.class.path\") #\":\"))" and saved his output and my output and the only difference (after adjusting paths from his workspace to mine) is

diff ~/projects/work/diffs/my.classpath ~/projects/work/diffs/his.classpath
3d2
< /Users/coworker/workspace/metabase/bin/build-mb/resources

dpsutton22:06:02

and we are both on Clojure CLI version 1.10.3.855

Alex Miller (Clojure team)22:06:07

did you -Sforce on both?

Alex Miller (Clojure team)22:06:35

does the directory exist on both?

dpsutton22:06:36

no. and i wish i would have told him to try that. are the cache's transitively used like that?

dpsutton22:06:33

yeah. dir exists, and the file it should find is there. it just seems to be a stale classpath. and i believe the resources addition was recent, so if there's a stale classpath around that would make sense. didn't think that it might use a cached one of a local root though.

Alex Miller (Clojure team)22:06:15

changes in local dep deps.edn are not automatically picked up - you have to -Sforce to pick up those changes

dpsutton22:06:25

at both locations?

dpsutton22:06:35

or just the dep or just the top level?

Alex Miller (Clojure team)22:06:41

top level is where it matters

Alex Miller (Clojure team)22:06:48

since it's that classpath you care about here

dpsutton22:06:09

awesome. thanks. think he's in an interview but we'll try that. thank you very much as always

Alex Miller (Clojure team)22:06:31

in the leaf local dep, any changes in deps.edn should make that project stale and recompute there in that project

Alex Miller (Clojure team)22:06:14

but things that depend on it will not automatically detect that change

dpsutton22:06:49

oh i see. so the root's cache includes the old classpath and it doesn't know to invalidate

dpsutton22:06:03

got a good model of it now. thanks

Alex Miller (Clojure team)22:06:14

for the wrapper to know to check that file, it would have to understand the deps model, which would mean running Clojure code, which would negate the benefit of the cache :)

dpsutton22:06:58

i guess the cache doesn't include a hash of the deps.edn files. i've never looked inside of cpcache really.

Alex Miller (Clojure team)22:06:05

I do think it could remember this info in the cache though and use that to determine the set of files to check for staleness

Alex Miller (Clojure team)22:06:23

there is a ticket for this, just has never been something I've gotten around to

dpsutton22:06:34

i'll go look and vote for your next pass through. thanks

dpsutton22:06:15

i think this is it but it's pretty verbose:

Alex Miller (Clojure team)22:06:09

the hash includes knowledge about the current project's deps.edn files, but not ones in the cp

hiredman23:06:40

we have a new build script at work, and with it I (and no one else, including the CI server) sometimes get weird errors when running the tests about the clojure.core.cache.wrapped namespace being missing. I changed the test runner to pop open a repl when it hit that error instead of exiting, and poked around and found that clojure.core.cache.wrapped had been added to *loaded-libs* but the namespace didn't actually exist, when I removed it from *loaded-libs* it loaded fine. That to me sounds like a concurrent code loading issue, and some digging and monkey patching of require to dump stacktraces when called off the main thread and I have provisionally tracked it down to https://github.com/clojure/tools.deps.alpha/blob/master/src/main/java/clojure/tools/deps/alpha/util/S3TransporterFactory.java#L48 , I have seen the s3 transport do this in the middle of my test run: call back into clojure, which causes core.async and a ton of other deps to load, which races with code running the tests.

hiredman23:06:08

is there some way to call calc-basis synchronously?

Alex Miller (Clojure team)23:06:50

you can pass -Sthreads 1

Alex Miller (Clojure team)23:06:42

that's one source of multithreading in the download process, but doesn't actually get to that thing in S3TransporterFactory

hiredman23:06:45

I guess I could require clojure.tools.deps.alpha.util.s3-transporter in our build script, which would no-op the transport factory requiring it

Alex Miller (Clojure team)23:06:59

I'm not sure exactly what your setup is but the S3 stuff is used to build a classpath via tdeps, so not sure why it's in process with your tests? (should be separate process)

Alex Miller (Clojure team)23:06:38

unless you're using tdeps programmatically

hiredman23:06:46

our build script is

Alex Miller (Clojure team)23:06:21

then yeah, I'd try to force the load earlier

Alex Miller (Clojure team)23:06:37

I guess I could use the require lock in the background load

hiredman23:06:38

dunno that would help, the test code being loaded is using require without the lock

seancorfield23:06:32

We don't need the S3 stuff -- is there some env setting that prevents t.d.a. from even trying to load it? Or is the fact that it's part of that Java class's initializer making that impossible?

Alex Miller (Clojure team)23:06:39

and there's a Named annotation on that class so ends in the META-INF in the jar etc

Alex Miller (Clojure team)23:06:02

nothing like dependency injection frameworks to ruin everything good about dependency injection

Alex Miller (Clojure team)23:06:30

it is intended to be tolerant of not finding the class (which is just a crutch for using it without compilation during dev)

Alex Miller (Clojure team)23:06:56

so you could just delete the class out of the jar if you were feeling crazy :)

Alex Miller (Clojure team)23:06:08

or use it as a git / local dep

Alex Miller (Clojure team)23:06:05

but if you want something better, file a TDEPS ticket and I'll think about it better in a week or two

hiredman23:06:29

it defeated me trying to to even find how it was initialized because I was grepping for 's3' not 'S3' 🙂

Alex Miller (Clojure team)23:06:24

if only I could just hand an instance to the thing that was using it rather than gen'ing a class and passing a class name

seancorfield23:06:31

Yeah, when I use t.d.a from source -- via the add-libs branch -- I get the warning about S3 not being found but works 🙂 so maybe we'll just change out dependency from a maven version to a git sha and just live with the warning about S3 Transport not being loaded 🙂

seancorfield23:06:55

@hiredman I sent you the :git/url coords, to save you looking it up.