Fork me on GitHub
#clojure
<
2024-02-15
>
valerauko06:02:02

clojure.tools.deps/create-basis doesn't merge paths? With an alias like {:foo {:extra-paths ["test"]}} I'd expect :paths in create-basis when run with -M:foo to have "test" but it does not

valerauko06:02:34

The :aliases :foo in the generated map does include the :extra-paths key just as in the deps.edn

valerauko06:02:27

Is there a way to get the merged paths list that includes extra-paths from currently active aliases?

seancorfield07:02:41

Ah, what key are you looking at in the basis? You probably want :classpath-roots (not :paths).

seancorfield07:02:03

(~/clojure)-(!2014)-> clj -A:deps
Downloading: org/clojure/clojure/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from clojars
Downloading: org/clojure/clojure/maven-metadata.xml from sonatype
Clojure 1.12.0-alpha7
user=> (require '[clojure.tools.deps :as t])
nil
user=> (:classpath-roots (t/create-basis {}))
["src" "/home/sean/.m2/repository/org/clojure/clojure/1.12.0-alpha7/clojure-1.12.0-alpha7.jar" "/home/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar" "/home/sean/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar"]
user=> (:classpath-roots (t/create-basis {:aliases [:test]}))
["test" "src" "/home/sean/.m2/repository/org/clojure/clojure/1.12.0-alpha7/clojure-1.12.0-alpha7.jar" "/home/sean/.m2/repository/org/clojure/test.check/1.1.1/test.check-1.1.1.jar" "/home/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar" "/home/sean/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar"]
user=>

seancorfield07:02:40

create-basis will create a new basis based on the options you provide -- it's not related to how your program has been invoked (I'm puzzling over your "when run with -M:foo" comment)...

seancorfield07:02:57

If you want the paths for the currently running Java process (not related to calling create-basis), you can do (System/getProperty "java.class.path") -- although that won't include any dynamically added stuff since the process started, I believe... By the time your program is running, there's no knowledge of the aliases used to invoke it: there is only the classpath.

valerauko07:02:57

With a deps.edn like below

{:paths ["src"]
 :main-opts ["-m" "foo"]
 :deps {org.clojure/tools.deps {:mvn/version "0.18.1398}}
 :aliases {:test {:extra-paths ["test"]}}}
And a main like this
(ns foo
  (:require [clojure.pprint :refer [pprint]]
            [clojure.tools.deps :as deps]))

(defn -main
  [& _]
  (pprint (deps/create-basis nil)))
When I run clj -M:test I expected that the printed map would contain "test" under :paths but it doesn't. (Neither under classpath-roots for that matter)

valerauko07:02:46

Is such a thing possible? Getting the merged :paths from the currently used deps.edn with the currently active aliases

seancorfield07:02:27

create-basis is nothing to do with how the current program is run

seancorfield07:02:28

See what I wrote above. The current program classpath is accessible, but you don't need tools.deps for that

valerauko07:02:46

This is for an analyzer/compiler tool that would look for target clj/c files under the final paths. That's why I'm trying to get a merged :paths from the deps.edn x current aliases

seancorfield07:02:35

By the time your program is running there is no such thing as "current aliases"

seancorfield07:02:13

All you have is the classpath

valerauko07:02:19

Yes you've made that pretty clear but I don't know how to express it better

valerauko07:02:06

When I call create-basis I get a map which includes the details from the current deps.edn. With the above example, even if I move the main-opts under :test, it works. But "test" is not included in paths, classpath-roots or under :classpath.

seancorfield07:02:13

Again, how you run the program is completely independent of the create basis call

seancorfield07:02:44

You can tell create basis to use specific aliases, as I showed above

seancorfield07:02:05

But your program will not know how it was invoked

valerauko07:02:20

So there is no way to get the clj uh, command? runtime? to give me that information? = CLI options used with the clj call

seancorfield07:02:58

This is why -X:deps tree accepts an :aliases argument - so you can tell it what aliases to use when it calls create-basis

seancorfield07:02:46

You'd have to do the same - let users tell your tool what aliases to use

👍 1
seancorfield07:02:19

Or, just walk the classpath looking for files

valerauko07:02:45

It'd be pretty convenient if the clj tool injected that information somewhere, env or something, but if that's not possible now I'll work around it

Alex Miller (Clojure team)13:02:27

Actually you can get the aliases that were used in the basis

Alex Miller (Clojure team)13:02:35

I had to add that to make sync-deps work properly in 1.12. Docs at https://clojure.github.io/tools.deps/#clojure.tools.deps/create-basis show this as :basis-config

Alex Miller (Clojure team)13:02:46

If you are in process you can use the new 1.12 clojure.java.basis api to get the basis and then look for :basis-config

seancorfield14:02:04

Hah, Alex beat me to it... When I woke up this morning, I realized that if you're willing to use 1.12 you can get the current process basis with https://clojure.github.io/clojure/branch-master/clojure.java.basis-api.html#clojure.java.basis/current-basis

lassemaatta07:02:24

Should it be mentioned somewhere (or is it?) that since clojure 1.11 you can directly destructure a map of a 1-length seq?

lassemaatta07:02:26

It was quite the surprise when a colleague accidentally wrote e.g. (when-let [{foo :bar} (some-function-which-returns-a-seq-of-maps)] ..) which worked fine during initial testing when the function returned a lazy seq with one element.

daveliepmann08:02:37

Is it on https://clojure.org/guides/destructuring ? I thought that was the one source of truth on the topic.

lassemaatta08:02:02

Nope, as far as I can tell. It does mention the "However, in Clojure 1.11 the capability was added to allow passing of alternating key→values, or a map of those same mappings, or even a map with key→values before it to functions expecting keyword arguments. ...", which relates to this through https://clojure.atlassian.net/browse/CLJ-2603

lassemaatta08:02:16

But yeah, that destructuring guide was the first thing I checked when I encountered this behaviour so it might be nice to mention this there.

1
daveliepmann08:02:30

seems like the "Approach" language in CLJ-2603 could be adapted

lassemaatta08:02:31

I think I'll create an issue for this

👍 1
lassemaatta10:02:34

Sure, it's intentional. But it's also a bit surprising and not really mentioned anywhere.

3
lassemaatta10:02:30

Of course, I can't really come up with a legitimate use-case where I would intentionally want to utilize this and perhaps(?) any such use case might be categorised as "you're doing it wrong'. But it still might be nice to know about this behaviour.

p-himik10:02:14

There's a lot of undocumented behavior that makes thing work in a particular way. :) There's no point in documenting it all. Alex explicitly says in that thread that it's undefined: > associative destructuring on lists/vectors/lazy-seqs is not defined

daveliepmann10:02:03

it seems like there is tension between "this is intended" and "this is undefined"

p-himik10:02:54

Yes, and those are not exclusive.

p-himik13:02:10

It's not the same though. What's documented at that link is "associative destructuring’s interpretation of seqs", so (let [{:keys [a b]} '(:a 1 :b 2)] ...). But what's asked is (let [{:keys [a]} '({:a 1})] ...). Same as in the thread I linked above.

lassemaatta13:02:41

Also, I noticed that the destructuring guide does mention "associative destructuring also works with lists or sequences of key-value pairs for keyword argument parsing", so I guess it is sort of mentioned there 🙂

p-himik13:02:12

"Of key-value pairs" is not "of maps".

p-himik13:02:27

What's meant is ([:a 1] [:b 2]).

lassemaatta13:02:43

but in any case, am I correct in summarizing that code like (let [{:keys [a]} '({:a 1})] .. in user code (excluding clojure itself) goes in to the same GIGO category like calling (keyword ":" ":") ? And the result of that is undefined and not nil, even though that might be preferred?

Alex Miller (Clojure team)13:02:31

GIGO is not the right framing, neither side is garbage. :) undefined in / undefined out would be more accurate. This was certainly undefined pre 1.11 and I remember we talked about whether to make it well defined in 1.11 but @U050WRF8X would maybe be a better one to answer that, I don’t remember. Programmatic keywords are neither garbage nor undefined - they make a keyword and you can use that keyword just fine, and that is useful in many programs. What you can’t do is rely on programmatic created keywords to be print / read roundtripped

lassemaatta13:02:23

right, thanks (I need to remember to choose my words more carefully in #clojure)

Aya12:02:38

Hey guys is there a way to read a large json file of say about 1gb in clojure?

Pavel Mugyrin12:02:24

Have you tried https://github.com/dakrone/cheshire? There is an api to read from a stream AFAIK

Aya12:02:21

currently slurp doesnt work with this

jpmonettas14:02:38

what problems are you seeing? slurp should be able to load a text file into your heap if it fits there. Is your heap max value > 1Gb? What happens when you do (def json-str (slurp "./bigfile.json")) ?

Pavel Mugyrin12:02:31

Hi! I've stumbled upon an annoying issue with the clojure.data.csv api. Basically, read-csv function does not allow to pass nil values as:

(csv/read-csv input :separator \, :quote nil)
While the write-csv on other hand allows such a usage. Why is it important? Imagine a system where you sometimes need store a custom separator and (or) quote symbols in a database and use corresponding one when creating\reading a csv file. In most cases you just want to keep these options to whatever the default behavior of the library is. So in this scenario you might expect to pull maps with nil values from the db like this.
{::my/separator \,
 ::my/quote nil  
That way you would have to dissoc \ filter out nils before passing the options to read-csv. Another solution would be duplicating default options in the db record which kinda goes against the idea of "keeping the default library behavior". This happens because read-csv destructures the arguments differently than write-csv does:
;; read-csv
let [{:keys [separator quote] :or {separator \, quote \"}} options]
    ...))

;; wirte-csv
(let [opts (apply hash-map options)
        separator (or (:separator opts) \,)
        quote (or (:quote opts) \")
        quote? (or (:quote? opts) ...)
        newline (or (:newline opts) :lf)]
The fix is trivial: https://github.com/coder11/data.csv/pull/1/files and is fully backwards compatible Do you think it's worth submitting a patch for that?

Bob B13:02:25

interestingly, there's an open ask out there to allow nil with a different behavior: <https://ask.clojure.org/index.php/6999/allow-read-csv-to-read-files-without-quoting?show=6999#q6999> This brings up the question of what the behavior should be... one could argue that if you pass nil, you might not be asking for the default.