Fork me on GitHub
#tools-deps
<
2018-05-04
>
dominicm05:05:12

Integrant with aero is basically how we solve some of this. We built something that looks like Integrant using component.

rickmoynihan08:05:09

Yeah, we use integrant too; in one project via duct (so weavejester can write all the middleware config layer for us) and in another older project we retro-fitted integrant with aero… which in many ways I prefer to duct; but there are pros and cons. What I was really getting at though with the integrant thought was that you could possibly imagine combining integrant and deps.edn into a hybrid that would give you dependency resolution on components and configure/wire them together at the same time. I find it interesting how tools like integrant/component essentially make your require tree composable and configurable, and wonder if there’s a way to use integrant for packaging too. i.e. to turn integrant components (modules) into individually resolvable artifacts that something like tools.deps could then fetch and resolve for you. It’s currently a very half baked notion, and most likely a terrible idea though 🙂

dominicm05:05:18

Our problem is pretty much that a namespace can blow up if deployed where it can't read certain environment variables and look at certain files. These files change per library, with the same path. I guess there's some convention for the path. This is particularly noticeable in localization.

sveri07:05:52

@rickmoynihan In our team using gradle we define every dependency explicitly, no matter if its an in-team-project dependency or an external one. So naturally I would try the same with leiningen / tools-deps / boot.

rickmoynihan08:05:41

@sveri: Not sure exactly what you mean by defining every dependency explicitly? Do you just mean that you produce mvn artifacts for each subproject?

sveri09:05:14

@rickmoynihan gradle has a concept of project dependencies. In the root build file you can define your projects which then can depend on another.

Alex Miller (Clojure team)14:05:59

this is kind of a wip / sneak-peek but I’ve implemented a dynamic add-lib that you can use at the repl (that also will download and add transitive deps)

Alex Miller (Clojure team)14:05:12

$ clj -Sdeps "{:deps {org.clojure/tools.deps.alpha {:git/url \"" :sha \"d492e97259c013ba401c5238842cd3445839d020\"}}}"
Cloning: 
Checking out:  at d492e97259c013ba401c5238842cd3445839d020
Clojure 1.9.0
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'org.clojure/core.memoize {:mvn/version "0.7.1"})
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.pom from 
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.pom from 
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.pom from 
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.jar from 
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.jar from 
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar from 
true
user=> (require 'clojure.core.memoize)
nil
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
true
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
false

👍 4
Alex Miller (Clojure team)14:05:32

^^ you can use that command at the top to try it yourself off the branch

Alex Miller (Clojure team)14:05:18

everything about this is subject to change as Rich and I are still discussing many aspects of naming and use

Alex Miller (Clojure team)14:05:17

there is also a new libmap namespace that tracks the libs currently in use (these can be loaded via a property that points to a file which is set by clj, or by a property set to edn data - hook for tools, or by just inspecting the classpath and trying to understand it)

Alex Miller (Clojure team)14:05:35

add-lib will add new libs in the context of the existing libs, so it won’t (for example) download and add libs of transitive deps that have already been loaded. when you add libs, those are added to the runtime lib map and considered in subsequent adds

dominicm14:05:09

This is very cool, and useful!

Alex Miller (Clojure team)14:05:31

the primary use here is to add a capability to the repl environment to dynamically add libs. this does not give you the isolation of something like pods, but it’s a small useful step into a more live repl experience.

dominicm15:05:39

@alexmiller this uses java properties you say?

Alex Miller (Clojure team)15:05:05

yes - clojure.libfile (will now be set by clj) and clojure.libmap

Alex Miller (Clojure team)15:05:40

although the fallback to classpath introspection actually does a reasonable job even if that’s not set in my experience so far

dominicm15:05:24

Pods may conflict with that, as java properties aren't localized to the pod. I don't have an alternative suggestion, but just a data point that occurs to me.

Alex Miller (Clojure team)15:05:34

if you’re using pods, you don’t need this

Alex Miller (Clojure team)15:05:02

classpath introspection:

dominicm15:05:02

I'm thinking of inspection of clojure.libfile for other purposes, e.g. finding gitlibs to ignore during a (reset)

dominicm15:05:49

That's a bad example in the context of pods, but I think it conveys my intent.

Alex Miller (Clojure team)15:05:42

the clojure.libfile points to the libfile corresponding to the classpath used when starting the jvm. that has a fixed answer.

Alex Miller (Clojure team)15:05:08

if you’re using pods, you’re in a bubble under that.

dominicm15:05:28

libmap then, rather than libfile?

dominicm15:05:37

https://github.com/RutledgePaulV/clj-embed wouldn't be able to convey clojure.libmap to it's runtimes?

Alex Miller (Clojure team)15:05:04

libmap is still jvm-wide so like libfile

Alex Miller (Clojure team)15:05:28

but I think it’s interesting to consider how to allow classloaders to build their own env

dominicm15:05:45

Maybe I don't understand what libmap is, what is it?

Alex Miller (Clojure team)15:05:57

it’s just libfile in edn form

Alex Miller (Clojure team)15:05:27

so a tool could start the jvm, build the classpath via means other than clj and pass the libmap via data rather than file

dominicm15:05:43

Given the definition of "jvm-wide", then java properties are always right. My thoughts are that it's really closer to being about the classloader though, what libmap generated the current classloader.

Alex Miller (Clojure team)15:05:44

the runtime libmap is basically stored in a var right now. maybe interesting to consider whether that should be dynamic to allow per-thread context or something per-classloader to handle pod/clj-embed type cases

dominicm15:05:05

Sorry, this probably sounds very pedantic. This is really exciting, I'm happy about the feature even as it currently is.

dominicm15:05:43

It's definitely notable that due to how Clojure works, embedding always carries a new copy of Clojure. So "clojure local" is perhaps sufficient?

Alex Miller (Clojure team)15:05:31

no, I appreciate the good pokes on this. we’ve thought a fair bit about the add-lib stuff but less about the runtime management of the libmap and so I’m still trying to work out the ideal shape of that. and we have other deeper plans for the libmap stuff so I want to get it right.

Alex Miller (Clojure team)15:05:51

it’s reasonably likely that part will actually float into Clojure itself

dominicm15:05:39

Which part, sorry?

Alex Miller (Clojure team)15:05:44

the runtime libmap management (not add-lib)

dominicm15:05:23

So clojure.core will be aware of the deps.edn format? :thinking_face:

Alex Miller (Clojure team)15:05:51

so, yes it will be aware of maps :)

Alex Miller (Clojure team)15:05:31

in terms of doing semantic things - that doesn’t have to be in core. core’s job is to ensure that it’s loaded and available in a way that can be replied upon

Alex Miller (Clojure team)15:05:52

I wouldn’t rule out core being aware of it but I don’t see a need for that yet

Alex Miller (Clojure team)15:05:22

well, maybe. we’ll see where things go. :)

Alex Miller (Clojure team)15:05:06

maybe I should push this stuff into a blog post

dominicm15:05:23

I'm a little confused. Will it be a state storage, which libmap will sit in as a key?

dominicm15:05:01

Blog posts are always exciting, even if marked as "these ideas are alpha"

Alex Miller (Clojure team)15:05:42

I don’t think I understand the question

Alex Miller (Clojure team)15:05:54

right now it’s a var holding an atom holding the libmap

Alex Miller (Clojure team)15:05:25

you can look at the code on the branch, it’s not that long

dominicm15:05:24

What will Clojure know about managing the runtime libmap though? I'm really confused by what it means for that to be in Clojure, sorry.

Alex Miller (Clojure team)16:05:53

it means that anything using Clojure could look at the libmap (not just things using tools.deps.alpha)

dominicm16:05:25

I see! So there would be a var which was mutated or something?

seancorfield16:05:15

The channel name is #tools-deps not #tools.deps BTW @alexmiller (last line of the blog post)

dominicm16:05:25

I see! So there would be a var which was mutated or something?

Alex Miller (Clojure team)16:05:35

it’s a var holding an atom, so would just swap! the atom

Alex Miller (Clojure team)16:05:20

but I think there are interesting potential uses that are merely read-only

dominicm17:05:33

I understand now. I was interpreting your words too broadly I think. This would solve the problem of pods. If I'm thinking right, this would complicate Clojure becoming aware of class loaders in the future. Admittedly, this is something I am a minority who longs for this.

Alex Miller (Clojure team)17:05:10

Clojure only works because of custom classloaders so I don’t think I agree at all with that

Alex Miller (Clojure team)17:05:29

I also don’t think this would solve the isolation aspect of pods

dominicm19:05:45

I mean that you can't require multiple versions of a namespace from different class loaders. No? The pod could be controlled to update the atom on start couldn't it?

Alex Miller (Clojure team)20:05:38

I think you’re misunderstanding what is happening here

Alex Miller (Clojure team)20:05:52

but I’m not sure that I understand what you think it’s doing enough to clarify

Alex Miller (Clojure team)20:05:14

the libmap is not affecting anything about how the Clojure runtime currently works

Alex Miller (Clojure team)20:05:57

it is assisting in opening a path to additively add jars to the DynamicClassLoader in use (this actually has existed in Clojure for a very long time with clojure.core/add-classpath which is deprecated and I think has not worked for a very long time)

Alex Miller (Clojure team)20:05:14

pods in Boot provide isolation (different Clojure runtimes) and already do allow you to load different versions of a Clojure namespace at the same time in different pods as far as I understand them

dominicm20:05:04

My use of from there was poor. You can't have multiple versions of a library loaded into a single Clojure. You have to load a whole new Clojure.

Alex Miller (Clojure team)20:05:07

yes and nothing is changing about that

Alex Miller (Clojure team)20:05:43

I’d say that problem is way down the interest list

dominicm20:05:14

I know. I was thinking about it that were ever to be "fixed". Yeah, I'm one of the few it bothers because I have interest in a constrained deployment environment, where dependencies aren't really possible.

Alex Miller (Clojure team)20:05:12

I totally get why and when isolation is useful, but Clojure was never designed with that intent and there seem to be more important problems to solve