Fork me on GitHub
#tools-deps
<
2018-05-04
>
dominicm05:05:12

Integrant with aero is basically how we solve some of this. We built something that looks like Integrant using component.

rickmoynihan08:05:09

Yeah, we use integrant too; in one project via duct (so weavejester can write all the middleware config layer for us) and in another older project we retro-fitted integrant with aero… which in many ways I prefer to duct; but there are pros and cons. What I was really getting at though with the integrant thought was that you could possibly imagine combining integrant and deps.edn into a hybrid that would give you dependency resolution on components and configure/wire them together at the same time. I find it interesting how tools like integrant/component essentially make your require tree composable and configurable, and wonder if there’s a way to use integrant for packaging too. i.e. to turn integrant components (modules) into individually resolvable artifacts that something like tools.deps could then fetch and resolve for you. It’s currently a very half baked notion, and most likely a terrible idea though 🙂

dominicm05:05:18

Our problem is pretty much that a namespace can blow up if deployed where it can't read certain environment variables and look at certain files. These files change per library, with the same path. I guess there's some convention for the path. This is particularly noticeable in localization.

sveri07:05:52

@rickmoynihan In our team using gradle we define every dependency explicitly, no matter if its an in-team-project dependency or an external one. So naturally I would try the same with leiningen / tools-deps / boot.

rickmoynihan08:05:41

@sveri: Not sure exactly what you mean by defining every dependency explicitly? Do you just mean that you produce mvn artifacts for each subproject?

sveri09:05:14

@rickmoynihan gradle has a concept of project dependencies. In the root build file you can define your projects which then can depend on another.

alexmiller14:05:59

this is kind of a wip / sneak-peek but I’ve implemented a dynamic add-lib that you can use at the repl (that also will download and add transitive deps)

alexmiller14:05:12

$ clj -Sdeps "{:deps {org.clojure/tools.deps.alpha {:git/url \"" :sha \"d492e97259c013ba401c5238842cd3445839d020\"}}}"
Cloning: 
Checking out:  at d492e97259c013ba401c5238842cd3445839d020
Clojure 1.9.0
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'org.clojure/core.memoize {:mvn/version "0.7.1"})
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.pom from 
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.pom from 
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.pom from 
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.jar from 
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.jar from 
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar from 
true
user=> (require 'clojure.core.memoize)
nil
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
true
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
false

👍 4
alexmiller14:05:32

^^ you can use that command at the top to try it yourself off the branch

alexmiller14:05:18

everything about this is subject to change as Rich and I are still discussing many aspects of naming and use

alexmiller14:05:17

there is also a new libmap namespace that tracks the libs currently in use (these can be loaded via a property that points to a file which is set by clj, or by a property set to edn data - hook for tools, or by just inspecting the classpath and trying to understand it)

alexmiller14:05:35

add-lib will add new libs in the context of the existing libs, so it won’t (for example) download and add libs of transitive deps that have already been loaded. when you add libs, those are added to the runtime lib map and considered in subsequent adds

dominicm14:05:09

This is very cool, and useful!

alexmiller14:05:31

the primary use here is to add a capability to the repl environment to dynamically add libs. this does not give you the isolation of something like pods, but it’s a small useful step into a more live repl experience.

dominicm15:05:39

@alexmiller this uses java properties you say?

alexmiller15:05:05

yes - clojure.libfile (will now be set by clj) and clojure.libmap

alexmiller15:05:40

although the fallback to classpath introspection actually does a reasonable job even if that’s not set in my experience so far

dominicm15:05:24

Pods may conflict with that, as java properties aren't localized to the pod. I don't have an alternative suggestion, but just a data point that occurs to me.

alexmiller15:05:34

if you’re using pods, you don’t need this

alexmiller15:05:02

classpath introspection:

dominicm15:05:02

I'm thinking of inspection of clojure.libfile for other purposes, e.g. finding gitlibs to ignore during a (reset)

dominicm15:05:49

That's a bad example in the context of pods, but I think it conveys my intent.

alexmiller15:05:42

the clojure.libfile points to the libfile corresponding to the classpath used when starting the jvm. that has a fixed answer.

alexmiller15:05:08

if you’re using pods, you’re in a bubble under that.

dominicm15:05:28

libmap then, rather than libfile?

dominicm15:05:37

https://github.com/RutledgePaulV/clj-embed wouldn't be able to convey clojure.libmap to it's runtimes?

alexmiller15:05:04

libmap is still jvm-wide so like libfile

alexmiller15:05:28

but I think it’s interesting to consider how to allow classloaders to build their own env

dominicm15:05:45

Maybe I don't understand what libmap is, what is it?

alexmiller15:05:57

it’s just libfile in edn form

alexmiller15:05:27

so a tool could start the jvm, build the classpath via means other than clj and pass the libmap via data rather than file

dominicm15:05:43

Given the definition of "jvm-wide", then java properties are always right. My thoughts are that it's really closer to being about the classloader though, what libmap generated the current classloader.

alexmiller15:05:44

the runtime libmap is basically stored in a var right now. maybe interesting to consider whether that should be dynamic to allow per-thread context or something per-classloader to handle pod/clj-embed type cases

dominicm15:05:05

Sorry, this probably sounds very pedantic. This is really exciting, I'm happy about the feature even as it currently is.

dominicm15:05:43

It's definitely notable that due to how Clojure works, embedding always carries a new copy of Clojure. So "clojure local" is perhaps sufficient?

alexmiller15:05:31

no, I appreciate the good pokes on this. we’ve thought a fair bit about the add-lib stuff but less about the runtime management of the libmap and so I’m still trying to work out the ideal shape of that. and we have other deeper plans for the libmap stuff so I want to get it right.

alexmiller15:05:51

it’s reasonably likely that part will actually float into Clojure itself

dominicm15:05:39

Which part, sorry?

alexmiller15:05:44

the runtime libmap management (not add-lib)

dominicm15:05:23

So clojure.core will be aware of the deps.edn format? :thinking_face:

alexmiller15:05:42

it’s just a map

alexmiller15:05:51

so, yes it will be aware of maps :)

alexmiller15:05:31

in terms of doing semantic things - that doesn’t have to be in core. core’s job is to ensure that it’s loaded and available in a way that can be replied upon

alexmiller15:05:52

I wouldn’t rule out core being aware of it but I don’t see a need for that yet

alexmiller15:05:22

well, maybe. we’ll see where things go. :)

alexmiller15:05:06

maybe I should push this stuff into a blog post

dominicm15:05:23

I'm a little confused. Will it be a state storage, which libmap will sit in as a key?

dominicm15:05:01

Blog posts are always exciting, even if marked as "these ideas are alpha"

alexmiller15:05:42

I don’t think I understand the question

alexmiller15:05:54

right now it’s a var holding an atom holding the libmap

alexmiller15:05:25

you can look at the code on the branch, it’s not that long

dominicm15:05:24

What will Clojure know about managing the runtime libmap though? I'm really confused by what it means for that to be in Clojure, sorry.

alexmiller16:05:53

it means that anything using Clojure could look at the libmap (not just things using tools.deps.alpha)

dominicm16:05:25

I see! So there would be a var which was mutated or something?

seancorfield16:05:15

The channel name is #tools-deps not #tools.deps BTW @alexmiller (last line of the blog post)

alexmiller16:05:35

gosh dang it

👍 4
dominicm16:05:25

I see! So there would be a var which was mutated or something?

alexmiller16:05:35

it’s a var holding an atom, so would just swap! the atom

alexmiller16:05:20

but I think there are interesting potential uses that are merely read-only

dominicm17:05:33

I understand now. I was interpreting your words too broadly I think. This would solve the problem of pods. If I'm thinking right, this would complicate Clojure becoming aware of class loaders in the future. Admittedly, this is something I am a minority who longs for this.

alexmiller17:05:10

Clojure only works because of custom classloaders so I don’t think I agree at all with that

alexmiller17:05:29

I also don’t think this would solve the isolation aspect of pods

dominicm19:05:45

I mean that you can't require multiple versions of a namespace from different class loaders. No? The pod could be controlled to update the atom on start couldn't it?

alexmiller20:05:38

I think you’re misunderstanding what is happening here

alexmiller20:05:52

but I’m not sure that I understand what you think it’s doing enough to clarify

alexmiller20:05:14

the libmap is not affecting anything about how the Clojure runtime currently works

alexmiller20:05:57

it is assisting in opening a path to additively add jars to the DynamicClassLoader in use (this actually has existed in Clojure for a very long time with clojure.core/add-classpath which is deprecated and I think has not worked for a very long time)

alexmiller20:05:14

pods in Boot provide isolation (different Clojure runtimes) and already do allow you to load different versions of a Clojure namespace at the same time in different pods as far as I understand them

dominicm20:05:04

My use of from there was poor. You can't have multiple versions of a library loaded into a single Clojure. You have to load a whole new Clojure.

alexmiller20:05:07

yes and nothing is changing about that

alexmiller20:05:43

I’d say that problem is way down the interest list

dominicm20:05:14

I know. I was thinking about it that were ever to be "fixed". Yeah, I'm one of the few it bothers because I have interest in a constrained deployment environment, where dependencies aren't really possible.

alexmiller20:05:12

I totally get why and when isolation is useful, but Clojure was never designed with that intent and there seem to be more important problems to solve