"Resource path" is an invention of bulid tools, championed perhaps by Maven... Clojure-the-language does not have any concept of it.

Exactly, as Phill says, there is only "classpath".

In my case I have a leiningen project and I dont want Clojure to compile files in my resources folder. I just found an elegant solution to my problem by specifying up-front what I actually want AOT compiled in the uberjar profile

So on one hand, build tools invented "resource path" with the intention that you would not put code there. On the other hand, Clojure's classloader redeems a "require" by looking at the classpath, -- and build tools typically fuse their "source path" and "resource path" to make a classpath -- so it is possible that a "require" might engage code that you had put in the resource path.

right, in my case I have pure clojure data that has to be in leiningen/wagons.clj ... I originally put it in the resources folder expecting it to not get compiled or touched by leiningen, but it looks like that is not the case

since the file is pure data there is no ns declaration so i cant just mark ^:skip-aot on it (it only contains a single map inside)

Leiningen, Boot, and the CLI all take different approaches to AOT compiling stuff... 🤷:skin-tone-2:

I am of mixed feelings as to whether AOT is an abomination.

I wrote a blog post about how they all view "paths" some years back. Oh, yeah, I'm with you on AOT!

In my opinion, the ONLY reason to AOT is to improve startup performance for an uberjar. That's it.

For gen-class stuff for interop, I think more focused, manual compile is safer and allows for more custom behavior at build time.

Using :impl-ns to lay a boundary that viral AOT won't cross

I have an actual use case for AOT compilation here, I think. I am implementing a Maven wagon and use gen-class so I can extend HttpWagon and override some methods, then use this wagon from leiningen

it probably would've been easier to write things in Java, but the gen-class documentation is pretty straightforward and I think I figured my way around it

One trick to prevent the viral nature of AOT is to not put any non-core requires in ns and to load stuff dynamically (via requiring-resolve).

I cling to Leiningen (probably for the next 5 years) because I have problems enough without a build tool that is still improving in any way. But implementing a new wagon for Leiningen in 2023 is an interesting investment! Could you accomplish the same with whatever the new build tool is?

Heh, I switched from leiningen to boot back in 2015 then to the CLI in 2018...

Does the CLI have a plugin way to connect new types of things extending the set (mvn, git)?

tools.deps supports multiple procurers but it isn't well packaged yet... I think ClojureCLR might be the first project to tackle adding a new procurer tho'...

Lingy might also deal with this soon I suspect...

In my case I need a way to provide OAuth Bearer tokens to fetch artifacts from a private repo. I couldnt find a lein plugin for it so I made one myself

I don't know if what I am doing is doable with tools.deps, but im assuming not because tools.deps doubles down on data

I assume one would have to program the oauth token logic themselves in tools.build and invoke tools.deps as a library there instead of using the built in deps tool

if im wrong please correct because, although i dont use tools.deps, nothing i do in my wagon is inherently tied to leiningen, so i dont mind porting this to other tools

Well, with tools.deps and tools.build it is "just" Clojure code. With Leiningen, you have to conform to the whole plug-in architecture and work within what lein provides. That's why we switched away from it: it was far too proscriptive. Boot was much easier to customize, and the CLI is even easier.

ive never used boot but i think i agree with your statement -- i see tools.deps as "all data, no code" and tools.builds as "code for your project infrastructure"

Is there a more specific channel for tools deps? The OAuth angle is buried kind of deep here and it is very interesting

since what i needed to do in my wagon is all code / project infrastructure, i assume id have to use tools.build and invoke tools.deps after i figure a way to get oauth tokens sent in headers

#C6QH853H8 and #C02B5GHQWP4 both exist

If your wagon.clj is just data, is it possible to use an EDN file instead of CLJ? This way, it will not be compiled and you may (or may not) read its content from you classpath if you need it. You than build any tooling you need with those data.

unfortunately not, relevant snippet of code from leiningen

(doseq [wagon-file (-> (.getContextClassLoader (Thread/currentThread))
                       (.getResources "leiningen/wagons.clj")
                       (enumeration-seq))
       :when (not (@registered-wagon-files wagon-file))
       [hint factory] (read-string (slurp wagon-file))]
  (aether/register-wagon-factory! hint (eval factory))
  (swap! registered-wagon-files conj wagon-file))

there may be a way to take in wagons.edn as well, but I expect this would be a breaking change and reserved for leiningen 3.0

Just import all the namespaces that extend multimethods in the main core namespace.

Should I rely on require order? Some IDEs sort them alphabetically

The order of defmethods doesn't matter, right? And each file that uses defmethod already require's the defmulti.

The order of namespaces is not important for multimethods. You only need to import all the namespaces once.

Hmmm @U0HG4EHMH my defmethods are scattered across files separate from the defmulti so we can't assume that if defmulti was evaluated that so was the defmethods. Kinda like defprotocol and extend.

@U1WAUKQ3E if you consider only the code that defines the methods order is not important, but the code that uses the methods is also imported in the main ns, so the order of code that defines and code that uses is important.

@U8RHR1V60 Do I understand correctly that you have some namespace A that invokes a custom multimethod (defined in namespace B) in a top level form (not inside a defn), so you have to ensure that namespace A is loaded after the namespace B? In that case, I suggest either delaying the invocation of the multimethod until everything is loaded, or making A require B explicitly.

@U06PNK4HG It seems you understood perfectly 😁 This is exactly what I'm looking for: How would you delay (or otherwise ensure the order of) the invocation of the methods until everything is loaded?

@U8RHR1V60 just avoid calling a multimethod on top of a module, e.g. not this:

(ns ...)

(some-ns/my-method ...)

(ns ...)

(defn do-something []
  (some-ns/my-method ...))

Or, and hear me out: Don't use them. (Unless you're building a library)

There's not usually a good reason to spread your dispatch table across files when you own all of the code. Just hardcode a hashmap of key->fn.

Lib...

> if I'm using a tool exactly for what it's for (polymorphism and extensibility) The advice here is not "don't use multimethods for polymorphism and extensibility" but rather "don't go for polymorphism and extensibility when you don't need them". If you need them, then that advice is not applicable of course.

Yes, I assumed that was the (good) intention behind that answer. In this case I prefer the assumption to be that I need them, this is exactly what my library explore, it's not a "hack-it-till-it-works" kind of problem.

Also, since you mentioned that it's a library - ensuring the right order probably isn't a task for your shoulders. It's something to document well but not something to solve on a library level when that library is intended to be working as a part of a bigger thing.

Protocols and multimethods actually behave the same way about this, just the looser structure of multimethods makes it more easy to run afoul of the rule which is: using an interface causes a dependency on the definition of the interface and a dependency on one or more implementations of the interface. The way protocols are implemented, the creator of the dispatch object is responsible for making sure an implementation of some interface exists For multimethods you could say: whoever creates a map with the key :foo with the value :bar is responsible for ensuring that when some multimethod is called, that multimethods has a method installed for such a map. However for some reason everyone seems to want to go the other way and demand that whoever defines the interface (the multimethod) is responsible for loading all implementations that might be used

Just in case it might be relevant - that last bit is especially problematic in CLJS since requiring and never using namespaces with defmethod in them will prevent DCE of those namespaces.

Actually, It's not surprising to me and that exactly what I assumed as well. My problem came to surface as a user of my own lib, and the solution I'm looking for is in the hat of a user. I.e as a user of such lib what techniques are recommended to ensure correct evaluation order. p.s @U2FRKM4TW this will probably go to the docs 🙂

@U2FRKM4TW thank you I believe it's quite relevant, what is DCE?

If you depend on some code in another namespace being present, require it

Dead Code Elimination. As an example, if you require clojure.pprint in your CLJS code and never use it, the build size will still be increased significantly because that namespace uses defmethod. defmethods themselves aren't a huge problem. But the defined implementations depend on pretty much the rest of the namespace.

oh, so any namespace that uses demethod is deoptimised so I should be careful with that.

For cljs with advanced compilation

Nah, not deoptimized. It's just that

(defn f [...])
(defn g [...])
(defmethod x :some-val [...] (f ...))

@U0NCTKEV8 I'm trying to do it centrally: i.e in my entry ns require the demthod nss and then require another ns that will depend on those defmethods. is this approach reasonable?

No

I mean you can make it work, and people do it, but just have namespaces required what they depend on

@U2FRKM4TW thanks, that's reassuring 🙂

@U0NCTKEV8 alright so it would be preferable to have every ns that depends on those demethods require them itself? I guess I can make a ns that require all the demethods and then have all the other nss require that ns right?

I think it is preferable, clearer, and less likely to break in weird ways when various bits of tooling tries to understand your namespace dependency graph

The big thing to watch out for is comingling definitions (protocols, multimethods)of interfaces with uses of interfaces, because you'll end up with a cyclic dependency between your definitions/uses and your implementations

@U2FRKM4TW @U0NCTKEV8 thanks for all your tips 👍 I already separated all my defmultis into their own protocol ns as they are my interface and only that.

@U8RHR1V60 re: your "don't do it" post I mean you gotta realize that I (and others) have done exactly the thing you're asking about in exactly the wrong situation, and we wanna help others avoid a pitfall. What's being said is, "Let me guide you around this hole." Nobody is saying, "Quit. Give up. Don't try new things." I'm not sure what else you would want me to say to suggest a tool that addresses the same issue. I did tell you: 1. When this advice doesn't apply (when you're making a lib) 2. What to do instead (use a hashmap of val->fn) You are spot on wrt what multimethods offer. It sounds like you've already considered the issue I'm bringing up. Good for you! You're walking around the big hole that I've watched people fall into time and again! I apologize that I came off short/snarky. That wasn't my intention, but I understand why it did. Chat is hard 🙂

Why isn't the namespace where you've defined the Multi-method also defining the default methods for it?

But I think overall maybe it's a wrong user expectation? The way I see it: 1. Each namespace that defines a method for a multi must require the namespace of the multi 2. Users must require the namespaces of the methods they want to use.

I wanted to separate protocol (in the abstract sense, not to be confused with clojure's protocols) from implementation

@U0K064KQV ideally you are correct we wouldn't want users to have to require scattered implementations and only require the protocol definition, however since defmulti + defmethods were implemented "imperatively" you'd have to make sure that all relevant defmethods are evalutated somehow. In clojure it must be through "manual" requires. The other option would be that somehow the defmulti form will "lookup" all relevant defmethods in some kind of resolution space (think class-path) and evaluate them as part of its evaluation. But it's quite complex and I can understand why the implementers didn't go with that, I'm sure there are also caveats I don't even see yet with that approach.

No I'm saying the other way around. That a user only needs to require the "protocol" definition is a wrong user expectation. As a user, you have to think of it that you choose what implementation you want to use of the protocol. And that's what you require or define yourself.

So in fact it should be you only need to require the namespace where a method is defined, not where the "protocol" is defined.

Ohh ok I think I understand you better now. Your thinking "dev-time" polymorphism. Think of run-time polymorphism, for example when you use get to access map or vector, do you choose the right get? No, get uses ILookup to choose at runtime the relevant lookup method based on the type of the data you use. In my case I need the same kind of polymorphism.

Ya, I understand. But I think you're expecting something more static. At runtime, you have to require or define a method for the kind of input you intend to provide. So for example, if you had a Cat namespace, you'd expect the Cat namespace to define a method for Speak. And you'd expect that when you require Cat, you can now call Speak on it.

(ns animal)
(defmulti speak ...)

(ns cat
  (:require [animal :as a]))
(defmethod a/speak ...)

(ns user
 (:require [animal :as a]
           [cat :as c]))
(a/speak (c/make-cat))

It's the wrong expectation to think that requiring animal will implement all possible implementation of animals.

And that way, the user can also extend further the multi-method by creating its own animals and defining its own implementation for it.

I'm not sure how to put it clearly. But basically, as a user, you should think of it as, you must decide the set of all methods you intend to be polymorphic about the multi-method and require them all.

Something like... Okay, I want things to speak (require the ns that defines speak) and here are all the type of things I want to be able to have speak (require all the ns that provide a speak implementation for those things or define my own)

That works nicely when you have the definition of cat and speak is co-located but when protocol, type and implementation is separated, you now have to require all three but the implementation is the only require that you don't actually require from

Well, then you have to require all three. The ns that defines the type you want to use, the multi-method you want to use, and the implementation of that multi-method for that type.

And I say type, but I mean "value" in the case of multi-method, since it dispathes on any value.

Yes, the extensibility has its trade-off

My point is, as a user of a multi-method, you should expect to have to require all these things. If you don't, you have the wrong expectation of multi-methods.

Then the order shouldn't matter, because everything requires what it needs. The ns of the defmethod requires the ns of the defmulti and of the type it will be using (if any). The user ns requires the ns of the type, the defmethod and the defmulti (in any order). Etc.

Well, if you're speaking ideally it might be arguable, I'm guessing that not the only solution and that there would be tradeoffs of control over experience. If you're talking about the actual implementation in clojure, yeah, that was my starting point...

I'm talking about the multi-method feature of Clojure. If you want to just have dispatch on a known set, you can just use a case/switch. The point of the defmulti is that it lets you externally extend the dispatch.

I think maybe one of those two things might be what you want? Option 1

(ns animal)
(defmulti speak ...)
 
(ns default-animals
  (:require [animal :as a]))
 
(defmethod a/speak :cat ...)
(defmethod a/speak :dog ...)
 
(ns user
  (:require [animal :as a]
            [default-animals]))

(ns animal
  (:require [cat :as cat]
            [dog :as dog]))
(defmulti speak ...)
(defmethod a/speak :cat ...
  (cat/speak-impl ...)
(defmethod a/speak :dog ...
  (dog/speak-impl)

(ns user
  (:require [animal :as a]))

Currently I'm with Option 1

clojure.spec is also in alpha, yet used in production It’s hard to tell what exactly is unfinished or unstable about spec or spec2

The current spec 2 work includes the select implementation

https://github.com/clojure/spec-alpha2/wiki/Schema-and-select

I only bring it up because it seemed, at a glance, like it had bugs and people weren't using it for hobby projects. It sounds like there are proposals to do this in Malli but nothing concrete.

It has not been released yet and there are definitely some known issues