Thanks! This is based on research from my PhD. The last 3rd of my dissertation (published 2018) speculates about how to add symbolic execution to Typed Clojure, and it's taken all this time to really figure out a way to implement it that makes sense. And also, the idea itself goes against the spirit of a type system in some ways, so it's taken a long time to let the idea settle in my stomach and decide that this will be, overall, an improvement. There's also this model I wrote with tons of tests that has other ideas I will try and port over, such as catching stackoverflows and then suggesting specific actions to the user via error messages https://github.com/frenchy64/lti-model

the section of my thesis is here if you're interested https://thesis.ambrosebs.com/#x1-110000IV

Sorry for the delay, my notification settings are not quite right here. Anyway, thanks for the insight! Was curious if there was a single "aha!" moment, some downtime, or a long process of learning and experimenting to get there

Given that it was part of your thesis, I've got a sense for which bucket that falls into 😅

I remember in 2017 I was just unconvinced that (map (fn [x] (inc x)) [1]) was not checkable using local type inference. That was the germ of this work. I drew a lot of pictures like the ones on my https://www.patreon.com/ambrosebs tiers that showed the data flow between types. The lti-model repo was a big step in confidence, but I really struggled to integrate it with polymorphic type inference. That's why symbolic execution has been in Typed Clojure for several years, but I disabled it by default and never really announced it. A recent insight that made it all click was that I needed to treat different occurrences of the same type variable differently---substitute occurrences that occur covariantly differently than those contravariantly.

Yes, except for primitives. Not sure how it would work with boxing, or whether there would be any in the first place.

(locking 'a) (or "a" , :a) might act funny due to interning, IDK Boxed integers from -128 to 127 are cached, the rest not, so outside of that range it would seem less reliable

The identity of boxed number types should not be used and relied upon. That is, they should never be treated "as objects" – used as locks, as keys in IdentityHashMap, etc. This might work for now but will stop working once Project Valhalla is released. The same goes for Optional and other thin wrapper types.

@U061KMSM7 Furthermore, locking (which is the same thing as synchronized in Java) should be avoided 99% of the time, as it's an old and inefficient approach to concurrency control. Instead, you should use explicit lock types like https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html and the likes.

@U06PNK4HG So what would be the ‘clojure way’ to simply make sure a data sync function is not re-entrant?

Or is it best to just drop down to Java here?

Making a ReentrantLock and locking/unlocking it via interop is the best way right now.

Got it, thanks!

Just in case, Reentrant here means that the same thread that owns the lock can enter the same critical section twice, not that two threads can do it at the same time:)

I agree with Alex but a basic understanding of locking can show that it's perfectly fine to use for a wide array of simple use cases. e.g. there's no foreseen concurrency at all, I just want to ensure that it's impossible that a resource is accessed concurrently. java synchronized is reentrant. I'd expect Clojure locking to also be?

> java synchronized is reentrant. I'd expect Clojure locking to also be? Yes, they are pretty much identical. I'd agree that it's maybe not worth sweating over in simple cases. But at the same time, the existence of synchronized has been a big mistake, and being rigorous about avoiding it even when it's not critical is a good practice.

tryLock (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html) is cool btw, it's a fine tool from time to time

If y’all don’t mind, what’s a simple code example of ReentrantLock?

Here’s what I’m doing:

(def ^:private event-sync-lock
  "Run only one sync process at a time."
  (ReentrantLock.))

and then…

(.lock event-sync-lock)

inside a try, and then a finally clause:

(.unlock event-sync-lock)

(.lock mylock)
(try <critical-section>
     (finally (.unlock mylock))

Obviously, the function I want to protect is inside the try as well

It's better to .lock outside of try because if that fails with an exception you don't want to attempt unlocking

Hmm, @U06PNK4HG you beat me to it

<fixing code>

Also @U45T93RA6 symbols are not interned, so locking on 'a would not be advisable

Symbol.intern has plenty of matches on the clojure repo, anyway I was not seeking to dive into a rabbithole that is most likely not practical.

I guess I must be missing something then, I'll look into it more before I make that strong statement again 😅

Generally my understanding has been that symbols aren't properly interned because they can hold metadata, and that interning is why keywords can't hold metadata, so I may be missing the common case of no-metadata-symbols. Alternatively it could just be a legacy name from early in development that stuck with symbols even though they don't have interning semantics. Hence, I need to research more.

Symbol has a method called https://github.com/clojure/clojure/blob/2a058814e5fa3e8fb630ae507c3fa7dc865138c6/src/jvm/clojure/lang/Symbol.java#L57-L59 , which just returns a new Symbol. Comparatively, Keyword has a method called https://github.com/clojure/clojure/blob/2a058814e5fa3e8fb630ae507c3fa7dc865138c6/src/jvm/clojure/lang/Keyword.java#L34-L53 that performs actual interning (creating or adding the new keyword to the cache) and then returns the new or existing Keyword

(identical? 'a 'a) ;;=> false

something to be aware of

you might as well just use an Object

Yeah, this matched what I expected

One reason a symbol is not identical to another symbol of the same name, is that individual symbols can carry metadata, unlike keywords

The way you describe it, it seems to be screaming that it's an XY problem. Even just the "deeply nested macros" part doesn't sound good. What exactly is the problem you're trying to solve with those macros?

I'm building a DSL which needs to internally produce some side effects (think API calls, DB queries). These side effects are produced through a function foo which I wrap around certain functions baz written in the macro. I bind some dynamic vars which contain tokens or connection strings which are arguments of foo. All is simple until I want to "package" multiple baz functions into a single function (to simplify my DSL). So I write a macro baz-simple . But now I can't wrap foo around baz-simple, I need to fully expand baz-simple (recursively continue expanding) first.

^^sorry if this is confusing.

The reason for needing foo in the first place is because this is a "user-facing" DSL which you can write in your web browser.

Yeah, I don't get what you are describing either. It would be helpful if you could come up with some small runnable example of what you are trying to achieve.

Here is a runnable example of what my macro is doing https://gist.github.com/S4G4R/d4bfb90da400cd73a35c6164dcf33d59 What I would like to know is if there is a good way of getting rid of nested macros such as purchase-items (and use it as a simple function/multimethod/etc instead)

Is the only reason to use macros here so that you can set/replace the purchase-item operation from outside?

Yes!

Well, you can use a dynamic variable for that, for example, which you already kinda use in your code.

(defonce wallet (atom {:money 5000}))

(def ^:dynamic *purchasing-fn*)

(defn purchase
  [wallet {cost :cost}]
  (swap! wallet update :money - cost))

(def items
  {:bicycle {:id 1 :cost 50}
   :car     {:id 2 :cost 100}
   :truck   {:id 3 :cost 200}})

(defn purchase-item [item-type]
  (*purchasing-fn* (get items item-type)))

(defn purchase-items [n item-type]
  (dotimes [_ n]
    (purchase-item item-type)))

(comment
  (binding [*purchasing-fn* (partial purchase wallet)]
    (purchase-item :bicycle)
    (purchase-items 5 :bicycle))

  @wallet)

If that's an option, I would avoid the dynamic var altogether and pass the fn everywhere you need to use it.

I would like to keep purchase-item a pure function (which returns the payload for the API call to purchase an item), and separate the actual execution from the data for easier testing. I could have two versions of the same function, one which uses the bound dynamic variable and calls the function, but it would lead to alot of duplication. I think that's a good idea though as a last resort. It would make other parts of the code simpler for me

I do need the dynamic var since this is a user facing DSL. Its not used internally. This could go be typed in your web browser somewhere

(shop
 (purchase-item :bicycle)
 (purchase-items 5 :bicycle))

BTW purchase-items should be a function and not a macro. There's 0 reasons to have it be a macro. That (shop ...) block can still be implemented without any dynamic vars or macros or any kind of magic, really. All purchase-* functions can return the command, and shop would then execute the list of commands. A common pattern.

That was my initial implementation until I wanted to add support for intermediate results/bindings. We may want to fetch an item, get its cost, and add 10 to it. And then update it. I'm sorry, my example doesn't convey this. It's a simpler version of my actual macro

This is quite possible

(shop
 (let [{id :id cost :cost} (get-item :bicycle)]
   (update-item id {:cost (+ cost 10)})))

> We may want to fetch an item, get its cost, and add 10 to it. And then update it. That all can be expressed as a command. The code above would get an item and create a command to update it.

I get your point, but we do need to actually execute (get-item :bicycle), which I'm not sure how to do without the shop macro and without the dynamic var.

Unless you mean creating a single command that would do both things.

I see. I would still make the users pass around an explicit arg, just like you do with e.g. next-jdbc when passing around the connection. Even if it makes users type extra 2 characters per operation.

I'd still need to work on binding the symbol to an actual var somehow when the code gets run. But I will explore that option

I think I could live with duplicating the functions, having a pure one that returns the command and another which actually executes it using the dynamic var.

This has been pretty helpful

If you need to capture the operations as data to be interpreted later, consider making a hidden state atom that contains a list of operations to execute. Each call to purchase-atom would append to that atom. Return that value of the atom as the result of shop wrapper. That way you would avoid atoms but preserve the operations for later.

I need the intermediate results for subsequent operations. So I can't really store the operations as a sequence and defer their execution. I had done something similar to your suggestion as an earlier implementation but it failed as complexity grew

@U01B1CZQ9PF have you looked at the clojure.walk namespace? prewalk and macroexpand-all sound like they would work with your usecase without requiring you to fully reconsider your approach.

Though you may get better code if you do reconsider your approach like the other people here have suggested.

@U5NCUG8NR Yes, I use a combination of clojure.walk/postwalk and macroexpand-1 to wrap commands and expand inner macros. It works quite well. My question was more about trying to find an alternate solution that doesn't require the expansion at all.

I see

foo> ( "net/sf/uadetector/service/UADetectorServiceFactory.class")
#object[java.net.URL 0x4cde00b4 "jar:file:/Users/vemv/.m2/repository/net/sf/uadetector/uadetector-resources/2014.10/uadetector-resources-2014.10.jar!/net/sf/uadetector/service/UADetectorServiceFactory.class"]
foo> net.sf.uadetector.service.UADetectorServiceFactory
Syntax error (NoClassDefFoundError) compiling at (*cider-repl foo:localhost:59782(clj)*:0:0).
Could not initialize class net.sf.uadetector.service.UADetectorServiceFactory
foo> *e
#error {
 :cause "Could not initialize class net.sf.uadetector.service.UADetectorServiceFactory"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "Syntax error compiling at (*cider-repl foo:localhost:59782(clj)*:0:0)."
   :data #:clojure.error{:phase :compile-syntax-check, :line 0, :column 0, :source "*cider-repl foo:localhost:59782(clj)*"}
   :at [clojure.lang.Compiler analyze "Compiler.java" 6825]}
  {:type java.lang.NoClassDefFoundError
   :message "Could not initialize class net.sf.uadetector.service.UADetectorServiceFactory"
   :at [java.lang.Class forName0 "Class.java" -2]}]
 :trace
 [[java.lang.Class forName0 "Class.java" -2]
  [java.lang.Class forName "Class.java" 466]
  [clojure.lang.RT classForName "RT.java" 2209]
  [clojure.lang.RT classForName "RT.java" 2218]
  [clojure.lang.Compiler resolveIn "Compiler.java" 7412]
  [clojure.lang.Compiler resolve "Compiler.java" 7375]
  [clojure.lang.Compiler analyzeSymbol "Compiler.java" 7336]
  [clojure.lang.Compiler analyze "Compiler.java" 6785]
  [clojure.lang.Compiler analyze "Compiler.java" 6762]
  [clojure.lang.Compiler eval "Compiler.java" 7198]
  [clojure.lang.Compiler eval "Compiler.java" 7149]
  [clojure.core$eval invokeStatic "core.clj" 3215]
  [clojure.core$eval invoke "core.clj" 3211]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1296$fn__1297 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1296 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
  [clojure.main$repl$fn__9215 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1329$fn__1333 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__1399$fn__1403 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__1399 invoke "session.clj" 217]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 831]]}

The versions of these 2 dependencies (as visible on my classpath) match with latest https://github.com/igrishaev/user-agent/blob/a75e124351e78c20c08f03e7e9aba0a4ab24899c/project.clj#L27-L28C34, https://mvnrepository.com/artifact/net.sf.uadetector/uadetector-resources https://mvnrepository.com/artifact/net.sf.uadetector/uadetector-core

% clj -Sdeps '{:deps {net.sf.uadetector/uadetector-resources {:mvn/version "2014.10"}}}' -M 
Clojure 1.11.1
user=> net.sf.uadetector.service.UADetectorServiceFactory
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See  for further details.
net.sf.uadetector.service.UADetectorServiceFactory
user=> 
%

Maybe its logging dependency complicates things

the first thing I would do would be restart you repl if you haven't done that yet, to make sure the jvm is started with that class visible to the root application classloader

the second thing I would is something like java -cp lein classpath` clojure.main` to get a non-nrepl/cider repl and try it there

Leaving aside those, I was wondering if one can extract actionable info from the exception/situation I posted (from time to time I do get completely unactionable errors from weird JVM corners, so I expect nothing)

resource is using a different classloader from the compiler for some reason is my "info" from that error

not 100%, but most likely

resource defaults to using the context classloader of the current thread, but has a a 2-arity where you can pass in a classloader

the figuring out which classloader the compiler is using tends to drive me nuts, but sometimes you can find it at clojure.lang.Compiler/LOADER, but that is usually a dynamic classloader child, so you have to walk up the parents to do any kind of comparison

you could try something like (resource ... (.getClassloader clojure.lang.RT)) to see if whatever classloader loaded RT, which is likely to be a parent of the classloader the compiler is using, can see the resource

I guess I am not sure, it might not be a classloader visibility issue, it might be a static init error

which Could not initialize class net.sf.uadetector.service.UADetectorServiceFactory seems like it might point to

but I guess I would expect a more detailed error in that case pointing to the static init that failed, but maybe I am misremembering

or maybe it is a state thing (you get the detailed error the first time, and afters just a generic "you tried this already" error)

Thanks for all the info. atm,

(= ( "net/sf/uadetector/service/UADetectorServiceFactory.class")
   ( "net/sf/uadetector/service/UADetectorServiceFactory.class"
                             (ClassLoader/getSystemClassLoader)))

If you need a class loader to load a resource, you might want the one from the current thread https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html#getContextClassLoader-- ?

The plot thickens...

user> (.getContextClassLoader (Thread/currentThread))
#object[clojure.lang.DynamicClassLoader 0x478893b1 "clojure.lang.DynamicClassLoader@478893b1"]

user> (.loadClass (.getContextClassLoader (Thread/currentThread))
                                          "net.sf.uadetector.service.UADetectorServiceFactory")
net.sf.uadetector.service.UADetectorServiceFactory

user> (.loadClass (ClassLoader/getSystemClassLoader) "net.sf.uadetector.service.UADetectorServiceFactory")
net.sf.uadetector.service.UADetectorServiceFactory

user> net.sf.uadetector.service.UADetectorServiceFactory
Syntax error (NoClassDefFoundError) compiling at (*cider-repl user:localhost:60058(clj)*:0:0).
Could not initialize class net.sf.uadetector.service.UADetectorServiceFactory

eval is using the clojure compiler classloader, which who knows, might be messed with by your tooling

that kind of thing is why I suggested getting the classpath from lein and using it to launch a vanilla clojure repl and see how that behaves

I know it can work, but that doesn't help me resolve when it doesn't work

I had tried

(with-bindings {clojure.lang.Compiler/LOADER
                (ClassLoader/getSystemClassLoader)}
  (eval 'net.sf.uadetector.service.UADetectorServiceFactory))

if it works in a vanilla repl that tells you it is some part of your cider/nrepl setup that is messing with it

which doesn't help me fixing it, which is what I'm interested in

then you can start looking at that, does it work with a vanilla nrepl setup (no plugins no cider,etc)

then you'll find the bit of tooling that is breaking it

then you can find the source for that

then you can see what nonsense it does to the classloader

(with-bindings {clojure.lang.Compiler/LOADER
                (ClassLoader/getSystemClassLoader)}
  (eval 'net.sf.uadetector.service.UADetectorServiceFactory))

I am not sure that does anything

I would have to look at the compiler, but eval might always push a fresh dcl on to loader, and I don't recall if it considers the current value of LOADER, of if it calls the RT make classloader stuff and all the different cases there

and I am not sure that loadClass calls static inits

yeah, loadClass doesn't run static inits

> or maybe it is a state thing (you get the detailed error the first time, and afters just a generic "you tried this already" error) This was the case. And the root cause was fairly tedious to debug given my huge classpath. Two of its items had a resource named org.apache.xerces.xni.parser.XMLParserConfiguration , with different contents. These apparently intend to choose a class at runtime. With the wrong class, UADetectorServiceFactory would fail at static init, but quite opaquely (it complained about empty data being loaded rather than having anything crash) Not sure if there was any lesson here 🙃 just a few aspects conspiring to make things difficult. Thanks for your time.

One of the solution I could find on the web suggests to create a jar file with an embedded manifest listing the classpath; did someone made a build tool for that?

I don't know whether the Scoop-supplied ClojureTools has been updated to match the very latest CLI version (which includes new features to avoid this limitation). Best to ask in #CFN4QDHPS which is where the Powershell stuff and alternatives is usually discussed.

Thank you for the pointers, I will check that

Indeed the clojure scoop version is not the last one. I switched to deps.clj and it works. https://clojurians.slack.com/archives/CFN4QDHPS/p1683209542279109?thread_ts=1683207794.496629&amp;cid=CFN4QDHPS

The CLI changes don’t generally fix this problem, only for programmatic invocation of tools, that was only for programmatic invocation of tools

You cannot

Darn, I suspected as much. Thanks anyway.

Ok, wait, by the above I assume you mean you have evaluated the above in the repl and then want to use alter-meta! after the above has been evaled

If you have read that form and want to add pre conditions before it is passed to eval that is possible, but not once it has been evaled

Right, I was looking to do the latter.

There are a few ways, one of them would be with rewrite-clj to navigate into the structures and add a new {:pre ,,,} node. Another could be simply vanilla clojure. (defn f ([x] ,,,) ([x y] ,,,)) is data: first you can parse its 'fntails' (each argv-body list), then you can (map whatever fntails)

I didn't realise that I suggested using rewrite-clj at macroexpansion time (e.g. in defmacro my-special-defn , and then rewriting the body being expanded), it's probably not idiomatic usage

I was looking for a way to add :pre after evaluation. 🙂 I’m aware I could do it between read and eval.