If you look at how the defmacro macro works, it expands into code that creates the def and then as a side effect a call to setMacro (which alters the vars metadata)

What you have attaches {:macro true} to the symbol foo, then copies the metadata from the symbol to var when creating the var

But when aot compiled the copying of the metadata from the symbol isn't done (I believe because the vars are created in a static init block before they are given a value)

Whoa, thanks!! Do you have any links/source code where I can read more?

(source 'defmacro)

Thanks, I've read that -- I'm mostly curious about the lack of metadata copying on AOT compiling

That I am being more hand wavy about

I am not sure what mechanism is causing that behavior, you might see it if you aot compiled with elide metadata set

I think that's a AOT optimization, there's a flag like Hiredman said, it's to make the bundle and runtime memory smaller by removing meta on vars which normally are useful only when developing

Also might minimally speedup startup maybe

Doc is here: https://clojure.org/reference/compilation#_elide_meta

Thanks! We aren't using metadata eliding though.. :thinking_face:

Actually, I think AOT doesn't remove metadata. The problem is that only {:macro true} gets removed. Here is what I think I understood. If you have a namespace

(ns example.lib)

(defmacro inc-macro [x]
  `(+ 1 ~x))

(def ^{:macro true :additional true} inc-macro2 #'inc-macro)

const__27 = (AFn)RT.map(new Object[] { RT.keyword((String)null, "macro"), Boolean.TRUE, RT.keyword((String)null, "additional"), Boolean.TRUE, RT.keyword((String)null, "line"), 13, RT.keyword((String)null, "column"), 1, RT.keyword((String)null, "file"), "example/lib.clj" });

final Var const__7 = lib__init.const__23;
        const__7.setMeta((IPersistentMap)lib__init.const__27);
        const__7.bindRoot((Object)lib__init.const__19);

alterMeta(dissoc, RT.list(macroKey));

//binding root always clears macro flag

{:additional true,
 :line 6,
 :column 1,
 :file "example/lib.clj",
 :name inc-macro2,
 :ns #namespace[example.lib]}

(defmacro inc-macro3 [& form]
  `(inc-macro ~@form))

(lib/inc-macro3 1)
=> 2

(lib/inc-macro2 1)
=> clojure.lang.ArityException
   Wrong number of args (1) passed to: example.lib/inc-macro

or just explicitly set the macro flag like the defmacro macro does

Or that... 🙂

because the code that explicitly sets the flag instead of relying on metadata copying will just get run again in aot'ed form

This works.

(ns example.lib)

(defmacro inc-macro [x]
  `(+ 1 ~x))

(def ^{:macro true :additional true} inc-macro2 #'inc-macro)
(.setMacro #'inc-macro2)

(lib/inc-macro2 1)
=> 2

Thanks all! Interesting read, @U04BP7T2GTA -- ended up going with that exact solution (just wrapping with a normal defmacro), which IMHO might be preferable for others' understandability since:

(def ^:macro foobar #'other-foobar)
(.setMacro #'foobar)

(defmacro foobar [& args] 
  `(other-foobar ~@args))

if you add macro falg after you don't need to also do ^:macro

Yep, but still slightly strange code to come-across, because then you might ask "why not add ^:macro metadata instead?" and the cycle continues! 😛

depends. if you have a call to setMacro it is less likely the automatic reflex will be to switch to ^:macro, if you have a call to alter-meta! then it is more likely

Actually I came to a slightly different conclusion why clj code works in the REPL after I cleared my head. It's not because of the reverse of .setMeta , .bindRoot. @U0NCTKEV8 was right all along. It's all about the implementation of defmacro. https://github.com/clojure/clojure/blob/b2366fa5c748f9d600879c3e0b549e631a5b386f/src/clj/clojure/core.clj#L489 defmacro will first to defn (essentially def) which does execute DefExpr.eval with the order of .bindRoot and .setMeta. But immediately afterwards it will execute .setMacro that overrides macro's metadata. Plus it's not like .setMeta will have {:macro true} anyway. --- To confirm

(defmacro ^{:macro false} dec-macro [x]
  `(- ~x 1))

(meta #'dec-macro)
=>
{:macro true,
 :arglists ([x]),
 :line 10,
 :column 1,
 :file "/home/mbezjak/workspace/use-uberjar/src/using/core.clj",
 :name dec-macro,
 :ns #namespace[using.core]}

In my opinion it's a bad practice to use declare altogether

Yes, though sometimes necessary

(declare ^:private foo)
(meta #'foo)
;; => {:private true, :declared true, :line 270, :column 1, :name foo}
(def foo 10)
(meta #'foo)
;; => {:line 273, :column 1, :name foo}

The meta is overwritten by the latter def evaluation. FYI.

So I'd say it might be best to define the meta on the def, and not on the declare. Or make sure to have it on both.

No.

You could delegate the task to a limited thread-pool.

> You could delegate the task to a limited thread-pool. That doesn't make it memory-constrained though. That task is also free to spin up it's own threads under the hood.

@ULE3UT8Q5 If you really want that, I would suggest running separate processes instead for better isolation. That said, a https://clojurians.slack.com/archives/C03S1KBA2/p1614110292087900 I did an experiment - it's far from bulletproof but it sort of works: https://github.com/jumarko/clojure-experiments/blob/develop/src/clojure_experiments/performance/memory.clj#L224-L229

@U06BE1L6T running separate processes is my go-to approach, but it has its limits..

I’m mostly concerned about limiting CPU time

You cannot do that easily. If you are trying to run untrusted code then you are asking for trouble. If you own the code and know what it's doing then the approaches suggested above might work.

I was expecting to see a single invocation (based on the API) but it seems I have to add a synchroniser or something myself. This is an issue with a web environment where you might get multiple requests for the same key.

Are you concerned about the anonymous function being called more often than you expected? It's probably used with swap! in the cache library, so keep in mind that swap! can run the updater function again (and again) if it hits a collision.

Yes, the anonymous function should only be called once per key. I read the documentation in the core.cache usage, it mentions it's safe from a cache stampede, but it's not 100% accurate.

The function is wrapped in a delay within core.cache so that swap! will not call it again. It's being called once per thread though.

I think that's why people recommend core.memoize. FOund this article I read a couple of years ago: https://quanttype.net/posts/2020-10-25-caching-http-requests.html

Ah, thanks @U06BE1L6T I will take a look there.

@U7PBP4UVA I'm not sure non thread safe are the words here, I guess look-up-or-miss is supposed to be about looking up data and not generating side effects?

Well you have to somehow generate the data in the case of cache miss.

yeah, but if it is all about retrieving, the worst thing is some extra calls, so not as efficient as it can get, but nothing should be wrong, so it should be safe

oh yeah in that sense it's thread safe as the result is eventually correct - but it's a version of cache stampede.

yeah, that I agree

but I think defining your own lookup-or-miss wrapper that locks on the cache object should do the trick

That's exactly what the article linked above mentions: > You could implement this yourself by doing some locking… but you could also use https://github.com/clojure/core.memoize, which does it for you.

Caches are notoriously hard to get right, so unless you're 100% certain in what you're doing and there's nothing available out there that suits your needs, you should definitely stick to the existing solutions, core.memoize in particular.

Yep, same sentiment here @U2FRKM4TW

Maybe consider caffeine

The library :-)

core.memoize seems to do exactly what we want. Thanks everyone for the suggestions!

(let [thread-count 20
        key-mod 3
        invocations-counter (atom 0)
        expensive-function (fn [k]
                             (swap! invocations-counter inc)
                             (Thread/sleep 3000)
                             (str "value-" k))
        cache (-> {}
                  (cache/ttl-cache-factory :ttl 120000)
                  (cache/lu-cache-factory :threshold 100))
        memoized-function (memoize/memoizer expensive-function cache)
        latch (java.util.concurrent.CountDownLatch. thread-count)]
    (doseq [i (range thread-count)]
      (println "starting thread" i)
      (.start (Thread. (fn []
                         (memoized-function (str "my-key" (mod i key-mod)))
                         (.countDown latch)))))

    (.await latch)
    [(memoize/snapshot memoized-function)
     (deref invocations-counter)]
    (assert (= key-mod (deref invocations-counter))))

FWIW, it took me several rounds to get it as "safe" as it currently is but I'm still a bit surprised that you're seeing a sort of stampede with cache-wrapped/lookup-or-miss but not with memoize/memoizer -- can you put this example up on http://ask.clojure.org so I can take a closer look -- and I'll probably create a JIRA issue against core.cache for it...

I have been using it with a delay inside of the lookup-or-miss! and then deref the outcome. Think this works fine:

@(cache-wrapped/lookup-or-miss cache-atom "my-key" (fn [k] (delay ...)))

Ah I see the issue. I’m not using the wrapper and I see the wrapper is wrong.. It’s dereffing the delay inside the swap. It should happen outside of the swap! https://github.com/clojure/core.cache/blob/master/src/main/clojure/clojure/core/cache/wrapped.clj#L67 @U04V70XH6

With many threads having a cache miss at the same time could mean that they all deref their delay during the compare-and-set loop of swap!, causing more than one execution. The fix is simple I think. Instead of

(c/lookup (swap! cache-atom c/through-cache
                             e
                             default-wrapper-fn
                             (fn [_] @d-new-value))

@(c/lookup (swap! cache-atom c/through-cache
                             e
                             default-wrapper-fn
                             (fn [_] d-new-value))

If you can write this up on http://ask.clojure.org, I'll take a look at it when I can... probably in a week or two (I'm planning to schedule time to work on core.cache documentation early next month).

@U7PBP4UVA do you want to write the problem statement in http://ask.clojure.org? I can reply with a solution (I think)

Sounds good. Thanks!

I had this in mind. I will do the writeup.

https://ask.clojure.org/index.php/12567/multi-threaded-cache-stampede-in-core-cache

Thanks!

For now we fixed it by using core.memoize, but I recall a colleague fixing something like this by using a promise - it would be eagerly assoc'ed in the cache, so that subsequent requests would get a single promise, and the first thread would actually do the work and deliver the value (details are hazy - this was not core.cache).

Yeah, ISTR core.memoize uses delayed values in the cache but core.cache deliberately does not -- because that would put the burden on the user to correctly deref things (and add confusion over what types of values were in the cache).

Thanks @U064X3EF3 for https://clojure.atlassian.net/browse/CCACHE-65

I added my answer with a potential workaround HTH

I've made it about this far

(ns togglz
  (:import (org.togglz.core Feature)
           (org.togglz.core.manager FeatureManagerBuilder)
           (org.togglz.core.repository.jdbc JDBCStateRepository$Builder)
           (org.togglz.core.spi FeatureProvider)
           (org.togglz.core.user SimpleFeatureUser UserProvider)
           (org.togglz.core.metadata FeatureMetaData)
           (org.togglz.core.repository FeatureState)))

(def features
  {::new-home-screen {}})

(defrecord KeywordFeature
  [kw]
  Feature
  (name [_]
    (name kw)))

(defn map->metadata
  [feature feature-metadata-map]
  (when (seq feature-metadata-map)
    (reify FeatureMetaData
      (getLabel [_]
        (:label feature-metadata-map))
      (getDefaultFeatureState [_]
        (FeatureState. feature (:enabled-by-default feature-metadata-map)))
      (getGroups [_]
        #{})
      (getAttributes [_]
        {}))))


(defn keyword-feature-provider
  [features]
  (reify FeatureProvider
    (getFeatures [_]
      (-> (keys features)
          (map ->KeywordFeature)
          (into #{})))

    (getMetaData [_ feature]
      (some->> (:kw feature)
               (get features)
               (map->metadata feature)))))

(defn user-provider
  []
  (reify UserProvider
    (getCurrentUser [_]
      (SimpleFeatureUser. "ethan" true))))

(defn start-feature-manager
  [db]
  (-> (FeatureManagerBuilder.)
      (.featureProvider (keyword-feature-provider features))
      (.stateRepository (-> (JDBCStateRepository$Builder. db)
                            (.tableName "togglz")
                            (.createTable true)
                            (.build)))
      (.userProvider (user-provider))
      (.build)))

where im getting stuck, conceptually, is the wiring up of the admin panel - how tf. do i put servlet handler under ring middleware?

oh...amazing > Caused by: java.lang.NoClassDefFoundError: javax/servlet/Filter Love the jakarta rename

honestly i would rather just hand roll flags :thinking_face:

the intent is to kind of keep the chunk structure (maybe that's what you mean)?

i think so. looking at the commit that added it, it says "add support for chunked seqs to 'for'". i think I just don't understand enough about the chunking system to get why this is needed.

I think it's so that it returns a chunked sequence as well, otherwise it would have the behavior of unchunking

Run clojure -Sdescribe on both machines and check that they are finding the deps.edn file(s) you expect.

(perhaps one machine is XDG and the other isn't and you have the dot-file in a place it is not expecting @mars0i?)

also, is Clojure CLI version the same? (`clj --version`)

@mars0i wrote:

:aliases {:nrepl-server {:extra-deps {nrepl/nrepl {:mvn/version "1.0.0"}}
                          :main-opts  ["-m" "nrepl.cmdline"]}}

Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate nrepl/cmdline__init.class, nrepl/cmdline.clj or nrepl/cmdline.cljc on classpath.

Interesting. Thanks @U04V70XH6! I just realized that I should have waited to ask when I had both machines present. At work and I only have machine A. -Sdescribe does say that I have a deps.edn in ~/.clojure, but it's just an empty map. I'll check machine B when I get home.

What is XDG?

@mars0i I think Alex might be right that it's a CLI version issue -- -M has changed its meaning over time.

@U064X3EF3 Oh, didn't think of that. i.e. clj could be one version, but it loads another version of Clojure that's listed in deps.edn? I'll check that.

The -M option has changed its meaning. With an old CLI it would not pick up extra-deps, just main-opts.

clojure -Sdescribe will show the version (as well as file paths)

I see about -M. OK, thanks. I'll check that stuff. It would be nice to understand what's happening. Great! Thanks.

@U04V70XH6 about using a thread--thanks. I was just trying to include the code blocks in the original question and couldn't remember how to do that in Slack without making a new post.

In the future I'll post a short description of the question and then add details in a thread.

Depends on your preference settings in Slack but ctrl-enter should allow you to enter newlines without sending the message I think...?

OK, will try that.

But, yeah, short problem statement in the main channel, details in a 🧵 is best.

When you're back at the non-working machine, let us know what version the CLI was and if updating it solved your problem.

OK, will do--thanks!

https://github.com/borkdude/lein2deps Not sure how good it is.

Interesting @U0K064KQV. I'll try that. Thanks.

@U04V70XH6, @U064X3EF3 Yes--it was the CLI version. The machine with the error had 1.10.1.478. The one without the error has 1.11.1.1208. Upgrading CLI fixed the problem. Thank you.

just looking at the examples in the readme for drift it doesn't look like it is intended to run from a jar with migrations as resources in the jar

it sounds like you're saying that within a jar files are accessible as 'resources'. what does this mean?

Files are things that exist in a filesystem, once you put things in a jar, they aren't files anymore, but the jar is

look at the http://java.io/resource fn. Many java / clj libs can work with resources that are on the classpath via this fn, but some simply expect a physical file on the FS

thanks @U0522TWDA I'll give it a look

can you reccomend any material for understanding what a resource is?

I guess https://mkyong.com/java/java-read-a-file-from-resources-folder/ ?

But if drift only supports file system and not classpath resources then you are out of luck. Personally I use https://github.com/weavejester/ragtime, which is maintained. There is also https://github.com/yogthos/migratus, which I haven’t tried

nice

i took a look at ragtime before but i got the impression that they leave a lot more work to the client. i'm going to switch to migratus

a jar file is a glorified zip. resources = files inside the zip (in a specific location). files = files outside the zip.

Resources are steams of data you retrieve via a class loader, entries in a jar file is just the most common place a class loader will retrieve them from

i'm finding the lions share of what's challenging about learning Clojure being not Clojure itself, but what Java I need to learn in order to use Clojure. especially all of the stream stuff, this type of stream and that type of stream. coming from a JavaScript and PHP background this stuff is mostly unfamiliar. is this an area of Java which is really complicated, or is it really simple and justs needs to be gotten the hang of?

Yeah, if you're not looking to do anything polymorphic there isn't much reason to use a protocol.

that was my intuition that I was nevertheless in the process of ignoring, so wanted to check 😅 thanks @U04V32P6U

I agree with @U04V32P6U but want to add that polymorphism is a spectrum and, if experimentation is cheap, exploring the implementation space could yield unexpected results 👍

What do you mean by "polymorphism is a spectrum"?

It's monomorphic -- operating on a single data type -- or it's polymorphic, no?

The features are upgradable, so there's no reason to start with a protocol just because you're not sure if you might need one in the future. You can have functions, then feel one of them would benefit being polymorphic, switch to a multi-method, then realize a set of them would benefit being polymorphic together, now switch to a protocol, all without ever breaking the call-sites

The function call would still always just be:

(fn-name operand paramerters...)

@U04V32P6U Sort of - in the case of concrete types and the usual usage of protocols, you’re correct. What I meant by “polymorphism is a spectrum” is that there’s a practical continuum between reaching for conditional logic on a map, multi-methods/hierarchies, and type-based dispatch. I apologize for the ambiguity and my comment’s possible irrelevance to the original question.

Oh no need for an apology! I was curious to understand more, thanks