I made an atom thing for Java and i ended up making my own swapComplex that could also return “context” inferred from the swap

(defn swap-complex! [atom f & args]
  (loop []
    (let [old-value               @atom
          {:keys [new-value 
                  derived-value]} (apply f (cons old-value args))]
      (if (compare-and-set! atom old-value new-value)
        {:new-value     new-value
         :derived-value derived-value}
        (recur)))))

so for this example

(let [last-executed (atom 0)]
  (fn wrapper
    [& args]
    (let [now (System/currentTimeMillis)
          {last-executed-changed :derived-value} (swap-complex! last-executed
                                                                (fn [last] ... {:derived-value true :new-value ...}]

I'd warn against using compare-and-set! for primitives (such as longs). See the comment by Favila at clojuredocs: https://clojuredocs.org/clojure.core/compare-and-set!

FWIW I wrote a library for rate limiting a while back: https://github.com/aeriksson/kilderkin

I think https://github.com/wilkerlucio/pathom3 might be designed to address this use case. If you don't want to embrace some kind of declarative dsl (kind of necessary to derive graphs that let the computer figure this stuff out) if you only have to deal with this a couple of times, you could just include some (documented) :skip-x, :omit-y type options to the function. Sometimes worse-is-better.

Why would it be 10000 * 1 MB? Suppose you have a function like

(defn get-data []
  {:a (generate-huge-random-blob)
   :b 1})

It is true, hat the 10000 calls share "some data" returned, but some not. (The function returns a sequence):

(defn get-data-vector [v]
 (map get-huge-data v)

I think there's some misunderstanding between us, but not sure in which direction. JVM won't hold on to the data that's not referenced, right? So, if the caller of that function gets all the data, all the time, and then simply selects the desired data after the fact, the unneeded data will not be retained. The only downside here that I can see is that the unneeded data will still be generated, time and time again - but from your description of the problem only the RAM usage seems to be the problem (which it's not, if unused data is not referenced) and not CPU.

It is more complex. The calculation is time consuming, overall minutes for each vector element. (so one million would take a week) So I decided to force non-lazy.... But in an other use case, it's only seconds . We talk about training of ML models. (some models are very fast, some tale minutes) for each call.

Let's say this function has eventually very different performance characteristics, depending on use case. Seconds to days. (short input seq and fast ML model vs long input sequence of slow ML model)

A "solution" is probably this:

(defn get-data-vector [v filter-fn]
 (map #(filter-fn (get-huge-data #)) v)

One very easy approach then is to wrap all the costly data in delay - but the downside is that the client of that function will have to wrap all concrete data access in force. So, borrowing from the example above, it will become

(defn get-data []
  {:a (delay (generate-huge-random-blob))
   :b 1})

ok, using delays. Thabgs will think about it.

My solution feels somhow non-idiomatic, so doing a function which allows to "configure" the return.

It's still a pure function, but ...

Justified by "reduction of RAM usage" ... and the "need" of non lazy due to "time consuming". maybe I should go to "actors" instead of a simple "map" function?

Not sure what you mean by "actors". But the query approach would work (after all, it's similar to SQL, which we have had for ages). Alternatively, if your data generation can be split into tiny parts that could be called separately, that would definitely be a more idiomatic approach, and for good reasons.

Clojure "agents" I ment. Maybe each calculation should be done by an "agent"

You can use them but I don't see how it would help. There's nothing about the agents that would make your task simpler. Also, I'm pretty sure that that functionality is left there for backwards compatibility. IIRC, Rich at some point stated that he overestimated their usefulness and that if he were to reimplement Clojure from scratch, he would start with core.async and would never implement agents. But don't quote me on that.

Maybe you can use a higher-order function and pass it to the get-data function. In the case you need all the data, you pass a fn that does all the expensive stuff. And in the case you want a small answer back, the fn omits all the expensive stuff

If your problem is that it's potentially expensive to get each piece of data, then why not just return it as a lazy sequence? Only small chunks will get realized at once, how long it takes is irrelevant, and you can use map or filter to get down to a subset of the data and produce another lazy sequence that when realized fits into the heap. Since the JVM is garbage collected if you iterate over this huge structure and don't retain the head, then you'll produce a lot of GC pressure which will slow things down somewhat, but as long as no single chunk of data blows the heap then you won't have memory issues. And if you map to only a subset of the data where it would fit in memory, you can retain the head without issue as well. @U7CAHM72M

Sets metadata of {ClassName true} on the var/symbol someObj, depending on the context.

but why would you do that ?

Maybe someone mistook it for just ^ClassName, as a type hint. Or maybe there's some tooling that supports it. Or maybe it's processed in the code itself at some point. Impossible to say for sure without having the full context.

yeah you're right just wanted to check if there was something in clojure I didn't know about. thanks for your help ! I think it was a mistake

Maybe performance is the reason.

user=> (let [args (seq [1 2 3])] (time (dotimes [i 10000000] (.applyTo ^clojure.lang.IFn + args))))
"Elapsed time: 779.743179 msecs"
nil
user=> (let [args (seq [1 2 3])] (time (dotimes [i 10000000] (apply ^clojure.lang.IFn + args))))
"Elapsed time: 1463.516917 msecs"

You could generate java source code with something like javapoet and feed it to javac

from a philosophical point of view, no by design

that is not a problem Clojure is trying to solve (arbitrary Java class making). it does have class making capabilities that mesh well with the Clojure approach to datatypes etc

okay, thanks

Why on Earth can't the JVM find org.jpl7.fli.Prolog? As a comparison, if I import a class that doesn't actually exist, I get an immediate error (not deferred until I try to use the class).

prologclj.core> (import 'org.jpl7.fli.Prologxxx)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:445).
org.jpl7.fli.Prologxxx

I've got this down to as minimal a case as I can, and I'm out of ideas.

i think this error doesn’t mean it cannot FIND the class but that it cannot initialize it > Could not initialize class org.jpl7.fli.Prolog

https://github.com/SWI-Prolog/packages-jpl/blob/master/src/main/java/org/jpl7/fli/Prolog.java#L71

Ah, that's extremely helpful! Thanks, that's undoubtedly it. I'll chase that next.

always print out the whole error with *e

you'll probably have seen that the ClassNotFoundException is chained to an ExceptionInInitializerError, or something like that

which would clue you into the native library issue

@U050ECB92 I imagine clj is wrapping the initialization error in a class not found exception? If so, is that worth a bug report and fixing on http://ask.clojure.org or jira?

i had this recently with a driver for vertica on the new m1 pro mac

the JVM is wrapping it

usually CNFE is chained to something

ah ok 👍

out of curiosity, @U054FA810 you should post the whole exception (`*e` ) in a slack snippet (CMD-Shift-Enter)

that way we all can see the unadulterated data

Though, I get a much more helpful error message if I try to do a lein repl from the command line.

But, I've also got this in my project.clj:

:jvm-opts ["-Djava.library.path=/usr/local/opt/swi-prolog/libexec/lib/swipl/lib/x86_64-darwin"]

So either lein is ignoring that option, or I'm missing something else.

what version of java are you running? do you happen to be running on an M1 mac?

it does look like the java.library path isn't being set. can you share your project.clj? I would double check to see if :jvm-opts isn't accidentally nested in the wrong place

It's pretty rudimentary. I had stripped out everything else, and extracted the clj deps from their JAR into /resources.

It doesn't appear to be nested, but java.library.path absolutely is not getting set.

how are you confirming that java.library.path is not getting set?

prologclj.core=> (System/getProperty "java.library.path")
"/Users/username/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."

just to double check something, if you use ["-foo"] as your jvm opts, does it complain?

Hm, no it doesn't.

And not using M1, by the way. Five-year-old MacBook Pro.

it should be unhappy about the ["-foo"]

I just changed the version of Clojure in the deps to 1.10.2, and it picked it up. So it's definitely reading this project.clj.

This seems like it has to be something really basic that I'm missing.

did you copy and paste the :jvm-opts arg from somewhere? sometimes a weird unicode character can sneak in

Well, from the CLI.

I just tried :jvm-opts ["-Dfoo=bar"], and there's no foo system property. So it does look like those jvm opts are not getting applied at all.

% lein version
Leiningen 2.9.8 on Java 17.0.1 OpenJDK 64-Bit Server VM

from terminal, what happens if you run, lein change :jvm-opts set '["-foo"]'

Nothing visible.

% lein change :jvm-opts set '["-foo"]'
% echo $?
0

it should have updated your project.clj

Ah, just saw that, had to revert the emacs buffer it was up in.

does lein repl show an error now?

Nope. The project.clj was changed, but still just the loading error.

have you tried lein clean and retrying?

Worth a shot.

Still no dice, same behavior.

well, I'm out of ideas. at this point, I would start a brand new project, check to make sure adding system properties work, and the re-add the dependencies again

🤷

Setting LEIN_JVM_OPTS and checking the process list does show the library path being set on the Lein JVM.

OK, so, setting the JVM_OPTS environment variable does, in fact, get the java.library.path set on the REPL's JVM.

But that would be really annoying to try to do with Cider...and this should just work in Lein.

yea, super weird

And, incidentally, with the library path set it does fix the loading issue, and I can instantiate org.jpl7.fli.Prolog just fine.

is there a way to get lein to print out the project it ends up seeing?

I guess one other thing to check is your ~/.lein/profiles.clj and see if there's a profile that overrides the :jvm-opts

Hm, that's a pretty likely candidate, in fact.

it's weird that it would have precedence over your local project, but we did check a bunch of other potential causes

and it probably isn't relevant, but :java-opts is an alias for :jvm-opts

Well, there is indeed an ancient :jvm-opts override in there...probably more than five years old, probably followed me from another machine in my dotfiles. It was in there to fix a problem with Overtone at the time, according to the comment.

sneaky

profiles are a menace. Thanks, that was a brilliant thought.

Yep, that seems to have sorted it! The problem was too surgically specific, and had to be something to do with my own machine's setup. Brilliant, thanks for the support.

glad it got resolved!

fresh variables generate linearly , disjunctions (or, conde clauses, etc) generate forks in the search tree. conjunctions (clauses in the same fresh or the same conde group) constrain. membero is a constraint, too

membero is also a linear search, slow

everything is both

membero generates branching searches as well

if the member is not ground but the list is, it will branch the search where the member is unified to every entry in the list

my mistake. It's a conde