~ % clj
Clojure 1.10.3
user=> (= 1.0 (long 1.0) (bigdec 1.0))
false
user=> (== 1.0 (long 1.0) (bigdec 1.0))
true

(== 1.0 (float 1.0) (bigdec 1.0)) is probably more indicative of the difference since the 1.0 literal will be a long from the reader

yes, but the important thing is that = behaves opposite the documentation.

thus the memoize function won't confuse (f 1) with (f 1.0) because hash equality if based on =, not ==

> Can someone help me understand the documentation for = ? https://clojure.org/guides/equality

I suspect the phrase "type-independent way" might be in comparison to how Java does it. Java equality between short and int and long is always "not equal" result. Clojure gives equality between any two integral types that are numerically equal.

Agreed that it is easy to misconstrue that sentence. Clojure built-in docs are infamously terse.

Probably won't change in Rich's lifetime.

Why do you say that, @U0CMVHBL2? (Not contesting it, just curious 🙂 )

Sorry, weird thing to say. Multiple Clojure core maintainers have expressed a strong preference for, and justification of, brief documentation

I don’t mind brief documentation as such, but OTOH, there are many times that I would have been lost without Clojuredocs. (Thank you, Clojuredocs!) And hopefully, ambiguity can be avoided while keeping the docs short, although I gather how these two aspects can challenge each other at times.

@U0CMVHBL2 I see you’re a clojuredocs contributor, so thank you! 🙏

memoize will be confused

{1 1
 1.0 1.0}

so it WONT be confused. unless I'm confused.

since 1 is different (not = ) to 1.0, they will be memoized separately. right?

did I misunderstand something here?

(defn foo [n]
  (prn n)
  n)

(def mfoo (memoize foo))

(mfoo 1)                                ;; print 1

(mfoo 1.0)                              ;; print 1.0

(mfoo (int 1))                          ;; no print

what does (int 1) do?

so the docstring for = is missing some explanation what is “type-independent manner” is

(= (long 1) (short 1)) this returns true

it cast Long to Integer in clojure every every number is either Long or Double.

(type 1) ;; => java.lang.Long
(type (int 1)) ;; => java.lang.Integer

is that a clojure thing, or does that mean my CPU represents short and long both as 64 bit integers?

no, short is short but 1 is Long

1 as you have it in written code

OK, so = thinks a (long 1) is equal to (short 1), this means that memoize will confuse them.

so perhaps = does in fact do SOME type-independent magic ???

yes, some. and explanation what sort of “some” is missing in the docstring

👍:skin-tone-2:

clojure-rte.rte-core> (defn foo [n] (prn n (type n)) n) #'clojure-rte.rte-core/foo clojure-rte.rte-core> (def mfoo (memoize foo)) #'clojure-rte.rte-core/mfoo clojure-rte.rte-core> (mfoo 1) 1 java.lang.Long 1 clojure-rte.rte-core> (mfoo 1) 1 clojure-rte.rte-core> (mfoo (short 1)) 1 clojure-rte.rte-core> (mfoo (long 1)) 1 clojure-rte.rte-core> (def mfoo (memoize foo)) #'clojure-rte.rte-core/mfoo clojure-rte.rte-core> (mfoo (short 1)) 1 java.lang.Short 1 clojure-rte.rte-core> (mfoo (long 1)) 1 clojure-rte.rte-core>

from that page - https://clojure.org/guides/equality

Clojure's = is true when called with two immutable scalar values, if:

Both arguments are numbers in the same 'category', and numerically the same, where category is one of:

 * integer or ratio

 * floating point (float or double)

 * BigDecimal.

so short, int, long falls into category integer but I’m surprised to see ratio and integer in the same one

(= (clojure.lang.Ratio. (biginteger 1) (biginteger 1)) 1) ;; => false

ahh and of course I forgot that (= [1] (list 1)) so I suppose memoize will conflate these as well.

An even weirder example:

user=> (= 1/1 (clojure.lang.Ratio. (biginteger 1) (biginteger 1)))
false
user=> (= 1/2 (clojure.lang.Ratio. (biginteger 1) (biginteger 2)))
true

I know that the reader turns 1/1 into just 1. But it still looks weird. :)

yeah %)

this is not the same but caught me few times )

(identical? (Boolean. true) (Boolean. true)) ;; => false

At least in the case of core.cache.wrapped/lookup-or-miss the wrap-fn is provided the value of e as well. (the default wrap-fn is #(%1 %2)) with %1 being value-fn and %2 being e )

Seems to be the same with core.cache/through :

(defn through
  "The basic hit/miss logic for the cache system.  Expects a wrap function and
  value function.  The wrap function takes the value function and the item in question
  and is expected to run the value function with the item whenever a cache
  miss occurs.  The intent is to hide any cache-specific cells from leaking
  into the cache logic itelf."
  ([cache item] (through default-wrapper-fn identity cache item))
  ([value-fn cache item] (through default-wrapper-fn value-fn cache item))
  ([wrap-fn value-fn cache item]
   (if (clojure.core.cache/has? cache item)
     (clojure.core.cache/hit cache item)
     (clojure.core.cache/miss cache item (wrap-fn #(value-fn %) item)))))

yes I’ve seen that too… I still don’t get the design choice here though, as value-fn also receives the value of e; so I could’ve just wrapped it myself.

Yeah, I recently stumbled on this because I gave lookup-or-miss a map and the default wrap-fn isn't documented so I got some pretty interesting behaviour.

But since it's in there, there probably is a use-case for having separate value- and wrap-fns?

Which is exactly why I’m asking 🙂

I think you could look at the soft cache directly and count its items or use SoftReference.get on each value to see how many have been garbage collected

Back in the day we used https://github.com/joegallo/robert-bruce which worked pretty well for retries with backoff but I'm wondering if there is something more recent that's better.

https://github.com/BrunoBonacci/safely is another option

Thanks Darin, that looks like a really good option.

I do some variation of

(trampoline
 ((fn f [sleep]
    (try
      (do-something)
      (catch Throwable t
        (if (> sleep (* 60 1000))
          (throw t)
          (do
            (Thread/sleep (* (rand) sleep))
            #(f (bit-shift-left sleep 1)))))))
  1))

Thanks Kevin, I figured you'd have some good input. 🙂

When I unpack the JAR, the folder structure is definitely there as I would have expected it.

It’s not a file when you jar it up. It’s inside an zip.

It should be an URL, but file should pass it on to the Coercions protocol for URL, should’t it?

URL is irrelevant. URL can encode lots of different things (files, zips, http, ftp, sftp)

You were able to turn it into a file in dev because it was a file in dev.

In prod, it’s not a file.

Ah, I get it. so it’s not a file:// protocol, and thus fails

It’s inside of a zip.

right

yeah you will have to enumerate zip entries basically

You can potentially look at the entries of the jar you’re in.

Or you can manually keep a manifest up to date as a static edn structure.

Looking up the jar entries seems pretty sus IMO.

It’s definitely not how Java expects you to use resources. I don’t immediately know how it might break things, but I suspect that it could break somehow.

New resources will be added to the lib from time to time, and I wanted to avoid creating a registry.

you can walk a zip/jar file and check the entries that match a certain directory - why not?

Can you automagically resolve jars from the classpath?

e.g. if you have 10 jars on your classpath, how do you know which one to walk? Just walk all of them?

you can infer that from what you get back through (io/resource "the-directory") ?

just make sure the directory is unique

so you won't have clashes, so use a reverse dns org name

(io/resource "the-directory") works?

in a jar?

hmm, actually not sure :) let's test

(io/resource "the-directory") was my try for the Jar, with aforementioned outcome 😉

user=> (require '[ :as io])
nil
user=> (io/resource "clojure")
#object[java.net.URL 0x5226e402 "jar:file:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure"]

yes, so there you get the jar file in question

and then you walk this jar to extract the jar entries from the relevant dir

it's a bit more work, but doable

Yeah, I’m still super skeptical face.

Like, the semantics of that are weird.

(io/resource "foo/bar.edn") and (io/resource "foo/baz.edn") - afaik bar.edn and baz.edn are allowed to be in separate jars.

this is true, but if you control the environment, this won't happen

so foo isn’t even really a directory

like org.acme.my-resources

as far as Java resources are concerned

Yeah, you definitely wanna fully namespace this sucker. But also know that you’re doing something that Java doesn’t really expect.

Like, resource directories aren’t really a thing to the JVM.

(I’m actually surprised they let you resolve directories at all.)

I'm doing a similar thing in clj-kondo where I walk through all jar entries and copy all clj-kondo.exports entries

but the use case is a bit different I guess

So here’s what Spring does: https://github.com/spring-projects/spring-framework/blob/ccb080f948df45c36b21135759795cd7e572b813/spring-core/src/main/java/org/springframework/core/io/support/PathMatchingResourcePatternResolver.java#L494-L523

so basically walk through all entries every time, dispatching on the type of the entry

That’s the other thing: How do you switch between dev and prod? You’d have to dispatch on the resource type.

Yeah that’s probably enough: 1. Use a fully-qualified path, 2. Dispatch on the protocol of the URL

yep

Thanks, you two, that was pointing me into the right direction. Now getting a seq of paths with this code (which is not pretty). Getting to late here today, but I think I will get this somewhere tomorrow.

Final result which is working now: https://github.com/lambdaschmiede/freitag/blob/master/src/freitag/core.clj

@U0N9SJHCH: That looks quite good, I’ve done this sort of thing in the past too. You might be ok here for your use case, but a word of warning for others. In my experience you need to be careful when using the java FileSystems API to read jars/zips. In particular you’ll find that internally it maintains a registry/cache of open file systems — which essentially means you can’t open and close the same zip file on multiple threads without extra coordination over the shared file system resource. I have some code in of our projects which wraps this properly to support safe concurrent access into the same zip file from multiple threads. For your code it looks also like you have a minor bug; which is that you never seem to close the filesystem once you’re done with it. I’d imagine this will mean in a REPL if you execute path-from-fs twice on the same zip/jar it will die with an exception the second time. To fix this you’ll need to put the file-system inside a with-open and make sure you eagerly consume it.

@U06HHF230 thanks for the heads up! You are right, I missed closing the Filesystem (which also meant the code can only run once).

I have some code in one of our projects which I should probably open-source as a library that properly wraps the FileSystems calls so it supports safe concurrent access to the same zip file across threads.

Will add a with-open in the afternoon. Multiple threads shouldn’t be an issue, as the load function is only called once for initializations, but I’ll look into that also and maybe add some words of warning to the Readme

yeah multiple threads shouldn’t be a problem here — though you may want to consider moving the side-effect so it only occurs when you call query, and caches it at runtime…. e.g. perhaps (def vacations (delay (load-all)) Just so any problems reading occur at the time of the first query, rather than initialisation time

Delaying the read would be also interesting for the library, if you do not need all of the resources, we will add more files over time and it would be more lightweight not to dump everything into memory in the beginning