Fork me on GitHub
#clojure
<
2021-10-07
>
Jim Newton09:10:09

Can someone help me understand the documentation for = ? https://clojuredocs.org/search?q=%3D it says numbers are compared in a type independent way. However, experimentation shows numbers are compute type-DEPENDENT. (= 1 1.0) returns false. I don't understand the claim in the documentation.

dpsutton09:10:04

~ % clj
Clojure 1.10.3
user=> (= 1.0 (long 1.0) (bigdec 1.0))
false
user=> (== 1.0 (long 1.0) (bigdec 1.0))
true
is an example of the difference

dpsutton09:10:39

(== 1.0 (float 1.0) (bigdec 1.0)) is probably more indicative of the difference since the 1.0 literal will be a long from the reader

Jim Newton09:10:56

yes, but the important thing is that = behaves opposite the documentation.

Jim Newton09:10:37

thus the memoize function won't confuse (f 1) with (f 1.0) because hash equality if based on =, not ==

vemv09:10:20

> Can someone help me understand the documentation for = ? https://clojure.org/guides/equality

andy.fingerhut13:10:26

I suspect the phrase "type-independent way" might be in comparison to how Java does it. Java equality between short and int and long is always "not equal" result. Clojure gives equality between any two integral types that are numerically equal.

andy.fingerhut13:10:48

Agreed that it is easy to misconstrue that sentence. Clojure built-in docs are infamously terse.

andy.fingerhut13:10:54

Probably won't change in Rich's lifetime.

rich 1
reefersleep09:10:01

Why do you say that, @U0CMVHBL2? (Not contesting it, just curious 🙂 )

andy.fingerhut13:10:37

Sorry, weird thing to say. Multiple Clojure core maintainers have expressed a strong preference for, and justification of, brief documentation

reefersleep22:10:05

I don’t mind brief documentation as such, but OTOH, there are many times that I would have been lost without Clojuredocs. (Thank you, Clojuredocs!) And hopefully, ambiguity can be avoided while keeping the docs short, although I gather how these two aspects can challenge each other at times.

reefersleep23:10:14

@U0CMVHBL2 I see you’re a clojuredocs contributor, so thank you! 🙏

Jim Newton09:10:44

is it a misprint/typo in the documentation? Or am I just misunderstanding what it is saying?

Jim Newton09:10:09

Thanks. by the way, for my application the current behavior is exactly what I want, despite the documentation. So I'm please, just confused about the docstring.

Jim Newton09:10:06

I want to make sure that if using memoize (or friends), that a call to the function with 1 is not mistaken for a call to the function with 1.0.

delaguardo09:10:45

memoize will be confused

{1 1
 1.0 1.0}
inside it is using a map to store arg -> res relation and 1 and 1.0 are not the same

Jim Newton09:10:27

so it WONT be confused. unless I'm confused.

Jim Newton09:10:55

since 1 is different (not = ) to 1.0, they will be memoized separately. right?

Jim Newton09:10:51

did I misunderstand something here?

delaguardo09:10:18

(defn foo [n]
  (prn n)
  n)

(def mfoo (memoize foo))

(mfoo 1)                                ;; print 1

(mfoo 1.0)                              ;; print 1.0

(mfoo (int 1))                          ;; no print

Jim Newton09:10:07

what does (int 1) do?

delaguardo09:10:20

so the docstring for = is missing some explanation what is “type-independent manner” is

Jim Newton09:10:21

(= (long 1) (short 1)) this returns true

delaguardo09:10:43

it cast Long to Integer in clojure every every number is either Long or Double.

(type 1) ;; => java.lang.Long
(type (int 1)) ;; => java.lang.Integer

Jim Newton09:10:44

is that a clojure thing, or does that mean my CPU represents short and long both as 64 bit integers?

delaguardo09:10:13

no, short is short but 1 is Long

delaguardo09:10:55

1 as you have it in written code

Jim Newton09:10:56

OK, so = thinks a (long 1) is equal to (short 1), this means that memoize will confuse them.

Jim Newton09:10:45

so perhaps = does in fact do SOME type-independent magic ???

delaguardo09:10:24

yes, some. and explanation what sort of “some” is missing in the docstring

Jim Newton09:10:36

👍:skin-tone-2:

Jim Newton09:10:55

clojure-rte.rte-core> (defn foo [n] (prn n (type n)) n) #'clojure-rte.rte-core/foo clojure-rte.rte-core> (def mfoo (memoize foo)) #'clojure-rte.rte-core/mfoo clojure-rte.rte-core> (mfoo 1) 1 java.lang.Long 1 clojure-rte.rte-core> (mfoo 1) 1 clojure-rte.rte-core> (mfoo (short 1)) 1 clojure-rte.rte-core> (mfoo (long 1)) 1 clojure-rte.rte-core> (def mfoo (memoize foo)) #'clojure-rte.rte-core/mfoo clojure-rte.rte-core> (mfoo (short 1)) 1 java.lang.Short 1 clojure-rte.rte-core> (mfoo (long 1)) 1 clojure-rte.rte-core>

delaguardo09:10:49

from that page - https://clojure.org/guides/equality

Clojure's = is true when called with two immutable scalar values, if:

Both arguments are numbers in the same 'category', and numerically the same, where category is one of:

 * integer or ratio

 * floating point (float or double)

 * BigDecimal.

delaguardo09:10:47

so short, int, long falls into category integer but I’m surprised to see ratio and integer in the same one

(= (clojure.lang.Ratio. (biginteger 1) (biginteger 1)) 1) ;; => false

Jim Newton09:10:40

ahh and of course I forgot that (= [1] (list 1)) so I suppose memoize will conflate these as well.

p-himik10:10:49

An even weirder example:

user=> (= 1/1 (clojure.lang.Ratio. (biginteger 1) (biginteger 1)))
false
user=> (= 1/2 (clojure.lang.Ratio. (biginteger 1) (biginteger 2)))
true

p-himik10:10:52

I know that the reader turns 1/1 into just 1. But it still looks weird. :)

1
delaguardo10:10:04

this is not the same but caught me few times )

(identical? (Boolean. true) (Boolean. true)) ;; => false

1
rickmoynihan10:10:24

What is the purpose of wrap-fn in core.cache? Could I not have just wrapped value-fn myself?

Lycheese10:10:25

At least in the case of core.cache.wrapped/lookup-or-miss the wrap-fn is provided the value of e as well. (the default wrap-fn is #(%1 %2)) with %1 being value-fn and %2 being e )

Lycheese10:10:59

Seems to be the same with core.cache/through :

(defn through
  "The basic hit/miss logic for the cache system.  Expects a wrap function and
  value function.  The wrap function takes the value function and the item in question
  and is expected to run the value function with the item whenever a cache
  miss occurs.  The intent is to hide any cache-specific cells from leaking
  into the cache logic itelf."
  ([cache item] (through default-wrapper-fn identity cache item))
  ([value-fn cache item] (through default-wrapper-fn value-fn cache item))
  ([wrap-fn value-fn cache item]
   (if (clojure.core.cache/has? cache item)
     (clojure.core.cache/hit cache item)
     (clojure.core.cache/miss cache item (wrap-fn #(value-fn %) item)))))

rickmoynihan10:10:54

yes I’ve seen that too… I still don’t get the design choice here though, as value-fn also receives the value of e; so I could’ve just wrapped it myself.

Lycheese10:10:12

Yeah, I recently stumbled on this because I gave lookup-or-miss a map and the default wrap-fn isn't documented so I got some pretty interesting behaviour.

Lycheese10:10:10

But since it's in there, there probably is a use-case for having separate value- and wrap-fns?

rickmoynihan10:10:36

Which is exactly why I’m asking 🙂

octahedrion12:10:33

is there a way to prevent gen-class creating implementations for default methods of an interface ?

Jim Newton12:10:09

I am using clojure.core.memoize and clojure.core.cache as following:

(defn gc-friendly-memoize
  [g]
  (m/memoizer g (c/soft-cache-factory {})))
Is there a way to ask the caching mechanism to give me any debug feedback? I'm not 100% sure what kind of feedback I'd like. But information such as how many values are cached, and how many get released during GC for example would be useful.

Alys Brooks18:10:07

I think you could look at the soft cache directly and count its items or use SoftReference.get on each value to see how many have been garbage collected

3G18:10:49

What would be the best channel to talk about Clojure training for experienced Developers?

borkdude18:10:56

@gregg.walrod #off-topic is a good channel to chat about anything.

Alex Miller (Clojure team)18:10:33

not sure how up to date this all is now but https://clojure.org/community/training has some resources

Alex Miller (Clojure team)18:10:10

(and if it's not up to date, issues/prs to https://github.com/clojure/clojure-site/issues welcome)

1
👍 1
danielglauser18:10:15

Any recommendations for libraries that support retries with backoff at a minimum, or full on service orchestration? We're using AWS Step Functions right now and they're getting rather expensive.

danielglauser18:10:30

Back in the day we used https://github.com/joegallo/robert-bruce which worked pretty well for retries with backoff but I'm wondering if there is something more recent that's better.

danielglauser18:10:49

Thanks Darin, that looks like a really good option.

hiredman19:10:04

I do some variation of

(trampoline
 ((fn f [sleep]
    (try
      (do-something)
      (catch Throwable t
        (if (> sleep (* 60 1000))
          (throw t)
          (do
            (Thread/sleep (* (rand) sleep))
            #(f (bit-shift-left sleep 1)))))))
  1))
which is basically robert.bruce. safely looks neat. https://www.youtube.com/watch?v=m64SWl9bfvk may be an interesting some what related talk (circuit breakers, retries, load balancing, etc)

1
danielglauser19:10:22

Thanks Kevin, I figured you'd have some good input. 🙂

javahippie19:10:49

I have an issue with loading data from a folder in my resources, the structure is as shown below. In my code I want to automatically discover all files present in the data folder with (.listFiles (file (resource "data"))) with file and resource being referred from http://clojure.java.io. This works in the REPL, but as soon as I pack my code into a library, install it in the local repo and call it from a different application, I get `Execution error (IllegalArgumentException) at libraryname.core/load-all (core.clj:14). Not a file: jar:file:/Users/zoeller/.m2/repository/path/to/jar/0.1.0/libraryname-0.1.0.jar!/data`

javahippie19:10:32

When I unpack the JAR, the folder structure is definitely there as I would have expected it.

potetm19:10:52

It’s not a file when you jar it up. It’s inside an zip.

javahippie19:10:25

It should be an URL, but file should pass it on to the Coercions protocol for URL, should’t it?

potetm19:10:14

URL is irrelevant. URL can encode lots of different things (files, zips, http, ftp, sftp)

potetm19:10:32

You were able to turn it into a file in dev because it was a file in dev.

potetm19:10:36

In prod, it’s not a file.

javahippie19:10:40

Ah, I get it. so it’s not a file:// protocol, and thus fails

potetm19:10:42

It’s inside of a zip.

borkdude19:10:08

yeah you will have to enumerate zip entries basically

potetm19:10:30

You can potentially look at the entries of the jar you’re in.

potetm19:10:52

Or you can manually keep a manifest up to date as a static edn structure.

potetm19:10:44

Looking up the jar entries seems pretty sus IMO.

potetm19:10:39

It’s definitely not how Java expects you to use resources. I don’t immediately know how it might break things, but I suspect that it could break somehow.

javahippie19:10:53

New resources will be added to the lib from time to time, and I wanted to avoid creating a registry.

borkdude19:10:32

you can walk a zip/jar file and check the entries that match a certain directory - why not?

potetm19:10:47

Can you automagically resolve jars from the classpath?

potetm19:10:18

e.g. if you have 10 jars on your classpath, how do you know which one to walk? Just walk all of them?

borkdude19:10:42

you can infer that from what you get back through (io/resource "the-directory") ?

borkdude19:10:06

just make sure the directory is unique

borkdude19:10:12

so you won't have clashes, so use a reverse dns org name

potetm19:10:33

(io/resource "the-directory") works?

borkdude19:10:54

hmm, actually not sure :) let's test

javahippie19:10:23

(io/resource "the-directory") was my try for the Jar, with aforementioned outcome 😉

borkdude19:10:30

user=> (require '[ :as io])
nil
user=> (io/resource "clojure")
#object[java.net.URL 0x5226e402 "jar:file:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure"]

borkdude19:10:41

yes, so there you get the jar file in question

borkdude19:10:58

and then you walk this jar to extract the jar entries from the relevant dir

borkdude19:10:35

it's a bit more work, but doable

potetm19:10:57

Yeah, I’m still super skeptical face.

potetm19:10:19

Like, the semantics of that are weird.

potetm19:10:01

(io/resource "foo/bar.edn") and (io/resource "foo/baz.edn") - afaik bar.edn and baz.edn are allowed to be in separate jars.

borkdude19:10:21

this is true, but if you control the environment, this won't happen

potetm19:10:30

so foo isn’t even really a directory

borkdude19:10:32

like org.acme.my-resources

potetm19:10:37

as far as Java resources are concerned

potetm19:10:03

Yeah, you definitely wanna fully namespace this sucker. But also know that you’re doing something that Java doesn’t really expect.

potetm19:10:20

Like, resource directories aren’t really a thing to the JVM.

potetm19:10:49

(I’m actually surprised they let you resolve directories at all.)

borkdude19:10:54

I'm doing a similar thing in clj-kondo where I walk through all jar entries and copy all clj-kondo.exports entries

borkdude19:10:09

but the use case is a bit different I guess

potetm19:10:05

so basically walk through all entries every time, dispatching on the type of the entry

potetm19:10:32

That’s the other thing: How do you switch between dev and prod? You’d have to dispatch on the resource type.

potetm19:10:34

Yeah that’s probably enough: 1. Use a fully-qualified path, 2. Dispatch on the protocol of the URL

javahippie20:10:42

Thanks, you two, that was pointing me into the right direction. Now getting a seq of paths with this code (which is not pretty). Getting to late here today, but I think I will get this somewhere tomorrow.

rickmoynihan09:10:30

@U0N9SJHCH: That looks quite good, I’ve done this sort of thing in the past too. You might be ok here for your use case, but a word of warning for others. In my experience you need to be careful when using the java FileSystems API to read jars/zips. In particular you’ll find that internally it maintains a registry/cache of open file systems — which essentially means you can’t open and close the same zip file on multiple threads without extra coordination over the shared file system resource. I have some code in of our projects which wraps this properly to support safe concurrent access into the same zip file from multiple threads. For your code it looks also like you have a minor bug; which is that you never seem to close the filesystem once you’re done with it. I’d imagine this will mean in a REPL if you execute path-from-fs twice on the same zip/jar it will die with an exception the second time. To fix this you’ll need to put the file-system inside a with-open and make sure you eagerly consume it.

javahippie09:10:09

@U06HHF230 thanks for the heads up! You are right, I missed closing the Filesystem (which also meant the code can only run once).

rickmoynihan09:10:27

I have some code in one of our projects which I should probably open-source as a library that properly wraps the FileSystems calls so it supports safe concurrent access to the same zip file across threads.

javahippie09:10:41

Will add a with-open in the afternoon. Multiple threads shouldn’t be an issue, as the load function is only called once for initializations, but I’ll look into that also and maybe add some words of warning to the Readme

rickmoynihan09:10:57

yeah multiple threads shouldn’t be a problem here — though you may want to consider moving the side-effect so it only occurs when you call query, and caches it at runtime…. e.g. perhaps (def vacations (delay (load-all)) Just so any problems reading occur at the time of the first query, rather than initialisation time

javahippie09:10:07

Delaying the read would be also interesting for the library, if you do not need all of the resources, we will add more files over time and it would be more lightweight not to dump everything into memory in the beginning

fadrian21:10:47

I'm using the aleph package (version 0.4.7-alpha7). According to the aleph project.clj, there's a 0.4.7-alpha8, but I haven't upgraded to that yet because when I try to I get: Error building classpath. Could not find artifact aleph:aleph:jar:0.4.7-alpha8 in central (https://repo1.maven.org/maven2/) So for now, I guess I'm using alpha7. Aleph uses manifold - in its case, version 0.1.9-alpha3. Of course, manifold is now up to version 0.1.9, so I've overrode aleph's version in my local deps.edn, by using deps: { ... manifold/manifold {:mvn/version "0.1.9"} ... } So far, so good. Everything seems to load. And from a versioning standpoint, all should be well (-ish). However, in aleph, I get to the point netty.clj:797 where I make a call to manifold: (manifold.executor/thread-factory #(str prefix "-" (swap! num-threads inc)) (deliver (promise) nil) nil daemon?) Note that this call has four parameters. Now for the odd part. When I look at the call thread-factory in manifold executor, there is a four-parameter argument defined (executor.clj:35): (defn ^ThreadFactory thread-factory ([name-generator executor-promise] (thread-factory name-generator executor-promise nil true)) ([name-generator executor-promise stack-size] (thread-factory name-generator executor-promise stack-size true)) ([name-generator executor-promise stack-size daemon?] (reify ThreadFactory (newThread [_ runnable] (let [name (name-generator) curr-loader (.getClassLoader (class thread-factory)) f #(do (.set executor-thread-local @executor-promise) (.run ^Runnable runnable))] (doto (if stack-size (Thread. nil f name stack-size) (Thread. nil f name)) (.setDaemon daemon?) (.setContextClassLoader curr-loader))))))) But when this call is made, I get the following error: ; Execution error (ArityException) at aleph.netty/enumerating-thread-factory (netty.clj:797). ; Wrong number of args (4) passed to: manifold.executor/thread-factory Anyone got an idea about what's going on? I'd say it's a versioning issue, but aleph out of the box is using an earlier version of manifold, and manifold has released a four-argument version of this function.

dpsutton21:10:41

in a repl you could type (source manifold.executor/thread-factory) and see what source is actually being used

fadrian21:10:09

source tells me there's a four-adic function binding.

hiredman21:10:42

I would double check your full dependency graph, if some library you are using was aot'ed with an older version you could be getting that instead

hiredman21:10:35

% clj -Sdeps '{:deps {manifold/manifold {:mvn/version "0.1.9"}}}' -M -e "((requiring-resolve 'manifold.executor/thread-factory) str (promise) 1000 true)"
#object[manifold.executor$thread_factory$reify__481 0x6d2dc9d2 "manifold.executor$thread_factory$reify__481@6d2dc9d2"]
%
suggests it is something with your dependencies

hiredman21:10:52

something like

(->> (requiring-resolve 'manifold.executor/thread-factory) deref class .getDeclaredMethods (filter #(= (name 'invoke) (.getName %))) (map #(count (.getParameterTypes %))))
will tell you the arities the fn object accepts

hiredman21:10:25

and if the arities it accepts don't match what you see in the source, that means the fn object didn't come from the source you are looking at

fadrian21:10:09

I've checked the versions and, according to clj -Stree |grep manifold, I'm getting: X manifold/manifold 0.1.9 :use-top manifold/manifold 0.1.9 X manifold/manifold 0.1.9-alpha3 :use-top X manifold/manifold 0.1.8 :use-top The ones to the right, headed with the X, are dependencies in the libraries. According to the X prepending the tree node and the :use-top annotation, these should be replaced with manifold/manifold 0.1.9 version that's getting loaded according to its leftmost position. However, the top dependence says to ignore 0.1.9, which seems odd to me. More troubling, when I run your last snippet, it's telling me there's only diadic and triadic versions of this function. So I guess my question is "How do I use my top-level deps.edn to tell the build system to override all local dependencies and use manifold/manifold 0.1.9?"

hiredman22:10:15

I don't have an aot'ed manifold to test on but something like

(-> (requiring-resolve 'manifold.executor/thread-factory) deref class (.getName) (.replaceAll "\\." "/") (str ".class") ((requiring-resolve ')))
will return nil if the function was created from source, otherwise a url pointing to where the class was loaded to on disk (likely in a jar file)

hiredman22:10:05

my guess is you'll find a dependency badly packaged, and including an aot'ed older manifold version

hiredman22:10:38

in which case you are kind of out of luck, if a dependency is including stuff from manifold in their artifact you cannot exclude it

hiredman22:10:26

oh, just noticed the 0.1.8 in the deps tree

hiredman22:10:34

whatever that is

hiredman22:10:41

that is likely the broken thing

hiredman22:10:38

if you find the jar file for in your m2 and use jar -tf you will likely get a listing of files that includes classes from manifold

Alex Miller (Clojure team)22:10:37

re "How do I use my top-level deps.edn to tell the build system to override all local dependencies and use manifold/manifold 0.1.9?" - top deps override all (that's what you're seeing above), so I think you are getting 0.1.9.

Alex Miller (Clojure team)22:10:18

(System/getProperty "java.class.path") would tell you your classpath at runtime and you should be able to support the actual path to the jar there

hiredman22:10:55

unless something is badly packaged

hiredman22:10:20

an uberjar, or aot'ed, and including source or classfiles from their transitive deps

Alex Miller (Clojure team)22:10:26

true, if you've got a dependency jar folded into something else, that would be a problem

hiredman22:10:55

alternatively something could be changing mutating the var

Alex Miller (Clojure team)22:10:10

you might be able to load the resource to the manifold class or source clj and tell which jar it's in

hiredman22:10:18

whatever there are getting in the repl doesn't match the source for 0.1.9 (looking at the arities of the defined invoke methods)

fadrian22:10:09

aleph is using manifold 0.1.9-alpha3. aleph also uses byte-streams 0.2.5-alpha2, which uses manifold 0.1.8. The current version 0.2.5 of byte-streams of course uses version 0.1.9 of manifold. If everything were up-to-date in the version of aleph available from maven, life would be good. But that's just me complaining. In any case, the manifold versions are not in those jar files. Why would tools not use the overriding manifold 0.1.9 from my top-level deps.edn? Or am I misunderstanding how :extra-deps works?

fadrian22:10:56

Or are things just not cleaned properly?

fadrian22:10:02

Is there any way to clear the cache so that I can be sure it's not something there?

fadrian22:10:02

It doesn' matter. I cleared them out manually (can't beat an rm -rf) and the pom files for the aleph and byte-streams keep dragging back the old versions. Can I manually edit the .pom files to point to the new version?

hiredman23:10:00

if the issue is what I think it is then no you cannot do anything about it, but it may not be what I think it is

hiredman23:10:42

if the issue is some other jar files (not the manifold jar) are including either source code or generated classes from manifold, then excluding the manifold dependency isn't going to do anything

hiredman23:10:56

and if that is the case cleaning won't do anything, it is a problem with how the dependency is packaged

hiredman23:10:39

but we don't know for sure if that is the case, you'll need to try and track down where the manifold.executor/thread-factory you have, that doesn't have the right arity, is coming from

hiredman23:10:00

which is what

(-> (requiring-resolve 'manifold.executor/thread-factory) deref class (.getName) (.replaceAll "\\." "/") (str ".class") ((requiring-resolve ')))
I mentioned earlier is for

hiredman23:10:53

interpreting the result is a little problematic, but if it returns nil that means the class was created by evaluating source, and if it doesn't return nil that means it is likely(but not certain) the class was created by loading a class file from disk

hiredman23:10:39

you also just want to look at

(-> (requiring-resolve 'manifold.executor/thread-factory) deref class)
in the repl, if the output is not manifold.executor$thread-factory then likely something is trying to monkey patch the manifold library

hiredman23:10:39

pom files, deps.edn, etc describe dependencies between blobs of stuff (code, resources, etc), and editing poms, exclusions in deps.edn, etc effect that dependency graph

hiredman23:10:34

but if the issue is one of the blobs of stuff A includes copies of data in blob of stuff B, even if you exclude B, you will still get the data from B via A

hiredman23:10:59

because A was badly packaged