Fork me on GitHub

This clears it up well, thanks Sean!


(POST "/emails" [] (fn [req] ...) ; emails-handler as anon fn -- do you mean (fn [req] (emails-handler req)) so that there's an indirection in there? I would just do (POST "/emails" [] #'emails-handler) for that indirection.

👍 3
Dave Nguyen03:10:16

Hi everyone, is there a framework or a set of libraries that can resemble next.js functionality in Clojurescript world?


that's probably a bettter question for #clojurescript but I think reagent is the most popular react wrapper, and last I heard luminus came closest to that pre-bundled out of the box experience (these are the things used in the excellent "web development in clojure" book, which is fairly up to date)


luminus doesn't do cljs / reagent / re-frame out of the box, but can be convinced to add those features via flags

thanks3 3

to start a repl with cursive or calva, it seems i need a deps.edn. I'm trying to tack on clojure to an existing maven project. is there a way to just have the deps.edn simply refer to the local pom.xml for dependencies, or do i need to copy them in to the deps.edn? More specifically I want other modules from the pom.xml accessible in the repl.


If the maven project is a Java project (or other JVM library), I'd suggest creating a separate Clojure project and include the jar from the maven project as a dependency on the Clojure project.


@joel380 I would sort of expect Cursive to know about pom.xml files, given that it's a Java IDE under the hood, but in general for Clojure projects, yes, you'll need a deps.edn that has the same :deps {...} content as the pom.xml file.


At least once you've done that, you can keep deps.edn as the source of truth for the project and re-sync the pom.xml <dependencies> from deps.edn via clojure -Spom.

👍 3

if cursive can figure out from pom.xml i'd prefer that... i'm having trouble figuring out how to specify a pom module in the deps.edn (specifying 3rd party no issue, but a submodule doesn't seem to work)


Ah, yeah, you can't use BOM-style deps in Clojure -- you have to specify all the individual components directly.


@joel380 if you can start the repl some whatever way, then you can connect to it with Calva. You'll need to provide the cider-nrepl dependencies to the running app to get things like peek definition and completions going.


(for [x *command-line-args*] (print x)) If I save above two lines to pa.clj and run. $ clj pa.clj Clojure is even more fun , why it does not print anything to console ?


if I put (println **command-line-args**) above for, it prints the list of args.


for is lazy


use doseq instead


I have a map in a -> thread, with multiple assoc/updates. I want to assoc/update a key using the value of another key from object being threaded , is there an idiomatic way that keeps it thread-like? Here's an illustration:

(-> {:foo 10}
    (update :foo inc)
    ;; A better way to do this while keeping it inside the threading macro
    ((fn [m] (assoc m :bar (:foo m)))) 
;; => {:foo 11, :bar 11} 


@dromar56 I would maybe use as-> for this:

(as-> {:foo 10} $
  (update $ :foo inc)
  (assoc $ :bar (:foo $)))


but probably I would just use let and not try to force everything in a thread. let over thread :)


I usually use as-> inside -> :

(-> {:foo 10}
    (update :foo inc)
    (as-> $ (assoc $ :bar (:foo $))))


I think that's the "cleanest" way in my use case, thanks

Jim Newton10:10:41

I have yet another question about semantics of Java classes and interfaces. The class clojure.lang.ISeq is an interface, and the class java.lang.Number is an abstract. Does this mean it is possible to have an object with is both a sequence and a Number? If not why? More precisely, according to refl/type-reflect we see that clojure.lang.ISeq has flags #{:interface :public :abstract} and we see that java.lang.Number has flags #{:public :abstract} . I am currently ignoring the fact that these are :public, does that matter?


> Does this mean it is possible to have an object with is both a sequence and a Number yes


user=> (def x (proxy [java.lang.Number clojure.lang.ISeq] []))
user=> (number? x)
user=> (seq? x)


Possible surprises me, as it is definitely unusual. But apparently there are no JVM rules against it. Interesting.


definitely unusual, not completely unreasonable. but if you ever needed to do something like this, you'd make it a seqable number, not a seq and a number


(i.e. implement clojure.lang.Seqable not clojure.lang.ISeq )


Sure, like a JVM String, making a number seqable would perhaps make sense in order to get a seq of its decimal digits, or something like that.


or the number as a list of bits


or the number as a range


not that I would ever do it like this


but I've seen it done in the wild (not in clojure tho)

Jim Newton11:10:46

I was thinking that the justification might be that someone could create a number which is a finite sequence of digits or of bits.


That might be a reason someone would want to do it, but I suspect that there are very few reasons that the JVM disallows a class from implementing two JVM interfaces, or disallows extending a JVM abstract class and also implementing any number of JVM interfaces, whether doing so in any particular case makes sense to a person or not.


Here is an article I found from a quick Google search, since I don't feel like reading the Java Language Spec to see what it says on which combinations are legal, and which are disallowed by the JVM:


you can represent average as both number and sequence


e.g. “loss-less” average produced from n numbers, which is a number with a metadata about it’s source


the JVM disallows a class from implementing two JVM interfaces


@andy.fingerhut the only thing that the JVM disallows is extending multiple classes


you can implement as many interfaces as you want


I put in a thread to my previous message one article showing a case where trying to implement two interfaces with same method name but different return value type seems not to be allowed, so there are at least a few rules restricting some kinds of combinations, if that article is correct.


right but that doesn't have much to do with the class implementing two interfaces


it has to do with the JVM disallowing a class to implement methods with polymorphic return types


agreed that absent such kinds of corner cases, a class can implement an arbitrary number of interfaces

Jim Newton12:10:40

if you're willing to accept numbers as ranges, then it would also make sense to consider matrices and vectors as numbers. However, it seems this is not the intent of java.lang.Number . Rather a number seems to be something which implements the following members. In particular numbers have an integer value and a double value, which range does not but, a bit stream representing the bit representation does. It appears from this definition that a bignum would not qualify as a number.

{:bases #{java.lang.Object},
 :flags #{:public :abstract},
 #{{:name byteValue,
    :return-type byte,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public}}
   {:name java.lang.Number,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public}}
   {:name floatValue,
    :return-type float,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public :abstract}}
   {:name longValue,
    :return-type long,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public :abstract}}
   {:name shortValue,
    :return-type short,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public}}
   {:name serialVersionUID,
    :type long,
    :declaring-class java.lang.Number,
    :flags #{:private :static :final}}
   {:name intValue,
    :return-type int,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public :abstract}}
   {:name doubleValue,
    :return-type double,
    :declaring-class java.lang.Number,
    :parameter-types [],
    :exception-types [],
    :flags #{:public :abstract}}}}


For what reason do you say that it appears from this definition that a bignum would not qualify as a number?

Jim Newton12:10:09

@andy.fingerhut I'm just trying to understand, but it seems a bignum cannot have a member intValue of type int , as the largest int is 2^64 - 1, whereas the largest bignum is larger than that.


Those methods are allowed to return useless values in such cases.


or, e.g. the least significant N bits of the bignum value


In most cases I know of where they can lose information, they do so in a fashion that is at least semi-useful.

Jim Newton12:10:18

hmmm. useless values, I laughed when I read that. 😉 I wouldn't have guessed that to be the case


It wasn't a carefully thought out answer -- the later ones are closer to my opinion 🙂


If those methods cannot lose information, then the only type that can implement all of those methods at the same time is an 8-bit integer, because byteValue is in the list of methods.


There are at least a few syntactic rules that the JVM enforces about which classes can be sub-classes of others, and which combinations of interfaces that a class can implement, but except for those, the language itself makes no restrictions. That allows a lot of combinations that a person might never think to do, because they have little or no utility.


Trying to look at method names and determining from the method signatures alone which combinations are allowed and which are not is going on too little information.

Jim Newton17:10:33

Actually the question I'm trying to solve is given two java classes (which might be :abstract, :interface, or :final) decide whether the set of all objects of one class intersects with the set of all objects of the other class. For example the set of Serializables intersects the set of Comparables. However the set of Strings is disjoint from the set of Integers.

Jim Newton17:10:25

Currently my rule is that if both neither class is final, and neither is explicitly Object, then they are not disjoint.

Jim Newton17:10:57

But in light of this recent discussion, there are times when two interface classes are necessarily disjoint.


If you ignore Java interfaces, then the answer is true if one class is a sub-class of the other (either directly, or through a chain of sub-class relationship).


If you are trying to answer the question "given to java classes, is there an interface that they both implement?", that is also pretty straightforward to answer, but I doubt that is the question you are trying to answer.

Jim Newton17:10:12

yes if 1 class is a subclass of another they are not disjoint, that's evident


I should mention that in the JVM, an interface will show up in the output of the Clojure reflection API whose output you copied and pasted a sample above as having the keyword :interface there. In Java source code and documentation, there is usually a sharper distinction made between Java classes versus Java interfaces than that.


That is, for something that has the :interface keyword in it, most people familiar with Java would never call that a class. They would call it an interface.

Jim Newton17:10:32

no, the question is given two classes are they disjoint? where classes includes any non-nil value which find-class returns. for which class? returns true.


If you want to know for two interfaces, are there any two JVM classes that both implement that interface, then the only way to check that, that I know of, is to somehow iterate over all class definitions and check whether there are two of them that both implement that interface.


The JVM is dynamic enough in creation of classes and interfaces at run time, that the answer can change over time. e.g. Clojure's compiler creates new classes when you eval a defn form, and some others.

Jim Newton17:10:28

By class I mean any value for which class? returns true


There are libraries I have searched for, and found, that attempt to iterate through all classes currently defined in a running JVM, but there are many caveats in how complete they can be.


class? returns true for things that most people call Java interfaces, as well as things most people call Java classes.


classes always form a tree structure of sub-class relationships, with java.lang.Object at the root.

Jim Newton17:10:48

exactly. I am calling those classes, inkeeping with the class? function

Jim Newton17:10:54

what does java call the set of all classes union with the set of all interfaces?


I don't know if that has a name.


Every JVM object has one and only one class, determined when it is created.


Such a class is never a Java interface.

Jim Newton17:10:40

for my purposes I don't care about the distinction. But to be sure, anytime I talk to a java person they get upset about this.


I am not upset. I am trying to determine if the question you are trying to answer makes sense or not.


It might not be a well-formed question.

Jim Newton17:10:14

😉 yes you're cool headed.

Jim Newton17:10:59

but the people in my office always start talking about the differences in the two, and they've never convinced me that it is something I need to care about.


You said the question you are trying to answer is this: "Actually the question I'm trying to solve is given two java classes (which might be :abstract, :interface, or :final) decide whether the set of all objects of one class intersects with the set of all objects of the other class."

Jim Newton17:10:36

Ok. let me test a definition on you.


Suppose I pick two Java interfaces, I1, and I2, as the Java classes you want to ask and answer that question about.


So the more restricted question becomes: "given two java interfaces I1 and I2, decide whether the set of all objects of one class intersects with the set of all objects of the other class"


What do you mean by "the set of all objects of class I1"? Do you mean the set of all objects, each of which is a particular non-interface class X, does X implement class I1?

Jim Newton17:10:24

now set s1 = the set of all objects, whose class has l1 in its ancestors list union with the singleton set containing the class itself.

Jim Newton17:10:41

now set s2 = the set of all objects, whose class has l2 in its ancestors list union with the singleton set containing the class itself.

Jim Newton17:10:59

is s1 guaranteed to be disjoint from s2 ?

Jim Newton18:10:32

or is it possible that those sets have a non-empty intersection?


So here is how I might restate that question, which seems like it might have the same answer as your question. Given interface I1, are there two Java classes C1 and C2, such that C1 implements I1, and C2 implements I1?

Jim Newton18:10:05

Currently my code concludes that if l1 and l2 are interfaces, then it is not guaranteed that s1 is disjoint from s2.

Jim Newton18:10:35

no, that's not the same question, but it is indeed similar.


OK, my question was not as close to yours as I thought. Let me try again.

Jim Newton18:10:49

can I restate your questionb ack to you?


I am pretty sure now my question is different than yours, but you are welcome to go ahead.

Jim Newton18:10:37

change "are there two java classes" to "are there guaranteed to never be two java classes"

Jim Newton18:10:46

either now or at some point in the future


It is always possible to define new interfaces and new classes at run time in the JVM, barring some kind of JVM security manager restrictions or running out of memory. Definitely not the common case if you are running Clojure.


So it depends upon what you are willing to allow happening in the future.

Jim Newton18:10:10

every time I define a clojure Record I define a new java class, right?

Jim Newton18:10:18

at run time.


If you are allowing the future to define new classes, except under fairly unusual circumstances of two interfaces having very similar incompatible method signatures, it is always possible to define a new class that implements both interfaces.

Jim Newton18:10:18

so my code is currently supposing that it cannot guarantee interfaces are disjoint.


whether it makes sense to a human to productively do so is an entirely separate question

Jim Newton18:10:18

The new information from our discussion today leads me to believe that I can tighten that restriction. If two interfaces declare the same member with two different?/incompatible? types, then we know the interfaces are disjoint regardless of future events.

Jim Newton18:10:56

indeed, i'm not asking what is useful, I'm asking what is possible

Jim Newton18:10:10

my program does not judge the usefulness of your application


So continuing in my habit of distinguishing classes vs. interfaces, you cannot guarantee interfaces are disjoint. Similarly, if you allow the future to define new sub-classes C2 of a given class C1, you cannot guarantee that an interface I1 and a class C1 are disjoint, unless C1 is final (in which case sub-classes of C1 cannot be created).

Jim Newton18:10:39

agree. however, Object IS FINAL


It isn't clear to me whether knowing this disjointness property of two Java classes/interfaces is, but I imagine you have ideas on that.

Jim Newton18:10:57

I'm not sure if that's a one-off or whether there are other final classes we can inherit from.


The class java.lang.Object is final?

Jim Newton18:10:43

correct me if I'm wrong


If it were, then no other classes could be created that are sub-classes of it, and the class hierarchy would consist of only that class java.lang.Object


unless I am completely misremembering the meaning of final as it applies to Java classes...

Jim Newton18:10:02

I think you're right. it is :public but not :final and not :abstract. I was remembering wrong

Jim Newton18:10:57

clojure-rte.rte-core> (:flags (refl/type-reflect Object))


no worries. If it is any consolation, I was quite surprised to learn that the Clojure and Java reflection APIs treat interfaces as classes with an extra :interface attribute.


(year ago, when I first encountered that fact)

Jim Newton18:10:38

I may have to revisit my code to see what I'm doing with classes which are not :final, and not :abstract, and not :interface


Is it easy to explain why you want to prove whether two classes are disjoint?

Jim Newton18:10:50

let me try in one paragraph.


Given how common it is for Clojure functions to care more about the interfaces that an argument or return value implements, rather than its Java class (often called the concrete class in discussion you and I have had in the past couple of days with others), as soon as you associate a Clojure parameter or return value as "should implement this interface", it becomes possibly-overlapping with almost every other object that implements another interface.

Jim Newton18:10:14

Imagine you have a regular expression, not of characters but of types. e.g., (:* Number) means a sequence of 0 or more numbers.


And actually, my earlier statement about two classes being related as either one is a subclass of the other, or not, if they are not final, then a future sub-class of both of them could implement a common interface, so they cannot be guaranteed disjoint either

Jim Newton18:10:54

(:* (:cat Number String)) is a sequence of alternating Number followed by String. 0 or more of them


So the only things you could ever guarantee disjoint are two final classes that don't implement a common interface I1, or maybe a handful of other cases.


stopping talking now, and reading instead, sorry .... 🙂

Jim Newton18:10:59

I think any two final classes are disjoint. There is never an object (as I understand) which is an instance of two different final classes. correct me if I'm wrong.


but if those two final classes both implement interface I1, do you consider them disjoint, or not?

Jim Newton18:10:06

they are disjoint. The set of objects of class-a is disjoint from the set of objects of class-b, even if the classes share an interface.

Jim Newton18:10:23

if class-a and class-b are both final

Jim Newton18:10:33

and class-a != class-b

Jim Newton18:10:00

note that String = java.lang.String

Jim Newton18:10:18

back to the discussion

Jim Newton18:10:00

(:* (:cat Number (:or String (:not Long))))

Jim Newton18:10:30

represents the set of squences of Number followed by either a String or a Number which is not a Long.

Jim Newton18:10:32

etc etc etc.

Jim Newton18:10:44

These can be combined infinitum.


So Clojure spec has similar regex-of-objects-satisfying-predicates kind of capabilities, so this looks familiar in that sense

Jim Newton18:10:26

they so-called regular-type-expression can be represented as a deterministic finite automaton, whose transitions are disjoint types.


It is only trying to do dynamic checks so far, no kinds of static analysis that I am aware of, although a few people have investigated in that direction.


OK, I see where the disjointness guarantee can come in there

Jim Newton18:10:09

if any state in the finite automaton has two transitions labeled with types which fail to be disjoint, then the automaton fails to be deterministic


If you want to avoid nondeterministic finite automata

Jim Newton18:10:35

if the automaton is determistic then we can answer lots of questions which we cannot answer if it is non-deterministic.

Jim Newton18:10:10

For example, we can ask is there a sequence which is recognized by two different automata


I mean, every non-deterministic finite automata can be transformed into a deterministic one, but since there is a potential exponential size blowup in the number of states, that fact is often not a practically useful one.

Jim Newton18:10:49

also given a sequence, we can ask "does it match the regular-type-expression" and we can perform the computation in LINEAR time WITHOUT backtracking

Jim Newton18:10:22

correction: every finite automaton over a finite alphabet.

Jim Newton18:10:34

the set of types is infinite.


sure. I often forget there are people studying infinite alphabets, as they seem a bit fairy-tale in practical uses, but I suppose in this case their use seems a bit more realistic.

Jim Newton18:10:00

the set of clojure objects is an infinite set. so if you wanted to make a finite automaton to recognize arbitrary sequences you'd need infintite alphabet and consequently infinitely many transitions.

Jim Newton18:10:35

my research handles a special case of infinite alphabets,


I know regular expressions can be transformed into a nondeterminstic finite automata (with epsilon transitions) with a linear number of states and transitions, in the size of the regular expression (at least for finite alphabets), but I thought that sometimes the deterministic finite automata was exponential in the size of the regular expression?

Jim Newton18:10:55

alphabets which can be partitioned into recursively-enumerable sets. (edited) where the membership function is always deciable.

Jim Newton18:10:37

So I have imposed a type-calculus over the Java type system, written in clojure.

Jim Newton18:10:08

a student of mine is writing the same thing in Scala

Jim Newton18:10:31

my PhD thesis was implementing this in Common Lisp, which already was equipped with such a type system.

Jim Newton18:10:42

so I didn't have to implment the type system.

Jim Newton18:10:15

I hope that sheds some light on my wierd questions all the time...


I found a reference to this Wikipedia article which might back up my belief that there are regular expressions whose minimal DFAs have exponential size blowup:


It does, and Rich Hickey when giving an early talk on Clojure spec mentioned the future possibility of comparing specs to each other as a function's API changed over time, to determine subset relationships between Clojure specs.


To my knowledge, no one has implemented such a thing yet.


If you are curious for that quote in context, search for "solved problem" in this talk transcript:

Jim Newton18:10:54

questions of inclusion (subsetness) and disjointness can always be answered with RTE (regular type expressions) provided the questions can be answered on the leaf level types.

Jim Newton18:10:22

do you know Rich?


I knew that there were algorithms for doing this over finite alphabets, but had not considered whether infinite alphabets, or Java interfaces vs. classes, changed how practical such decision algorithms might be.


Not personally, no. We have had the occasional on-line and in-person interaction, but probably only a few hours over 10 years.

Jim Newton18:10:28

i'm sure there are issues/problems/errors in my theory and implementation. But I'd like to get it into a state where I can release it or publish it.


So does the inclusion problem remain solvable if you don't know the disjointness properties of pairs of types?

Jim Newton18:10:42

one important property (at least I think it is important) of my type system is that it is extensible. The hope is to create a type which represents the set of all objects which match a given spec

Jim Newton18:10:46

but I don't yet know how to do that


Or maybe a more practically useful question for software developers: suppose you consider a collection of classes and interfaces that are defined right now, and leave out of consideration possible future classes and interfaces that might be created. That seems potentially more practically useful to answer such inclusion questions for RTEs, if the answer for most pairs of interfaces is "can't be proved disjoint" (when arbitrary future classes are allowed)

Jim Newton18:10:49

Good question indeed. I believe the answer is yes. Why, given two types A and B, if you don't know their subtype relation nor their disjoint relation, then you can partition (union A B) into {(and (not A) B) (and A (not B)) (and A B)}. but any of those three might be empty, we just don't know whether it is.


There are a whole bunch of Java classes that aren't final, and imagining that some one some day might create a sub-class that implements some interface somewhere, isn't a practical future possibility.

Jim Newton18:10:23

such a partition causes the automaton to be larger and potentially have transitions which will never be taken. For example, if A is a subset of B, then (and A (not B)) is always false.


And if people knew that a system answered certain questions for the current set of classes and interfaces, and you gave them an example of how the answer might change if certain classes were defined later, they could either re-run the algorithms later as the software changed with new class/interface definitions, or at least use their judgement about the software to determine how likely such future classes would be.

Jim Newton18:10:58

you make a good point. a practical issue I'd like to avoid is that someone loads my code, then loads his application using it. that user application invisibly builds an automaton which assumes interface-A and interface-B are disjoint (because no class currently inherits from both).

Jim Newton18:10:46

then the user loads a third library which declares a class which inherits from both. That means the automaton is now invalid. and there is nothing to trigger it to be re-computed.


Right, hence my phrase about informing them about the assumptions on which the answers are based, and what could change them in the future.


Many software developers can understand and handle such qualifications, especially with examples.


I am just wondering if your type system, and how common it is for Clojure functions not to rely on particular Java classes, but on Java interfaces, will commonly lead your RTEs to never be deterministic, or only rarely.

Jim Newton18:10:12

its a good point. i'm open about it. but cautious because i'm not sure the gain is worth the risk.


There are a lot of commonly used Clojure functions that take something that implements clojure.lang.ISeq, and returns something implementing that, for example.


but make no promises about which classs

Jim Newton18:10:36

in my system they are always deterministic, but may contain unsatisfyable transitions.

Jim Newton18:10:32

any state in the automaton has some number of transitions leading to that number of next states.


if your system says "clojure.lang.ISeq is an :interface, therefore is not provably disjoint from something implementing another thing that is an :interface, therefore I can't say much useful about it", then I suspect most Clojure functions will lead to that situation.

Jim Newton18:10:54

the transitions are labeled with disjoint types (some of which might be empty types).

Jim Newton18:10:59

I compile the transitions into a data structure sort of a decision network which is guaranteed to never make the same type test twice.

Jim Newton18:10:46

so given an object at runtime I ask whether it is Comparable, and then possibly whether it is serializable, then possibly whether it is a Number etc.


I've got to go do some other work for a bit, earn my pay and all, but been interesting chatting. I would be interested to see what kinds of results you end up with.

Jim Newton18:10:16

and depending on the answers to those questions transfer control to the appropriate next state.

Jim Newton18:10:49

Thanks for the chat. Can you suggest the best way for me to present my code to the public. There are no conferences anymore


especially if it results in code I could run that compares two specs (or a useful subset of them) to see if one is a subset of the other.


If you had something that could be used to answer that kind of question about Clojure specs, I suspect the Clojure conj conference might be interested, although it is more developer-focused and less theoretical. I do not know which of the academic conferences might be interested, but suspect there would be some -- perhaps. you know better than I. I thought several conferences were still having on-line presentations?


Only a subset, I know, but still some.

Jim Newton18:10:39

many thanks for your valuable feedback, especially finding that weird bug with ldiff

Jim Newton18:10:02

i'm going to eat supper now

Eric Ihli12:10:36

I have some data that's 144mb in EDN. It's taking a significant amount of time to serialize/deserialize (60+ seconds as EDN). I'm trying Nippy now. Nippy appears to have serialized it just fine with nippy/freeze-to-file, but when I try (nippy/thaw-from-file) I get an error: GC overhead limit exceeded. It also churns 100% CPU for over a minute before throwing that error. I thought Nippy was faster to deserialize than EDN, so this is surprising. My assumpsions are that 144mb is not an unreasonable size and Nippy should be faster than EDN and it shouldn't throw this error. Are my assumptions wrong? If my assumptions are right, any thoughts on how to troubleshoot?


I haven't used Nippy, so don't have any information for you specific to that library, unfortunately. Creating an issue on its Github repository, and/or looking through existing issues, might glean some useful information.


You can try increasing your JVM's max heap size with a java command line option such as -Xmx1g to see if that lets it go through for you, but if it is a bug where the library is trying to allocate an unlimited memory, it would only lead to GC overhead limit exceeded later, rather than earlier. Still, it might be a useful bit of information in your investigation.


There is a commercial tool called YourKit Java Profiler that has a 15-day free trial version, and I believe free licenses for use on open source projects, that can attach to a JVM started with a couple of extra command line options it gives you, that can analyze the set of allocated objects in various ways, e.g. by class, sorted by most memory occupied, that might help. There are free tools for doing that, too.

Eric Ihli13:10:19

Thanks for the ideas. I think I found it in a closed issue from last year.


Yeah my first guess is what andy said -- increase the jvm heap size via the command line option or in your project.clj ... but I don't know .. maybe Nippy is not paging data ... but isn't that supposed to be the job of the JVM?

Eric Ihli14:10:43

Hmm. I don't think the heap size is the problem. It might be "a" problem. But it's not "the" problem. The EDN file is ~144 mb and gets processed by read-string in about a minute. Nippy is supposedly faster. Nippy serializes the EDN in about 10 seconds to about 80 mb. That all seems reasonable. It just blows up when desrializing. Maybe something specific about the type of data. I'll open an issue.

Eric Ihli15:10:04

I tried a few more things trying to make this easily reproducible and I think I've narrowed it down to something specific about the 140mb of edn data that I have. If I generate similar-looking data, I don't experience any issues.

Eric Ihli23:10:08

An update for future readers: this appears to be a heap size issue. Confusing, because I never saw my used heap approach max heap, and it would usually churn indefinitely withut ever throwing an OOM exception, and I was only reading into memory from a few files of < 150mb and didn't expect (incorrectly, I now presume) that process to require 4+ gigs of heap. But running my repl with -Xmx6g let me deserialize the file that was previously causing issue.

Jim Newton12:10:36

w.r.t a class implementing two different interfaces, is it possible to look at the members of two interfaces in search of contradictory type definitions. For example if one interface requires member foo to have type int and another interface requires a member of the same name to have type double, does that mean it is impossible to have a class which lists both interfaces in its ancestors set?


I sent to myself recently above, linking to an article that seems to say "yes" to your question:


It looks like some Java developers run into this restriction when they have actual desires to have a single class implement multiple interfaces with conflicting method signatures, and find workarounds for that:


That StackOverflow answer shows that the C# language has a way to disambiguate which method you are implementing, by also supplying the interface name in the definition of the method. Java does not have that.

Jim Newton12:10:05

I started looking into this question of whether interfaces are disjoint. and now I don't know how to figure out whether two members are the same.

Jim Newton12:10:19

apparently a java class (or interface) can have multiple members of the same name. So when two different interfaces both have a method of the same name, they might be a conflict or not. Is it the arity that determines whether two members of the same or different, or is it both the arity and somehow the parameter types? For example, java.util.List has many methods named of. Three of which are:

{:name of,
  :return-type java.util.List,
  :declaring-class java.util.List,
  :parameter-types [],
  :exception-types [],
  :flags #{:public :static}}
 {:name of,
  :return-type java.util.List,
  :declaring-class java.util.List,
  :parameter-types [java.lang.Object<>],
  :exception-types [],
  :flags #{:varargs :public :static}}
 {:name of,
  :return-type java.util.List,
  :declaring-class java.util.List,
  :exception-types [],
  :flags #{:public :static}}
if a second interface has a member named of with different arity, I suppose it is NOT a conflict. If it has the same arity and same types, I suppose it. IS a conflict. But if it has the same arity, and related types, such as types which are all subtypes or all super types, is that allowed?


I don't know the precise rules for conflict off the top of my head, but I suspect that even methods with the same name will not arise very often in interfaces defines in Clojure's implementation, since those interfaces were all created by a single author.


I'd recommend for now throwing an exception, or returning some kind of "unknown-answer" result, if you actually come across two interfaces where you don't already know the rule.


It is arity and the parameter types, but I don't know the rule Java uses for distinguishing parameter types when they are both classes or interfaces.


You could also ask in the #java channel here on Clojurians Slack, and someone else may have a good reference to an official answer, or know it.

Jim Newton14:10:11

I experimented with one of my students. It seems that if two interfaces have a method with the same name and the same input parameter types, it is an error, regardless of the return types. That's pretty easy to check.

Jim Newton14:10:06

e.g. interface Ia can have method m(int) and interface I2 can have method m(double) and there's no conflict.

Jim Newton14:10:47

but if they both have a method m(int) there's a conflict

Jim Newton14:10:43

cool! I didn't know there was a #java challenge on Clojurians


For what reason do you say above: "It appears from this definition that a bignum would not qualify as a number."


super duper om computer


I am having a project using Clojure but there is Java in it as well. How do I compile the java with clojure tools?


@timok Are you using Leiningen or Clojure CLI? If the latter, the answer is: compile the Java manually, put the classes on your classpath. Or put the Java in its own project and build a lib JAR you can depend on from Clojure.


Leiningen has a facility to compile Java code and incorporate it into your Clojure project.


mvn compile then?


(but it shells out to javac so it's not exactly magic)


Correct, it is clojure cli tools


I mean clojure cli tools create a pom anyway so I use maven correct?


the cli tools have no opinion or restriction on how you use java


you could use an ant project and import a jar made via deps.edn for example


but the fact that deps.end can make a pom (and uses the maven ecosystem regardless) does make using maven convenient


ok thanks to you both! Have it working.


Is there any sanctioned way to make new reader macros in clojure?

Alex Miller (Clojure team)19:10:01

for data, you can use tagged literals


It seems like tagged literals (understandably) cannot expand like macros, is this correct?


(Basically I'm trying to do this in clojure: )


(Ideally without having to fork the language anyway)


@teodorlu It looks like tagged literals can hold values, but not expressions (or macro references), yes?


Basically, I'm trying to do this in clojure:


this is really neat! I thought there something familiar and I realized I had also seen your talk, "Movies as Programs". I think there's a lot of interesting approaches that have yet to be tried.

Alex Miller (Clojure team)19:10:45

I mean they could theoretically hold code (which is data) and do something with that but I'm not sure you'd be happy with where you'd end up.


Okay, another option then is a regular (textual) macro, whose invocations can be turned into source positions and let the IDE hide them.


That 'might' be doable with attached metadata....maybe...I'll have to think about that.


Actually, ya, that would there any way to 'tag' arbitrary expressions in clojure?


Like you can with functions.


Maybe something like (func ^meta-info args)?


Its okay if func has to be a macro and can even specify that it requires ^meta-info


Basically I'm just looking for something an IDE can hook into.


You can attach metadata to any var:


@teodorlu In this case though, I'm looking to attach the metadata to the variable's use, rather than just its definition.


I see. Not sure how you'd achieve that.


Alright, thanks.


I'll look into this more.


Thanks. 🙂


I do admit I am sort of stretching the language here, although I would like to avoid contorting the language too much... 😉

Alex Miller (Clojure team)19:10:58

if func is a macro, you don't metadata there - you can just pass whatever and the macro can do whatever it wants with it


I think I'm starting to see how I might go about doing this.

Louis Kottmann21:10:23

hello, can anyone explain to me why regexes as keys get duplicated in a map? i.e:

myapp.core> (assoc (assoc cmds #"cake" "hello") #"cake" "hello")
=> {#"cake" "hello", #"cake" "hello"}
myapp.core> (assoc (assoc cmds "cake" "hello") "cake" "hello")
=> {"cake" "hello"}

Louis Kottmann21:10:50

should I avoid using regexes as keys if I want them to be unique within the map?

Alex Miller (Clojure team)21:10:16

regexes don't compare for equality

Alex Miller (Clojure team)21:10:23

one workaround is to use the regex string instead (which has pros and cons)

Louis Kottmann21:10:07

sooo, I should avoid them for this use?

Alex Miller (Clojure team)21:10:16

well they won't work well as keys in a map or values in a set for sure

Alex Miller (Clojure team)21:10:49

this is one of very few things in Clojure that don't have equality as value (function instances being another)

Louis Kottmann21:10:12

mmmh when I try:

(= #'message #'message)
=> true


that isn't a regex

Louis Kottmann21:10:36

do you mean 2 same functions with same name, arguments and body don't compare?


function equality is undecidable

Louis Kottmann21:10:43

that makes more immediate sense to me than regex equality

Louis Kottmann21:10:02

as 2 same regexes casted to String should be comparable on their String representation

Louis Kottmann21:10:12

but I'm probably missing something


regex equality is not the same as a regex's representation equality

Louis Kottmann22:10:17

Two regexes could have the same string representation ?


no, if they have the same representation then they are the same regex


but two regex with different representation could be equal


so equality of representation is not equality of regex


in general regex equality is decidable, but since most regex implementations (like java's) have extended operator support (e.g. backreferencing), the complexity of equality is at least NP-complete, and I believe possibly PSPACE

Louis Kottmann08:10:19

I see, I just do not understand how the basic case of 2 regexes written exactly the same way are not equal. The problem of finding out if 2 regexes written differently evaluate to the same string rules is indeed very hard and I would not expect that equality to work out of the box ^^


(= #"message" #"message")


that isn't a function


#' is var quote


it means give me the var with this name


they are sort of functions, because you can invoke vars as functions, similar to how you can call keywords or collections as functions


but they are not function objects


and to say regexes don't compare for equality is not entirely accurate, you can call = on them, just like you can call = on function objects


but they have identity equality, not value equality

Alex Miller (Clojure team)21:10:39

they compare with identity is what I meant


basically you are comparing pointers

Louis Kottmann21:10:26

I'll try a few things on the REPL to play with regexes equality

Louis Kottmann21:10:38

but I'll remove them as keys of the map and find a better way

Louis Kottmann21:10:56

thanks for the info, this is a nice community


there's a very simple logic to it. if they are the exact same object they are equal, else they are not equal


every usage of a regex literal creates a different regex object, and different regex objects will never be equal regardless of the value of the regex


#"..." being a regex literal


it is a kind of interesting connection between regexes and functions, because they can both define something that produces the same result in more than one way, and we tend to think of equivalence for them as based on their results


Is there any good way to get source locations out of read?


(And if not, is there a better way to read a file and get source locations?)


if you use a clojure.lang.LineNumberingPushbackReader with read it will add position metadata to forms it reads (certain forms cannot have metadata attached)


check out (source source) clojure.repl/source

hiredman21:10:07 (a clojure reader written in clojure) is a thing that also exists

Eric Ihli21:10:28

This is an image of me starting a REPL session in Emacs/Cider, loading a namespace, letting it sit around for ~30 minutes, then calling (def data (read-string (slurp "resources/some-data.edn"))) some-data.edn is only 140mb. Can anyone explain why my heap jumps by almost 2gb when slurping/reading a 140mb edn file?

Alex Miller (Clojure team)21:10:58

well obviously, it needed 2 gb of objects to read that file

Alex Miller (Clojure team)21:10:43

whether it's 2 gb of data is a separate question - you might have a lot of garbage that can be collected (or you might not). hard to say without knowing what the data is

Alex Miller (Clojure team)21:10:14

or invoking gc at this point and seeing what the used heap is afterwards

Eric Ihli22:10:28

Ah. I see. I assumed 140mb of edn would take up ~140mb of memory. I'm gathering now, that's not the case. nil might get encoded as edn to a certain number of bytes but take up a different number in memory. Same thing with datetimes, etc... And in the process of coercing, a bunch of intermediate objects get created as well.

Alex Miller (Clojure team)22:10:54

All objects have overhead of on the order of dozens of bytes


Clojure strings are JVM strings (in Clojure/Java), and tend to take about 40 bytes per string overhead, plus the storage for the characters in the string, which is often 8-bits per ASCII character if a string is all ASCII, but 16-bit per character if any of them are not ASCII


^ What Andy said above - especially for very short strings you can have a lot of overhead when compared to their "size" in the file. Also hashmaps has a lot of overhead per element:


@UJP37GW2K Here is a transcript of a Clojure session that you can try out on your own machine for analyzing the in-memory size of Clojure data structures compared to their EDN representations, for small parts of your Clojure data:


It averages about 10 bytes of memory per character of EDN on the JVM version I tested with. There is a utility called cljol I wrote that you can see example use of in the file I linked that is good for showing the JVM objects and sizes for small Clojure collections.


That 10 bytes of memory per character of EDN is for your data file -- it could vary for different classes of data structures represented in EDN. For example, long keywords with many repetitions of the same keyword will share the storage of that one keyword name in Clojure/JVM.

💯 3
Eric Ihli12:10:51

Thanks everyone. This has been a huge help. Great to have cljol in my toolbox!


Okay. Do any of these also work in clojurescript, or are they exclusive to clojure?


tools.reader works in both clojure and clojurescript


@bronsa cool, thanks


Cool. Although these only seem to give line numbers and not also column numbers, is that correct?


it gives both


Ah, okay, thanks.


the data is attached to the object as metadata (if possible)


but you can also ask the reader object manually for line/col/file info before/after reading


AHHH, okay, thanks.


(Sorry, still a bit new to the clojure docs. Thanks again.)