Hi Clojurians, I'm having issues with clojure.tools.namespace.repl/refresh and jdk versions > 8, I'm using Amazon correto JDK and was able to confirm that clojure.tools.namespace.repl/refresh doesn't work in any correto version > 8: • Amazon correto 8 -> working • Amazon correto 11 -> not working • Ubuntu OpenJdk 11 -> working • Amazon correto 16 -> not working Iv also found https://clojure.atlassian.net/browse/TNS-54 which seems to have been resolved a while ago, any idea what the issue can be? Thanks

“Doesn’t work” is not too helpful

“I do x and expect y but see z” would be more useful

Does the code load without reloading on those different jdks?

Hi @alexmiller happy to add more details 🙂 iv used this project https://github.com/re-ops/re-share and ran the following:

$ git clone 
$ cd re-share
$ lein repl 

; In cases it did work the expected result (Amazon correto 8 and Openjdk 11)
re-share.config.secret=> (clojure.tools.namespace.repl/refresh)
:reloading (re-share.oshi re-share.wait re-share.core re-share.zero.keys re-share.config.core re-share.schedule re-share.es.common re-share.encryption re-share.config.secret re-share.spec re-share.zero.common re-share.log user re-share.zero.events)
:ok

; In cases it didn't work (Amazon correto jdk 11/16)
re-share.config.secret=> (clojure.tools.namespace.repl/refresh)
:reloading ()
:ok

$ lein deps :tree | grep namespace

....
 [org.clojure/tools.namespace "1.1.0" :scope "test"]

lein deps :tree | grep classpath 
...

   [org.clojure/java.classpath "1.0.0" :scope "test"]

The above did re-produce for me in one other project

I don't using reload a ton, but I have used it on a number of different correto installs just fine, and it isn't a particularly complex thing,

So my guess would be the behavior is the result of something else

Hi @hiredman did it work on versions 11/16?

I don't mind trying a clean project from scratch it could be that I'm missing something

have you tried it having performed set-refresh-dirs beforehand? that way it won't try traversing the classpath

(set-refresh-dirs is very recommendable for other reasons anyway)

My immediate guess is it has something to do with the user.clj you have in dev/ without listing dev/ in the project.clj anywhere

But it is tricky, with lots of lein plugins, and potentially more in your profile.clj to untangle this kind of thing

So the first thing is to disable all the plugins, both in the project and any user level stuff. Then get rid of the user.clj file, then see what is going on

Ok iv confirmed this on a clean project (https://github.com/narkisr/test) Using two different jdk versions:

$ lein repl 
1
nREPL server started on port 33397 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.8.3
Clojure 1.10.1
OpenJDK 64-Bit Server VM 11.0.10+9-Ubuntu-0ubuntu1.20.04
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

test.core=> (require 'clojure.tools.namespace.repl)
nil

test.core=> (require 'clojure.tools.namespace.repl)
nil

test.core=> (clojure.tools.namespace.repl/refresh)
:reloading (test.core test.core-test)
:ok

test.core=> Bye for now!

$ lein repl 
nREPL server started on port 36563 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.8.3
Clojure 1.10.1
OpenJDK 64-Bit Server VM 11.0.10+9-LTS
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

test.core=> (clojure.tools.namespace.repl/refresh)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
clojure.tools.namespace.repl

test.core=> (require 'clojure.tools.namespace.repl)
nil

test.core=> (clojure.tools.namespace.repl/refresh)
:reloading ()
:ok

I will remove my profile.clj next

> So the first thing is to disable all the plugins, both in the project and any user level stuff. Then get rid of the user.clj file, then see what is going on fwiw, it looks fairly safe to me those plugins aren't exactly wild "dev" seems missing from the :source-paths and there's set-refresh-dirs missing

The above project doesn't use user.clj just empty project calling the function directly

The confirm on a clean project is not correct

You didn't do the same thing in both repls

Your second attempt is missing the require to load tools.namespace

https://github.com/stuartsierra/reloaded uses exactly my advice and comes from the t.n author. I recommend you to save time and not go overboard with scientific debugging

Ok, I think I got a lead removing my profiles.clj did work:

{
    :user {
    :plugins [
       [mvxcvi/whidbey "2.2.0"]
       [io.aviso/pretty "0.1.37"]
       [cider/cider-nrepl "0.22.4"]]
  
      :middleware [
        cider-nrepl.plugin/middleware
          io.aviso.lein-pretty/inject
          whidbey.plugin/repl-pprint]
  
      :source-paths  ["dev" "test"]
  
    :dependencies [[io.aviso/pretty "0.1.37"]]
   ...
  }

$ lein repl 
nREPL server started on port 45839 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.8.3
Clojure 1.10.1
OpenJDK 64-Bit Server VM 11.0.10+9-LTS
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

test.core=> (require 'clojure.tools.namespace.repl)
nil
test.core=> (clojure.tools.namespace.repl/refresh)
:reloading (test.core test.core-test)
:ok
test.core=> 

my bet is cider-nrepl :) https://github.com/clojure-emacs/cider-nrepl/pull/668

I'm using some library that gives names that include unicode symbols to some of it's vars. It works fine when I run it on my own machine but elsewhere it throws a ClassNotFoundException when it tries to invoke one of these variables. Is there an environment variable I should be setting to ensure this works everywhere?

I want selmer.parser/render (https://github.com/yogthos/Selmer) to act like the following:

(render "{{input}}" {:input ["a.txt" "b.txt"]}) => "a.txt b.txt" ;; does not work
(render "{{input.1}}" {input ["a.txt" "b.txt"]}) => "a.txt" ;; works

(render "{{input}}" {:input ["a.txt" "b.txt"]}) => ["a.txt" "b.txt"] 

Is there a way to know during macro expansion if a symbol is expected to be primitive by the compiler even though it's not type-hinted?

the compiler doesn't look at things till after macroexpansion

which is to say, no, because there is no expectation at the point where macro expansion runs

When expanding a macro inside the scope of a function with a type hinted primitive I can see in the &env it's mapped to a Compiler$LocalBinding which has a jc field (Java Class) which contains long

how unwise is it to take advantage of that?

Besides the pending execution for black magic?

it depends what you mean by "take advantage", but it will likely break any other macro expanders that are not the compiler

(so if your editor provides some kind of macro expander functionality, or core.async's go macro, etc)

I'm experimenting with opportunistic inlining of equality checks

For example, (= "foo" bar) can be inlined as (.equals "foo" bar). Same for keywords. Numbers are trickier

With numbers, the only thing I can be sure of is if the argument to the function is a type hinted primitive, it would throw a runtime exception before reaching any equality check in the scope of the function if it wasn't a number anyway

the compiler already does a fair bit of that kind of thing if it can determine the types

and if it throws or not actually depends on how things are type hinted

in the general case type hinted code doesn't throw an exception if the type doesn't match

user=> ((fn [^String a] a) 1)
1
user=>

Generally, no, but primitives do, because invoke calls invokePrimitive

(only for double and long)

correct, and that is the only case

and there the compiler knows the type, so it is already doing a bunch of "inlining"

I think ClojureScript does more of that than Clojure

Huh, here's something weird

the call to clojure.core/= is replaced by a call to the Util/equiv static method

which is the compiler may then replace with some byte code

Decompiling:

(defn foo [^long n] (== n 1))
;;
    public static Object invokeStatic(final long n) {
        return Numbers.equiv(n, 1L) ? Boolean.TRUE : Boolean.FALSE;
    }

On the other hand:

(defn foo [^long n] (if (== n 1) true false))
;;;
    public static Object invokeStatic(final long n) {
        return (n == 1L) ? Boolean.TRUE : Boolean.FALSE;
    }

Why does the second get inlined and the first doesn't?

it is because jvm bytecode is not expression based

the decompiled result looks the same, but one ternary operator is the result of boxing, and one is the if

Yeah, but the first case has an extra method call

I can share the bytecode, too, if you'd like

the "extra" method call is what I am referring to

ok, sorry, missed your intention there

there are two different sets of byte code intrinsics the compiler has

one is purely replace this static method call with this sequence of bytecodes

so the bytecodes need to produce the exact same result as calling the static method would

the other is when compiling an if, if the test is a a matching static method, replace it with this series of byte codes

Pretty much every bytecode VM (and CPU) has that thing where comparisons are branches instead of producing a value

correct

so when inlining if tests, you don't actually have to match what the static method would do, you just need to match the branching behavior

which is why those two sets of intrinsics are different

But e.g. the Lua compiler will emit the same bytecode for (== n 1) as for (if (== n 1) true false) (too lazy to write the Lua equivalents)

I am not saying it couldn't do the swap

I am saying there is a reason there are two different sets of intrinsics

the if intrinsics have an match for Numbers.equiv, the non-if intrinsics don't

Because the value produced by Numbers.equiv would be different?

no

It would not be as straightforward to use the branch intrinsics for returning the boolean instead of just when the source has a branch but it would be correct and feasible

the non-branching intrinsics just need a different sequence of bytecode to match the behavior of Numbers.equiv outside of an if

but, the question as always is "why?"

I haven't looked at the inlining specifics in a while, it may also be the case that the if intrinsics and the non-if intrinsics can conflict

if you inline the non-if version, then the if inlining can't be applied

The intrinsics are more a case of "it would be silly not to use these specialized bytecodes if the types are known" than "let's add a bunch of optimizations, it will be awesome"

In Lua they have a bytecode that implements polymorphic equality directly instead of the equality operator being a function, so they must do the transformation

Given:

(defprotocol Eligible
  :extend-via-metadata true
  (make [this] :bar))

(defn eligible? [x] (satisfies? Eligible x))

(def x (vary-meta () assoc `make (fn [_] :made)))

(eligible? x) => false

There are several discussions about how to use the full qualified keywords, and even in Clojure Spec Rationale[1] there is a quote: > These are tragically underutilized and convey important benefits because they can always co-reside in dictionaries/dbs/maps/sets without conflict It makes me feel somewhat guilty of not understanding something very important, a thing that I must understand. I see a lot of experimentation on how to use it, with a lot of learning, but it's hard to interpret what core Clojure maintainers expect when they created the full qualified keywords. It's much easier to get a glimpse of how it helps to solve real issues with examples, so my question is: Where can we find more substantial resources with examples that cover how and when to use full qualified keywords. What kind of practical problems full qualified keywords solve, beyond the abstract notion? 1: https://clojure.org/about/spec#_global_namespaced_names_are_more_important

I think, basically, it is rdf

which I guess is clear as mud

to some degree, my impression, is that rhickey's preferred data model is something like rdf (or a triple store or whatever, I mean look at datomic)

so you can image a map a given map is a set of rdf triples, with an implicit subject

a big difference between rdf and something like sql is column names in sql are implicitly namespaced by the table the column appears in

What is there to say about it? Sufficiently qualified attributes allow you to assign global meaning to attributes

rdf doesn't have that, everything is just a soup of attibutes, so you have to do things like use urls as attribute names to avoid two "name" attributes of the same thing conflicting

so similarly with maps, if you want to merge two random maps together, the odds of that working with namespace qualified keys is much higher than without

It'd be nice if there are some books or blog posts with real examples @alexmiller, I ask for the whole community. I make an exhaustive search and just found toy examples

or experimentations, but it'd be nice to see if someone produced something along the lines of a real use case. Videos are welcome too.

I’m not sure what needs to be said that’s interesting

it is the same reason spec is the way it is, you globally say "such and such a key is such and such a thing", you don't say "in some context such and such a key is such and such a thing"

Yes, I think that you already said all about it, I'm looking for good references of real use cases in application code.

I know that @wilkerlucio uses it a lot in pathom, but he maintains library code

being global, context free, requires disambiguation

In application code you can totally control the data, so it is not as useful.

I don't know that I agree

large applications deal with a lot of data

not only from multiple sources, but from different eons of development of the same application

being able to mix and match all that is very useful

yeah, also, its hard to live complete isolation, namespaced attributes (when big enough) remove the question of "what is this data about?", IME small programs hardly ever even think about this concern, but as it grows, and them you have many services with many teams integrating stuff, having those unambiguous tractable attributes can help tremendous

I think its sad our industry got so used with local (short) names everywhere, makes a pain to compose systems at large

I need to medidate more about it, it's like a koan that I'll suddenly solve and eventually get the fqn nirvana 😄 :person_in_lotus_position:

@marciol I think it’s one of those things that you don’t really see the value of until you’ve been bitten by the problem it solves. At work, we use qualified keywords a lot.

Yes @seancorfield, seems that we need to get the feeling after using it in anger to see the benefits and where it falls better

If everything is just a bag of properties, what next, a bag of bytes? "Use text, that is a universal interface". Just wire up some AWK and Perl scripts.

Perhaps this example will be motivating for you? next.jdbc qualifies column names by the table they come from when returning query results. If you join across two tables, both with an ID column and a name column, you will get two different qualified keywords and no conflict: {:author/id 1 :book/id 23 :author/name "Alex Miller" :book/name "Clojure Applied" :book/author_id 1}

Without the qualification, the author ID and book ID would collide and so would the book name and author name.

One issue with that approach is what if there are more than one book or author in one record? Just having a "type" namespace wouldn't scale. (e.g., a book with a person that wrote it, and a person that edited it)

It's a nice application @seancorfield

@isak Then there would be multiple column names — a database cannot have more than one name column in a table.

That next.jdbc feature is the must relevant application thing I have seen but it kind of just exports the namespacing you get inside SQL

@nilern By default, yes. But you can provide your own (result set) builders that do something more sophisticated if you wish.

@seancorfield At my work we just do JSON SQL queries instead, so what comes back from the database would be {:author {:id 1, :name "Alex"} :editor {:id 2, :name "Rich"}}

Usually you don't SELECT * on a join anyway because probably that data is going to a JSON response so you want to control what gets sent out and namespaced keys break down with JSON and random client code

We do not send qualified keys back to client code. But we do a LOT of queries internally that stay entirely inside our apps and the qualified names help avoid conflicts.

To be honest, I was quite happy to use next.jdbc in a project... and to be honest, the experience was not that good. Most of the time I spent converting qualified keywords to JSON and back, and in the end I tried to make a builder-fn that would return the query exactly how @isak sent me. How did you do it? It would help me a lot on my code here 😄

I have a hard time following code where the set of keys keeps changing although it is kind of idiomatic to be flexible with that

afaik JSON has no problem with "foo/bar" keys

@borkdude except that nobody uses it 😅

A lot of Clojure programmers allocate way too much mental overhead by using different names for the same thing. This is one of the appealing and powerful attributes of spec. Global semantics + global names. I have seen a lot of CLJ + CLJS applications hurt themselves by using JSON on the wire.

I think to interpret your data correctly, you almost always need to know 1) which view you are looking at, and 2) which path into that view you are looking at. And if you know both of those things, do you still need namedspaced keywords?

we use it at work in our (third party facing) APIs, we never had a complaint about it

Overall the namespaced keywords hype smells of RDF and the Semantic Web, not a good smell. They solve some naming clashes etc. but not everybody is doing Data Lakes or whatever

lots of weak arguments in here

the Semantic Web, while a failed effort, got it very right to use names that indicate shared meaning

I think my issue with namespaced keywords is that nobody (except Datomic and Datahike, maybe) use it. Once I tried to force myself to use it, and in the end I felt like Typed-OO again: I had a JSON or XML, had to convert to a map, then re-convert to qualified keywords, then convert back to non-qualified to persist in SQL... after some time I would have to query that DB, get non-qualified, re-qualify, work with it, de-qualify to send to an API...

s/JSON/Transit s/DB/Datomic qualified names through the stack

having explicit translation layers between input data, storage, and app logic is good actually

(forcing the db to hold OO logic etc. is not)

If my client don't use Transit and my company don't use Datomic, then what? It's not that easy, specially Datomic being closed-source

my point is that there is a unified way of thinking here to create a stack that respects qualified names

we're trying to move the state of the art forward, not start from whatever broken substrate is the status quo

Yes. But should an open-source language as pragmatic as Clojure be opinionated to use a thing that virtually don't exist outside that language? Specially when one of the pieces is closed-source?

have you not noticed that Clojure's author is opinionated? :)

To be fair, it supports the other way very well also

trying to make the edges of your program look like its implementation is a pathology, that's how we got ORM, CORBA, etc - the data interchange format / storage do not need to look like the format your application uses internally

the goal is not similarity, it's solving the problem of disambiguating and describing data

But is building the whole stack around qualified names worth it just to avoid mixing up :book/name and :author/name? I would rather just use static typing, it catches so many more errors with less effort...

another strawman argument

static typing doesn't extend to the db or data interchange either

(disambiguation is not the only benefit)

Saying “qualified keywords aren’t worth the effort because they don’t solve 100% of the problem” is missing the point.

They add value wherever you need names to have a larger context than “just inside this one piece of data” — which means they add value in a pretty large space.

You can choose not to use them and justify it however you want, but you don’t have to work with me so we don’t need to argue about it 🙂

It seems like people are interested in using qualified keywords, but finding good guides, resources, and examples is still a challenge. Hopefully, that will improve over time.

Going in and out of the DB using next.jdbc means I can use qualified keys in both directions (`next.jdbc` ignores the qualification going into the DB, for convenience, and qualifies with the table name coming out — again for convenience), and you can explicitly map those names to whatever you need. clojure.set/rename-keys and/or select-keys means this is a “one-liner” at the boundary between naming — which I would expect at a boundary returning JSON since you’re unlikely to be exposing your internal domain data 1:1 anyway.

I wish we’d used qualified keywords from day one, at work, to be honest.

right - I think this points to the right focus - having good translation layers between input, logic, and storage, and namespacing of keys helps

If we rewound time and made it so that every json library for clojure didn't have an option to keywordize keys I think more codebases would be cognizant of it

I suspect if Clojure books and tutorials had used qualified keywords from day one, we wouldn’t be having this conversation 😉

exactly @seancorfield

@emccue Do you mean that JSON libs kept data structures with string keys?

yeah - unconditionally interning external input strings so that you can write (:key map) instead of (map "key") or (get map "key") blurs the line between "parsed data format" and "internal domain representation"

I may be reading it wrong, but I took it to mean that they "keywordize" option in the json lib leads to codebases without an explicit data transformation layer between input and app logic - the keywordizing is like a classic 80/20 of that

I can see the argument for drawing a starker line between JSON format and Clojure hash map format.

^not that its definitely a bad thing - but in the absence of educational materials that aren't either a rant on the clojure website or a rant published as a book how libraries/existing code works is probably the strongest driver of how people use a language

and namespaced keys is where that fallback really becomes a painpoint, since json doesn't have namespacing

(i guess and assert without evidence)

like, the better clojure books are "for the brave and true", which does a good job but it is just mechanics - how not what and why

There was a time when "Cheshire and MongoDB, no more translation layers" was the attitude

and the clojure for web development book - which isn't prescriptive about much beyond how to glue together the libraries in a way that works

not sure where that was the attitude?

I worked with the maintainer of cheshire at a clojure shop for a bit, and the general attitude there (not sure what his attitude was) was virulently anti-monodb

Thats because MongoDB is less a DB and more a long running bit thats going to end with someone taking a bow and shouting "The Aristocrats"

yeah, that

so there are lots of attitudes

I get that "things that will exist in a global context should be qualified as such (ie full domain)" and "it can also be useful to qualify things categorically/by domain in our app" such as :user/email but I'm kinda lost at the idea of "json in clojure land should have used string keys by default". Does this imply that we should add qualifiers to everything at the boundaries of a program? Or is the point to keep keys as strings until making a decision about what qualifier a particular keyword should have?

I read it as an unfortunate side effect of a convenience - since keywordizing makes the incoming data look "native", people start structuring applications such that they lack the translation layer from the interchange format to the application

and that works until eg. you want to use namespaced keys inside the application

and you might get confused and think the solution is that the interchange format needs namespacing

there definitely is a tendency to look at json as data structure and not a serialization format

really, the same with edn

Lately I've been experimenting with, when getting a stream of a lot of different k/v pairs from a source, just namespacing everything :source/x as a reminder, but I'm not sure how good an idea this is, because a :source/type won't always have the same meaning

Right to the point @emccue, it'd be nice to have more resources about the Clojure way to solve real problems, at least, as the maintainers imagine should be this way, because the other way, we watch this fragmentation. Maybe, it's a tragic fate of Lisps.

I think there are a wide range of resources about this already

As much as it’s possible to do, given that “real” systems are their own universes that live in the context of companies and groups of developers with histories, and thousands of micro decisions

I don't think resources are the problem

There are tons of resources for learning and putting clojure into practice

and a bunch of conference talks about what the intent is behind stuff

But there is no effective aggregation of that information

we don't have a "The Rust Programming language"

if that makes sense

"Programming Clojure" is as much that as anything

I don't mean in terms of a language tutorial

wow @alexmiller, I have the 2nd Edition here

the 3rd covers spec and Clojure CLI

hmmmmm

In the rust community the language is a complicated beast

so thats basically all it covers

but I means in terms of a collaboratively built central resource

programming clojure is the only book about clojure on rich's clojure bookshelf(this is, of course, not fair because I think it was the only book about clojure when the bookshelf was created) https://www.amazon.com/ideas/amzn1.account.AFAABBRGIVOWVKTHP5NOJU5LMROQ/3BSKWCYM12RBZ

imo, there is no one answerable answer to this. everyone comes to Clojure with a different background, wants a different thing, is building different things. there are resources to answer almost any question you might have. it's impossible to organize them in all some way that makes sense to every person at every point in their path of knowledge. I've never seen that in any language community I've intersected with (other than communities so small that a single person can write a single definitive resource)

in the last decade, there have been at least dozen, probably many more attempts to create such a thing. they are all helpful. they are all missing important topics. they are all useful to some extent.

Agree!

the Clojure web site is open for issues and PRs. I have helped people edit a variety of things into a published state there over the years and would be happy to help do more of that (as time allows). most of the guides https://clojure.org/guides/getting_started were contributed

I'd be happy to entertain organizational and expanding ideas in the community area

I actually did a big overhaul of the getting started area a couple months ago that is still pending a few things so has not yet been merged, but that's coming

The last Clojurists Together survey indicated they might want to explore more creative things to fund... maybe "The Encyclopedia of Qualified Keywords: from Theory to Practice" could be funded into existence

Thanks @marciol (and everyone who subsequently weighed in) for posting https://clojurians.slack.com/archives/C03S1KBA2/p1617654390371500 that led to such an prolific and informative thread, ending with https://clojurians.slack.com/archives/C03S1KBA2/p1617663102017400. It reinforces how important qualified keywords are. The term "tragically underutilized" jumped out at me the very first time I read the rationale.