You can still use something like io/resource in CLJC, but it will have to be in a macro on the CLJS side so it gets executed at compile time (it will of course use compile time classpath as well), so it might be a hassle given that you can't use it as a macro on the CLJ side since it would lead to the same error. Another approach would be to split that string into a bunch of defs or, since that might not work as well, a bunch of defns, and then make a new def by combining them. That way, the large string will be split into smaller ones, each one in its own method. To ease the second approach, especially if the chance of you changing that string is high, you can write a macro that reads a file and generates a bunch of defns where each contains, say, at most 10kB string, and an extra def that uses them to come up with the resulting string.

@U028BUU1P3R can you provide a brief example of the code that was failing? I think that I've seen this sort of problem before; the tldr is that one should favor thin code that points to big data, over large code that maps 1:1 to data. Method code too large! says it all - Java classes (which is what all clj code ultimately compiles to) aren't meant to be arbitrarily large.

Sure. I made myself a cli wordle clone to satisfy my wordle-cravings. I've got a two wordlists in separate files - one is 14K, and the other is 67K. I wrote in clj but I have vague intentions of making a web version so my family can play together, so I changed it to cljc.

;; old way
;(def targets (str/split-lines (slurp (io/resource "targets.txt"))))
;(def dict (str/split-lines (slurp (io/resource "dict.txt"))))

;; cljc way - this blows up with Method code too large!
(def targets ["word1" "word2" ,,, "word2500"])
(def dict ["word1" ,,, "word12972"])

I've just changed it back to the slurp for now, but I think I'll take a a swing at the macro approach sometime soon.

Yeah the old way is the one way to go in general. Normally I'd add a delay so that one avoids the compile-time side-effect - those are yucky in general :) For js compat, maybe I'd look into how to "slurp" from cljs (which surely varies from web to cli) and make a protocol, with jvm and (possibly two) cljs implementations.

There is, but it's private, clojure.core/root-resource. And it's so small - you basically already know what it does.

Slightly mitigated by weak references but too quickly you can probably oom

Yep, this message and the following two: https://clojurians.slack.com/archives/C03S1KBA2/p1643147571081700

ok, I'm asking because I noticed that when using keywords as keys in a map, the lookup becomes much quicker than when using symbols

and those symbols could be the result of auto-generating in macros

but this danger is already present if you evaluate random programs, as they can contain random keywords, so maybe it's not a huge deal

You could put them behind a bounded cache

so probably: if the user was malicious, he/she could cause an OOM, but in normal usage, it shouldn't be a problem

How would you use a bounded cache for this?

What you want to do is prevent the creation of too many keywords, any mechanism that will keep track of the number of keywords created and would throttle or force the cleanup of old keywords can be set up. You can set a policy which biases known keywords, dig in the implementation of Keyword to see how you can force the cache to evict unreferenced keywords, and generally wrap keyword in something which will bound its behavior in that context

@UK0810AQ2 I could also convert the symbols (which are a limited set/scope) to some number, or maybe a keyword like :k1 etc instead of :g__auto_001

I notice that a long lookup is much slower than a keyword lookup:

user=> (let [m {:a 1 :b 2 :c 3 'd 6 'x 7 1 1}] (time (dotimes [i 10000000] (.get m 1))))
"Elapsed time: 348.475696 msecs"
nil
user=> (let [m {:a 1 :b 2 :c 3 'd 6 'x 7 1 1}] (time (dotimes [i 10000000] (.get m :a))))
"Elapsed time: 31.343012 msecs"

I would just use numbers if I could make that as fast as the keyword lookup

maybe int-map is faster?

Seems so!

user=> (let [^java.util.Map m (into {} [[1 2] [3 4] [:b 2]])] (time (dotimes [i 1000000000] (.get m :b))))
"Elapsed time: 3000.653457 msecs"

user=> (let [^java.util.Map m (into (i/int-map) [[1 2] [3 4] [4 2]])] (time (dotimes [i 1000000000] (.get m 4))))
"Elapsed time: 500.608064 msecs"

I wonder if such a thing also exists for CLJS, or if that would even make sense there

It seems in CLJS using an int as a key is faster than a keyword anyway:

cljs.user=> (let [m (into {} [[1 2] [3 4] [:b 2]])] (time (dotimes [i 100000000] (.get m :b))))
"Elapsed time: 2821.658472 msecs"
nil
cljs.user=> (let [m (into {} [[1 2] [3 4] [:b 2]])] (time (dotimes [i 100000000] (.get m 1))))
"Elapsed time: 794.211123 msecs"

@U04V15CAJ Your comparisons are not entirely correct. Those are small maps so they are array maps and not hash maps - the lookup is sequential. When the long is the last item, of course its lookup will be slow.

Try hash-map, the lookup speed should be around the same for all the keys - as long as you don't have some bad case of hash collision.

good point!

Thanks for having this conversation, it reminded me that I needed to get #farolero to stop interning keywords at runtime. I was able to cut a new release today fixing what would probably have been a pretty hard-to-trace bug a while from now when I deploy farolero to production on something.

@U5NCUG8NR Aha! Are there any performance repercussions when disabling that?

should be about the same on unwind because I've exchanged an indirect keyword identity (comparison was still using = at a top level) to a boxed long comparison, but it should be faster when entering a block that needs to construct a jump target can do so via a simple fetch_add on an atomic rather than needing to do a gensym, get the string out of it, and then construct a keyword with that and a fixed namespace string.

The gensym one clearly does an atomic fetch_add somewhere in its body anyway, so I'm just going down to only the fetch_add, rather than needing string ops as well

interesting... I ended up not using int-map yet, since assoc seemed to be way slower, so in total performance would be actually worse than just a regular hash-map.... A hash-map + keywords are in total the fastest for the problem I have, but the interning of keywords still could be a problem for some environments. Maybe I'll make the keywordizing configurable or so. :thinking_face:

making it configurable makes sense when you can know at app development that you're receiving only trusted data. Another thing to consider would be to have your own interning deftype that allows you to have "epochs" where you say "all the keys from this epoch will get GCed together"

If you're doing tons of stuff that needs efficiency through a single request but across requests it won't give much benefit, then maybe just wrap an epoch around your handler, but if you're doing something where there's savings to be made across requests, you could just make it like a ttl cache basically.

ofc doing it this way brings problems if your users want to get data out by writing literal keywords into their code

Could make a custom associative data structure that just wraps a map basically but compares keywords as equal to the new key type.

But that loses a lot of the performance benefits if you're doing that constantly.

Hmm.

(let me know if my ramblings are useless, I can stop)

oh please continue ;)

So if the intention is to keep things convenient while being performant, I think that means you basically need a global "initial epoch" that would have all the interned items that are literal values in your code, and you could provide an alternative to the get function that would map a keyword to something in this initial epoch at compile time, and then you make interning be a two-step process of first looking things up in the initial epoch and then if it's missing falling through to the non-static epoch, that way you get fast lookup at all your callsites, and construction of keys is also not that much more expensive.

Personally I'd opt for making this alternate get actually be a function and just have an inline function definition that will do the epoch registration. It would mean that depending on if you're using it as an argument to a HOF you'd get different performance characteristics, but it'd be as performant as possible in all cases.

I think that mostly covers the main usecases for "I want to make my own interning", although I'd love to hear if you have any thoughts about holes in this idea. There's probably other good ways that don't go down this route though.

That's what I basically did with the int-map, symbols were converted to a number

but instead of a number I could just create Objects

I think they'll be about the same size seeing as boxed numbers have their size dominated by the object header. Although I think being able to use identical? would help performance.

I want to move to mutable arrays eventually anyway. Using keywords was a nice cheap way to get some free speed-ups without changing much

but as it's only safe to intern keywords in environments that aren't long-lived or do not process code untrusted input, making a configuration option for this in the meanwhile makes sense I think

yeah, I agree

so with your int map of symbols being converted to a number, what did you do to allow easy lookup from the data without losing the benefits of a simple type like integers for comparison? Were you able to move the mapping from symbol to integer to compile time or was that a runtime lookup? Or am I missing something and there's no lookup needed with known-at-compile-time keys?

compile time

the lookups were faster

but the changing of the map using assoc was slower

so in total it was slower

ah, fair enough

Same on Linux, but works in Chrome.

a random stack overflow mentions that firefox will error if any resource on the page (eg. favicon) but other browsers just silently fail to load it

but that's just from a quick google

https://clojurians.slack.com/archives/C053AK3F9/p1643478725715269?thread_ts=1643477894.308129&cid=C053AK3F9

On that note, cljdoc is looking for ops-savvy folks for advice/help. Drop by #cljdoc if you have time/interest/love-in-your-heart.

On both Safari and Firefox I get messages that secure connection can’t be established/revoked certificate.

Thanks @UTFAPNRPT, known issue.

@U050TNB9F has resolved the certificate issue.

Yes I was trying to add meta to the clojure.core/list function

I was believing that every IObj could take meta

Erm, this might just be what set/join does I glean from reading the docs

Ugh, I feel stupid now XD I had never used set/join before

user=> (def db #{{:genome "hg19" :sample "A"} {:genome "hg38" :sample "A"} {:genome "hg38" :sample "B"} {:genome "hg19" :sample "B"}})
#'user/db
user=> (def search #{{:genome "hg38"}})
#'user/search
user=> (clojure.set/join)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
clojure.set
user=> (require '[clojure.set :as set :refer [join]])
nil
user=> (join search db)
#{{:genome "hg38", :sample "A"} {:genome "hg38", :sample "B"}}

I was just typing up that you would need to put single k/v entries into a hash-map to do the query, clearly you beat me to it

Regarding your first question - if that's a frequent operation, just create an index with group-by or clojure.set/index. If it's a one-off thing, just reduce over the db and accumulate the matching records. Regarding the second part - so a query map represents an OR query? Note that clojure.set/join would join the query together with the results - probably not what you want, given that then all results will have :sample "A".

Note also that join may work incorrectly if you have maps with different keys. Also, it may easily be slower than a hand-rolled alternative because it indexes all the data each time you call the function.

Thanks for pointing me to the index function. It will speed up my code but only add one line AFAICS.

LGTM. If the :else branch is potentially the most frequently hit one, it could make sense to make it the first one in cond. But perhaps JVM itself will do that in runtime.

You could also make it a tiny bit faster by replacing destructuring with manual calls to first and rest/`next`, given how the first two branches of cond don't use tail-2 and tail-1, respectively.

I guess it’s not possible to predict which branches are hit the most. Calling first and rest explicitly is a good point, I changed to destructuring for better readability, but did not think of this 👍

It isn't lazy if that is a concern

Currently we always need all values from the lists, but if the system grows it might become a concern, you also raise a good point there