Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2021-06-30

Channels

#clojure
<
2021-06-30
>
mwolf00:06:18

I'm looking for a clojure library that implements a disk-backed LRU cache -- any suggestions? I'm reading many millions of records from a database and need to manipulate/mangle them and I'm running out of memory in the jvm process. (option B is mucking around with jvm memory settings, but if there's a clojure-friendly caching solution available I'm guessing it might be the more scalable and potentially simpler solution.)

markaddleman00:06:36

Not a cache directly but Chronicle Map might be helpful: https://github.com/OpenHFT/Chronicle-Map

mwolf00:06:29

looks like it has potential, will look through the readme a bit. thanks!

Adrian Smith11:06:11

What kind of manipulation are we talking about?

jcsims15:06:18

Are you able to deterministically paginate your queries from the database? Not sure what kind of mangling you're doing (e.g. if all that data is required to be in memory/cache at once)

mwolf00:06:45

(btw, was looking at org.clojure/cache.core -- did I overlook an implementation in that lib that reads from/writes to disk?)

seancorfield00:06:21

core.cache is all in-memory.

mwolf00:06:16

rats, thought so. thanks for confirming.

dsp13:06:41

I have a question about java interop. Do typecasts happen automatically? Am trying to understand how it is handled under the hood. Say, char-array, which calls (. clojure.lang.Numbers char_array size init-val-or-seq) and thus maps to static public char[] char_array(int size, Object init){ in Numbers.java. One can pass a Long as the size, but the signature for this method expects an int, which > Integer/MAX_VALUE results in an overflow. Do all interop calls attempt to coerce the inputs?

dsp13:06:00

(This is just a followup on a nasty bug I encountered, where I was using arbitrarily-sized byte-arrays to store temporary data, and did not know that JVM arrays were int-indexed. I am wondering if this behaviour is general with interop. Also where is the best place to put a note or warning about such things. I don't have commit access, but it might be useful for int-indexed to be referenced in the docstring at least. Not sure who to ask about this sort of thing.)

Alex Miller (Clojure team)13:06:46

The best place to ask questions like this is at https://ask.clojure.org

Alex Miller (Clojure team)13:06:35

in general, most Java things are int-indexed (arrays and collections) and Clojure defaults to longs for integer values

Alex Miller (Clojure team)13:06:25

most people don't have more than max int number of things in either of those, so this difference is not obvious

dsp13:06:34

Thanks. I'll read more there.

jaide23:06:29

What's that Clojure talk where Rich discusses semver and how any API or breaking changes should be a separate file, function, or library?

hiredman23:06:10

something with speculation in the title maybe