Fork me on GitHub
#beginners
<
2020-07-20
>
neo255107:07:17

With core.memoize, would it be possible to to combine cache strategies? I especially think about TTL and LRU. My problem are some results are valid for (say) one hour but I still would like to control the memory usage with a maximum number of element. I guess I could just compose them and it should work, but I wonder if that would be recommened.

jsyrjala08:07:30

I think something like this works

(def my-func (memoize/lru my-func* (cache/ttl-cache-factory {} :ttl 60000) :lru/threshold 200))

andy.fingerhut14:07:06

@ recently dug into the core.cache library and its uses, and wrote this article about it: https://dev.to/dpsutton/exploring-the-core-cache-api-57al. Hopefully he does not mind too much being at-ed here for his attention, in case his fresh-in-memory investigation of core.cache means he has an answer to your question.

dpsutton14:07:08

not at all. I did not investigate how the caches compose as the suggestion here. I would be nervous but I would check it.

dpsutton14:07:31

and thinking on it, the caches do not use the cache protocol on the underlying cache object that holds the information in the deftypes. so i don't believe there's any way that the caches will compose like this. so the ttl cache will be unbounded but by the ttl expiration. That sounds like a good structure though so a package that adds that would probably be welcome in the community at large

dpsutton14:07:13

and adapting the lu-cache but using the ttl as the usage count for eviction might be a strategy to write one that combines the two

neo255114:07:30

:exploding_head: even after reading your post, it is still too complicated for me haha.

dpsutton14:07:42

the gist is setting up a cache which imitates the usage found in projects on github and then hammering it with 20 threads making 20000 accesses and seeing what breaks.

dpsutton14:07:06

looking for race conditions when checking a cache has? a value and then subsequently checking the cache for the value and not finding it

seancorfield16:07:30

Caches should compose. I think there are even examples of that in the docs. I remember a bug fix going in that only surfaced when you composed caches.

seancorfield16:07:54

(but composing caches as part of memoization is non-trivial to get correct, I suspect)

dpsutton17:07:36

i don't think that's possible. when you hit a cache entry in a ttl it does not propagate that hit into an underlying lu or lru cache as far as i can tell

dpsutton17:07:53

same for all of them. i don't see any checks to see if the cache object is a CacheProtocol itself. They are just associative objects as far as i can tell

seancorfield17:07:26

The associative operations are implemented in terms of the cache.

dpsutton17:07:42

with lookup. but i don't see hit in there

dpsutton17:07:40

(but i'll read some more and play with it tonight)

seancorfield17:07:09

hit is a no-op in most cache implementations. has? checks TTL expiry (since we're talking about the TTL cache).

seancorfield17:07:17

It's the caller that is supposed to use the has? / hit / miss strategy. If you use that on all caches, it behaves as expected (and you can compose caches).

dpsutton17:07:29

i guess that means some can compose but not all. stencil in the article just used get and assoc and the lu and lru cache's predictably didn't work

seancorfield17:07:52

Stencil uses core.cache incorrectly.

seancorfield17:07:28

That was a surprise to me, when I read your article -- that misuse is widespread 😞

dpsutton17:07:52

yeah. but the way stencil uses caches is the same way a TTL cache uses an underlying cache. just assoc and dissoc and get

dpsutton17:07:12

so a TTL cache can never wrap an LU or LRU cache because it won't propagate the hit into that underlying cache

seancorfield19:07:40

Yeah, the hit method is very problematic. And different caches also have different behavior on lookup -- TTL cache can succeed on has? and then fail on get, for example. I'm fairly certain that almost no one uses the more esoteric caching strategies (and almost no one at all actually tries to compose them)...

aviv09:07:22

hey, is there any clojure-channel here for community-open-source projects? new projects, etc if not: any web/forum that has?

nfedyashev15:07:23

Check out #announcements

gtzogana14:07:45

is there some kind of process viewer i can use to see threads created by go?

noisesmith18:07:48

strictly speaking core.async creates N threads, go just lines up to use them for a slice at a time core.async/thread creates threads (in a pool for reuse)

noisesmith18:07:07

IOW all the threads are created when you load core.async, whether you use go or not

noisesmith18:07:00

the thread inspection tools will show you the core.async thread, and their stack traces, but won't tell you directly which go block they are running

noisesmith18:07:21

it's rather opaque once it's running in the VM

gtzogana18:07:26

ok, got it, thanks

alexmiller14:07:01

ctrl-\ (*-nix) or ctrl-break (win) will dump the stack of all threads in your current process JVM

alexmiller14:07:22

jstack is a jvm tool that can be used externally (or kill -3 the pid)

alexmiller14:07:47

profiler type tools let you connect with a ui and see this kind of stuff - jconsole (comes w/the jvm), jprofiler, yourkit

imparato.andrea15:07:58

hello everyone! is there a way to do something like this:

(loop [x '(1 2 3)]
 (for [y x]
  (recur (next x)))

imparato.andrea15:07:24

basically i want to use recur inside a for loop

smith.adriane16:07:28

loop and for don't really work together like this. why not just:

(let [x '(1 2 3)]
  (for [y x]
   x))

imparato.andrea16:07:52

yeah that was just a dummy example.

imparato.andrea16:07:12

My actual problem is to translate this function

(defn all-sentences [chain current-key sentence]
    (let [words (get chain current-key)]
      (if (not= #{nil} words)
        (concat sentence
                (reduce (fn [result w]
                                   (all-sentences chain w (concat result current-key)))
                                 '()
                                 words))
        (concat sentence current-key (list "END$")))))

imparato.andrea16:07:28

into another function that uses tail recursion

imparato.andrea16:07:42

chain is a huge markov chians map

imparato.andrea16:07:59

and with this implementation i hit a stackoverflow unfortunately

smith.adriane16:07:33

related to your stackoverflow, someone recently shared this article that seems relevant, https://stuartsierra.com/2015/04/26/clojure-donts-concat

gtzogana18:07:15

also on the topic of core.async, what’s the best way to mock out a function that will be called in a go block?

gtzogana18:07:33

with-redefs doesn’t seem to work, even though docs say it should be visible across all threads

noisesmith18:07:16

the problem with with-redefs is that the original def is re-established as soon as the with-redefs block exits

noisesmith18:07:33

to use with-redefs with a go loop, you need to block exit of with-redefs until after the loop exits

gtzogana18:07:34

ahh. so maybe i can block with a take or something

noisesmith18:07:52

also, with-redefs is racy, if you use it on the same var from two threads, you can lose your initial definition

noisesmith18:07:15

it's not safe for real code (and also blocks you from being able to safely run tests in parallel)

gtzogana18:07:55

this is a single test right now so i guess i’ll bite the bullet for the moment

ericihli23:07:02

Running into some unexpected behavior with spec generators that I don't understand.

(s/def ::user
  (s/keys :req-un [::name ::email ::password ::followed-states]))

(gen/sample (s/gen ::user))
;; => ({:name "x",
;;      :email "",
;;       ...

(s/def :crux.db/id
  (s/with-gen uuid?
    #(s/gen (into #{} (take 10 (repeatedly
                                (fn [] (java.util.UUID/randomUUID))))))))
(gen/sample (s/gen :crux.db/id))
;; => (#uuid "c99f13e9-7e54-44e1-9a4f-9466ce2afc89"
;; ...

(s/def ::user+
  (s/and ::user (s/and (s/keys :req [:crux.db/id]))))

(gen/sample (s/gen ::user+))
;;1. Caused by clojure.lang.ExceptionInfo
;; Couldn't satisfy such-that predicate after 100 tries.
;; {}
Everything works fine until that last spec ::user+. I'm able to generate every sub-spec that ::user+ relies on. But when I try to generate samples of ::user+, it can't satisfy a predicate. I'm going to search to see if there's a way to get more information about specifically which predicate it failed to satisfy and I'm also going to try (s/def ::user+ (s/with-gen (s/and ::user ,,,) #(merge (gen/sample (s/gen ::user)) {:crux.db/id ,,,_)) (defining ::user+ with a with-gen and created a generator that calls gen/sample on ::user and then merges it with the new key; on the assumption that the problem lies somewhere in there, but I don't actually see why/where). Aside from what I'm going to try, does anyone know why that last gen/sample isn't working?