Fork me on GitHub
#clojure
<
2018-09-22
>
crankyadmin07:09:16

Morning all, I have a requirement where I need to alternate between items in a list: (def my-list [1 2 3 4 5]) My requirement is that I need to pull 1 then the next time pull 2 and so forth, but once I've pulled 5 I need to go back to 1 and loop through it again.

crankyadmin07:09:28

I can't wrap my head round how to do this...

crankyadmin07:09:14

I suspect I need to use cycle in some form.

schmee07:09:14

so you want a function that gives a different result each time you call it?

schmee07:09:59

then you need an atom or something similar

crankyadmin07:09:52

Thats it basically...

crankyadmin07:09:04

Its the dynamically building the function I'm having difficultly with.

crankyadmin07:09:36

I fetch a list of things thing need to loop over them, but I don't know whats going to be in the list till I fetch it (from Elasticsearch)

crankyadmin07:09:23

Something like this but with out knowing what I passed in last time.

schmee07:09:26

hmm ok, why do you need to loop through it again?

crankyadmin07:09:04

Whats happening is I have what we call a tag,

crankyadmin07:09:41

thats then links to a list like ["11235" "456345" "234234"]

crankyadmin07:09:02

(they're marry up to images, but thats not important for this)

crankyadmin07:09:27

each time I see the tag I need to drop in one of the above numbers.

crankyadmin07:09:32

The tag could appear 100's of time but I only I have three items in the list above so I need to loop over it repeating alternating which item I use.

crankyadmin07:09:54

As a example: 1st time I see the tag i need "11235" 2nd time I see the tag i need "456345" but on the forth time I see it, I need "11235" and so forth.

schmee07:09:06

are the tags in a list?

schmee07:09:17

the items with the tag field

crankyadmin07:09:22

Nope, its just one tags.

crankyadmin07:09:12

{:tag-name "foo"
  :list-of-creatives ["11235" "456345" "234234"]}

schmee07:09:42

ahh, but there is some sort of sequence of tagged items then?

schmee07:09:44

such as that one

schmee07:09:18

thing I’m getting at is that if you have two lists with items, then cycle is your buddy

crankyadmin07:09:39

yeah... I have another file that contents foo and I need to replace foo each time I see it with the numbers.

schmee07:09:40

if you get each thing one at a time then you might need some state

crankyadmin07:09:00

Yeah... this is exactly the bit I'm stuck on 😞

crankyadmin07:09:20

I have potentially hundreds of tags to deal with...

schmee07:09:09

can you load up all the tagged items into memory, or are there too many?

crankyadmin07:09:51

For this PoC yes.

schmee07:09:58

ok, then something like this could work: • load up all tagged items in a list * group-by :tag to get a map of tag -> items * for each tag/items pair in the map, loop over it with something like (map mapping-fn items (cycle list-of-creatives-for-item)

schmee07:09:22

does that make any sense? 😄

crankyadmin07:09:22

Yeah I think so!! Let me see what I can hack up myself following your logic and see what happens! No doubt I'll be back for more advice 😂 Its a great learning experience!

schmee07:09:47

np and gl!

schmee07:09:44

if that doesn’t pan out, a stateful option is to have an atom with a map from tag -> array index, and update the index for each tag everytime you it

crankyadmin07:09:28

yeah that was my initial thought, it just didn't "feel" right 😐

schmee07:09:39

I feel ya 😄 in my experience it is often beneficial to find a stateless solution first and only go for a stateful solution if the stateless one ends up being too complicated and/or slow

crankyadmin08:09:16

Thats exactly it, I'm trying to remain as stateless as possible...

roklenarcic14:09:12

I'm writing a macro and I'm using a function that the user passes into the macro in the macro itself (i.e. to generate things). Of course I only get an unevaluated symbol. What's the best way to get the fn itself?

roklenarcic14:09:00

so far I've got:

(defn- load-from-symbol [sym]
  (let [sym (if (namespace sym) sym (symbol (str *ns*) (name sym)))]
    (var-get (find-var sym))))

roklenarcic14:09:16

although this leaves holes for imports

roklenarcic14:09:24

I'm an idiot, it's just (var-get (resolve sym))

orestis15:09:59

So I’m trying to write some guide on using core.cache, and I’ve realized I don’t fully get the semantics of atoms and race conditions:

(defn get-key [cache-atom k]
    (let [value (cache/lookup @cache-atom k ::not-found)]
       (when-not (= value ::not-found)
         (swap! cache-atom cache/hit k)
         value)))

orestis15:09:39

Isn’t there a race-condition between the dereferencing of the cache-atom and the swap! call there?

orestis15:09:24

I mean, there’s not a Thread-based race condition in that something will be corrupted etc. But the value I’m getting might be stale — there’s no transaction ensuring read consistency etc. swap! will retry until it can atomically update the cache, but dereferencing is uncoordinated, right?

mg15:09:36

As a cache, I think the idea is that multiple cache hits on the same key should be idempotent

orestis15:09:49

When you’re dealing with an LRU cache, timing matters though.

orestis15:09:06

(I realize that for a caching point of view, there shouldn’t be any assumptions of true transactions, I just realize how I’m missing something in my understanding of atoms)

mg15:09:28

Oh, also cache/lookup is where the value is fetched, including if it was missing in the cache. the swap! call doesn't use the value, it's just updating the hit information on the key

orestis15:09:17

Cache lookup doesn’t try to fetch anything, cache/miss will put a value in the cache.

mg15:09:40

Sorry, I'm misremembering that, been a while since I looked through core.cache source

orestis15:09:02

Yep, it can be confusing.

orestis15:09:41

I guess a concrete scenario would be: a) (cache/lookup :a) succeeds, giving me a value. b) Meanwhile, the cache atom is updated by someone else in such a way that :a is evicted. c) I call (cache/hit :a), but :a is no longer in the cache.

mg15:09:48

Anyway, regardless I suppose in the case you posted above, cache/hit will need to account for the possibility that the key is no longer in the cache, since it could have been evicted between the deref and the swap!

orestis15:09:53

Note that the documentation isn’t actually suggesting the pattern I posted above — and instead does:

(defn get-data [key]
  (cache/lookup (swap! cache-store
                       #(if (cache/has? % key)
                          (cache/hit % key)
                          (cache/miss % key (retrieve-data key))))
                key))

orestis15:09:18

This overcomes the atom “race condition” — but then retrieve-data might be called multiple times if the atom’s CAS operation has to retry.

orestis16:09:17

The LRU cache does handle hit being called when the key is not present in the cache: https://github.com/clojure/core.cache/blob/master/src/main/clojure/clojure/core/cache.clj#L225

orestis16:09:26

Here’s the term that describes what I’m talking about: https://en.wikipedia.org/wiki/Cache_stampede

kulminaator17:09:32

in my own designs i always lean towards the chance that caches will fail me, that fetching entries from backend possibly fails (network timeout etc.) and that i probably come around fetching for stuff multiple times, i never run single nodes of anything anyway so the according backends just have to be ready for it (and that's usually how my design ends up, slow part is either s3 or any database). on nodes that don't survive a cold start too well i do a warmup before i add them to the loadbalancers ... but all this is heavily usecase based 🙂

kulminaator17:09:18

occasionally it's also nice if your hard hammered machine always takes entries from cache, even if they are slightly stale, and orders a fresh data to be fetched in async on the background.

orestis17:09:21

Have you found any patterns that help with these designs?

orestis17:09:51

E.g. core.async, promises etc?

andy.fingerhut17:09:26

It has been a while since I read the core.cache code at all, but I remember the few times I tried it always confused me, I think partly because it was intended that you could "layer" any combination of LRU, etc. behaviors on top of each other (or at least more than one such combination).

roklenarcic17:09:24

Easiest thing is to add a varnish in front of a service and it will hold all http connections and make only a single request upstream

roklenarcic17:09:29

if request is cacheable

roklenarcic17:09:55

even a single second max-age will work wonders in that regard

orestis17:09:05

Yeah, not having to handle this is he best, unfortunately our service is private hence never cacheable.