This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-07-20
Channels
- # announcements (7)
- # babashka (16)
- # beginners (58)
- # boot (12)
- # calva (3)
- # cider (11)
- # clj-kondo (9)
- # cljs-dev (8)
- # clojure (82)
- # clojure-europe (9)
- # clojure-italy (11)
- # clojure-losangeles (1)
- # clojure-nl (8)
- # clojure-uk (8)
- # clojurescript (5)
- # css (2)
- # cursive (5)
- # datomic (20)
- # docker (2)
- # emacs (4)
- # figwheel-main (16)
- # fulcro (53)
- # graalvm (17)
- # jackdaw (2)
- # jobs (4)
- # kaocha (6)
- # lambdaisland (2)
- # luminus (2)
- # meander (1)
- # off-topic (146)
- # re-frame (4)
- # releases (1)
- # rum (12)
- # sci (71)
- # shadow-cljs (26)
- # test-check (22)
- # vim (1)
- # xtdb (9)
I've sworn off all blogging systems where the content is not sitting in a text file in a github repo
ideally i could upload an org file and it would do all the stuff for me. but i don't want to set all that up now so i'm looking for a blog as a service
well, jekyll / gh pages
Yeah, I do all my blogging via http://github.io (and a custom domain). I write markdown, run a process to create HTML, and push to GH.
or whatever to gh pages
I use Octopress (a variant of Jekyll)
I'm also still using this but still on Octopress 2. Getting a lot of deprecation warnings from Ruby last time I ran it..
@U04V15CAJ Yeah, last time I tried to update Ruby to get rid of those warnings, I ended up breaking Octopress and it took me days to get it all working again 😞
Yeah 😞
We're in the same boat then. Creating a Docker container crossed my mind, so at least it keeps working consistently
Luckily, I don't use Ruby for anything so I can mostly just ignore it and leave it all "old"...
same here. I don't use any fancy stuff so if needed I can port it to my own Clojure based solution (maybe even using babashka) but that's work
There are a couple of Clojure-based site generators already... I keep meaning to investigate them and maybe port my blog to one of them... but, hey, Octopress works and I have more important stuff to work on (and I don't blog as often as I used to).
Yep. There's been a few people taking up babashka as a blogging tool as well: https://www.mxmmz.nl/blog/building-a-website-with-babashka.html It's a bit more DIY, but at least you can understand every part easily.
thoughts on the api for core.cache: https://dev.to/dpsutton/exploring-the-core-cache-api-57al
this is great :thumbsup:
> Operating on just a cache value I would say the main issue with the ttl cache is that it's technically not a value
even if it's not in an atom
> When we used the through-cache
function we computed our value 19 million times, but had we not cached we would only have computed it (* 20 20000)
= 400,000 times!.
I would also point out that both look-or-miss
and core.memoize
wrap the computation in a delay so that you don't compute the value more times than you request the value from the cache
That's a great article -- I'll probably update the README on core.cache
to point to it. Yes, absolutely, the clojure.core.cache.wrapped/lookup-or-miss
function is the "best" API -- and it's how we use all the caches at work, but it didn't exist when I took the library over from Fogus.
@U7RJTCH6J Yup. That's a vector for a DoS attack on core.cache
, unfortunately. Someone specifically raised that JIRA issue, which is where the wrapped
API came from.
thanks for reading @seancorfield
The README has a new section "The core.cache API is hard to use correctly. That's why clojure.core.cache.wrapped/lookup-or-miss exists: it encapsulates all the best practices around using the API and wraps it in an atom from the get-go. Read this article about the core.cache API by Dan Sutton for why it is important to use the API correctly!"
Are these caveate still valid for core.memoize?
@UEQGQ6XH7 core.memoize
was designed to use the core.cache
API correctly, although it did have a nasty bug with TTL cache handling (which I fixed last year).
So we could say that memoize
is a higher level and safer API?
core.memoize
is for memoizing functions. It uses core.cache
behind the scenes, but just as an implementation detail.
Perhaps a stupid question about clojure.core.cache.wrapped/lookup-or-miss
and caching in general. Why does it attempt to put a newly computed value in the cache 10 times and why does it return nil if it fails? Why not just compute the value once, "try to cache and forget", and return the value?
@U2FRKM4TW In the normal case, it would only attempt that once. There are some caches that have pathological edge cases where this repeated attempt is necessary. TTL caches, for example, have an edge case where they can return true
from has?
and then immediately fail on lookup
-- and return a nil
value even when you supply a "not found" value.
This is a problem that core.memoize
works around in the same way.
You do not want the computation done at all if it is already in the cache.
Thanks. I'm afraid I still don't understand how having retries with value reevaluation could help TTL caches.
But after reading some docstrings in clojure.core.cache
, I realized that they make me even more confused. So maybe some other time. :)
@U2FRKM4TW It is not reevaluating the value. It uses a delay
to only evaluate it once. But, yes, core.cache
code is very complicated 😐
i avoid core.cache and wrap one of the many caching libs for java. Keeping the whole cache in an atom can give perf. issues when contention is high due to retries caused swap!. I've been bitten badly by this a couple of time.
i go through that in the article. ended up computing something 20 million times when caching it over 400,000 calls. There is a lovely function (and which i conclude is the only api you need for core.cache) in clojure.core.cache.wrapped/lookup-or-miss
which will compute at most once and try to set the cache at most 10 times before abandoning
not saying don't take your strategy but the solution i found there should be the starting point of core.cache when evaluating
i don't really see the point of keeping the cache in an atom, caches are mutable anyway
so you might as well use something backed by ConcurrentHashMap, or one of the java libs, which scale much better w.r.t. the number of threads, and are easier to use correctly
i guess the composition of different cache types is a nice feature of core.cached, but i never needed it in practice
ah, one thing i'm noticing now that you mention it is that if this function retries 10 times and never gets to pull the value out of the cache it returns nil rather than the delayed computed value. which seems off to me. i had misread it earlier
yeah. i'd prefer a function that did the single swap with through cache, and if the lookup succeeds, return that value else return the computed value without looping
kind of a fire and forget once into the cache. if there's so much churn on it, don't care if your value has already been evicted.
you can use compare-and-set!
to manually bound the number of attempts to update an atom
i guess you could hack it in by throwing an exception after n invocations of the function
for production usage, i also want the hit/miss rate exposed for metrics, last time i used core.cached I had to add this myself
you can use compare-and-set!
to manually bound the number of attempts to update an atom
another problem with using swap! to update the cache is that you should not use functions with side-effects
I believe the technique adopted by core.cache is to represent the cache as an immutable collection, and updating the cache is thus replacing the value that is the contents of the atom with a new immutable collection.
The only side effect is the one that swap! is designed for
I am not saying that there is no other way to design a cache -- just that it does not have the problem of using functions passed to swap! that have side effects.
most implementations that use core.cache with an atom wrap the calculation of the new value in a delay to ensure that the calculation is done at most once
or at least, they should
I believe what core.memoized does (layered on top of core.cache) is wrap everthing in a delay, and the delay may be swapped in multiple times, but then derefing the delay the computation is run only once
causing more reads the to the database than calls to this expression due to retries
anyway, i dont want to be to negative, there are plenty of valid use cases for core.cache. My point is that under contention core.cache can behave in unexpected ways, so be careful.
(cache/through-cache C1 my-key (partial jdbc/get-by-id db-spec :storage))
and this returns a cache that might not even have your value. so you need to call the function again
It is interesting that clojure.core/memoize mentions that it only supports referentially transparent functions, but the core.memoize library seems to make no mention of that restriction.
Yeah, no occurrences anywhere in the core.memoize repo files for "referential" "pure" "side" or "effect". And examples in docs/Using.md show wrapping of functions whose names strongly imply side effects, like fetch-user-from-db
in core.memoize it's not really a problem as the function is wrapped in a delay, so it will run only once despite retries
Does anyone know if there is an exemption for US citizens travelling to emigrate or take work in a european country?
@chrisblom That's why clojure.core.cache.wrapped
exists with lookup-or-miss
, to ensure the value-fetching function is only called once (at most).
Using clojure.core.cache
correctly is hard. Using clojure.core.cache.wrapped
is a lot easier. @dpsutton and I were just discussing composition of caches being broken in several combinations due to the way most caches rely on the hit
method for tracking 😐
If you just use a single (non-composed) cache and you pretty much only use wrapped/lookup-or-miss
then you're pretty safe.
Yeah, someone raised a JIRA issue ages ago about the through
function allowing for cache stampede -- and there's no way to fix it inside the cache itself, only in a well-behaved caller -- which is what that wrapped function is for.
All the other stuff in the wrapped ns is just to allow for it to be a drop-in replacement for the original API, to make it easier to switch over.
I very much doubt there is extensive use of core.cache
outside the most basic cache types (basic, TTL -- which has its own "interesting" gotchas since has?
can return true
and then lookup
can return nil
/ not-found).
We use core.cache.wrapped
very heavily at work, but only with basic and TTL caches.
the wrapped namespace is great, i don't think it was there yet the last time I used core.cache
i also used the delay trick, but had problems with edgecases if the function would throw
what does the lookup-or-miss in core.cache.wrapped do when the wrapped function throws an exception?
You can provide a wrapper function (in the 4-arity version of lookup-or-miss
) to control how that behaves.
(and, of course, you're already passing the value function which can also control that)
So there are multiple levels of control.
(although I've never found a use for the wrapper function, as I recall)
In the general case, I would expect folks would not want a value added to the cache if the value fn throws an exception tho'...?
There's no doubt that core.cache
is very hard to use correctly 🙂
(we stopped using the main API completely at work and switch to the wrapped API instead)
i think highlights the fact that even though you want tools and constructs made out of simple stuff, you don't always want to directly use the simple stuff
I am not sure, but some of the hard-to-use API calls exposed from the core.cache and core.memoized library seem like they might best be categorized as "not simple", i.e. "complex", because using them in a correct fashion requires a sequence of calls, and/or calls made in the correct context (e.g. inside of a swap! function, or not), that intertwines them quite a bit.
I don't think "simple" means the same thing as "short implementation, by count of lines of code"
and what drove me to look into it was because the api didn't make it obvious how it should be used. and thus there are lots of different styles of using it in the wild with many being wrong
several of the pitfalls emerge from using caches in conjunction with atoms and the fact that there are cache implementations that aren't idempotent (ie. the ttl cache)
one element of being "complex" is that it can be broken down into simpler pieces. I'm not sure how you would further simplify the cache protocol: lookup
, has?
, hit
, miss
, evict
, and seed
. I would be curious if someone has some thoughts here.
my thesis is that so many pieces make it far more likely for it to be misused. and using those is almost always wrong
to me the consumer api is the single lookup-or-miss
. and even that can return nil. but everything else is a footgun
personally, I want tools and constructs made out of simpler pieces (which usually means more pieces) even if I don't use the simpler pieces directly
I think think it's good that swap!
packages the CAS model in a way that makes it easier to use, but I'm also happy that it is built from simpler parts (eg. I can use compare-and-set!
when it makes sense).
i use a countdown latch in my testing. and i would never want its constituent pieces exposed. i want a single dumb api exposed of decrement and wait. same thing with a cache. i don't want to be in charge of hitting on successful lookups
There is also a notion of trying to break thing up into pieces that are orthogonal, or independent, and can be composed in fairly arbitrary ways. Breaking something up into pieces that are small, but can only be correctly combined in about 1% of the possible ways that one can imagine combining them in, definitely seems low on a measure of independence/orthogonality
it seems like most(all?) of the issues directed at the cache protocol are really incorrect usages of atoms
eg. (using mutable cache implementations like ttl and sending side effecting functions to swap!)
I love the idea of pluggable, composable caching strategies, but I don't know how much people really need that much flexibility. We use just TTL and basic at work (mostly TTL) and we mostly just use wrapped/lookup-or-miss
everywhere (although some places still use an explicit wrapped/lookup
and wrapped/miss
separately -- and a few places use wrapped/seed
, mostly in testing contexts). We could live with a much, much simpler caching library.
I should say that I've done my fair share of defining and using APIs that have significant dependencies on in what order they must be called in, or else things can go wrong. For some kinds of functionality one wants to implement, it can be challenging to think of a way to create an API that does not have those properties.
i think of ghadi's admonition to use a crypto library that doesn't let you do it wrong
I have the same feelings as @seancorfield about the caching stuff
a guiding principle that I return to often is "make the common case easy and the complex case possible"
agreed. there are many different reasons to use caches that have different requirements
I am guessing SPI here does not mean "Serial Peripheral Interface", but Google isn't helping me find a better expansion right now
API is a top-end interface. SPI is a bottom-end interface. or at least that's how I think about it