This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-07-06
Channels
- # aws-lambda (8)
- # beginners (49)
- # boot (13)
- # braveandtrue (2)
- # cider (12)
- # cljs-dev (15)
- # cljsrn (2)
- # clojure (143)
- # clojure-italy (8)
- # clojure-nl (17)
- # clojure-spec (22)
- # clojure-uk (21)
- # clojurescript (145)
- # code-reviews (1)
- # cursive (33)
- # data-science (14)
- # datomic (25)
- # emacs (3)
- # events (1)
- # fulcro (48)
- # graphql (1)
- # onyx (15)
- # perun (5)
- # play-clj (2)
- # protorepl (1)
- # re-frame (27)
- # remote-jobs (3)
- # ring (3)
- # rum (7)
- # shadow-cljs (87)
- # specter (4)
- # test-check (14)
- # testing (2)
- # tools-deps (9)
I want to make a composable query system — starting from a root query that enforces core business logic, add extra filtering, sorting, paginations (limit/skip) etc. I’m pondering whether to make the result of this query to be something Seqable/Countable/Others — so consumers of the API would be able to call normal count/map/filter/reduce operations, rather than custom functions to do things like that. I think the way to do it is use a Record and implement the relevant interfaces right? Also, does this seem like a good idea or should I force consumers of the API to explicitly “realize” the various results, making this ultimately side-effectful operation explicit?
Ideally also I’d like to be able to cache the results and reuse them on later invocations — but discard them if the query itself changes.
I’ve made a thread in Clojureverse here: https://clojureverse.org/t/lazy-sequences-deferred-results-composable-queries-oh-my-please-help/2427 — it’s hard to capture nuance on Slack…
I had this idea literally 10 minutes before leaving work and I know it’s going to torment me all weekend 🙂
I'd caution against using laziness for deferred results if timing of realization or scope of some resource (eg. db connection) is in play. It's relatively common to use laziness to implement streaming in Clojure code, but it's a very common source of errors.
also It's idiomatic to avoid creating new data types until there's some feature of a data type you actually need
so yeah, use hash-maps, vectors, lazy-seqs etc. until you can prove they are not sifficient
why would a future help?
also the way to get a lazy-seq from an iterable is iterator-seq
@orestis from your link > Consumers of the API can just call seq if they want to execute the query. what does that mean?
The point was to force execution of the query, so you get back a plain non-lazy collection.
OK - so the idea is that it's something that acts like a lazy-seq and executes some communication with a remote server when realized? I was confused because seq doesn't force lazy results.
also that's not how existing laziness works - forcing it still gives you a lazy thing back, it's just been realized already
Rather than targeting full lazyness on each item with lazy-seq maybe realizing it in chunks is best
Perhaps I should narrow the context — this is for a relatively simple web API, where you never want to stream data, and the number of items you get back is kept relatively small, e.g. in the hundreds.
In my experience laziness to delay interactions with a stateful API is a source of many problems. Especially when mixed with chunking.
So the laziness part is not specifically needed — just that you can still modify the query but also execute it — and if you’ve executed it once, you don’t have to execute it again.
do you actually want a caching layer?
Perhaps I’m still influenced by mutability — e.g. I realize now that this is how Django’s QuerySets work — you create a QuerySet, that once executed caches the results. But in Clojure, executing my query would just give me a collection back.
by "modify" do you mean make a new slightly different immutable object, or actually mutating an object representing a query?
And then I could package the original query with the collection back together if I need to modify the query and execute again.
I think an actual caching layer would make this all a lot simpler
the problem with memoizing is it can balloon memory - with an actual cache you can control that
for sure, use memoize, if you are OK with every query you ever send staying in memory until reboot
clojure core.cache is good - it plays nicely with immutable things and lets you decide how to implement the actual mutable storage part
The more I think about it, the more I realize laziness is completely irrelevant to what I actually wanted to ask.
and it includes multiple caching algorithms, including LRU
In the sense that I can’t think of where I would find it useful, and I was just influenced by the inherent laziness of lazy-seqs etc.
@orestis yeah, if you squint at it laziness can look like a cache, but when you need a cache, I'd say use an actual cache :D
<dependencies>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>core.cache</artifactId>
<version>0.7.1</version>
</dependency>
</dependencies>
right, most of what it adds to core.cache is implicitly deciding to use an atom
where core.cache lets you use anything
What about the idea of implementing the Seqable/Countable interfaces so that the query gets executed when seq/vec/count
is called — in contrast to calling something like (q/execute query)
or (q/count query)
(vs. (vec query)
, (count query)
)
once again, seq doesn't force lazy things
https://github.com/clojure/core.memoize/blob/master/src/main/clojure/clojure/core/memoize.clj#L342
Ah, but it wouldn’t be a lazy thing — it would be my record, that when asked to return a Seqable
thing, it would just execute the query and give back the results of that.
well, incidentally it forces 1 chunk because it checks the head, but...
honestly, even though it brings you into core functions q/execute
is easier to read than vec
@orestis does it do the same query again when you ask for a seq again?
@noisesmith Good question — I’d like to say no, it caches the results, but in all honesty, I can’t see a use case where that was ever going to be needed — so perhaps, the answer is yes, when you call seq, the query is executed again.
also how do you manage the cache? who owns it?
right
why is this better than just a function that gets the results and then a plain collection of the results? why combine them in this way?
It would be nice to have a per-request cache to ensure at least consistent results. It might even be needed with Mongo’s lack of joins (this is a legacy thing so I can’t switch databases just yet).
@dpsutton This is the question I’m asking 🙂 Perhaps I’m overthinking things, Clojure gives you enough tools that you start to think “how would it look if I applied protocols here”.
The whole idea results out of trying to figure out what something like (find-widgets)
should return. Does it return a list of widgets, hitting the database? Or does it return something that can be further filtered before hitting the DB? And how would the API look like for these use cases?
There are some core rules relating to permissions and other things (e.g. filter things which are soft-deleted), but there are other rules that are mostly presentation related and must be added on top of the core business rules.
when I used mongo with clojure we had a function that put all queries through a ttl cache, due to the domain logic we knew that caching in an N minute timespan was always OK
the ttl cache was done with core.cache and an atom as the in-memory store
the same in-memory cache was used for all queries (simple enough since atoms keep things concurrency safe, and we had low enough throughput of fresh queries that mutating the atom wasn't a bottleneck)
Clojure lets you make powerful assumptions about what you can do with immutable data. Unless you can enforce the same semantics on a DB level (eg. Datomic), don't try to blur the distinction of data operations in clojure with query logic.
Hehe, that’s an option — but then you end up with something like (find-widgets db user filter sort paginate)
which kinda looks off to me.
because it will lead to bad bugs when devs assume "data is sane and works in functionally / mathematically meaningful ways" and your query layer can't back that up
We will have one of those find-widgets
for every entity in our system, and we will need to compose them to do joins, so if you want to eventually change the signature of these functions you’re in a lot of pain.
(find-widgets db user query-map)
and then you just build an interpreter for query-map
. A "signature change" is just a change to your query interpreter
@noisesmith Right, I agree with that — when I realized laziness is not worth it and is irrelevant to my API design 🙂
@dpsutton and I guess my mini-DSL could just be a few functions with meaningful names that construct that query-map
?
that's what i would do. it's just data so no macros involved, transform maps and vectors into the final query that mongo will interpret
One thing that I might need to do is to switch from a plain query to an aggregation pipeline, according to the specific query. You can do some joins with a pipeline that I could leverage.
Perhaps the correct thing to do is use find-widgets-query
vs find-widgets
, then just have an execute-query
. Then the intent of the functions is clear.
Thanks for all the input everyone, I need to go but I will update my Clojureverse topic if you’re interested in following up there. Much appreciated @noisesmith @emccue @dpsutton
Do you have any suggestions how can I implement --verbose
switch for clojure.tools.logging
? That is, enable log/debug
output to console.
What’s the underlying logging engine that you’re using? (eg log4j, logback etc)
(and, does it have to be --verbose
or can it be any command line param?)
Looking through here (http://clojure.github.io/tools.logging/) it looks like the *logger-factory*
var holds the factory, which will return one of these: https://github.com/clojure/tools.logging/blob/fec4503b253ee1dab47f8613f6ef35e08b7e4707/src/main/clojure/clojure/tools/logging/impl.clj#L214
how do I use fnil
with java functions, I forgot.. need something like (fnil .intValueExact)
java methods are not functions, to use fnil you need a function
(fnil #(.intValueExact %) n)
after my edit?
thank you @noisesmith
ins)user=> (def n-fnil (fnil #(.intValueExact %) (biginteger 0)))
#'user/n-fnil
(cmd)user=> (n-fnil nil)
0
(ins)user=> (n-fnil (biginteger 88888888888888))
ArithmeticException BigInteger out of int range java.math.BigInteger.intValueExact (BigInteger.java:4550)
(cmd)user=> (n-fnil (biginteger 8888888))
8888888
:thumbsup:
If I have a map, say {:a 1 :b 2 :c 3 :d 4}
, and I have a set #{:a :b}
, what’s the best way to get the first key existing in the map and the set?
Let’s say no
(first (set/intersection s (set (keys m))))
or maybe (key (first (select-keys m s)))
but the second one could NPE on the key call
(first (keys (select-keys m s))) - no more npe
but "first" isn't really a meaningful thing for sets and maps
(beyond the fact that first implicitly calls seq which gets you a thing that acts like a list that is...)
(select-keys {:a 1 :b 2 :c 3} #{:a :b})
you can just select keys to see the "intersection"
often, I type faster than I think
not to mention faster than a reasonable person reads for comprehension
Thanks for the options
@colindresj what are you trying to achieve? often times when you're asking for the first of a set or map you need a simple change one stack frame up
I need a function that will take any arbitrary map and return the first matching key from a set of keys
Actually doesn’t need to be a set, could be any iseq, I just like that it’s represented visually in a set
Yeah this is pretty straightforward, albeit kind of uncommon
well, select-keys means you no longer get the first match
First in my case is just first to be encountered, not necessarily in a specific order
if the input might be a seq, and the order matters
(first (filter #(contains? m %) keylist))
Yeah above is what I had originally, just wanted to see other options
if nil/false are never the val, (first (filter m keylist))
also works