This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-11
Channels
- # announcements (6)
- # babashka (7)
- # beginners (145)
- # biff (2)
- # calva (9)
- # cider (4)
- # circleci (9)
- # clj-commons (22)
- # clj-kondo (26)
- # cljs-dev (70)
- # cljsrn (4)
- # clojure (46)
- # clojure-australia (9)
- # clojure-europe (62)
- # clojure-nl (5)
- # clojure-norway (4)
- # clojure-spec (11)
- # clojure-uk (3)
- # clojurescript (18)
- # copenhagen-clojurians (1)
- # core-async (1)
- # cursive (13)
- # datahike (6)
- # datomic (47)
- # emacs (5)
- # events (2)
- # fulcro (13)
- # google-cloud (2)
- # gratitude (2)
- # helix (5)
- # honeysql (5)
- # hyperfiddle (31)
- # jobs (1)
- # jobs-discuss (6)
- # london-clojurians (1)
- # lsp (5)
- # off-topic (9)
- # polylith (12)
- # portal (18)
- # re-frame (5)
- # reagent (29)
- # releases (2)
- # shadow-cljs (43)
- # specter (1)
- # test-check (8)
- # vim (1)
- # xtdb (66)
Does the datomic cloud client api cache connections? We have a poorly implemented caching mechanism for our datomic connections and I'm wondering if it's even necessary for us to hold onto these objects.
We have a multi tenant architecture. In production, a single server can connect to one of 350+ datomic databases when fulfilling an http request. I assume the people who implemented the connection cache wanted to make sure that getting a connection was as fast as possible
I'm making guesses though. The people who implemented the connection cache (a la memoize) are no longer with the company, commit history has nothing, chat history was lost in an acquisition.
Once a connection is acquired, it can be used indefinitely without perf penalty. Connections are also thread safe.
So they are safe to cache, I figured that much since we've been doing it for years without much issue. I'm guessing the datomic client api doesn't cache them then, and if we want them cached we should continue managing that ourselves.
Right on, thank you Joe. I'll be changing the code to use something like core.cache instead of memoize to manage these things.
An atom may be sufficient as well. FWIW, although it may be quick to acquire a new connection, that does not mean the compute-group the connection object routes the request to has the DB Spun up and there may be a small delay while that DB is loaded. If you wish to avoid this, you can create query-groups specific to a DB or group of DBs to ensure they always remain loaded (e.g. by querying that DB occasionally). The tradeoff here is that all DBs in that compute-node compete for resources.
Yeah, we think we have observed this behavior (delay from the query group loading the DB). Right now all our databases use the same query group. We're also working on changes that will allow us to spread our database connections across a number of query groups instead funneling them into one. There is a good bit of work we have to do to make that happen but it is underway.
@UDVJE9RE3 the last point under connections here indicate that connections are indeed cached and creating them is inexpensive https://docs.datomic.com/cloud/client/client-api.html#connection
Yeah, that documentation contradicts the behavior I've seen though, and what was said in this thread. The connection objects returned are for sure not cached. The instances are not identical across calls to db/connect
.
It also doesn't feel inexpensive. There are network calls being made. I see a significant delay when I call d/connect. Sometimes as much as 1 full second.
I'm going to ask this as a separate question but it relates to the question above about connection caching. Hopefully this all makes sense. We cache the connections in our system but I've found today that when I used d/sync
on my cached connection using a dodgy value for t
(in my case I mistakenly put a tx in there, which is a bigger int) datomic.client.impl.shared/advance-t*
has changed the t
value of the connection state but the stale connection checking which is done in datomic.client.impl.shared/recent-db
will never correct that state because the dodgy value is always greater than the status received from the remote call. I get Database does not yet have t={...}
and it will keep giving me that error until I flush that connection from the cache.
So, my question is, is there a way that connection can be brought back to life or is this just something we need to be very careful about when caching connections?
@neilprosser Passing a future t
to d/sync
is UB, be sure to avoid doing that.
one simple technique I've used when interacting with untrusted clients, is to make an authenticated 'cookie'
server knows some secret...
cookie = HMAC(some_secret, t
)
Thanks @joe.lane and @ghadi. I figured it might just have to be a 'be more careful next time and don't do that'. I hadn't realised that it was a big problem until I broke it today.
I heard a scary thing that cloud’s entity id structure isn’t guaranteed, so masking out the partition bits like on-prem d/tx->t
doesn’t work. So what’s the alternative?
(bit-and eid 0x3ffffffffff)
That is what we use in cloud. Cognitect told us about it
It has not failed us so far, but we do not use this in production code. Only when debugging or auditing our databases
@ghadi Oh, why is that? I wanted to build a consistency mechanism in my application by returning t
to my client and then using it for the following queries which need to be in a consistent state with the data the client has send before.
(I assumed t is exaclty meant for that purpose 😅)
they could supply a T far in the future, or discover and exploit some bug in datomic with invalid values
We do this too (on-prem api though), but 1) we parse the T. 2) we limit its range 3) we check that it isn’t too far “head” of the current T 4) we put a timeout on the deref to wait for the sync to complete.
I see, thank you for pointing that out. I didn’t even think about higher values :man-facepalming: Ok, I imagine that’s manageable. -> Wanted to go for a low timeout retry mechanism (with exponential backoff) anyway.
Hi guys! I am trying to create a datomic schema for a data structure that looks something like this: (some fields omitted, but not important)
{:url url
:negotiable? ""
:agency? false
:raw {:title "asdasd"
:description "asdasd"}}
I am having trouble figuring out how I am supposed to represent the nested map.
Assuming my top-level entity is called a "posting", should I have something like...
:posting/url
:posting/negotiable
...
:posting/raw.title
:posting/raw.description
(I don't even know if the dot syntax is a thing, I'm just guessing)
Or perhaps something like making :posting/raw
a ref type with isComponent? but then I'm not sure how I'm supposed to define its fieldsIf you know the keys and value types you'll need in the nested map, a component entity would make sense. You'd only have to install each attribute and could then transact the nested maps.
@U06FTAZV3 I think I've got it, something like this?
[(create-attr :posting/url :string)
(create-attr :posting/raw :ref {:db/isComponent true})
(create-attr :raw/description :string)
(create-attr :raw/phone :string)])
(excuse my constructor function)(d/transact conn [{:posting/url "http" :posting/raw {:raw/description "desc" :raw/phone "0723"}}])
I've used a convention for nested things that worked quite well in the past. The raw bits (assuming they only show up inside postings) would be called :posting.raw/description
etc.
Yeah, I was going to say that as a newbie, it feels a bit weird to make :raw/something
a "global" datom, when it only makes sense in the context of a :posting
calling it :posting.raw
is just a naming change, or will it actually let me operate on the "raw" as a map when transacting/querying?
If you prefix things with posting.raw/…
Clojure will help with destructuring but Datomic itself will just see these as names. I can't think of a place where any sort of formal hierarchy shows up on the database side of things. They really are just names.
In Clojure it's nice when you can do things like this:
(let [{:posting.raw/keys [description]} posting]
(str "Description is " description))