Fork me on GitHub
#datomic
<
2021-09-07
>
Faiz Halde06:09:51

could someone answer this (forgot to first check if there was a channel dedicated for datomic here)? https://stackoverflow.com/questions/69078793/datomic-durability-guarantees-under-storage-failure thanks2

favila11:09:02

The “durability” page you link explains how datomic uses storage. Most of what it writes is immutable (ie the entry is never updated in storage), so there’s nothing to invalidate. The mutable parts are few, tiny, and updated by only a single writer (the transactor) and reference immutable trees. (In fact the transactor is the only process which writes anything to storage—peers are read-only). These make datomic very tolerant of storage misbehavior-it just really doesn’t depend on updates to the same entry from multiple sources, and that’s where most of the tricky code (and bugs) are. You will get an error rather than an incorrect query result. That said it’s unreasonable to expect datomic to work with completely broken storage.

Faiz Halde12:09:35

ok, my understanding might be very limited here. I was talking about inserts only the scenario i was referring to was 1. let's say there's an empty datomic database 2. writer inserts a fact (Cassandra acknowledged it) 3. client immediately reads the fact (I'm guessing the client/peer node caches the result) 4. some failure happens on Cassandra and the acknowledged writes disappear (which seems to be possible with Cassandra as per jepsen, not sure if it's fixed in the newer versions) So now there's a disparity between the peer/client cache (unsure who caches) and Cassandra storage. Am I making sense in this scenario? Datomic does recommend to replicate the data 3 times for Cassandra so there's certainly a high degree of reliability

favila12:09:40

Peers cache, but the cache may be pre-filled by a transactor if they share the same cache. In your scenario, a key is simply missing. The moment a peer needed that segment and it is not in a cache layer, datomic would fail with an error.

favila12:09:08

The jepsen article says these lost writes were from the “limited transaction” mode. I don’t see why datomic would use that.

favila12:09:11

I have no knowledge of datomic’s Cassandra implementation nor have I ever used it, but I know datomic doesn’t need cross-key transactional writes.

Faiz Halde14:09:20

ok thanks2 I'll try to read further about its internals

favila14:09:54

datomic uses storage as a pretty-dumb key-value store of binary blobs where the vast amount of data (key count and byte-count) is immutable

favila14:09:18

there’s just not much that can go wrong. The datomic cloud product doesn’t even bother with a database: the primary datastore is s3 (with dynamo for that handful of mutable keys I mentioned) with multiple cache layers on top

😄 2
favila14:09:48

the mutable keys are never cached

Jakub Holý (HolyJak)12:09:23

Is this the right approach to finding all the people that NEVER owned a Tesla, i.e. this combination of d/history (to filter out also people that had a Tesla in the past but do not have it anymore) and not ? 🙏

(d/q '[:find ?e
       :where [?e :person/id]
       (not [?e :person/car ?car]
         [?car :car/make "Tesla"])]
  (d/history db))

favila12:09:05

Seems right

Jakub Holý (HolyJak)12:09:30

Hi @augustl @magnars ! You have used on-prem in the past, any idea whether it is possible / how to use with it - from my application = peer process - the library https://github.com/hden/ragtime.datomic, which builds on the datomic.client.api instead of the Peer datomic.api? I would rather not have to start https://docs.datomic.com/on-prem/peer/peer-server.htmlhttps://docs.datomic.com/on-prem/peer/peer-server.html so that the first peer instance can use it when running Ragtime... 🙏

augustl12:09:37

I've never felt compelled to wrap the datomic API so I have little experience to share on that front :)

😿 2
Fredrik13:09:05

Replacing API calls such as (satisfies? client-protocols/Connection conn) with`(instance? datomic.Connection conn-mem)` , and (datomic/transact conn {:tx-data tx-data}) with (datomic/transact conn tx-data should probably suffice.

👀 2
souenzzo13:09:33

Hello. Can I use #"regex" as a parameter of a query in datomic.client.api ? I used to do that in on-prem. Testing in dev-local it works, but i suspect that if I run it in cloud, it will have some isssue in serialization/parsing. (example in thread)

souenzzo13:09:40

(let [client (-> {:server-type :dev-local
                  :system      "hello"}
               d/client
               (doto (d/delete-database {:db-name "hello"})
                     (d/create-database {:db-name "hello"})))
      conn (d/connect client {:db-name "hello"})]
  (d/q '[:find ?n
         :in $ ?re
         :where
         [?e :db/ident ?ident]
         [(name ?ident) ?n]
         [(re-find ?re ?n)]]
    (d/db conn) #"^[a-t]+$"))
=>
[["add"] ...]

grzm15:09:23

good morning, all! Looking at the REBL page on https://docs.datomic.com/cloud/other-tools/REBL.html#nREPL I see that the id for the nREPL section is spelled wrong (`<h3 id="nRPEL">nREPL</h3>`) which makes the anchors fail.

👍 2
octahedrion17:09:29

I'm using datomic.ion.dev :as dev for Ions deployment and I can successfully (dev/push {}) , deploy etc. However I can only do it once -- on subsequent times I get:

Execution error (RejectedExecutionException) at java.util.concurrent.ThreadPoolExecutor$AbortPolicy/rejectedExecution (ThreadPoolExecutor.java:2055).
Task java.util.concurrent.FutureTask@668a219d[Not completed, task = com.amazonaws.services.s3.transfer.internal.CopyMonitor@15698dd0] rejected from java.util.concurrent.ThreadPoolExecutor@7734f8b0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]
and the only way I can find to fix it is to restart my REPL which is something nobody wants to have to do. How can I make it work repeatedly ?

tony.kay17:09:15

I'm working on this datomic streaming backup library, and I am seeing a Datom from our production database that I do not understand. Perhaps someone can fill in a gap in my knowledge, or perhaps I just have corruption in the database. I have a Datom [E A V] where A is a ref attribute. If I look (using the datoms) at the source database on the :vaet index I definitely have this Datom; however, when I try to pull this "target" entity I get nothing...well, technically I get this:

(d/pull db '[*] 69409970038386856)
=> {:db/id 69409970038386856, :inventory/_product ...}
when I try to find an EAVT with that V as the E...nothing. There is no "target" entity for this ref attribute.

tony.kay18:09:34

Actually, it looks like creating a ref to nowhere is perfectly fine in Datomic...

tony.kay18:09:47

I can just make up a random int and point a ref at it

tony.kay18:09:14

So, probably a mistake in the code somewhere made that...I guess I'll just elide those in restores

tony.kay17:09:29

is that corruption, or did I miss something in Datomic class?

tony.kay17:09:29

I did queries with as-of databases as well, and don't see the target at the t just before the transaction that asserts this ref datom

stuarthalloway18:09:13

@tony.kay A better way to look at it is "there is no such thing as nowhere". An entity is logically derived from all datoms with a common e. There are many valid modeling reasons that the number of such datoms could be zero.

tony.kay18:09:55

But if I try to transact a new entity with that explicit E later, Datomic refuses to...so how would I ever "add facts"?

tony.kay18:09:32

It just seems to me that the lack of a ref should be tied hand-in-hand with a lack of facts to point to

stuarthalloway18:09:56

at all points in time?

stuarthalloway18:09:00

at one point in time?

tony.kay18:09:23

I add EAV where A is a ref and V is a long. Later, I add EAV with E as that original V...Datomic refuses

stuarthalloway18:09:39

That is not always true.

stuarthalloway18:09:06

'Entities do not "exist" in any particular place; they are merely an associative view of all the datoms about some E at a point in time. If there are no facts currently available about an entity, Database.entity will return an empty entity, having only a :db/id.' -- https://docs.datomic.com/on-prem/overview/entities.html

stuarthalloway18:09:24

@tony.kay I think the most helpful doc improvement in this area would be a table enumerating the validity constraints enforced by Datomic at transaction time. With that, at least it would be clear that what you are imagining differs.

stuarthalloway18:09:01

Second most helpful would be more explanation of 'why'.

tony.kay18:09:05

Yes, such documentation would be helpful...but for the present case: How would I ever go about making that ref point to real facts in later time? I'm not able to manage it

tony.kay18:09:38

Datomic just refuses to let me use that number (I put in as V for a ref datom) as an E in a new entity

tony.kay18:09:03

oh...do I have to resolve the E of that datom and tie it together in the tx?

stuarthalloway18:09:10

This is still about copying an existing db, right?

tony.kay18:09:42

eventually...should just elide those datoms or not I guess.

tony.kay18:09:50

but also about a hole in my understanding as well

stuarthalloway18:09:03

what error do you get when Datomic rejects your datom?

tony.kay18:09:03

I guess in the usage of such a thing, I can always re-transact the edge with a new entity...and the old random long that was the V will be replaced. So, no big deal. On restore, how would I go about picking a value to restore this edge?

tony.kay18:09:41

(d/transact c {:tx-data [[:db/add "thing" :invoice/items 82387432]]} )
=>
{:db-before #datomic.core.db.Db{:id "5b35b695-b633-4376-b899-b28c97fdd596",
                                :basisT 85,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "5b35b695-b633-4376-b899-b28c97fdd596",
                               :basisT 86,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533398 50 #inst"2021-09-07T18:56:20.982-00:00" 13194139533398 true]
           #datom[79164837200686 316 82387432 13194139533398 true]],
 :tempids {"thing" 79164837200686}}
(d/transact c {:tx-data [[:db/add  82387432 :invoice-item/precise-quantity 3.4]]} )
Execution error (ExceptionInfo) at datomic.core.error/raise (error.clj:55).
:db.error/invalid-entity-id Invalid entity id: 82387432

tony.kay18:09:59

I picked 82387432 out of thin air

stuarthalloway18:09:51

Whenever you do not have an id yet, tempids are the right answer -- then update your knowledge from the tempid map after the transaction

stuarthalloway18:09:34

so picking "82387432" (note the quotes) out of thin air would be fine, albeit not very semantic

tony.kay18:09:38

but my point was to test what happens if someone has put an arbitrary value as a ref's target...like how you would heal it...and as I said, it looks like you just re-transact the edge and get a new ID

tony.kay18:09:26

but from a "restore" perspective, what is the right thing to do? I cannot jsut use a tempid as a valud

tony.kay18:09:29

that is rejected as well

stuarthalloway19:09:03

How could "someone put an arbitrary value" -- they would have the same problem you did, being rejected by Datomic.

tony.kay19:09:15

That first transaction WORKS

tony.kay19:09:28

even though that is a random, unused value in Datomic

tony.kay19:09:34

and :invoice/items is a ref many

stuarthalloway19:09:40

But who cares? No subsequent transaction can say anything else about the entity.

tony.kay19:09:59

right, so I should elide that datom in the restore, right?

stuarthalloway19:09:12

So it is dead on arrival, and your restore process can know that the code that made it was buggy, and just drop it.

tony.kay19:09:31

OK, that is the conclusion I had come to...was just making sure I wasn't missing something

tony.kay19:09:45

like somehow an edge to nowhere was a useful "fact" in itself

stuarthalloway19:09:01

Re: elide -- yes, but I would be worried that that original program was losing data you did not know about.

tony.kay19:09:15

that was my worry, thus my question...

stuarthalloway19:09:37

An edge to nowhere can be a useful fact -- maybe it was not nowhere at some other point in time.

tony.kay19:09:55

Datomic removes refs on the referent being removed, no?

tony.kay19:09:07

but only for retractEntity

stuarthalloway19:09:16

The problem you are describing is not merely an edge to nowhere, it is an edge to neverwhere

simple_smile 4
Daniel Jomphe19:09:18

(feels like a fairy tale) - I'm watching your convo as I'm going to try out Tony's lib on our db.

tony.kay20:09:44

feel free to help out 🙂

Daniel Jomphe20:09:57

Will try, although I'm only starting out with Datomic!

tony.kay20:09:54

I hope these last two bugs I just fixed will be the last...but this is my 15th attempt at a restore.

tony.kay20:09:20

at least the "resume" an interrupted restore seems to be working well, so I'm not having to start over from scratch

Daniel Jomphe20:09:34

That's a great property!

tony.kay20:09:01

at this speed of restore I'm looking at 44 DAYS to do one...having to start over isn't a thing I can stand

tony.kay20:09:48

Make sure you're using 0.0.15, which is bw compatible with backups made since I had to rework the tempid remapping logic.

2
tony.kay20:09:02

So, if you made a backup with a version older than that, it is no use, and never was. Still working out the kinks.

tony.kay20:09:33

Don't expect the backup to change again. The streamed data was fine...I just wasn't storing the right metadata.

2
Daniel Jomphe20:09:25

So this is going to be a permanent process running alongside the apps. The operational characteristics of the solution are a bit unclear to me for now. Each of our own environments is its own AWS account with its own Datomic stack. So it looks like I'd need my prod ion to include the backup process, pushing to a shared s3 bucket, and my e.g. staging ion to run the restore process, but then this staging db should be read-only, etc. etc., lots of sub-questions appear.

Daniel Jomphe20:09:03

Since our db is so small for now, I should just start experimenting in a single environment, with in-memory storage, than scale-up the ops gradually from there, I suppose.

tony.kay20:09:38

The primary use-case is streaming replication: Yes, a node in your production stack continuously runs backup-segment! (after a delay, or some number of txes have occurred..up to you). That puts the next tx-range into storage (typically s3). Then on a different system (in my case it needs to be in diff region) I run a simple app that does nothing but run restore-segment! in a loop, which adds those txes to the target db.

tony.kay20:09:05

And yes, if you transact something new against that target, it is now no longer capable of restoring from the original, because the tx-time will be too far in the future

tony.kay20:09:38

The only time you'll ever write to that db other than restore is in a disaster recovery, where it is now the new "master"

Daniel Jomphe20:09:42

So we should probably restore to 2 dbs in case we need to stand-up one of those!

tony.kay20:09:25

sure, can restore to as many as you want (to pay for 😉 ) at a time

tony.kay20:09:55

but if you "failed over" that means your original is trash, and you probably don't or cannot get any more data from it

tony.kay20:09:13

but you might more than one restore running just in case one of your restores has a critical failure...because it takes so long to get a restore "caught up". At least you wouldn't have a DR time gaps of weeks/months waiting for the new restore to get caught up

Daniel Jomphe20:09:43

It does feel like we're going to have to invest weeks/months of careful planning and trials into this!

tony.kay19:09:24

yeah, I did scan the history looking for related facts, but found none...coworker is certain it is from a bad data fix in the past

tony.kay19:09:41

useful thing to have 58M transactions to test your OSS library against 😄

🙂 2
stuarthalloway19:09:49

"Alexa, tell @jaret to make a task for me to make a table documenting tx constraint enforcement."

picard-facepalm 4
datomic 2
Daniel Jomphe19:09:39

To be pronounced Alexslack

ghadi19:09:59

have encountered the same situation in my own transaction replayer script

ghadi19:09:29

I think the strategy I took was to drop all heretofore unseen eids that appeared in :v position without ever appearing in :e position

Daniel Jomphe19:09:56

What to think of this... 4 cognitects in the last 10 minutes chiming in about Datomic Cloud backup & restores. 🙂

2
2
2
🙏 2
jaret19:09:41

Haha! well, to be fair it's been on our minds for 4 years 😉.

😄 2
☺️ 2
Daniel Jomphe19:09:27

Haha! Let it fly, who knows this time if it'll finally materialize into something concrete. 🙂

stuarthalloway19:09:44

@tony.kay important non-obvious detail for your work -- You should not assume that Datomic's built-in attributes have the same entity ids in all databases, therefore you should maintain a table from attribute keyword to entity id.

stuarthalloway19:09:16

It might be very tricky to discover this assumption in testing, as all dbs made in your account might very well have common set of attribute entity ids.

tony.kay19:09:50

right, I did not expect automated tests to catch that kind of thing, since there is no version diff

tony.kay19:09:44

Yeah, I track it via:

(id->attr (d/as-of db #inst "2000-01-01"))

tony.kay19:09:19

don't remember why I didn't want the user's schema in this mapping...but I am tracking them

tony.kay19:09:06

Oh, right, I don't need their's because I'm tracking the real-time remappings of IDs, including those of user schema ident entities

tony.kay19:09:27

so I just pulled the built-in ones since those don't get "restored"

tony.kay19:09:40

thanks for the tip, but yeah, covered that one

Joe Lane19:09:13

@tony.kay That approach assumes all Datomic attributes are present at the birth of the database, which isn't true for databases which upgraded their schema via https://docs.datomic.com/client-api/datomic.client.api.html#var-administer-system . For databases that were upgraded, the attribute identifiers for tuple related idents will be introduced both at a later t than #inst "2000-01-01" and they will also have numbers similar to user defined attributes. Where this can become extremely complicated is that the attribute identifiers for tuple related idents in the new destination database will be present at the birth of the database and have a lower, different number than from the source db.

tony.kay19:09:34

@U0CJ19XAM so my restore algorithm treats any schema after that point in time as something to restore on the target, which you're right, could present a difficulty since that schema will be there in a "new db" but would also be in the txn stream from the old one.

tony.kay19:09:11

won't the :db/ident cause an "upsert" in that case, though? I'll have to analyze it

tony.kay19:09:56

(we actually have a db that has this exact situation, so if my restore completes it will be a good indicator that what I did "works" for that case)

tony.kay19:09:13

ideas on how I might write a test around that?

tony.kay20:09:30

guess I could find what that looks like in the backup and add that to a test to restore to a db that has it from birth

tony.kay20:09:23

So, I tried the upgrade txn against a db using my algorithm, and it seems OK. It turned the install into an upsert, was OK with the install attribute against a tempid that already existed, etc. I think the way the lib is written, it will naturally "just work" against that scenario @U0CJ19XAM, but thanks for pointing it out so I could try it to see.

tony.kay20:09:12

I'm storing the "original ID" on the entities that have been created on the new db, so I can use that to resolve the remappings. So, this "upgrade schema" transaction put the mapping on the pre-existing schema in the new db, so that the future transactions in the stream should automatically remap the upgraded schema IDs to the new database's "original" schema IDs. I was worried that "install attribute" would be special, but it seems to have handled it ok.

tony.kay19:09:12

right...pretty sure I'm doing that, even for the built-ins

tony.kay19:09:57

The a's always come across as integers, so I had to include that mapping in the backup info