Fork me on GitHub
#datomic
<
2019-11-07
>
bartuka11:11:33

hi, what is the appropriate way to perform setup & teardown of datomic databases when using datomic cloud? I am using mount and creating a {:db-name "my-db-test"} before testing facts with midje and when it finishes I have a function call to d/delete-database which seems perfect fine for me. However, very often I got an error in the subsequent tests saying:

#error {
repl_1           |  :cause :db.error/db-deleted 463c4ecf-0733-4afd-a41a-16449265372a has been deleted
repl_1           |  :data {:datomic.client-spi/context-id cc5b3341-b4c8-40e9-8cb6-2c7b1fec2f4d, :cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message :db.error/db-deleted 463c4ecf-0733-4afd-a41a-16449265372a has been deleted, :dbs [{:database-id 463c4ecf-0733-4afd-a41a-16449265372a, :t 108, :next-t 109, :history false}]}

bartuka11:11:55

I have a retry logic for that, but it seems not alright and very often the max-retries is reached.

ghadi16:11:11

use ephemeral names for your database -- don't delete and recreate a db with the same name everytime

bartuka17:11:30

yes, I just did that and worked out ok! Thanks!

dmarjenburgh16:11:19

We are hitting the limit of the 4kb bytes per string value in datomic. What are the limitations/consequences of transacting datoms with, say an 8kb string?

dmarjenburgh16:11:12

By hitting the limit, I mean our users really want to store bigger text fields. I’m trying not to have to build something that splits the string it and combines it when querying. We already treat the string as opague (it’s gzipped and base64Encoded before it goes into datomic)

ghadi16:11:56

@dmarjenburgh since you already treat it as opaque, it wouldn't be a big stretch to store it elsewhere

ghadi16:11:29

[:db/add e a (content-hash text)]

ghadi16:11:47

then store the text somewhere else, keyed by content-hash

dmarjenburgh16:11:28

I'm trying to avoid that to keep latency down and the application simpler. I'm wondering why the limit exists.

marshall17:11:38

Datomic is not a BLOB store and does not support storing large opaque objects in datoms We understand this use case and are considering options, but for now, the suggestion from @U050ECB92 to store them out of band is definitely the best aproach

henrik22:11:56

We’ve reached for DynamoDB for smaller stuff, and S3 for file-sized things, and it’s worked out OK. DDB adds something like 10ms on top, worst case. Not excellent, but good enough for our use case. Of course, you miss out on the automatic disk/in-memory caching that Datomic otherwise handles, and may end up hitting DDB quite a lot unless you explicitly handle it in some custom manner.

dmarjenburgh12:11:46

I understand it’s not blob store and we already use s3 for binary data with a reference in datomic. I’m also not saying there shouldn’t be a limit, I’m just trying to understand why the limit is set at 4kb and what the tradeoffs are for storing text that happens to be a bit larger. As it stands datomic will actually happily allow larger strings (I’ve tried strings up to 16kb) and the transaction succeeds and can retrieve the values back seemingly without issue. I see 2 obvious cons: - You can cache fewer datoms in memory - Queries filtering against the large string value will be slower. Maybe I’m missing something else. DynamoDB allows transactions up to 10MB so I don’t view that as the limit. I need to weight this against the cost of storing string of, say 8kb somewhere else from an business/development perspective. Introduced complexity in the application logic, losing transactionality, adding latency and costing development time. I hope you understand where I’m coming from

calebp17:11:25

If we are already subscribed to Datomic Cloud, do we need to go through the marketplace interface to create a new system? Can we just create the system directly in Cloud Formation using the appropriate templates?

marshall17:11:39

Yes, you can absolutely get the templates directly from our releases page and launch that way @calebp https://docs.datomic.com/cloud/operation/new-system.html

calebp17:11:07

Thanks @marshall. That makes life easier

Luke Schubert17:11:13

do rules with multiple definitions evaluate in order?

Luke Schubert17:11:38

or phrased differently, do they short circuit like ORs?

hiredman18:11:14

They are not ors, I don't know the internals, but it is like a logic program, both branches are taken