Fork me on GitHub
#datomic
<
2020-10-08
>
zilti01:10:15

Is it a known bug that when there's a bunch of datums that get transacted simultaneously, it can randomly cause a :db.error/tempid-not-an-entity tempid '17503138' used only as value in transaction error?

favila01:10:56

The meaning of this error is that the string “17503138” is used as a tempid that is the value of an assertion, but there is no place where the tempid is used as the entityid of an assertion; the latter is necessary for datomic to decide whether to mint a new entity id or resolve it to an existing one

zilti01:10:46

Well, as you can see in the actual datums I posted, it clearly is being used as :db/id.

zilti01:10:29

I had my program dump all datums into a file before transacting, and I copied the two that refer to this string over into here

favila01:10:52

In your example, I see the second item says :account/accounts “17503138”. Are both these maps together in the same transaction?

favila01:10:43

(Btw a map is not a datum but syntax sugar for many assertions—it’s a bit confusing to call it that)

zilti01:10:21

Yes, they are both together in the same transaction. True, I mixed up the terminology... Entity would be more fitting

favila01:10:48

If they are indeed both in the same tx I would call that a bug. Can you reproduce?

favila01:10:10

Why is each map in its own vector?

zilti01:10:26

Yes, reliably, every time with the same dataset. Both locally with a dev database as well as on our staging server using PostgreSQL.

zilti01:10:08

Conformity wants it that way, for some reason

favila01:10:27

Conformity for data?

zilti01:10:32

I had that same issue a while back in a normal transaction without conformity as well though

favila01:10:45

Separate vectors in conformity means separate transactions...

zilti01:10:46

The migration library called conformity

favila01:10:01

I’ve only ever used conformity for schema migrations; using it for data seems novel; but I’m suspicious that these are really not in the same transaction

favila01:10:51

See if you can get it to dump the full transaction that fails and make sure both maps mentioning that tempid are in the same transaction

zilti01:10:20

It is often caused by one single entry that is the same structure as many others. Everything is fine, but for some reason, Datomic doesn't like it. Removing that one entry solves the problem.

marshall12:10:41

why are both of those entity maps in separate vectors? If you’re adding them with d/transact , all of the entity maps and/or datoms passed under the :tx-data key need to be in the same collection

marshall12:10:17

based on the problem you described, I would expect that error if you transacted the first of those, and then tried the second of those in a separate transaction

marshall12:10:26

if they’re asserted in the same single transaction it should be fine

zilti01:10:18

Ordering of the entries in the transaction vector doesn't seem to matter either

zilti01:10:53

The two datums causing problems:

[{:account/photo
   "REDACTED",
   :account/first-name "REDACTED",
   :account/bio
   "REDACTED",
   :account/email-verified? false,
   :account/location 2643743,
   :account/vendor-skills [17592186045491],
   :account/id #uuid "dd33747e-5c13-4779-8c23-9042460eb3f3",
   :account/vendor-industry-experiences [],
   :account/languages [17592186045618 17592186045620],
   :account/vendor-specialism 17592186045640,
   :account/links
   [{:db/id "REDACTED",
     :link/id #uuid "ea51184c-d027-44d0-8f20-df222e58daf3",
     :link/type :link-type/twitter,
     :link/url "REDACTED"}
    {:db/id
     "REDACTED",
     :link/id #uuid "c9577ca4-332d-41f0-b617-c00e89fc94b4",
     :link/type :link-type/linkedin,
     :link/url
     "REDACTED"}],
   :account/last-name "REDACTED",
   :account/email "REDACTED",
   :account/vendor-geo-expertises
   [17592186045655 17592186045740 17592186045648],
   :db/id "17503138",
   :account/vendor-type 17592186045484,
   :account/roles [:account.role/vendor-admin],
   :account/job-title "Investor"}]
and
[{:account/primary-account "17503138",
   :company/headline "REDACTED",
   :account/accounts ["17503138"],
   :tenant/tenants [[:tenant/name "REDACTED"]],
   :company/name "REDACTED",
   :company/types [:company.type/contact],
   :db/id "REDACTED",
   :company/id #uuid "ee26b11f-53ba-43f9-a59b-f7ad1a408d41",
   :company/domain "REDACTED"}]

Adrian Smith09:10:00

During a meetup recording that I haven't uploaded yet I recorded my own maven private token from https://cognitect.com/dev-tools/view-creds.html is there a way I can regenerate that token?

marshall13:10:39

Can you send an email to <mailto:[email protected]|[email protected]> and we will help with this?

Adrian Smith21:10:55

thank you, I've just sent an email over

Black11:10:00

Hey, I just missing something and can't figure out what. I am calling tx on datomic:

(defn add-source [conn {:keys [id name]
                        :or {id (d/squuid)}}]
  (let [tx {;; Source initial state
            :db/id                    (d/tempid :db.part/user)
            :source/id                id
            :source/storage-type      :source.storage-type/disk
            :source/job-status        :source.job-status/dispatched
            :source/created           (java.util.Date.)
            :source/name              name}]
    @(d/transact conn [tx])))

;; and then later API will call
(add-source conn entity-data)
After I call add-source entity is created, but after another call is made old entity is rewritten, only if I call transact with multiple transactions I can create multiple entities, but other than that old entity is being rewritten. I am new to datomic, and I can't find any resources about that, can anyone help?

favila12:10:50

tempids resolve to existing entities if you assert a :db.unique/identity attribute value on them that already exists. Are any of these attributes :db.unique/identity? are you sure you are not supplying an id argument to your function?

favila12:10:25

(btw I would separate transaction data creation into a separate function so it’s easier to inspect)

Black12:10:18

{:db/doc                "Source ID"
           :db/ident              :source/id
           :db/valueType          :db.type/uuid
           :db/cardinality        :db.cardinality/one
           :db/id                 #db/id [:db.part/db]
           :db.install/_attribute :db.part/db}

Black12:10:45

this is schema for source/id, I am not using :db.unique/identity

Black12:10:51

And I agree with separation tx and creation but first I would like to get it work

Black12:10:43

Id I removed :db/id from transaction, I shoud still be able to create new entity, right? But everytime first one is rewritten

favila12:10:13

can you give a clearer get/expect case? maybe a repl console?

favila12:10:10

something that shows you calling add-source twice with the returned tx data, and pointing out what you think is wrong with the result of the second call?

Black12:10:43

Ok I had unique on other parameter:

{:db/doc "Source name"
 :db/ident :source/name
 :db/unique :db.unique/identity
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/id #db/id [:db.part/db]
 :db.install/_attribute :db.part/db}
If I removed it, all entities are created and it works how I expected. So I will read more about unique attribute, thanks @U09R86PA4 I would not noticed it without your help!

zilti14:10:41

Well, I guess I am going to do my migrations using a home-made solution now. I just lost all trust in Conformity. It doesn't write anything to the database most of the time I noticed.

zilti14:10:22

Or are there alternatives?

ghadi14:10:23

can you describe your problem with conformity in more detail?

zilti15:10:15

I have a migration that is in a function. Conformity runs the function normally, but instead of transacting the data returned from it, it just discards it. The data is definitely valid; I made my migration so it also dumps the data into a file. I can load that file as EDN and transact it to the db using d/transact perfectly fine.

zilti15:10:55

Conformity doesn't even give an error, it just silently discards it.

ghadi15:10:12

is this cloud or on prem?

zilti15:10:02

On prem, both for the dev backend and the postgresql one

ghadi15:10:19

not sure what to tell you. you need to analyze this further before throwing up your hands

favila15:10:45

Conformity does bookkeeping to decide whether a “conform” was already run on that database. If you’re running the same key name against the same database a second time, it won’t run again. Is that what you are doing?

3
favila15:10:12

Conformity is really for schema management, not data imports

3
zilti15:10:45

No, that is not what I am doing.

zilti15:10:13

Well, the transaction is changing the schema, and then transforming the data that is in there.

zilti15:10:23

Or at least, that is what it is supposed to be doing.

favila15:10:52

We’re pointing out a case where it may evaluate the function but not transact

favila15:10:09

you can use conforms-to? to test whether conformity thinks the db already has the norm you are trying to transact

favila15:10:19

that may help you debug

zilti15:10:47

Well, what is the second argument to conforms-to? ? It's neither the file name nor the output of c/read-resource

zilti15:10:40

It wants a keyword, but what keyword?

favila15:10:55

the keyword in the conform map

favila15:10:32

{:name-of-norm {:txes [[…]] :requires […] :tx-fn …}}

favila15:10:42

the :name-of-norm part

favila15:10:12

that’s the “norm”

Filipe Silva15:10:01

heya, coming here for a question about datomic cloud. I've noticed that while developing on a repl, I get exceptions as described in the datomic.api.client api:

All errors are reported via ex-info exceptions, with map contents
as specified by cognitect.anomalies.
See .
But on the live system, these exceptions don't seem to be ex-info exceptions, just normal errors. At any rate, ex-data returns nil for them. Does anyone know if this is intended? I couldn't find information about this differing behaviour. A good example of these exceptions is malformed queries for q . On the repl, connected via the datomic binary, I get this return from ex-data
{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Query is referencing unbound variables: #{?string}\", :variables #{?string}, :db/error :db.error/unbound-query-variables, :dbs [{:database-id \"48e8dd4d-84bb-4216-a9d7-4b4d17867050\", :t 97901, :next-t 97902, :history false}]}
But on the live system, I get nil.

marshall15:10:56

@filipematossilva are you using the same API (sync or async) in both cases?

👋 3
Filipe Silva15:10:01

think so, yeah

Filipe Silva15:10:32

have a ion handling http requests directly, and the repl is calling the handler that's registered on the ion

Filipe Silva15:10:48

so it should be the same code running

Filipe Silva15:10:23

we can see on the aws logs that the error is of a different shape

Filipe Silva15:10:28

let me dig it up

Filipe Silva15:10:14

on the aws logs, logging the exception, shows this

Filipe Silva15:10:16

{
    "Msg": "Alpha API Failed",
    "Ex": {
        "Via": [
            {
                "Type": "com.google.common.util.concurrent.UncheckedExecutionException",
                "Message": "clojure.lang.ExceptionInfo: :db.error/not-a-binding-form Invalid binding form: :entity/graph {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Invalid binding form: :entity/graph\", :db/error :db.error/not-a-binding-form}",
                "At": [
                    "com.google.common.cache.LocalCache$Segment",
                    "get",
                    "LocalCache.java",
                    2051
                ]
            },
            {
                "Type": "clojure.lang.ExceptionInfo",
                "Message": ":db.error/not-a-binding-form Invalid binding form: :entity/graph",
                "Data": {
                    "CognitectAnomaliesCategory": "CognitectAnomaliesIncorrect",
                    "CognitectAnomaliesMessage": "Invalid binding form: :entity/graph",
                    "DbError": "DbErrorNotABindingForm"
                },
                "At": [
                    "datomic.core.error$raise",
                    "invokeStatic",
                    "error.clj",
                    55
                ]
            }
        ],

Filipe Silva15:10:47

(note: this was not the same unbound var query as above)

Filipe Silva15:10:20

printing the error on the repl, we see this instead

#error {
                                      :cause "Invalid binding form: :entity/graph"
                                      :data {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form, :dbs [{:database-id "48e8dd4d-84bb-4216-a9d7-4b4d17867050", :t 97058, :next-t 97059, :history false}]}
                                      :via
                                      [{:type clojure.lang.ExceptionInfo
                                        :message "Invalid binding form: :entity/graph"
                                        :data {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form, :dbs [{:database-id "48e8dd4d-84bb-4216-a9d7-4b4d17867050", :t 97058, :next-t 97059, :history false}]}
                                        :at [datomic.client.api.async$ares invokeStatic "async.clj" 58]}]

marshall15:10:51

that ^ is an anomaly

marshall15:10:54

which is a data map

Filipe Silva15:10:49

more precisely, (ex-data e) returns the anomaly inside that exception

marshall15:10:23

ah, instead of ex-info ?

Filipe Silva15:10:48

I imagine the datomic client wraps the exception doing something like (ex-info e anomaly cause)

Filipe Silva15:10:08

we're not wrapping it on our end, just calling ex-data over it to get the anomaly

Filipe Silva15:10:27

but on the live system, ex-data over the exception returns nil

Filipe Silva15:10:58

which I think means it wasn't created with ex-info

Filipe Silva15:10:34

I mean, I wouldn't be surprised if this is indeed intended to not leak information on the live system

Filipe Silva15:10:54

that anomaly contains database ids, time info, and history info

Filipe Silva15:10:42

just wanted to make sure if it was intended or not before working around it

ghadi15:10:13

@filipematossilva are you saying that you are not able to get a :cognitect.anomalies/incorrect from your failing query on the client side?

Filipe Silva15:10:20

if by client side you mean "what calls the live datomic cloud system", then yes, that's it

ghadi15:10:27

@filipematossilva so what's different about your "live system" vs. the repl?

ghadi15:10:45

clearly it's an ex-info at the repl

Filipe Silva15:10:01

I really don't know, that's what prompted this question

ghadi15:10:10

perhaps print (class e) and (supers e) in your live system when you get the error

ghadi15:10:14

or (Throwable->map e)

ghadi15:10:53

sync api or async api?

Filipe Silva15:10:35

regarding printing the error

Filipe Silva15:10:02

I'm printing the exception proper like this:

(cast/alert {:msg "Alpha API Failed"
                   :ex  e})

ghadi15:10:14

do you have wrappers/helpers around your query? running it in a future?

Filipe Silva15:10:15

on the live system the cast prints this

ghadi15:10:52

oh, yeah that's a com.google.common.util.concurrent.UncheckedExecutionException at the outermost layer

ghadi15:10:04

then the inner exception is an ex-info

Filipe Silva15:10:05

on the repl, when cast is redirected to stderr, the datomic binary shows this

Filipe Silva15:10:42

just realized that the logged response there on the live system wasn't complete, let me fetch the full thing

Filipe Silva15:10:15

ok this is the full casted thing on aws logs

ghadi15:10:03

understood

Filipe Silva15:10:41

now that I look at the full cast on life, I can definitely see the cause and data fields there

Filipe Silva15:10:02

which leaves me extra confused 😐

ghadi15:10:11

let me clarify:

ghadi15:10:47

in your REPL, you are getting an exception that is: * clojure.lang.ExceptionInfo + anomaly data in your live system you are getting: * com.google.common.util.concurrent.UncheckedExecutionException * clojure.lang.ExceptionInfo + anomaly data

ghadi15:10:09

where the Ion has the ex-info as the cause (chained to the UEE)

ghadi15:10:43

make sense? seems like a bug @marshall

ghadi15:10:07

to work around temporarily, you can do (-> e ex-cause ex-data) to unwrap the outer layer

ghadi15:10:17

and access the data

Filipe Silva15:10:54

I can see that via indeed shows different things, as you say

Filipe Silva15:10:50

but the toplevel still shows data and cause for both situations

Filipe Silva15:10:27

I imagine that data would be returned from ex-data

Filipe Silva15:10:13

let me edit those code blocks to remove the trace, I think it's adding a lot of noise and not helping

Alex Miller (Clojure team)15:10:05

I think it's important to separate the exception object chain from the data that represents it (which may pull data from the root exception, not from the top exception)

Alex Miller (Clojure team)16:10:10

Throwable->map for example pulls :cause, :data, :via from the root exception (deepest in the chain)

Filipe Silva16:10:57

@alexmiller it's not clear to me what you mean by that in the current context

Filipe Silva16:10:22

(besides the factual observation)

Filipe Silva16:10:26

is it that you also think that the different behaviour between the repl+datomic binary and live system should be overcome by calling Throwable->map prior to extracting the data via ex-data?

ghadi16:10:27

root exception is the wrapped ex-info

ghadi16:10:08

you could do (-> e Throwable->map :data) to get at the :incorrect piece

Alex Miller (Clojure team)16:10:38

I’m just saying that the data you’re seeing is consistent with what Ghadi is saying

Alex Miller (Clojure team)16:10:50

Even though that may be confusing

Filipe Silva16:10:14

ok I think I understand what you mean now

Filipe Silva16:10:22

thank you for explaining

ghadi16:10:52

but the inconsistency is a bug 🙂

Filipe Silva16:10:29

currently deploying your workaround, and testing

marshall16:10:49

@filipematossilva this is in an Ion correct?

Filipe Silva16:10:37

the workaround is fine enough for me, but maybe you'd like more information about this?

marshall17:10:27

nope, that’s enough thanks; we’ll investigate

marshall20:10:07

I’ve reproduced this behavior and will report it to the dev team

Filipe Silva16:10:41

in a handler-fn for http-direct

Filipe Silva16:10:31

@ghadi I replaced my (ex-data e) with this fn

(defn error->error-data [e]
  ;; Workaround for a difference in the live datomic system where clojure exceptions
  ;; are wrapped in a com.google.common.util.concurrent.UncheckedExecutionException.
  ;; To get the ex-data on live, we must convert it to a map and access :data directly.
  (or (ex-data e)
      (-> e Throwable->map :data)))

Filipe Silva16:10:49

I can confirm this gets me the anomaly for the live system

Filipe Silva16:10:09

slightly different than on the repl still

Filipe Silva16:10:44

live:

{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Invalid binding form: :entity/graph", :db/error :db.error/not-a-binding-form}
repl:
{:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message \"Invalid binding form: :entity/graph\", :db/error :db.error/not-a-binding-form, :dbs [{:database-id \"48e8dd4d-84bb-4216-a9d7-4b4d17867050\", :t 97901, :next-t 97902, :history false}]}

3
Filipe Silva16:10:32

which makes sense, because in the live exception the :dbs property just isn't there

Filipe Silva16:10:41

but tbh that's the one that really shouldn't be exposed

Filipe Silva16:10:49

so that's fine enough for me

Nassin16:10:20

is there are an official method to move data from dev-local to cloud?

Chicão19:10:07

Does anyone know how I get the t from tx (d/tx->t tx), but my tx is a map and the error in the conversion?

{:db-before datomic.db.Db@ad41827d, :db-after datomic.db.Db@d8231b67, :tx-data [#datom[13194139534369 50 #inst "2020-10-08T19:30:19.852-00:00" 13194139534369 true] #datom[277076930200610 169 #inst "2020-10-08T06:00:59.275-00:00" 13194139534369 true] #datom[277076930200610 163 17592186045452 13194139534369 true] #datom[277076930200610 165 277076930200584 13194139534369 true] #datom[277076930200610 170 17592186045454 13194139534369 true] #datom[277076930200610 162 277076930200581 13194139534369 true] #datom[277076930200610 167 #inst "2020-10-08T19:30:19.850-00:00" 13194139534369 true] #datom[277076930200610 168 17592186045432 13194139534369 true] #datom[277076930200610 166 #uuid "5f7f68cb-08f0-4cb2-964b-4e811a34a949" 13194139534369 true]], :tempids {-9223090561879066169 277076930200610}}
java.lang.ClassCastException: clojure.lang.PersistentArrayMap cannot be cast to java.lang.Number

csm19:10:48

You need to grab the tx from a datom in :tx-data , in your case 13194139534369. I think something like (-> result :tx-data first :tx) will give you it

csm20:10:29

I think also (-> result :db-after :basisT) will give you your new t directly

steveb8n23:10:45

Q: I want to store 3rd party oauth tokens in Datomic. Storing them as cleartext is not secure enough so I plan to use KMS to symmetrically encrypt them before storage. Has anyone done something like this before? If so, any advice? Or is there an alternative you would recommend?

steveb8n23:10:00

One alternative I am considering is DynamoDB

ghadi23:10:32

how many oauth keys? how often they come in/change/expire?

steveb8n23:10:54

I provide a multi-tenant SAAS so at least 1 set per tenant

steveb8n23:10:50

Also looking at AWS Secrets Manager for this. Clearly I’m in the discovery phase 🙂

steveb8n23:10:57

but appreciate any advice

ghadi23:10:57

interaction patterns within KMS are not supposed to be for encryption/decryption of fine granularity items

ghadi23:10:24

usually you generate key material known as a "DEK" (Data Encryption Key) using KMS

ghadi23:10:42

then you use the DEK to encrypt/decrypt a bunch of data

steveb8n23:10:47

ok. I can see I’m going down the wrong path with Datomic for this data

ghadi23:10:01

that's not the conclusion for me

steveb8n23:10:06

it looks like Secrets Manager with a local/client cache is the way to do

ghadi23:10:11

you talk to KMS when you want to encrypt/decrypt the DEK

ghadi23:10:45

so when you boot up, you ask KMS to decrypt the DEK, then you use the DEK to decrypt fine-grained things in the application

ghadi23:10:12

where to store it (Datomic / wherever) is orthogonal to how you manage keys

ghadi23:10:49

if you talk to KMS every time you want to decrypt a token, you'll pay a fortune and add a ton of latency

ghadi23:10:17

the oauth ciphertexts could very well be in datomic

steveb8n23:10:31

if I am weighing pros/cons of DEK/Datomic vs Secrets Manager, what are the advantages of using Datomic?

ghadi23:10:05

secrets manager is for service level secrets

steveb8n23:10:05

it seems like the same design i.e. cached DEK to read/write from Datomic

ghadi23:10:22

you could store your DEK in Secrets manager

steveb8n23:10:28

the downside would be no excision c.f. Secrets Manager

ghadi23:10:58

you cannot put thousands of oauth tokens in secrets manager

steveb8n23:10:10

excision is desirable for this kind of data

ghadi23:10:12

well, depending on how rich you are

steveb8n23:10:26

I’m not rolling in money 🙂

ghadi23:10:27

if you need to excise, you can throw away a DEK

steveb8n23:10:54

hmm. is 1 DEK per tenant practical?

ghadi23:10:02

I would google keystretching, HMAC, hierarchical keys

steveb8n23:10:16

seems like same scale problem

ghadi23:10:19

you can have a root DEK, then create per tenant DEKs using HMAC

ghadi23:10:27

deteministically

steveb8n23:10:40

ok. that’s an interesting idea. a mini DEK chain

ghadi23:10:15

tenantDEK = HMAC(rootDEK, tenantID)

steveb8n23:10:26

then the root is stored in Secrets Manager

steveb8n23:10:39

where would the tenant DEKs be stored?

ghadi23:10:42

need to store an identifier so that you can rorate the DEK periodically

ghadi23:10:50

you don't store the tenant DEKs

ghadi23:10:56

you derive them on the fly with HMAC

steveb8n23:10:09

ok. I’ll start reading up on this. thank you!

ghadi23:10:28

sure. with HMAC you'll have to figure out a different excision scheme

ghadi23:10:38

you could throw away the ciphertext instead of the DEK

ghadi23:10:49

because you can't throw away the DEK (you can re-gen it!)

ghadi23:10:12

but yeah db storage isn't your issue :)

ghadi23:10:17

key mgmt is

steveb8n23:10:28

interesting. that means Datomic is no good for this i.e. no excision

steveb8n23:10:55

or am I missing a step?

ghadi23:10:59

are you using cloud or onprem?

steveb8n23:10:08

cloud / prod topo

ghadi23:10:31

stay tuned

steveb8n23:10:45

now that’s just not fair 🙂

steveb8n23:10:00

I will indeed

ghadi23:10:15

how often does a tenant's 3p oauth token change?

steveb8n23:10:48

It’s a Salesforce OAuth so the refresh period is configurable I believe. would need to check

steveb8n23:10:21

i.e. enterprise SAAS is why good design matters here

steveb8n23:10:40

I’ll need to build a v1 of this in the coming weeks

steveb8n23:10:01

now that I think about it, I could deliver an interim solution without this for a couple of months and “stay tuned” for a better solution

steveb8n23:10:12

I’ll hammock this…