datomic 2018-06-27 | Slack Archive

donaldball02:06:32

I’m sure this something most have in their bag of tricks already… what’s a good fn to produce a txn that will transform a given db into an older version of itself, assuming their schemae are unchanged, ignoring txn datoms?

val_waeselynck06:06:57

Maybe this? https://stackoverflow.com/a/25389808/2875803

donaldball18:06:51

Yeah, have seen that before, but this operates on a tx, not the delta between two dbs, and has that obviously loud WARNING

val_waeselynck18:06:58

I'm afraid there's no surefire way, this is as good as it gets... what's wrong with asOf in your use case?

donaldball18:06:02

I’m preparing a moderately large data change, and if there’s an unforeseen problem, I’d like to be able to restore the db to a given t. I can, of course, use the datomic restore facility, but as a matter of both principle and practice, it seems like applying a txn to restore to a given state (of the user partition, at least) is more correct.

donaldball21:06:36

(defn restore-to
  [db t]
  (let [db' (d/as-of db t)
        user-vs-by-ea (reduce (fn [accum [e a v]]
                                (cond-> accum
                                  ;; only look at entities in the user partition
                                  (= 4 (d/part e))
                                  (update [e a] (fnil conj #{}) v)))
                              {}
                              (seq (d/datoms db :eavt)))]
    (into []
          (mapcat (fn [[[e a] vs]]
                    ;; vs are whare are currently true
                    ;; vs' are what we want to be true
                    (let [vs' (set (map :v (seq (d/datoms db' :eavt e a))))]
                      (concat
                       (for [v (set/difference vs' vs)]
                         [:db/add e a v])
                       (for [v (set/difference vs vs')]
                         [:db/remove e a v])))))
          user-vs-by-ea)))

seems to be okay?

val_waeselynck08:06:44

Seems much less foolproof than the SO link I gave you IMHO. This may fail for bytes values due to their equality semantics, and forces you to load the whole dataset in memory instead of just the diff

jwkoelewijn12:06:36

HI there, is this the place to ask questions about datomic ions as well? If so: how can i do any logging? according to aws docs everything that is println’d should end up in cloudwatch logs, however, this does not seem to be the case for me 😞 does anyone have any experience with this?

stuarthalloway13:06:32

hi @jwkoelewijn ! Ion code does not run inside a Lambda, so the AWS docs are not relevant here.

jwkoelewijn13:06:43

aaah, that explains

stuarthalloway13:06:01

you can use any ordinary AWS logging you like, BUT:

stuarthalloway13:06:27

you have to add a policy authorizing your nodes to do it

stuarthalloway13:06:12

we can and will provide something that is easier and more integrated, coming soon

jwkoelewijn13:06:16

hmmmm…seems a bit much for dev/debug purposes

jwkoelewijn13:06:31

sweet, i’ll wait for that then 🙂

stuarthalloway13:06:38

agreed, working on the better thing right now

jwkoelewijn13:06:51

quite impressed with the rest of ions i have to say

jwkoelewijn13:06:06

thanks for clarifying!

stuarthalloway17:06:13

@currentoor we certainly understand the value of tx-report-queue, and will probably do something similar (or better) in Cloud at some point. That said, you should plan and build your app around what exists today.

currentoor17:06:09

glad to hear this from the horse's mouth!

currentoor17:06:39

thank you, for the amazing tech and answering my question

stuarthalloway17:06:40

depending on your specific use, you might be able to fake the tx-report-queue with polling or with a transaction function

currentoor17:06:31

polling seems like it would have a lot of lag

currentoor17:06:58

the transaction function approach, pushing to a queue, could you elaborate on that?

currentoor17:06:31

that would push all transactions right? vs only the ones that succeeded?

currentoor17:06:58

and what's an example of something the transaction function could push to?

stuarthalloway17:06:18

the tx fn idea is very tricky, I would shy away from it

stuarthalloway17:06:45

since fns run in-tx before the tx succeeds

stuarthalloway17:06:57

in fact forget I ever said anything about tx-fns

eoliphant11:06:26

@currentoor Onyx supports datomic cloud and might address your need to ‘stream off’ changes

currentoor17:06:26

interesting, seems complicated though, i've never used anything with that many buzz words in it 😅

sekao18:06:13

i have noticed that if you try to return a large amount of data in the response body of a web ion (say, 1MB+) it fails with a 504 error: “Timeout connecting to cluster node”. is this configurable somewhere?

stuarthalloway18:06:25

@sekao have you tried the timeout per the ns doc? https://docs.datomic.com/client-api/datomic.client.api.html

sekao18:06:40

actually the route in question is not making any calls to the db. it’s just returning a big chunk of binary data. it definitely is size-related. if i make the data smaller, it generally succeeds.

stuarthalloway18:06:16

my bad @sekao, how about the lambda timeout at https://docs.datomic.com/cloud/ions/ions-reference.html#lambda-configuration

sekao18:06:42

i’ll try that. however, it’s definitely failing before 60 seconds, which appears to be the default lambda timeout. i just tried and my route fails after about 5-15 seconds

eraserhd18:06:06

I thought I saw some papers about datoms - the 5-tuple structure of facts - but now I can't find them. Does anyone have the link (if I'm not imagining them)?

sekao18:06:23

doesn’t look like the lambda timeout is the issue — i doubled it to 120 but same result. maybe API gateway has its own timeouts….

okocim19:06:21

is the :datomic.ion.edn/api-gateway-request considered a stable part of the api when the lambda proxies via the :api-gateway/proxy integration? I’d like to pull out some of the cognito identity information that is in the json under that key, but there is no mention of this data in the reference docs.

stuarthalloway20:06:11

@okocim not stable! Next release will this information in a documented place

viesti20:06:56

I haven't had time to play with Ions yet, but was wondering about the plain HTTP integration that Api Gateway offers. That would require an Ion to be a http server, instead of a function that serves a request, right?

okocim20:06:42

@viesti, no. Your ion is just an (fn []) that receives a request with a ring-like parameter and it should return a ring-like response map. The details are documented under “Developing Web Services” here: https://docs.datomic.com/cloud/ions/ions-reference.html. All of the http server machinery is taken care of by api gateway

stuarthalloway20:06:36

@folcon and others have asked questions about how to authorize ion code to use other AWS services. We have added a new section to the docs covering this: https://docs.datomic.com/cloud/operation/access-control.html#authorize-ions

okocim20:06:34

@stuarthalloway ha 🙂, Thanks Stu. I just spent the past couple of hours poking at that and finally figured that out. I’ll take a look and see how I did 😅

viesti20:06:51

@okocim yeah, but after Api Gateway there's proxy integration to a proxy Lambda, before hitting Ion, if I understood correctly :)

viesti20:06:17

which gives glue possibility to whatever a Lambda can be used for

okocim20:06:23

@viesti, yes that’s right, and the code you write executes somewhere inside your datomic cluster, not in the proxy lambda. I don’t know the details of how that all works, but your code will have the same permissions that are allowed by the instance profile that the Ec2 instance inside the datomic cluster has. Or so it would seem.

viesti20:06:58

was just thinking about going directly to Ion from Api Gateway with http integration, but what Ion takes in is probably not plain http. Was thinking about long lived connections, beyond Lambda timeout.

viesti20:06:09

should play with Ions, not just contenplate on playing :)

stuarthalloway20:06:11

@viesti we certainly contemplate additional integration points, with minimal (or no) code changes from those that exist today

viesti20:06:15

yeah, I think that Ions really make the serverless Datomic story become true, just got hungry on thinking about the "run your code" story for other AWS services that sit in front of the app/fn :). I might be chasing a non-existing problem though, if I recall right, Api Gateway can now be exposed to private vpc only also, which allows to make apps for companies that have extended on-premise networks to aws via tunneling+firewalls.

PB21:06:46

Curious - If I were to have my app transact schema to create new attributes during runtime; How much of an antipattern is that?

mg21:06:40

Not an anti-pattern at all

PB21:06:19

Interesting. Thanks

mg21:06:55

don’t go crazy with it or anything, but that kind of flexibility is what Datomic is for

PB21:06:53

I'm thinking through ways that I could do so without resorting to that. I do enjoy being able to look at my schema.edn and having a pretty good idea of what is there

2018-06-27

Channels