This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-06-27
Channels
- # aleph (10)
- # beginners (139)
- # cider (47)
- # clara (19)
- # cljs-dev (2)
- # cljsjs (9)
- # clojure (94)
- # clojure-berlin (1)
- # clojure-dev (23)
- # clojure-greece (5)
- # clojure-italy (5)
- # clojure-nl (14)
- # clojure-uk (36)
- # clojurescript (85)
- # cursive (2)
- # datomic (56)
- # emacs (3)
- # events (1)
- # fulcro (33)
- # garden (3)
- # graphql (6)
- # hoplon (53)
- # jobs (1)
- # leiningen (4)
- # mount (46)
- # nrepl (7)
- # off-topic (8)
- # om (3)
- # other-languages (4)
- # pedestal (7)
- # portkey (7)
- # re-frame (1)
- # reagent (16)
- # remote-jobs (3)
- # ring-swagger (2)
- # shadow-cljs (16)
- # slack-help (2)
- # tools-deps (2)
- # yada (1)
I’m sure this something most have in their bag of tricks already… what’s a good fn to produce a txn that will transform a given db into an older version of itself, assuming their schemae are unchanged, ignoring txn datoms?
Yeah, have seen that before, but this operates on a tx, not the delta between two dbs, and has that obviously loud WARNING
I'm afraid there's no surefire way, this is as good as it gets... what's wrong with asOf in your use case?
I’m preparing a moderately large data change, and if there’s an unforeseen problem, I’d like to be able to restore the db to a given t. I can, of course, use the datomic restore facility, but as a matter of both principle and practice, it seems like applying a txn to restore to a given state (of the user partition, at least) is more correct.
(defn restore-to
[db t]
(let [db' (d/as-of db t)
user-vs-by-ea (reduce (fn [accum [e a v]]
(cond-> accum
;; only look at entities in the user partition
(= 4 (d/part e))
(update [e a] (fnil conj #{}) v)))
{}
(seq (d/datoms db :eavt)))]
(into []
(mapcat (fn [[[e a] vs]]
;; vs are whare are currently true
;; vs' are what we want to be true
(let [vs' (set (map :v (seq (d/datoms db' :eavt e a))))]
(concat
(for [v (set/difference vs' vs)]
[:db/add e a v])
(for [v (set/difference vs vs')]
[:db/remove e a v])))))
user-vs-by-ea)))
seems to be okay?Seems much less foolproof than the SO link I gave you IMHO. This may fail for bytes values due to their equality semantics, and forces you to load the whole dataset in memory instead of just the diff
HI there, is this the place to ask questions about datomic ions as well? If so: how can i do any logging? according to aws docs everything that is println’d should end up in cloudwatch logs, however, this does not seem to be the case for me 😞 does anyone have any experience with this?
hi @jwkoelewijn ! Ion code does not run inside a Lambda, so the AWS docs are not relevant here.
aaah, that explains
you can use any ordinary AWS logging you like, BUT:
you have to add a policy authorizing your nodes to do it
we can and will provide something that is easier and more integrated, coming soon
hmmmm…seems a bit much for dev/debug purposes
sweet, i’ll wait for that then 🙂
agreed, working on the better thing right now
quite impressed with the rest of ions i have to say
thanks for clarifying!
@currentoor we certainly understand the value of tx-report-queue, and will probably do something similar (or better) in Cloud at some point. That said, you should plan and build your app around what exists today.
glad to hear this from the horse's mouth!
thank you, for the amazing tech and answering my question
depending on your specific use, you might be able to fake the tx-report-queue with polling or with a transaction function
polling seems like it would have a lot of lag
the transaction function approach, pushing to a queue, could you elaborate on that?
that would push all transactions right? vs only the ones that succeeded?
and what's an example of something the transaction function could push to?
the tx fn idea is very tricky, I would shy away from it
since fns run in-tx before the tx succeeds
in fact forget I ever said anything about tx-fns
@currentoor Onyx supports datomic cloud and might address your need to ‘stream off’ changes
interesting, seems complicated though, i've never used anything with that many buzz words in it 😅
i have noticed that if you try to return a large amount of data in the response body of a web ion (say, 1MB+) it fails with a 504 error: “Timeout connecting to cluster node”. is this configurable somewhere?
@sekao have you tried the timeout per the ns doc? https://docs.datomic.com/client-api/datomic.client.api.html
actually the route in question is not making any calls to the db. it’s just returning a big chunk of binary data. it definitely is size-related. if i make the data smaller, it generally succeeds.
my bad @sekao, how about the lambda timeout at https://docs.datomic.com/cloud/ions/ions-reference.html#lambda-configuration
i’ll try that. however, it’s definitely failing before 60 seconds, which appears to be the default lambda timeout. i just tried and my route fails after about 5-15 seconds
I thought I saw some papers about datoms - the 5-tuple structure of facts - but now I can't find them. Does anyone have the link (if I'm not imagining them)?
doesn’t look like the lambda timeout is the issue — i doubled it to 120 but same result. maybe API gateway has its own timeouts….
is the :datomic.ion.edn/api-gateway-request
considered a stable part of the api when the lambda proxies via the :api-gateway/proxy integration
?
I’d like to pull out some of the cognito identity information that is in the json under that key, but there is no mention of this data in the reference docs.
@okocim not stable! Next release will this information in a documented place
I haven't had time to play with Ions yet, but was wondering about the plain HTTP integration that Api Gateway offers. That would require an Ion to be a http server, instead of a function that serves a request, right?
@viesti, no. Your ion is just an (fn []) that receives a request with a ring-like parameter and it should return a ring-like response map. The details are documented under “Developing Web Services” here: https://docs.datomic.com/cloud/ions/ions-reference.html. All of the http server machinery is taken care of by api gateway
@folcon and others have asked questions about how to authorize ion code to use other AWS services. We have added a new section to the docs covering this: https://docs.datomic.com/cloud/operation/access-control.html#authorize-ions
@stuarthalloway ha 🙂, Thanks Stu. I just spent the past couple of hours poking at that and finally figured that out. I’ll take a look and see how I did 😅
@okocim yeah, but after Api Gateway there's proxy integration to a proxy Lambda, before hitting Ion, if I understood correctly :)
@viesti, yes that’s right, and the code you write executes somewhere inside your datomic cluster, not in the proxy lambda. I don’t know the details of how that all works, but your code will have the same permissions that are allowed by the instance profile that the Ec2 instance inside the datomic cluster has. Or so it would seem.
was just thinking about going directly to Ion from Api Gateway with http integration, but what Ion takes in is probably not plain http. Was thinking about long lived connections, beyond Lambda timeout.
@viesti we certainly contemplate additional integration points, with minimal (or no) code changes from those that exist today
yeah, I think that Ions really make the serverless Datomic story become true, just got hungry on thinking about the "run your code" story for other AWS services that sit in front of the app/fn :). I might be chasing a non-existing problem though, if I recall right, Api Gateway can now be exposed to private vpc only also, which allows to make apps for companies that have extended on-premise networks to aws via tunneling+firewalls.