Fork me on GitHub
#datomic
<
2018-11-07
>
val_waeselynck10:11:57

Documentation suggestion: document the :next-t key of Database values in clients https://docs.datomic.com/client-api/datomic.client.api.html#var-db It's pretty useful for working with the Log API in particular

quadron12:11:16

a stupid question: what is the clojure way of maintaining a certain relationship between datomic entities? Say there are three entities in the db, $PUBLIC_KEY $MESSAGE and $SIGNATURE such that the latter enjoys a mathematical relationship with regard to the first two. can datomic know about such a relationship or does it have to be programmed independently of datomic?

val_waeselynck14:11:58

Has to be enforced externally by the writers - there are no equivalent to e.g column constraints in sql. See https://stackoverflow.com/questions/48268887/how-to-prevent-transactions-from-violating-application-invariants-in-datomic

✔️ 4
chouffe14:11:03

What is the story around integration testing and unit testing with datomic cloud? I would like to be able to run tests without internet connection. The only reference I found was this: https://forum.datomic.com/t/integration-testing/465

kenny17:11:17

Hi @UDDUSGGKB. Because this became a blocker for us, we ended up writing datomic-client-memdb [1] to facilitate local development and CI testing with Cloud. It’s a small library that wraps Datomic Free in-memory databases with the Datomic Client protocols. I wrote the library so let me know if you have any questions. [1] https://github.com/ComputeSoftware/datomic-client-memdb

chouffe19:11:43

Sweet! Thanks for the github link and the lib. How do you deal with backups and restores with datomic cloud? Let say I want my in memory dev db to be a backup from prod?

kenny19:11:46

I haven't needed to do that. I don't believe there's any built-in way to do that with Cloud 😞 Certainly would be helpful if the Datomic team provided that.

chouffe19:11:42

Do you know if its possible to connect to one of the peer instances from the bastion server and run the datomic bin restore command?

chouffe20:11:45

Also, how can one restore a datomic db from the S3 buckets? I am still unsure how this all fits

kenny20:11:36

I don't think there is anyway to backup/restore with Datomic Cloud.

joshkh16:11:02

for those of you who use ions / api gateway, do you ionize a handler for every REST end point? for example, a mostly 1-1 map: (/example/auth -> my-auth-ion, /example/colors -> my-colors-ion). my lambdas go cold after a while (to be expected), and so each visit to the web app is painfully slow as each supporting lambda heats up. just wondering in practice how you all deal with this? is this acceptable in your production applications? does anyone use one big lambda and then dispatch internally?

joshkh16:11:57

in something like nodejs a cold start is barely noticeable, but as i fetch various assets i'm looking at 4-6 seconds per resource if it hasn't been touched in a while.

Joe Lane17:11:19

I’m enjoying using one lambda, then dispatching with multimethods.

Joe Lane17:11:26

Then it stays warm.

Joe Lane17:11:35

I can always scale my lambda concurrency up, then set a timeout where if it takes more than 200ms (due to cold start when scaling up during a burst), retry and hit the other warm instance.

joshkh17:11:56

so you have a single /{proxy+} integration?

Joe Lane17:11:59

No, not neccessarily.

Joe Lane17:11:15

Some apps I do, some I dont.

Joe Lane17:11:38

But for the most part, a lot of the endpoints, even with different routes, all end up calling the same lambda.

Joe Lane17:11:09

So /some/route/foo calls my-ion-lambda and /some/route/bar ALSO calls my-ion-lambda

Joe Lane17:11:25

Internally in my ion I can route based on the path.

Joe Lane17:11:34

but then my-ion-lambda stays warm.

joshkh17:11:02

ahhh i see. so do you dispatch on the url path?

joshkh17:11:21

using something like bidi?

ro617:11:19

yes, I'm doing the same. Just like a traditional Clojure webapp setup.

Joe Lane17:11:20

I can. One of my apps uses AppSync + GraphQL so there I dispatch based on graphql stuffs.

Joe Lane17:11:43

I know pedestal is coming out with an Ion based interceptor chain.

joshkh17:11:58

interesting

Joe Lane17:11:30

However, I’ve also succeeded in calling an ion lambda directly from the web without fronting it with API Gateway.

ro617:11:33

There's some tricks to getting things like CORS to work with API Gateway, but once you wrap your head around it it's pretty simple. I'm actually going to write a quick post at some point about the little stuff I ran into and how to fix it.

Joe Lane17:11:46

Ill say it again. Call a lambda. From the Browser.

joshkh17:11:56

yeah i ended up writing some custom wrappers to support cors and various mime types

Joe Lane17:11:09

browser -> json -> ion -> json -> browser

Joe Lane17:11:15

No web layer.

joshkh17:11:33

ah, so you're not apigw/ionizing

ro617:11:34

Ha, I still need to learn from @joe.lane how he's doing that. Basically the way the AWS SDKs do?

Joe Lane17:11:08

Here is a scratch project. Its a bit under documented and not totally cleaned up. https://github.com/MageMasher/ion-web

Joe Lane17:11:15

However after listening to rich’s talk at clojure nyc (theres a video on youtube) he made a comment that makes curious about sticking with apigw

Joe Lane17:11:12

I’ve been toying with wrapping calls to ions with reagent atoms 🙂

ro617:11:55

you mean what he said about going direct from APIGW to your code w/o Lambda in between?

joshkh17:11:52

i think my brain was warped by the datomic ions tutorial. it's just an odd place to leave the user: a single greedy end point pointing to a ring handler. i'm fairly new to AWS though, so maybe it's just me.

ro617:11:35

by "user", you mean the developer?

joshkh17:11:07

er yeah, trying to multitask in a meeting. 😉

joshkh18:11:00

by the way, thanks for the discussion. finding best-practices or "real life" example setups of ions is tough. i'm planning to blog a few of my trials and tribulations along the way. might spare a few people some trouble.

ro618:11:32

yeah, it's early days. The community sharing is a big help.

joshkh18:11:56

(it's mostly me not understanding what are probably basic principles but i'm sure i'm not alone)

joshkh18:11:15

i'm used to whacking stuff on heroku or dokku and calling it a day

ro618:11:06

you have to keep in mind that API GW can be configured as an edge network and Lambdas probably have similar elastic scaling properties (though I'm not sure if they're bound to one AWS region), so even though it looks like you're piping everything through one "place", it's not really a bottleneck.

ro618:11:22

haha, me to. Heroku was always my goto for prototyping.

ro618:11:53

Ions isn't quite as ironed out (yet), but it's getting close, and with a much more powerful model

joshkh18:11:14

agreed, and i'm certainly not complaining. it's a huge win for us to be able to push clojure code.

ro618:11:43

yeah, and as usual (in my experience), the difficulties come more from facts about how the JVM works (eg classloading and such), not Clojure proper

joshkh18:11:29

yup. or in my case AWS configuration and infrastructure.

ro618:11:45

right, all the non-Clojure-itself stuff, ha

ro618:11:09

even still, it's much less AWS stuff than other deployment solutions

joshkh18:11:30

for instance i still get worried about co-authoring our ions project. if another developer is working on some newly exposed ions in their code base while i do the same, and we're both pushing and deploying, then are we removing each other's ions?

joshkh18:11:51

(based on what's in ions-config.edn)

ro618:11:44

if you're pushing to the same system, I think so. I think the answer is for each developer to have their own Solo system for dev purposes

4
joshkh18:11:29

so multiple compute stacks sharing a storage stack?

ro618:11:18

I was thinking an entire storage+compute stack for each

ro618:11:37

but maybe what you're saying would be cheaper and more efficient, if it's possible

joshkh18:11:04

i guess so long as you're all in communication about schema changes that could work

joshkh18:11:40

anyway that wasn't a huge concern. just an example of my unknowns. 🙂

ro618:11:47

right. I think if devs are doing a good job with schema attribute namespacing, and your project is organized sensibly, you're much more likely to have clean "schema merges" when you go to integrate than you would with eg relational schema, or even schemaless. This is all speculation on my part though, since I haven't gone through any of that yet.

ro618:11:18

Honestly though, Datomic's notion of schema, and the possibility that it has properties like that, is the main difference-making reason that I'm betting on it

joshkh18:11:36

we found that our schema changed quite a bit during the very start of the project. in a normal db we'd "take the good parts" and maybe start a fresh one, but we still haven't solved data dumps / loads in Cloud.

joshkh18:11:32

but we've also found that having such a factual and flexible schema means we can continue working with the same instance and chalk up changes to historical events

joshkh18:11:41

which is great.

joshkh18:11:21

anywho, gotta run. thanks for the chat!

ro618:11:22

I see. Again, I haven't gotten far enough to speak from experience, but my sense is that following the rules for non-breaking growth that Rich has been laying out across the Clojure ecosystem allows you to live with your past decisions/data without having to do massive structural migrations along the way. I think that makes us devs uncomfortable because we like to clean up our messes (eg cleaning up Git history, refactoring, schema migration, etc...), but that discomfort with messes is totally trumped by the pain of breakage at integration points.

souenzzo21:11:31

Hey I'm trying to create a datomic cloud from awsmarketplace and after some waiting on CloudFormation I get this error The following resource(s) failed to create: [DhcpOptions, EnsureEc2Vpc]. Then it auto-rollback to ROLLBACK_COMPLETE status

souenzzo21:11:58

Now with The following resource(s) failed to create: [LambdaTagDdbPolicy, DhcpOptions, EnsureEc2Vpc].

eoliphant21:11:07

Is your account EC2-VPC? I think I ran into that for that reason

souenzzo21:11:25

Failed to create resource. See the details in CloudWatch Log Stream: 2018/11/07/[$LATEST]c6878226f3da4b06b3a803c8873b0262

souenzzo21:11:42

The following resource(s) failed to create: [LambdaTagDdbPolicy, DhcpOptions, EnsureEc2Vpc].

eoliphant14:11:36

Yeah that's super weird

eoliphant14:11:32

when I had that issue it was because my account in that region had both the EC2 (classic) & VPC under supported platforms. Not sure what else it might be, maybe hit the datomic guys with a support ticket

souenzzo14:11:21

I found that my AWS is really old and has this VPC/EC2 problem. I will try to solve it next week.