Fork me on GitHub
#datomic
<
2019-12-17
>
alidcastano17:12:11

if we follow the datomic ion tutorial we're interacting/changing with live database, correct? so we should delete it afterwards in order to start from fresh slate (since there's no way to rollback changes)?

alidcastano17:12:30

also, does starting/stopping a datomic-gateway effect billing? (so i shouldn't forget to stop it after developing?) there's not a lot of info on implications of these commands in docs

shaun-mahood17:12:04

@alidcastano Yes, the ion tutorial transacts data to a live database. If you want to start over, deleting and recreating the database shouldn't cause any problems. The API gateways are billed based on the number of requests, so you should only see a cost if you are using it a lot (mine is $3.50/million requests, not going to break the bank from development use).

alidcastano17:12:48

so it doesn't matter whether we leave it running or not, just the amount of requests we do during development

lanejo0117:12:12

You two are talking about different things.

lanejo0117:12:48

APIGateway is billed by the request, however, @alidcastano is talking about what used to be known as the bastion server.

shaun-mahood17:12:07

Did they change the name on me and I didn't notice?

lanejo0117:12:24

It does more things now though so I see why they changed it.

lanejo0117:12:59

@alidcastano The datomic-gateway is an ec2 machine which you will be billed for like normal unless you shut it off.

alidcastano17:12:29

@ so starting/stopping is required for ion development correct?

shaun-mahood17:12:32

Oh yeah, it's called "Access Gateway" in https://docs.datomic.com/cloud/operation/operation.html now Thankfully, if you forget to turn it off it's still not going to cost you much - looks like mine cost $3.74 to run the EC2 instance for an entire month

alidcastano17:12:18

oh yeah that's not bad at all. my first time interacting with aws I once left a service running that costs me a few hundred dollars so that has scarred me 😆

alidcastano17:12:42

thanks for looking into it @

lanejo0117:12:27

@alidcastano "developing", no, there is no need to bounce the machine (except maybe with some of the new analytics stuff, i'm not 100% on that...). If you want to deploy an ion for development (like a transaction function) the push/deploy commands from the cli go against the primary node. The access gateway is an evolution of the bastion machine which is used for secure access into the vpc. It now does more than secure access but if you're just getting started you can treat it like a "jump box".

alexmiller17:12:02

I am not an expert, but I think generally you should not need to bounce the datomic gateway for anything with analytics - it picks up changes in the metaschema dynamically afaik (but I could be wrong)

marshall18:12:33

That’s an old docs page

marshall18:12:45

i will fix the links/pages

cjsauer00:12:01

@alidcastano I usually scale down the auto-scaling group when not actively developing in order to save money. This section might be useful to you. https://docs.datomic.com/cloud/operation/planning.html#turning-down

eoliphant18:12:03

are cross-db joins supported in cloud?

alidcastano18:12:11

what's the recommended testing approach for ions? any repos or articles that demonstrating it?

alidcastano18:12:13

(not referring to repl development, but unit tests ideally with some sort of rollback per test)

lanejo0118:12:36

What exactly do you want to unit test? You can find code in the helpers here (https://github.com/cognitect-labs/day-of-datomic-cloud) that you can adapt to creating/destroying databases, but you may not need it.

alidcastano18:12:26

just basic tests to make sure resolvers are doing what's expected

alidcastano18:12:48

at least that's what im used* to doing with graphql + postgres. is not how it's done with datomic?

lanejo0118:12:01

So this is testing graphql resolvers using (I assume) Lacinia?

alidcastano19:12:32

that would be the equivalent setup, yea, but I haven't even gotten that far yet. I just like to wrap my head around how the dev/test flow works so I can have more confidence in what im doing.

alidcastano19:12:49

are the ions themselves not tested too?

lanejo0119:12:20

Well, an "ion" is an aws lambda function which acts as a proxy for your clojure code, so, you can just invoke your clojure code in a test and not involve invoking actual ions.

alidcastano19:12:29

ah have to learn the terminology better, thanks. yes, I mean testing the actual the clojure code. what I have in mind is an example setup that shows how I can test API with an the in memory client database (has same API ions use) and rollback changes per test hopefully that makes sense in context of datomic, otherwise I'll have to dig more into the code before commenting further 😅

mauricio.szabo16:12:47

I also want to know, what we're doing right now is creating a database then destroying it after each test, but its awfully slow and "seems wrong"... 😄

lanejo0116:12:57

Eh, it's not wrong, but you might be able to optimize. You can use d/with-db to try several tests after creating a db with some common fixture data.

mauricio.szabo20:12:17

Is there any alternative? Like, to run local tests , the need to connect to a cloud instance, create a database with a unique name that does not conflict with anyone else running tests at that time seems strange...

lanejo0120:12:59

It only matters if you work with the database. Otherwise no need to connect.

mauricio.szabo20:12:48

How do I do an integration test, for example, for a CRUD application without working with the database?

alidcastano20:12:04

do only peer database have an in-memory alternative? (so that approach can't be used with ions / client api)

matthavener21:12:50

can :db/tupleAttrs refer to idents with :db.type/tuple? (ie tuple ident containing tuples)

markbastian22:12:52

Any idea how to get past this issue with stack creation: "Embedded stack arn:aws:cloudformation:us-east-1:...:stack/vcl-Storage.../... was not successfully created: The following resource(s) failed to create: [InternetGateway, Vpc, AvailabilityZones]."? This is a solo instance.

steveb8n22:12:07

Q: I just went live with a prod system today. I think I still have a slow memory leak but it there’s so much headroom that its stable anyway. Is it ok to occasionally kill one of the instances to flush this out? I was thinking of doing this each night until I can find the problem

kenny23:12:04

Do you happen to be using the aws-api lib?

steveb8n00:12:39

I have been putting all the calls in try/finally blocks to “stop” the client

steveb8n00:12:53

but still leaking a bit I think

steveb8n00:12:03

do you have a story here?

kenny00:12:43

Ah. We had that same problem. We still create a client on all function calls but have a

(def default-http-client
  (delay (aws/default-http-client)))
statically declared. That is then passed to the client function via :http-client @default-http-client . That solved the memory leak for us.

kenny00:12:35

The behavior you described sounded very similar. It would take ~1 day before an OOM would occur.

steveb8n01:12:24

Did the system detect the situation and cycle the node when it hit OOM? Or did you have to do this manually?

steveb8n01:12:32

Thanks BTW!

kenny01:12:18

We don’t use Ions so don’t know.

steveb8n03:12:13

ok. thanks. I’ll keep a close eye