I found this question a few above my own https://clojurians.slack.com/archives/C03RZMDSH/p1635985753218600 - does this mean I cannot use ensure-transactor for SQL?

Hey @U02LH3SBBEJ, the information you're looking for is on the https://docs.datomic.com/on-prem/overview/storage.html page. For your storage you're going to want to follow https://docs.datomic.com/on-prem/overview/storage.html#sql-database. bin/ensure-transactor currently supports DynamoDB.

thank you 🙂 Will give that a go!

Start from the top and once you finish the SQL section don't skip the other non-storage sections, you will need them.

so, we already have our database, the connection all works and what I want to do is turn on Cloudwatch metrics, I’m not clear on how specifically to turn on Cloudwatch metrics? (this is a pre-existing app, been running about 3 years, it just needs Cloudwatch 🙂 ) Please excuse my ignorance, this is day 0 for me of trying to work with Datomic 🙂

I guess that I don’t need ensure-transactor because I created the AWS role myself?

ensure-transactor is a convenience script, but since you're already running in prod, you probably don't need it 🙂

Before we get into the Cloudwatch metrics effort, can you describe how you know the "Clojure+Datomic" application is performing poorly? What have you observed to lead you to that conclusion?

ahh, right Should I expect the metrics to be appearing? Like, I assume there’s a heartbeat metric or something? OR does my app have to explicitly say “send this metric to Cloudwatch”?

> can you describe how you know the “Clojure+Datomic” application is performing poorly? What have you observed to lead you to that conclusion? Hoo boy, that’s a long story 😄 I’ll condense! Customers reported degrading performance over time, as the data set has grown, we get operations like “get 1000 items into a <select> list” taking either 5ms or 10s under load. So basically, we measure the page with a stopwatch 😄 it’s not very sophisticated, we need to get observability in place. So I’m guessing some kind of CPU or memory contention, so the first step is to get some instrumentation in there and measure what is going on. The peer has some instrumentation via statsd so I can see that in Cloudwatch. However, the transactor is not currently outputting anything to Cloudwatch, so today I am trying to add that.

do I need to have the S3 log copying bucket as an intermediary for Cloudwatch logs? i only added the aws-cloudwatch* lines, not an aws-s3* line… I don’t know if I need the S3 or if I can go straight to cloudwatch?

Are any of the issues related to transactions? If not, then your problem squarely sits inside the peers (not trying to dissuade you from adding metrics to the transactor).

If you want transactor logs to go into a cloudwatch logstream you will need to install and manage the cloudwatch agent yourself.

The built-in logging facilities solely upload to S3 (I think it's daily by default)

I have no idea what the issues relate to yet, I’m sorry to say. I don’t really want the logs at this stage, all I want is things like heap usage etc to be pumped to Cloudwatch

the problem may well be the peers, unfortunately we have no visibility at all yet. I was hoping I could just enter a few lines of config and get the JVM/datomic metrics pushed to Cloudwatch?

https://docs.datomic.com/on-prem/overview/aws.html#other-storages ^^ This should be all you need.

OK, thanks. I have been through that and unfortunately I have failed to see any metrics in Cloudwatch. My background is in AWS rather than in Clojure/Datomic. I believe I have set the IAM role correctly and I have redeployed my transactor image with the correct config to send metrics to Cloudwatch. I have not set up anything for S3. Does the Cloudwatch bit rely on the S3 bit in some way? I’d have assumed not but maybe there’s some dependency? I see no logs even acknowledging Cloudwatch - when the transactor starts, I don’t see any mention of Cloudwatch, would I expect to?

Hey @U02LH3SBBEJ I noticed that you have a metrics callback configured are your metric lines reporting there? If the s3 dimension is not setup it will create a bucket for log copying. Did an s3 bucket get made on your system? In non-DDB systems customers typically setup an s3 bucket and point #aws-s3-log-bucket-id= to it. I'd recommend setting that up with an exisiting bucket just to see if you get logs copied to the bucket. The piece that always trips up non DDB users is the permissions on the transactor role. https://docs.datomic.com/on-prem/overview/aws.html#other-storages. Can you confirm you have those perms set?

Also @U02LH3SBBEJ, do you see metrics if you comment out the metrics-callback=<REDACTED>.datomic-logging.core/report-metrics?

I know you walked up to this system, but do you know if this worked previously? Does this work for another system? Where did this system live previously and are you copying it to test performance or tweaking it?

YESSSSSS! IT WORKS! Thank you all! ❤️ Removing the metrics-callback means I am now seeing datomic stats in Cloudwatch. What a relief! Thank you all so much

> jaret  [4:05 PM] > I know you walked up to this system, but do you know if this worked previously?    Does this work for another system?  Where did this system live previously and are you copying it to test performance or tweaking it? Great questions! I have been at $ORG for a year, this app is about 4 years old. It runs fine for other clients with smaller data sizes, which are in separate AWS accounts which are identically configured via Gruntworks. Our largest client recently onboarded a bunch of new customers and that seems to have degraded performance for “poor” to “intermittently unacceptable”. I am not moving the system, it remains hosted in ECS, I’ve just been parachuted in like “hey Gav you’ve got (expired!) AWS certification can you make this perform more gooder and stuff until we can EOL this application in 6 months” So I suspect I may have more questions, but now I can see what the transactor is up to, I feel like I’ve got a fighting chance! (the peer was already emitting metrics to Cloudwatch)

@U02LH3SBBEJ When you have questions feel free to reach out to https://www.datomic.com/support.html. Cases can be created by e-mailing <mailto:[email protected]|[email protected]> or <mailto:[email protected]|[email protected]>. We'd be happy to help with any performance issue or tuning and perhaps we can have a call to give you some basic tips for reviewing/understand what your bottlenecks are likely to be.

Do you have a REPL into prod or a connection to the prod database? I've used https://github.com/ptaoussanis/tufte/ for profiling Clojure code with decent success. My guess would be that your peer query is slow, but that is just a guess of course.

thanks, we will reach out to Datomic support and our Clojure guy is looking into Tufte :-)

> Is datomic.api only for on-prem

yes

Thanks

Btw, why is this api not part of the cloud api? It looks that it has some convenient functions.

The client api is designed for low bandwidth (ie possible out-of-process implementation) few client dependencies, and little client state. Many portions of the peer api simply can’t be implemented in a way that meets those goals

I see. Thanks again

tuple values are not inspected for entity refs; this won’t work

Ok, so I need to query to be able to resolve this right?

You can use [:example/id+other [1552 (d/entid db [:other/id 520832909])]]

d/entid resolves any entity reference to an entity id if it can

nil if it can’t

Oh, this is better, it should suffice. Thanks!

on-prem only. I don’t know how the cloud folks manage this without a query

Well, this is a query right? It will be done before calling d/pull, or am I misunderstanding?

it’s not a d/q query, i.e. it doesn’t invoke a datalog query engine

but it will do IO if it needs to, if that’s what you mean by query

Yeah, I see what you mean. Thanks, this was really helpful. I could not find that information on docs/searching the web.