Hey folks. Could anyone point me in the right direction for a Busy rebuilding index exception on a transaction? Can’t seem to find much in the docs or via search, though it’s possible I’ve missed something.

@chrjs that happens when data is coming in faster than processor can keep up, and will resolve on its own. Apps should plan for it and slow down

Were you doing a bulk load? I would not expect to see that from a single person’s interactive use.

Yep, bulk backload.

We can split the load to give it more time, no problem. Only running on a t2.micro, could lack of memory contribute to this?

@chrjs check out https://github.com/Datomic/mbrainz-importer/blob/master/src/cognitect/xform/batch.clj#L70-L92

don’t copy and paste that directly, but you should decide what you want to do in the face of various things that can fail during an import

Sure, that’s great. Thanks!

I also gave a talk on this topic https://www.youtube.com/watch?v=oOON--g1PyU

@chrjs if you switch from Solo to Production you will be able to import substantially faster

also: the topologies dictate EC2 instance sizes, so you cannot just switch instance sizes

I’ll check out that talk. We’re spinning up a production instance right now :thumbsup:

And thanks for being so responsive on datomic questions with the rollout of cloud. It must be your 6am or something right now (!?).

East Coast US. I am plenty responsive while waiting for kids to get dressed 🙂

🙂

@chrjs how much data are you loading?

@stuarthalloway We’re loading in loading from a few hundred flat files, each of which translates to about 12MB in memory, with approximately 500k datoms each. We partition into 50k ish datom chunks for each transaction. We’re seeing a rebuilding index error after the first three files on a t2.small. Spun up a prod instance now which delays the error to a few files later. We think a suitable variant of you backoff code will fix it though, probably just too much throughput - it’s a backfill job that is way more write intensive than our expected production load.

@chrjs can we get the exception from the CloudWatch logs?

also would be interested in seeing how your CloudWatch dashboard looks during the import

Give us just a few minutes to generate the error again so we can pinpoint the time for the corresponding CloudWatch log.

@chrjs you should not have to track that down yourself, just search for the pattern “Alert - Alerts” in the log group named datomic-{your-system}

@stuarthalloway I direct-messaged you the exception

and I've DM'd you the dashboard capture

(am working with @chrjs )

@chrjs @sleepyfox the dashboard capture shows 0 alerts in the Events/Alerts chart, so does not explain the nodes being unhappy. What client-side problem did you see along with this?

Exception! Backing off for  3200  :  #error {
 :cause Busy rebuilding index
 :data {:cognitect.anomalies/category :cognitect.anomalies/busy,
        :cognitect.anomalies/message Busy rebuilding index,
        :dbs [{:database-id 3109e63e-2a3b-44c9-92a1-83015c326b07,
               :t 31,
               :next-t 32,
               :history false}]}

@chrjs gotcha — that is to be expected, and should work after backoff and retry

@chrjs also, it probably takes DDB autoscaling 30 min or more to fully “notice” that you are doing an import

Yeah, we’ve got it working now using exactly your async backoff trick.

Ah, that last is interesting, noted.

it’s cool to check that out in the DDB console — go to the table, capacity tab, and open up the scaling activities

I’m puzzled by this sentence from the docs: Datomic apps that do not run on AWS must target On-Prem in the docs https://docs.datomic.com/on-prem/moving-to-cloud.html For cloud, does this mean even servers using the client API must be on AWS or only the database system? If it’s the former, what’s the rationale?

@denik Yes, clients of Datomic Cloud should be in AWS. Datomic Cloud runs in a private VPC with access controlled through IAM roles. You can run your client applications in another VPC and connect them using VPC Peering

https://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/Welcome.html

you can access Datomic Cloud through the Bastion server (via SOCKS proxy), but that route is intended for dev use, not production

(i.e. for accessing from your local dev laptop)

@marshall Why? Security? Speed? I’m not that familiar with VPC. If I have a server outside of AWS, is there any way to use the client API to talk to Datomic? I’m looking for an experience that’s more akin to a serverside firebase.

Security

primarily

having your database of record open to the internet isn’t a great position to be in

Will we have something like console for Datomic Cloud? It’s a tool we’ve come to appreciate a lot on the pro version 🙂