Congratulations for launching Datomic Cloud!

Also struggling with the "Authorize a Group" bit. Am I in the right place?

Yeah, totally works. Nice.

I had not taken too much interest in Datomic Clients (and thus Datomic Cloud) so far because I assumed db.with() could not be supported - but I now see that it is. How does it work? Does it hold a with-ed database on the server-side, or does it re-apply the speculative transaction on each subsequent query?

@olivergeorge glad you got it sorted

@val_waeselynck Datomic Cloud holds the with db on the server

does anyone know of a library for managing users & permissions using datomic?

quick question: I've noticed that Cloud variant is offering a different feature set than OnPrem. Are you planning to make two very divergent products? I see that you're excited by cloud, but there are some applications where AWS is not an option.

divergence is not an objective 🙂

but AWS provides a much richer shared baseline on which to build

so there will be continue to be differences

I understand that there will be options (like CloudSearch integration), which integrate with services offered by AWS, so obviously you can't use those without access to AWS.

Is there any chance that on-prem will get the “generic node” notion? (i.e. no explicit txor needed, per-database dispatching for a node group)

I do hope the Peer model will continue to be well supported though. In my case / opinion, that's where most of the leverage lies, and I don't think I would have made the switch to Datomic if there were only clients (however comfortable Datomic Cloud makes them).

Yeah my perception is the same. I’ve gotten a lot of leverage from on-board client caching/lazy entity crawling.

But I’ve not used the Client API. Perhaps the difference isn’t as stark as I imagine.

But the “node” thing, multiple durable locations, and encrypted at rest are all pretty rad. Would love to see at least a few of those end up in on prem.

@potetm yes, On-Prem may get nodes, encryption at rest, etc.

and conversations like these help us prioritize, thanks!

and Cloud will have a more complete story for keeping code and data colocated, although not necessarily the peer model

for cloud solo, why only two options for the node? (t2.small and i3.large) $30 vs $224, more middle ground would be nice, also, why an i3.large, what is the NVMe SSD used for?

solo uses only a t2.small production uses only i3.large instances

the configuration on the marketplace page that shows them both is a limitation of how Marketplace listings work

ah i3.large shows as an option for solo

ok

the SSD provides a large local cache

oh, per the docs I though only EFS was used for that

Is the datomic-socks-proxy the only means of accessing datomic in the cloud? We use a VPN tunnel to connect to our VPC when developing locally -it supports accessing AWS services transparently as though our local machine were in the VPC. Having to run the SOCKS proxy as well seems like a waste.

Docs say To run Datomic Cloud, currently you must have an AWS Account that supports only EC2-VPC in the region in which Datomic Cloud runs. When is that requirement expected to be lifted?

as long as you can resolve the endpoint that should work

OK. That is promising.

Say, random question but can anyone point me to some good open-source datomic databases / schemas similar to the mbrainz data set used in the tutorial?

I did find the seattle sample data included with the distro, too...

@timgilbert there are a bunch of small examples at https://github.com/cognitect-labs/day-of-datomic-cloud/tree/master/tutorial

could running backups somehow cause this failure on the transactor?

Critical failure, cannot continue: Critical background task failed
ActiveMQInternalErrorException[errorType=INTERNAL_ERROR message=AMQ119000: ClientSession closed while creating session]

(on-prem)

@uwo running backups can overwhelm your storage depending on config

particularly if you are running on DynamoDB which is provisioned

that backup is continuing, it’s only the transactor that’s falling over with a failed heartbeat. we’re using sqlserver as a store

I haven’t seen sqlserver get so overwhelmed that this problem happens

thx!

@uwo if heartbeat interval goes wonky just before failure, be suspicious of storage

@cch1 We originally endeavored to have the system work with both EC2 Classic and EC2-VPC. There is an inherent limitation between EC2 Classic and CloudFormation. We raised this issue with AWS Support and received the following response: “In EC2-Classic, Fn::GetAZs returns all the available AZs including the ones that you do not have access to. In EC2-VPC, Fn::GetAZs returns only AZs you have access to and which have a default subnet. So if a customer removes all but one of their default subnets, Fn::GetAZs will only return the AZ where the remaining subnet resides. Then if you try to use Fn::Select to get the second and third subnets, you will get an error because Fn::Select will try to reference an index that doesn’t exist. This is the downside of this approach. Unfortunately, i checked internally and we do not seem to have a workaround in place to fix this. So if you have a mix of EC2-Classic and EC2-VPC enabled for your account this approach may not be ideal for you” The CFTs we provide for Marketplace are generic and use discovery to set up VPCs/AZs. We’re considering options for you. I’ll get back to you by next Wed.

Does anyone have any idea why I might be seeing a 10-15 second response time for the first request after restarting my Peer? After that response times are sub-second. I have a small dataset so I'm trying to distinguish between whether the initially slow response is due to the cache not being full yet, which would be concerning as the data grows, or due to the Peer establishing a connection with the Transactor, which I wouldn't really care about.

running on dynamo

@captaingrover peer has to reload your database

that is a bounded cost, won’t get bigger as data grows

in a load-balanced deployment setting, you could load databases you want hot in the peer before telling the load-balancer you are ready for requests

@stuarthalloway great! that's what I wanted to hear.

I don't need the fancy load-balancer setup yet but I will definitely keep that in mind

Anyone else having the following problem when creating a Datomic Cloud Solo?

> The following resource(s) failed to create: [ExistingS3Datomic, ExistingTables, ExistingFileSystem, EnsureEc2Vpc].

@luchini are you able to see any additional details from the CFT errors?

@marshall this is the whole error:

The following resource(s) failed to create: [StorageF7F305E7]. . Rollback requested by user.
Embedded stack arn:aws:cloudformation:us-east-1:332243152968:stack/datomic-cloud-solo-test-StorageF7F305E7-1NVIQEOD7DDRO/7036b990-fc74-11e7-be75-500c28635c99 was not successfully created: The following resource(s) failed to create: [ExistingS3Datomic, ExistingTables, ExistingFileSystem, EnsureEc2Vpc]. 

Ah. So you can go look at the nested Storage Template and see if it reports an issue

Is there a way to see the Storage Template logs separately? (I’m a complete noob in CloudFormation)

under failed & deleted stacks

you can see the storage one

I did found the Storage Template itself (from your S3) and was reading it through… but it will take me on a tangent 🙂

You should see the Storage stack in the CloudFormation stack list

https://console.aws.amazon.com/cloudformation/home

Found it. It says the failure log is on CloudWatch

it may say Failed or RolledBack in status

then go to Events

and you can find the first event that failed

ExistingS3Datomic Failed to create resource. See the details in CloudWatch Log Stream: 2018/01/18/[$LATEST]d0fb81bad8dd4f83917b5428f051f03f

have you tried to launch unsuccessfully prior to this attempt?

or succesfully for that matter?

yup… a few times… same result always

I can’t find any left over S3 bucket from the previous runs

so i suspect that the first time you had some kind of failure, but now when you try to re-create with the same name you’re hitting a separate issue; if some of the parts of the system were created the first failed attempt they may interfere with creating one with the same name

Even if I can’t find those resources? Like in some kind of delayed naming cache

You are partially right @marshall. Tried with a different stack name and it got me a bit further but still failed with The following resource(s) failed to create: [EnsureEc2Vpc].

aha!

one sec

^ in the storage template

https://docs.datomic.com/cloud/setting-up.html#aws-account

as was discussed above with cch1, Datomic Cloud requires EC2-VPC

for now you can either create a new AWS account or start up in a new region in the same account that supports EC2-VPC

ah…. the keyword here is “only EC2-VPC”, correct?

yep

because I assumed my region did support EC2-VPC (but it also has classic for some old stuff that has not migrated yet)

right; if it’s a classic it won’t work

and now I see in the template FailIfEc2ClassicLogGroup 😄

thanks @marshall. This was tremendously helpful

yep; i suspect that log would say the same thing as I just did ^

no problem

tip here: maybe rename EnsureEc2Vpc to EnsureEc2VpcOnly (I know, a bit pedantic, but semantics always help 😄 )

congrats to the Cognitec team… we are stoked over here with Datomic Cloud

@marshall I have a large import I’d like complete to Datomic Cloud. Using local peer against local data store - I can configure transactor to speed things up a bit - and I understand one can adjust DynamoDB for import using On-Prem. Are there similar adjustments that can be made for Datomic Cloud? I’m a new to CloudFormation - but if you have pointers to docs - or advice that’s be appreciated. Local import currently takes several hours to a datomic:dev database - and I’m doing all the batching/pipelining that is recommended for large imports.

@donmullen Cloud will autoscale DDB for you, so the import will start slow and speed up, then scale DDB back down automatically

@stuarthalloway magic. Congrats on the launch.

@donmullen Cloud will make much less use of DDB than an equivalent On-Prem import

no DDB writes for indexing

Timing was perfect - was just getting ready to spin up On-Prem on AWS.

@donmullen how big is the dataset? Do you hope to stay in the Solo topology?

Hoping to stay for short term as we work on queries and data analytics - but the data is very large - millions of rows five primary data sets and about 6 GBs of raw data from csv’s for import data. That something Solo can handle?

@stuarthalloway just did a restore to local dev database from backup - it’s 12 GB in datomic/data.

Got thru video one - on to video two… 🙂

@donmullen I have imported full mbainz (100 million datoms) into Solo. It takes a while, especially after the AWS burst ends and you get only a fractional CPU

@stuarthalloway How long does mbainz take running locally against datomic:dev storage? Then compared to Solo? My import locally takes about 7.5 hours (I don’t create the indexes until after the import).

I’m assuming I should leave indexes off the schema until post data import.

ever since we added adaptive indexing (http://blog.datomic.com/2014/03/datomic-adaptive-indexing.html), the index thing matters less

@donmullen Cloud always builds all indexes, :db/index is not even a thing there

@stuarthalloway interesting

@donmullen your local setup likely has more CPU horsepower than i3 large (Prod), which in turn has way more CPU than t2 small

so if you are already sweating making imports faster, you will likely end up on Production

yeah - I figure we’ll land there - though once we get the import done ‘right’ we won’t be doing that much (at all?) - and will be incrementally adding data on a weekly basis to the core data set.

I am having trouble getting the datomic-socks-proxy to work. It gives me the error “Datomic System not found” although running the ‘aws ec2 describe-instances’ query seems to find it without issue..

I also get an “aws: error: argument command: Invalid choice, valid choices are:” prior to the datomic sys not found error.

@fingertoe you need a newer AWS CLI client

@stuarthalloway That did it! Thanks…