Has anyone compared Datomic's basic write performance against other SQL databases? I wrote some tests that pit PostgreSQL against Datomic and in most cases Datomic is 2-5x slower - even with tests that always insert new records. Is that other people's general expectation also, or do I need to do some tuning (I run Datomic with defaults) to make write performance somewhat comparable with other SQL databases?

Datomic Cloud is now available! http://blog.datomic.com/2018/01/datomic-cloud.html

https://docs.datomic.com/clojure/ docs seem to be down

refresh your cache

@petrus where did that inbound link come from?

A google search for "datomic.api/resolve-tempid": https://www.google.co.za/search?q=datomic.api%2Fresolve-tempid&oq=datomic.api%2Fresolve-tempid

Looks like the docs moved to sub-path /cloud

thanks @petrus ! — investigating

The pricing layout on the AWS Marketplace is confusing. It's not obvious that the The "Infrastructure Pricing Details" is not a line total at bottom although it's laid out like an invoice. I didn't realise I could click between the "Datomic Cloud Bastion" and "Datomic Cloud" (do I need both? Is Bastion like a "lite" version?) so I went ahead and accepted thinking the total cost only that of a t2.nano, but later on it shows I will be billed for both separately? Link: https://aws.amazon.com/marketplace/pp/prodview-otb76awcrb7aa

@petrus we agree — that layout is compelled by AWS, and we are working with them to implement improvements

morning

@petrus happy to explicate here

@petrus you want the bastion so you can connect to your system from the internet

@petrus if you have not yet, you might watch https://www.datomic.com/videos.html

with Datomic Cloud, would I automatically get any updates, e.g. if a bug or vulnerability is discovered?

@petrus depends. AWS can auto update you for e.g. meltdown

cool - I mean Datomic versioning, not so much OS/VM/hardware level

But for a Datomic bug you would need to update your CloudFormation stack after notification from us.

You would not need to go back to the marketplace site — you could just grab an upgrade per https://docs.datomic.com/cloud/operation/upgrading.html

Are "On Prem" upgrades also rolling? I manually set up a transactor recently. No way to "migrate" my existing system to Datomic Cloud?

For the production topology of Datomic Cloud, are writes scaled across the number of nodes in the tx group or is there still only a single transactor per database?

@petrus you can do rolling upgrades to On-Prem. Migration is an ETL job, we will provide tools but have not done so yet

@sleepyfox each db will have a preferred node for writes. Cloud will not make writing to a single db faster, but you can have many more dbs on a system as you scale horizontally. See https://docs.datomic.com/cloud/operation/scaling.html

@petrus notes for anyone considering On-Prem to Cloud migration: https://docs.datomic.com/on-prem/moving-to-cloud.html

OK, thanks.

@stuarthalloway dude 🙂 btw “On-Pre” heading typo in “Other Differences” section in your moving-to-cloud doc

@stuarthalloway, what does “Symbol magic” mean? (nm found it)

conversion of strings to symbols to help languages without a symbol type

I am looking at you, Java

gotcha!

how much sleep have you had, Stu? 🙂

does the transactor and peer server run in the same ec2 node for the solo version?

Really looking forward to the cloudsearch integration, we would totally love to move over to the cloud offering

in the docs it says future.. Would that mean month's or should I think in that it might take a year?

@lockdown- the arch change is bigger than that — there are no transactors or peer servers, just cluster nodes. That said, on Solo there is only one node. 🙂 See https://docs.datomic.com/cloud/whatis/architecture.html for more.

@stuarthalloway does this mean that cloud could potentially handle more writes somehow?

@robert-stuttaford more per system: yes, more per db: no

right

1 transactor + 1 for HA <= N cluster nodes

ok - so the solo topology is actually a transactor and a peer-server rolled into one, and adding node 2 takes you to HA writes + load balancing and adding more nodes adds more read scale after that. right?

of course, this summary ignores all the changes on the storage layer

there are no transactors, any cluster node can handle any write

that won’t allow more writes per db because the underlying CAS in DDB is still the gatekeeper

aha, gotcha!

that’s rad

but you can have many more dbs

is there a different theoretical datom limit for cloud?

the on-prem limit is due to peer memory to hold the roots. i guess peer-server has a similar concern?

dbs and query still use the same data structures, so nothing is really different there

right

@stuarthalloway I (and some of the people I'm working with) think your comment about just having cluster nodes is pretty darn cool. It might be worth considering putting that on the main marketing page. I'm excited to try this thing out!

@bmaddy thanks! Will consider.

@stuarthalloway can we now control read-only vs read/write at the connection level? (with client / cloud)

@mitchelkuijpers TBD on search integration, user demand will drive

That makes sense

@stuarthalloway re "there are no transactors or peer servers, just cluster nodes" -- the move away from a single-transactor model seems huge (and like it must have been extremely challenging to do while maintaining same guarantees).

@robert-stuttaford IAM integration is pretty deep, so you can give a client IAM creds that are e.g. read only for a db https://docs.datomic.com/cloud/operation/access-control.html#sec-2

perfect

@eggsyntax at the risk of making it sound less cool, that part was actually pretty easy

Awww 😉

transactors always had that guarantee — if we removed the code that manages HA, you could have N transactors. Semantics would be fine but perf would be terrible

Interesting, I never realized that.

Congrats to the whole Datomic team - really glad you managed to get through all the AWS hoops!

Seconded! Exciting stuff 🙂

thanks! it has been intense working through the process

Hello! Just using your Cloud Formation template now - is there a way of setting up the template to use an existing VPC?

@cp not at present. We did a bunch of testing and found too many variables to support

@stuarthalloway ok, thanks

@cp happy to discuss in more depth if that is a blocker

Note that the bastion lets you have dev access from the internet if you want

We’re just figuring out how we’d connect our running services from different VPCs into the nodes

yes, that seems to be the marketplace preferred way

I’m guessing we do that via the bastion

https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-peering.html

^^ that makes sense

use peering (not the bastion) for prod

your security auditor will tell you the same thing 🙂

cool - that makes sense thanks

congrats on the launch!

one more quick question: why is the API endpoint a http://datomic.net url? (cf https://docs.datomic.com/cloud/getting-started/connecting.html#creating-database) - I’m wondering how the plumbing all fits together

@cp that name lives only locally inside Datomic Cloud’s VPC, allowing clients to have a stable name to connect to

[aws/client - off] I'm on peer API. as-of db's can't be with-ed? I'm trying to do that, but it's not working

(let [db-as-of (d/as-of db t)
      ;; :my-attr is "isCompomnent true"
      {:keys [db-after]} (d/with db-as-of [[:db/retract :my-attr :db/isComponent true]])]
  (:db/isComponent (d/entity db-after :my-attr))
  ;; => true
  )

@souenzzo that is correct, you cannot with an as-of db

intuition: as-of is not time travel, it is a filter

It's on docs? It fails with not error. Even tx-data on return is "correct". Very difficult to debug.

Hm, it should be, I know it has come up before.

(Feature request: https://receptive.io/app/#/case/26649 )

@favila until then it should at least throw an exception. Logging it for a future release.

@souenzzo sorry that threw you, we will make it more failfast

@stuarthalloway re: cloud, is there a transition path from solo to production? Also the pricing (since it only shows EC2) is the same. At equivalent usage (what solo can handle) is prod expected to be more expensive? If so, how much?

@denik you can transition from Solo to Production with a CloudFormation upgrade https://docs.datomic.com/cloud/operation/upgrading.html

@denik if you are looking at the marketplace pricing, their site is currently not capable of telling you which instance types go with which topologies.

Solo runs on a single t2.small, and Production runs (typically at least 2) i3.larges

I'm going through the setup right now, and for Oregon I got $21/month on solo and $233/month on production as the estimated costs. I think it said that was for 2 i3.large.

But there is no fixed equation: pricing is per hour, and running >1 instance for availability is up to you. So Production cost will vary substantially with use

@shaun-mahood that sounds reasonable — start with Solo until you need more

Of course. Thanks @stuarthalloway. I’m very excited!

Oh yeah, solo will be overkill probably for what I'm doing - I was just curious since the estimator was kind of impenetrable to understand.

“kind of” is generous

@shaun-mahood belay that, the sales dept informs me you should run Production for everything, and forget you turned it on 🙂

with an ASG size of 12

at least

Still cheaper than an on-prem license 🙂 (I mean, without Marshall's "help")

@shaun-mahood yes. That text is inside the CFT, so triggers a deeper review process. We will fix it the next time we navigate that process.

OTOH, we can change http://docs.datomic.com with an S3 push, so if the text-about-the-text could be better that is an easy fix 🙂

So I'm guessing you have something you can point people to when they complain that requiring a jira ticket to change a docstring is too onerous? 🙂

@shaun-mahood btw did you watch the video walkthrough? It is short https://www.datomic.com/videos.html

Got a solo stack launched and transacted some movies. This is awesome 🙂

Not yet, the setup so far has been exceptionally straightforward and easy from the setup instructions. Just waiting for the stack to finish creating in the cloudwatch dashboard right now.

@cjsauer hooray! Out of curiosity, are you running as AWS owner or did you do the “authorize a Datomic admin with IAM” step?

@stuarthalloway I used the admin group setup route and tested it with my non-root account. Worked like a charm.

super cool!

fabulous!

that’s great to hear

“easy” vs “securious” always a challenge

btw I think I just made that word up, it means “serious about security”

i thought it meant curious about security

Short for supersecuriousticexpialadocious

Have moved past the movie example and moved to importing real data. We used existing IAM users authorised with the datomic policy.

Have found it to be pretty straightforward, although AWS's console UI is (as usual) awful.

The blog post should have info like this in it: https://clojurians.slack.com/archives/C03RZMDSH/p1516209093000142

Right now, it looks like the ~$30 estimate for production with t2 in the AWS console is real.

@sleepyfox awesome! Glad to hear it

little typo at https://docs.datomic.com/cloud/getting-started/connecting.html Install the Clojure

thanks @denik

pushed a fix for that typo

stuck trying to run the socks proxy. aws ec2 describe-instances... returns the system name, aws iam get-user returns the user in the iam group, yet running the script returns Datomic system <my-system-name> not found, make sure your system name and AWS creds are correct.

Hi @denik. Triple check your spelling of the system name and AWS region wherever they appear

@stuarthalloway already did, before the final message it also prints

To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument command: Invalid choice, valid choices are:

acm                                      | apigateway
autoscaling                              | cloudformation....

@denik that sounds like an argument is not getting to an invocation of the AWS CLI inside the script

@denik can you check the version of your aws cli?

aws --version

@marshall aws-cli/1.10.24 Python/2.7.10 Darwin/16.7.0 botocore/1.4.15

i think your CLI might be too old

aws-cli/1.11.170 Python/2.7.14 Darwin/17.3.0 botocore/1.7.28

is what i’m using

i’m wondering if the version you have doesnt have the required aws sub-command(s)

err that should be $ pip install awscli --upgrade --user

🙂

Is this channel logged anywhere

https://clojurians-log.clojureverse.org/datomic/

Shoot, logs at clojurians-log are busted, the don’t index anymore

but that only has to 11-16

I got everything up and running - the getting started docs and videos are excellent, thanks for putting in the effort to make them so solid.

Glad to hear it @shaun-mahood!

@denik did upgrading the CLI work for you?

What is a query-group? https://docs.datomic.com/cloud/getting-started/connecting.html#creating-database

Query groups are coming soon as a means for scaling reads https://docs.datomic.com/cloud/operation/scaling.html#sec-3

But they are not yet fully implemented 🙂

We’ll have more documentation when they are

@denik ^

thanks @jaret for two separate projects, is the idea to create two different databases in the same system or two different systems each with one db?

No the intention is more for An AutoScaling Group (ASG) of Nodes used to dedicate bandwidth, processing power, and caching to particular jobs. Unlike sharding, query groups never dictate who a client must talk to in order to store or retrieve information. Any node in any group can handle any request.

Oh apologies, I misunderstood your question and thought we were still discussing query groups.

@denik Generally, you should create two separate stacks for separate projects. But can you tell me more about the projects? will they share data?

@denik important to note that two databases in the same system is totally doable in cloud. But its the sort of thing that I’d need more details on in order to provide a full recommendation.

@jaret I see, they wouldn’t share data. I’d just like to have a system to quickly spin up durable experiments. Most of them will be deleted eventually (after months), but I’d want to migrate the ones that get traction into a new system eventually. It’s a bit like a (into new-system (select-keys databases [proj-specific...]))

Yeah, that use case would be fully supported in one system. And definitely recommended.

@jaret as well as migration to a new system?

TBD

there is not currently an API to “move” a db from one system to another

likely to be some interesting requirements there ^^ e.g. re-encrypting

Does Datomic Cloud include support equivalent to what we get with the $5k on-prem license?

hmm, appearance of Datomic Cloud made me look again on this (announced at last reInvent so quite new): https://aws.amazon.com/about-aws/whats-new/2017/11/aws-privatelink-on-aws-marketplace-now-available/

@ljosa Support is covered here: https://www.datomic.com/pricing.html (bottom of page)

we will be enabling AWS Marketplace PSC (basically opt-in to sharing your support contact info) asap

at which point all users who are subscribed will have access to submit tickets/etc 24x7, shorter SLAs, etc are available as a separate support contract

Yes, I saw that but I wasn't sure how to interpret it. There was a question internally whether support was a reason to stay with the on-prem edition. I wasn't sure whether the support included with Datomic Cloud was equivalent to what we have now with Datomic Pro or whether we'd have to add a separate support contract.

gotcha

the included support with On-Prem (nee Pro) carries a shorter SLA (2 day IIRC)

if you’re already using on-prem, I’d highly suggest you look at: https://docs.datomic.com/on-prem/moving-to-cloud.html

Cloud is a new product, with different underlying structures and requirements, so it’s not “plug and play” WRT moving from pro to Cloud

Thanks! Don't worry, it would take us some time to be ready to switch over … for one thing, we'd have to turn our peers into clients.

cool. just wanted to make you (and everyone) aware of the diferences

do you have plans to eventually release client libraries for other languages, such as javascript?

@ljosa We’re tracking which languages users want on our “suggest a feature” page located by following the link at the top right on http://my.datomic.com. We’re prioritizing issues here and looking for input on which clients are most desired.

So far we have a request for python (https://receptive.io/app/#/case/18126), elixir (https://receptive.io/app/#/case/17919), and javascript (https://receptive.io/app/#/case/17965).

does it matter whether I'm logged in with the paid company account or my own unpaid account?

erlang here: https://receptive.io/app/#/case/17908

All votes count, but we weigh larger organizations that represent teams of developers over individual users. I’d recommend voting from your company account or an email with the same domain.

ok, I voted for javascript and erlang from <mailto:[email protected]|[email protected]>!

@jaret: Any idea on the possibility of a CLJS client? I couldn't find an issue on receptive but I have this feeling that someone talked about it at some point.

Its on our radar, but we should log a receptive request so we can gather customer feedback. I am going to log that one right now.

Re: clients, I'd personally favor the work being put into a documented wire protocol versus like three language-specific clients. But I guess once the first one is released the protocol will be de facto documented

Yeah it’s always been the intention to create more language libraries for Client and enable our customers to create their own. So that approach is being considered.

There are 2 questions that I can't find guidance on in the docs - I've got hunches for both but they may be totally out to lunch. - Is there a recommended way to connect to datomic cloud from an application not running on AWS? - Is there a story for how to backup and restore a DB?

@shaun-mahood re q2, there will be N stories for different purposes

e.g. disaster recovery vs. moving db somewhere else vs redundant copies for safety

… but not done yet, stay tuned

@shaun-mahood re q1, for dev you should connect through the bastion server. Is that what you mean?

I'm thinking of running our app locally rather than on AWS (at least for a while).

I'm planning to integrate it with existing locally hosted storage and systems and gradually replace all the non-datomic and non-clojure bits, then pop the app hosting out to AWS.

@shaun-mahood we intend to make tooling to help with that, but it is not ready yet

This is my first real foray into AWS (outside of really S3 and really simple things), but I've lost my socks proxy connection once already so I assume it's not a good production connection.

I'm glad I'm at least asking questions that make sense and are on the radar 🙂

@shaun-mahood correct, socks proxy is for dev