are all features of ion available for solo?

has anyone tested AWS App-Sync using an Ion Lambda? i.e. graphql api for Ion without any code

ehm, wait no

if its like ring, :protocol should be something like “The protocol the request was made with, e.g. “HTTP/1.1".” and :scheme is :http or :https

@chris.blom thanks! You are right, :protocol should be like Ring

Ok, it was not immediately clear to me what ion is about: My understanding now is that: - its an application server integrated with datomic - has build in tooling based on deps.edn to deploy based on git revisions - it integrates with AWS Lambda and API Gateway to handle http requests, in a mostly ring compatible way Its not clear to me what “deploying your code to a running Datomic cluster” entails: - What exactly runs on Lambda, and what runs on the datomic cluster? - What are the limitations of running code in a datomic cluster? Can I access the local disk and other AWS services? - How does the autoscaling work? - Is it possible to develop and test ions locally? - Can I run some sort of test environment for CI testing?

The picture at https://docs.datomic.com/cloud/ions/ions.html might help

Essentially all of your code is running on the d cluster. Being there you can access all the aws services. For storage, I think you’d use aws storage services, not disk. Stu or Rich can probably answer some of the others better than I can but generally the answers will be to use the aws functionality for autoscaling, ci, etc.

ok, so the Lambda functions for an Ion are just glue to interface with the outside world, and delegate the actual work to the Datomic cluster?

@chrisblom yes https://clojurians.slack.com/archives/C03RZMDSH/p1528317660000673

thanks, good to know

I care a lot about local dev (Give me REPL or give me death!)

Client API now supports :server-type :ion, which connects remotely when you dev on your laptop, but connects in memory when you deploy the same code to Datomic: https://docs-gateway-dev2-952644531.us-east-1.elb.amazonaws.com:8181/cloud/ions/ions-reference.html#server-type-ion

stu, the transaction producing functions will run on whatever node in datomic cluster, right? But the final transactions are still directed to the node acting as transactor? (so essentially, it’s like the peer model)

@andrewhr the tx fns run where the txes do. There isn't a dedicated transactor per se as with on prem

but you will be able to have independent clusters running app/query code and handling txes

I remember something about “avoiding contention”, but in retrospect doesn’t make too much sense giving ddb could probably just autoscale in response. Maybe this image make me a little confused https://docs.datomic.com/cloud/whatis/architecture.html#production-topology

as far as I understand (together with you previous explanation), query groups aka “extra clusters” will tunnel their transactions thought the primary tx group

or when do you say “extra clusters” you’re really meaning “one set of storage resources” + “multiple sets of primary compute resources”?

@steveb8n I gave a talk last month at Serverless Chicago about using Datomic Cloud with AppSync, you may expect some kind of preliminary blog post or code sample extending that talk to Ions like …today?

I am finding it very difficult to focus on my day-job work right now, knowing that I could be spinning up a Cloud instance in my personal account and exploring getting AppSync to run, looking at modeling what $day-job does with the txn report queue in Ions callbacks, etc. etc. so …I don’t think it will be too long before I have a trip report re: AppSync ready for people to read hehe

@chris.blom

- What exactly runs on Lambda, 
a generic proxy. We call it 'Ultimate, the lambda'

- and what runs on the datomic cluster? 
everything

- What are the limitations of running code in a datomic cluster? Can I access the local disk and other AWS services?
AWS services sure. It is *your* instance, running in *your* VPC. That said, local disk, probably not a great idea. 

- How does the autoscaling work?
You can trigger autoscaling of the cluster on any of various metrics we or AWS produce.

- Is it possible to develop and test ions locally?
As Stu said, sure! The db API you'll see in the ion is the same as the client sync API, and the :ion server type dynamically loads the right back end.

- Can I run some sort of test environment for CI testing?
Yes. You can run a solo instance that is a target of the same application, deploying early revs to it and tested revs to prod.

"We call it 'Ultimate, the lambda'" That's awesomely horrible 😂

I suppose if everything runs in the cluster, then there's no way to restrict certain functions to certain operations, as you can do with Lambdas?

We had some bad experiences with AWS lambda in the past: 1. It has a global queue per account (our solution: never use AWS lambda for anything of high throughput as it will block other tasks unexpectedly). 2. AWS requires node updates sometimes (one time within 3 months from launch). @richhickey Are these issues taken into account? Is the ultimate lambda free of these concerns? Thank you.

@dominicm There are distinct instances of ultimate the lambda and each is an independent AWS Lambda and proxies to a particular fn on a particular Datomic compute group. From there you have all the ordinary wiring up of Lambdas available, to particular events etc.

@jeroenvandijk AWS has improved #1 with per-Lambda concurrency reservations/limits (which we expose as a knob). Not sure I understand 2 - lambda's internal nodes?

@richhickey Thank you, interesting. Regarding 2 sorry i meant the node.js runtime (assuming you use this, but might apply to jvm runtime too). We were forced to upgrade the node.js runtime and didn't have a choice to leave it as is (like how you can choose to stick with an old AMI version)

@jeroenvandijk we care very little about the lambda runtime as we're not doing much there, and your ion code does not run there so cares not at all

I guess you might occasionally have to roll for things like that

Ok understood. Thank you. I guess it could have been more like a one time occurrence.

One other thing I noticed in the architecture of Datomic Cloud (https://docs.datomic.com/cloud/whatis/architecture.html) is that the storage of record is S3 and Dynamodb is used as transaction log. Is it possible to clean up the transaction log every now and then and rely on S3 for older data in order to save data costs? (With a self-hosted Datomic setup we have a big dynamodb table and this would be a potential cost saver)

Can ions be used for light/hobbie stuff with solo?

@lockdown- yes, or even medium/moonlighting stuff 🙂

@jeroenvandijk yes, getting the log into S3 is an optimization we intend to implement

@jeroenvandijk for many use cases, Cloud is already cheaper than On-Prem with DDB just because indexing doesn’t have to hit DDB

@stuarthalloway Nice 🙂 TBH we have been really abusing Datomic for things that is advised against. We have also reached the 10 billion datom limit times 4.. Is this something that Cloud is also addressing?

https://docs.datomic.com/cloud/transactions/transaction-functions.html#calling

is says to omit the first argument (the database) but the example is including it?

In practice how are classpath functions most commonly invoked? [:find (pull ?f [:db/id *]) :where [(partial contrib.datomic/datomic-entity-successors $) ?succ] [(loom.alg-generic/bf-traverse ?succ :db/ident) [?f ...]]] vs (->> (loom.alg-generic/bf-traverse (partial contrib.datomic/datomic-entity-successors $) :db/ident) (d/pull-many $ [:db/id *])) The first has constraints on the complexity of the clojure form, but is more structured

@jeroenvandijk out of curiosity, how are you abusing it?

Are :cookies supported in the return map? https://docs.datomic.com/cloud/ions/ions-reference.html#web-code

is there a way I can pull attributes that are not :db/valueType :db.type/ref ?

without knowing what their names are in advance ?

Also, is there a story for web apps that have real-time requirements (i.e. Web sockets or SSE)? I don't believe AWS API Gateway has built-in support for either of those.

when i try to clj -Spom the deps.edn in the guide of ion i get

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.datomic:ion:pom:0.9.7 from/to datomic-cloud (): Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 82E950FAE7704579; S3 Extended Request ID: Su8rZMB9c+Z65RQbBwA8K1mxjVkjX0JU5NJRpbrT6k2/rRN8ubyMA3SJO9TucAlKFVf0fQUVZ3w=)

{:paths ["src" "resources"]
 :deps {com.datomic/client-cloud {:mvn/version "0.8.50"}
        com.datomic/ion {:mvn/version "0.9.7"}
        org.clojure/data.json {:mvn/version "0.2.6"}
        org.clojure/clojure {:mvn/version "1.9.0"}}
 :mvn/repos {"datomic-cloud" {:url ""}}
 :aliases
 {:dev {:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.160"}}}}}

@kenny right, API gateway has no websockets/sse

@gabriele.carrettoni do other things (like clj -Spath) work?

➜  datomic clj -Spath
Error building classpath. Failed to read artifact descriptor for com.datomic:ion:jar:0.9.7
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.datomic:ion:jar:0.9.7
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:276)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:192)
	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:253)
	at clojure.tools.deps.alpha.extensions.maven$eval668$fn__670.invoke(maven.clj:77)

same AccessDenied

I don’t have any issues doing that on my own box, not sure why you’d see that

I assume you’re not proxied or anything

@alexmiller i was inside the work vpn, just tried outside of it and i get the same error

@alexmiller maybe it's picking something from my .aws/credentials? :thinking_face:

maybe

can you try with com.datomic/client-cloud {:mvn/version “0.8.52”} instead ?

@alexmiller

I think prior is a bug in the ion-starter deps.edn, but it’s not this problem

aws s3 cp . - does that work for you?

download:  to ./ion-0.9.7.pom

yup

the plot thickens

I guess I’d start looking for other environmental variables - default AWS creds, repository settings in ~/.m2/settings.xml, etc. But I’m not sure what in any of those would actually cause a problem.

the fact that the s3 call works makes me think it’s not the aws creds

I think my question got lost 🙂 Are :cookies supported in the return map for web code? https://docs.datomic.com/cloud/ions/ions-reference.html#web-code

@gabriele.carrettoni I was able to repro it by monkeying with my aws creds, we will check into this

Ion is closed source, correct?

@kenny yes, it's part of Datomic

Any plans for a cloud agnostic Datomic Cloud?

@kenny nope, part of the value prop is getting maximum leverage and agility.

are tx functions aws lambdas?

@lockdown- no, some ions are just used by txes and queries, not behind lambdas

@richhickey your customers aren’t worried about vendor lock-in?

Application Load Balancer (ALB) supports Websockets and lately it got a feature to delegate authentication (https://aws.amazon.com/blogs/aws/built-in-authentication-in-alb/). Do you see possibility to front Ions with ALB?

@richhickey k, cool, just want to use ions for tx functions for now with solo and app code running on a ec2 node

Thinking that API Gateway is public (might change in future though), ALB could be exposed internally only, to allow say access from corporate network only

@kenny lockin has to be balanced with other objectives. If it dominates ones thinking you'll never maximize your leverage of the platforms. I can't say what's most important for anyone, but there are tradeoffs. We are supporting the 'leverage AWS' strategy. Lots of people succeeding there while others worry and don't ship.

@richhickey Does ion auto scaling apps essentially supersede the client model? Meaning there is no datomic client in play? Just a client-like local api

@dustingetz right, client-free apps totally possible

Idiomatic, right?

well, there's zero reason for clients inside ions, but you may have legacy architecture that dictates use of clients. They're not deprecated or anything

I think many apps can be written as ions

and of course you can mix and match

Are there technical reasons for no entity API or is it just historical at this point, and is the ion "local client api" equivalent in power to the entity api

@dustingetz there are semantic differences with client (vs peer) that yield 'no entity'. The 'local' API of ions matches the client API so people can do local REPL dev and testing (against cloud). So semantically, it's compatible with 'over wires'. That said, the perf of the same API in local mode within ions trounces clients.

but keeping wire semantics means if you need to make an architectural shift that requires moving some parts to clients you're not screwed

thus 'client' is the only API of cloud, ion or out

where does the performance gain comes from? eliminating the network calls?

@lockdown- right, no wires, same process

Remote client requesting query for Ion cluster [:find (pull ?f [:db/id *]) :where [(partial contrib.datomic/datomic-entity-successors $) ?succ] [(loom.alg-generic/bf-traverse ?succ :db/ident) [?f ...]]] Ion application local query (->> (loom.alg-generic/bf-traverse (partial contrib.datomic/datomic-entity-successors $) :db/ident) (d/pull-many $ [:db/id *]))

if I am an Ion app, I can do both, is the former considered idiomatic? Because it is equivalent and works over wires

the former has the overhead of the network, which of the two you use depends on your needs/architecture

@dustingetz the portability of the former seems like a flexibility win, but you may not care

@richhickey I like the former but that

thats a pretty wild "partial" in there

you would consider that idiomatic? Im ok with that if you are

idioms take time to develop

these are some brand new power tools, would hate to pour concrete advice 🙂

ok:)

To what extent is it reasonable to query other data stores (e.g a remote SQL or ElasticSearch server) from ions?

@val_waeselynck you can do anything as long as you give the datomic cluster node's role the necessary permissions

The ion will autoscale though and the foreign store will not

you'd need a queue

your instances, your VPC

@dustingetz not everyone scales or needs to

but in any case the important point is, as @richhickey said, it is your instances, use them as you will

@richhickey thanks, it may be a good idea to make that obvious in the docs - it helps assess the power / limitations, especially since we've been accustomed to "be careful of the code Datomic runs for you" with tx fns

This is very exciting - it seems to give you the getting started experience of Firebase or Lambda with the scalability of advanced hand-rolled systems

@val_waeselynck that still applies, you can hold up your txes

If I write to a durable kv store from a transactor fn and it takes 1ms to return, will this in practice slow down transactor throughput? Or are transactions processed in parallel up until the dynamo conditional put which i understand to be batched

it's going to be serial

Oh I see, because if there is a cas in the same tx, that needs a dbval

because e.g. tx fns can query and expect to see prior txes

yes, stuff like that

but remember you will likely be initiating your txes from ions, so there's another opportunity there for coordinated (if not transactional) work

I dont understand that, can you give me another hint

put in elasticsearch, transact

means it might be in elasticsearch but tx fails, but not holding up txes

Oh ok, so if the other store supported two phase commit that might work too

or whatever, cleanup on tx fail

you are likely retrying

thanks

why web ions and not just a ring app in a lambda function?

the point is, you will be in your VPC, running in a role, able to do cool things

@lockdown- a) ease, b) not having to run in lambda execution container, c) running in db context vs making a client call, d) speed

the important value prop is API Gateway. Putting a lambda behind it is just one option (hint)

ok, pretty cool

@richhickey Since it is our instance, why is eval blacklisted in :where clause evaluation

Or is/could that be different in Ion

there is an :accept list in ion, put stuff there to allow it

clojure.core/eval was excluded from cloud for security?

security of your own jars or something

nothing will run not on the list

i.e. nothing will be an entry point in tx/query/lambda

pants on by default

lol

people inadvertently expose clients, query etc

Oh, ok makes sense

are there docs that show where web ions might be incompatible with the ring spec?

trying to push

{:command-failed "{:op :push}",
 :causes
 ({:message
   "You must either specify a uname or deploy from clean git commit",
   :class IllegalArgumentException})}

@alexmiller may i suggest to update https://docs.datomic.com/cloud/ions/ions-tutorial.html#sec-5-4 changing the curl call from

curl https://$(obfuscated-name).

curl https://$(obfuscated-name). -d ':hat'

Can I specify an AWS profile when pushing or deploying?

https://docs.datomic.com/cloud/ions/ions-reference.html#deploy 🙂

How are people dealing with schema changes in development mode. I’m using conformity, but there is this problem: When I’m developing I might be “fiddling” with some bit of schema. I’d like to put it in a migration, but as soon as I do that it is “fixed” in the database, and I have to “hand unroll” it if I change my mind. It seems: 1. make an “up”/“down” function to use during dev, and drop the “up” one into the migrations only once it is stable. 2. It occurs to me that if you were to look up the tx time of the “up”, you could “undo” all of the changes that had been made since that time (which could be your automatic “down”…`d/as-of` to get the old values for the things the history could tell you have changed since then) Given those two, seems you could just add something like a “SNAPSHOT” versioning in conformity that would automate that on startup in dev mode (e.g. undo everything that has happened since the earliest SNAPSHOT was conformed, then re-conform). Anyone aware of existing code for doing that, or an alternative that is as convenient? Otherwise, we’re probably going to write it.

i’ve used datomock to develop migrations on top of an existing db

using forked connections to develop the migration, and transact it to actual db when finished

https://github.com/vvvvalvalval/datomock

nice..I’l look at that

i’ve even used it to test migrations in production, by connecting to the production db, forking the conn, and running the migrations and app with the forked conn

Seconded on datomock, it makes datomic development workflow a joy ❤️

yeah, I think that looks pretty good, actually.

Super cool idea…and only about 100 LOC…it’s all about finding the right abstraction on top of your tool

glad I asked. Thanks!