Fork me on GitHub
#datomic
<
2018-08-19
>
Oliver George02:08:29

G'day from down under. Seems like Datomic Cloud is coming to Sydney - should make a big difference to latency for Australian users. https://docs.datomic.com/cloud/releases.html#409-8407

🍾 4
Oliver George02:08:35

I didn't think this was possible because Sydney doesn't seem to provide AWS Auto Scaling based on this page: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/

dominicm05:08:41

I don't know what datomic uses, but auto scaling groups are part of ec2, not the auto scaling service, afaik.

Oliver George05:08:41

Right, perhaps I'm reading the marketplace details wrong. here's what it says:

Required AWS Services: APPLICATIONAUTOSCALING, AUTOSCALING, CLOUDFORMATION, CODEDEPLOY, DYNAMODB, EC2, EFS, ELASTICLOADBALANCINGV2, IAM, LAMBDA, LOGS, ROUTE53, S3

dominicm06:08:37

I guess it uses it then...

dominicm06:08:59

I might be wrong about auto scaling anyway 😊

dominicm06:08:20

Amazon EC2 Auto Scaling has moved You have several options for scaling with AWS. Do you only need to manage Amazon EC2 instances?

dominicm06:08:52

It's probably there for dynamodb scaling

Oliver George02:08:45

I can see ap-southeast-2 in the RegionMap of the compute templates but Sydney isn't a region I can pick on the AWS Marketplace page for Datomic - wondered if that was related or just an issue of getting marketplace updates approved by Amazon.

stuarthalloway13:08:10

@U055DUUFS you should be able to use ap-southeast-2 with the 409-8407 release. I will check into what is happening on the marketplace page.

Oliver George10:08:37

Thanks @U072WS7PE. Look forward to trying it out.

jaret13:08:43

@U055DUUFS Where were you not seeing Ap-southeast-2? I checked on marketplace and was able to see the region when I had 409 selected…

Oliver George16:08:36

Sounds like I got punked by "estimating your costs" here https://aws.amazon.com/marketplace/pp/prodview-otb76awcrb7aa

Oliver George10:08:32

Works like a charm.

vladex09:08:38

Hi! What would be an idiomatic way to query last asserted datom? My idea is to add :created-at and use max on it. does this sound reasonable?

vladex09:08:28

or get [?e ?created-at] relations and sort-by…

henrik11:08:46

Is pull-many a thing in Datomic Cloud?

stuarthalloway13:08:13

hi @U06B8J0AJ, pull is a convenience over query, you can always do what you want with a query.

stuarthalloway13:08:37

This gets asked a lot, it may be worth it to add pull-many just to spare confusion.

henrik13:08:36

I see, thank you. I figured I could probably create my own pull-many, but I saw it in an example and went looking for it. I guess if for nothing else, for backwards compatibility with existing guides that may be out there, it might be good to have it in.

eoliphant12:08:52

@vladex I haven’t had to do this personally, but you should be able easily drive this from the datomic log apis. If you do something like -> (tx-range {:start <some num> :end nil}) (last) (look at datoms asserted by tx) that should get you what you need

vladex13:08:09

@eoliphant do you think that this approach is more performant assuming that I have an index for :created-at? `

eoliphant13:08:02

not sure, I’d maybe (time) both approaches

favila14:08:32

@vladex you said “last asserted datom”, that’s impossible with a created-at attr because it can only be asserted on entities not datoms. Can you be more precise about what you are trying to do?

vladex14:08:33

yeah, you are right 🙂. Hypothetically, I use datomic for newborns registry at a hospital. I assert an entity with :name and :born-at. What is the most idiomatic way to answer what was the name of the last newborn?

eoliphant17:08:13

ah ok.. That’s clearer. So you want the ‘domain’ most recent time, not the ‘system’ time when the transaction took place. There may be a more idiomatic approach, but right off the top of my head, you can just do a :find (max ?ba) :where [ .. :born-at ?ba] ... then use that value to query for the entity you want

vladex17:08:07

yeah, that was my initial idea. “My idea is to add :created-at and use max on it”. Should be more clear next time 🙂

vladex17:08:39

but anyway seems strange, with SQL semantics is like ORDER BY LIMIT 1… maybe I should look also into raw indexes…

favila18:08:40

If you want to be really efficient, you should store the reverse index of what you want

favila18:08:03

Eg if you care about “newest” timestamp, store an attr with the negated ms value

favila18:08:23

Then it’s always first item in d/datoms

favila18:08:13

Datomic doesn’t have efficient reverse walking of its indexes, so even if you use raw indexes you will have to do some array bisection to find the last value

vladex18:08:00

interesting approach, thank you!

vladex18:08:35

but I see this pattern quite often: latest news, most recent posts, most liked comments… is datomic a good fit in this case? Or should I use third party indexing tools?

favila23:08:03

Not a good fit for a naive implementation, which is why it’s even a bit difficult to express succinctly in datalog

favila23:08:16

There’s lots you can do short of an external index, but if you can’t afford an index scan you do need to do something

favila00:08:58

Eg you could make a derived attr that orders more naturally; you could cache your “top x” list; you could read the Tx queue and keep the “top x” values incrementally in memory; you could write a bisection algorithm that uses index-scan to find the latest values without a full scan

vladex04:08:48

thank you, do you know some useful resources where I can dig into this topic?

joshkh23:08:17

Maybe someone can offer advice on modelling permissions in Datomic? I have a Permission entity that represents how any one entity can interact with another. It's essentially an edge with some data stored on it (source, target, access level). Works just fine, but a little hard to manage when either its source or target ref is retracted. The Permission entity still hangs around. I don't think I can solve this with components - that would require every "type" of entity to have a component reference to some Permission entity. Is there something tricky I can do during the retraction of some entity to also retract any related Permission entities?

4
Chris Bidler23:08:43

Could you not write a transaction fn that looks at every transaction (via :tx-report-queueor by registering a fn in Cloud) and when seeing a retraction, does a query to see if any Permissions reference the retracted thing, then does some cleanup?

Petrus Theron23:08:02

Does Datomic Cloud support DB functions yet?

Chris Bidler23:08:04

I don’t guarantee that this would be uh, performant but you could try it and see if it works well enough

joshkh23:08:42

Ah, I haven't looked into registering transaction functions in Cloud (which is where I'm operating).

Chris Bidler23:08:53

I haven’t done it in anger yet - I’m still in the “hey let’s get this existing infrastructure to scale successfully before we rewrite all our code to be Cloud-aware” phase of life - but I know you’re supposed to be able to do so

joshkh23:08:05

Is that a purely Ions thing? Or something I could use with the Cloud client?

Chris Bidler23:08:51

might be purely Ions, I haven’t really looked at the Cloud client at all tbh

Chris Bidler23:08:25

I am the living embodiment of that Squidward “fuuuuuuutuuuuuurrrreeee” .gif about Datomic Cloud

Chris Bidler23:08:47

I only care about the stuff you can just barely trick it into doing, things that are “supported”? Meh!

joshkh23:08:16

I'm living in the Cloud environment. Somewhere between the old and the new. 😉

joshkh23:08:21

Which I think is lacking in transaction functions? Or only supports a subset of what on-prem and ions does? Still wrapping my head around it.

joshkh23:08:04

I know I can do this easily in Cypher...