hm; I’m trying to see if I can keep my stuff in datomic or if I should use core.match instead. I have data describing the general structure of something in terms of data (mostly predicates); I have something that’s maybe a specific instance of one of those things, and want to efficiently query if it is and if so, which one, e.g. kinds of things:

[{:id :xyzzy :path [“a" odd? “b” even?]}
 {:id :iddqd :path [“c” “d"]}]]

{:path [“a” 3 “b” 2]}

(I appreciate that I can call arbitrary fns from datalog, but I guess core.match can probably do that path dispatchy bit faster :))

Hi all, is there any ref on setting up datomic with postgres, specifically the URI for the peers and how it hangs together ?

http://docs.datomic.com/storage.html#sql-database

I'm not sure about the ref, but it is simple and not really PG specific: Each peer needs to know the location of the storage and how to read it. When starting, it reads from the storage to determine the address of the current transactor.

thanks hans that was what was confusing me

since I only had the the storage in the uri

This is how failover works - A normal peer reads the transactor location and fails if it cannot contact it. The transactor tries to contact the transactor that's mentioned in the storage and if it cannot establish contact, registers itself as the new transactor in the storage.

nice and simple !

Given a parameter that is either an entity id (integer) or and identity uuid, I want to use the same query for both. I tried

(or [?e :entity/ex-id ?id]
    [?e :db/id ?id])

@yonatanel: I usually do something like this:

@luposlip thanks. I wanted to use pull inside the query but ended up using if as well. Separating the identity from the pull could be nice though. Thanks.

You could also do something like this:

(or [?e :entity/some-ident ?obj-ident]
    [?e :entity/some-other-ident ?obj-ident]
    (and [?e :entity/some-attr-that-is-always-present-for-this-entity-type _]
         [(= ?e ?obj-ident)]))

Which is best/fastest depend on your style and use case 🙂

@luposlip I remember trying that but couldn't make it work. Next time I'll remember to try this pattern again

Just tried it, it works fine.

In my case I use it for finding users.

But for performance reasons I usually check if the user-ident is a number, and then use a different (faster) query if that is the case. Then only if the ident is a String, I use the above, but without the (and …) clause.

@yonatanel (or [?id] [?e :entity/ex-id ?id])

but, let me introduce you to the wonder of lookup refs

Yes, I know lookup refs 🙂

[:user/username “some-user”]

(d/q '[:find ?e :in $ ?e :where [?e]] db a-long)
(d/q '[:find ?e :in $ ?e :where [?e]] db [:entity/ex-id a-uuid])

those two work the same

@robert-stuttaford for some reason I cannot get your example to work.

If I remember correctly, if the ex-id doesn't belong to any entity, the lookup ref method explodes instead of returning nothing.

AssertionError Assert failed: All clauses in 'or' must use same set of vars, had [#{?id} #{?id ?e}]

@robert-stuttaford Also, you must use the same vars on both sides of or

that is indeed a pity

didn't realise it threw when LR doesn't resolve

doesn't do that for missing ids

@luposlip: your query pattern works if the data type is the same for the first two clauses, but i think it may throw if not

@luposlip: curious; why return [?e ...] and not ?e .?

you’re probably right @robert-stuttaford

Just because in my case I simply want the user :db/id, not anything else.

?e . will give you the first result

[?e ...] will give you a vector of the results, no matter how many

ahh, didn’t realize that, cool! 🙂

you want ?e . 🙂

@luposlip @yonatanel these are pretty handy https://gist.github.com/robert-stuttaford/39d43c011e498542bcf8

will look at that @robert-stuttaford, seems sane.

(def uri "datomic:....") (as-conn uri) (as-db uri) (as-db (as-conn uri)) etc

oh, it's out of date, i've got more to add

added the log

:thumbsup::skin-tone-4:

Love the ?e . notation @robert-stuttaford. wonder why I haven’t seen this before 🙂

When an entity has multiple identity attributes, which one identifies the entity if I change some of them in a transaction to values already in other entities?

@yonatanel what do you mean by identity attr -- indexed, or unique?

curious, and eager for input from the Cognitects amongst us, aside from d/query's timeout, what other protections do we have against queries that OOM the process cc @marshall @jaret?

by OOM, i mean the result set can't fit in the available ram. we've had it a couple times now where some query by someone somewhere will consume all the ram and then peg the CPU with a GC moshpit

sometimes it recovers, sometimes not

i'd like to know, conceptually, what the api offers to protect us from this. with traditional dbs you can issue a COUNT to the box 'over there' and protect the client app, but in this case, the peer will download everything when issuing a count

In Datomic there isn’t an “over there"

i know 🙂 which is why i ask the question

The results have to be fully realized in the JVM. So if the result set can’t fit in ram...

this is a growing pain, of course; things that were small a while ago have gotten bigger. we're already auditing our code and our queries, and finding ways to further partition and break down the work, but i just thought i'd ask what's in the peer library to help us

aside from d/query, which certainly does help somewhat

Yeah, I’d say you might consider breaking up queries if you’re trying to realize large result sets.

Possibly split into “get entites” then populate subsets of entities

alternatively, there’s the direct datoms approach

i fully acknowledge that the answer may be 'nothing'

we're going to season everything we suspect with d/query and log the timeouts, and see where the fires start

are you having OOM as the only issue or are you also looking for queries with perf issues?

we're having OOM on a box that only has Datomic as a ram sink. the rest is http + websockets, and we have way too little active connections to cause an OOM

we suspect that some query or queries for certain users (due to large source datasets) are consuming all the ram

so the diagnostic vector for us is query

if you have things set up with timeouts and the ability to record them that sounds like a good approach. particularly if you can log the query parameters along with the timeouts

yes!

Does sync-index include fulltext indexes?

And if not, is there any way to ensure that fulltext indexes are fully created for a particular t before querying?

Fulltext indexes are eventually-consistent by definition, so I expect there is no way to ensure they are up to date. http://docs.datomic.com/release-notices.html#0-9-4699

Thanks @stuartsierra.

@robert-stuttaford By identity attr I mean :db.unique/identity. I have several per entity. Perhaps I should have used only one as identity and the rest as unique, but I wanted to use flexible lookup refs, and even if I didn't have this problem it's still interesting to know which of the identity attributes will determine the entity.

The docs say that in case of conflict the transaction will throw IllegalStateException, but in my case it didn't, since my "conflict" is between identities in the same entity, e.g asserting that [[email protected], "new unique name"] while [[email protected], "old name"] already exists will overwrite the existing name. Now I know.

@yonatanel you asserted both of those datoms in one transaction (what that section of docs applies to)? It looks more like you’re running into the difference between unique identities http://docs.datomic.com/identity.html#unique-identities and unique values http://docs.datomic.com/identity.html#unique-values and how they behave when you assert different information.