the only thing I can see is the indexer memory is high but normally it drops down when it reaches its ceiling. after the restart it jumped straight back to the prev level. Is this normal or should it restart at a min level (similar to jvm free)?

other than that, I’m still combing through logs to try and find a signal that explains it

just found “Too many open files” warnings in the logs. I am using aws client lib in Ions. could this be some kind of leak?

pretty sure it’s a socket leak. probably due to aws client mis-use on my part. I wonder if there is a way to measure open “files” as a metric so I can monitor this

any advice you might have on how to find the leak would be much appreciate

another useful idea would be a way to cause cloud instances to be killed automatically when the “Too many open files” point is reached. suggestions for this would also be much appreciated

I will try and find the leak myself but suggestions are welcome

I’m assuming this is an aws client problem but is there any other reason that “Too many files” warnings can happen in cloud?

Some folks encountering this issue on the forums have been attempting to isolate by using a file leak detector on the box and isolating the box to a standalone query group. https://forum.datomic.com/t/transactor-stops-responding-with-too-many-open-files-error/1863/3

I'd also put in that you have access to me with Datomic support. I'd be happy to help diagnose and troubleshoot. If you want to log a case shoot an e-mail to <mailto:[email protected]|[email protected]> or <mailto:[email protected]|[email protected]>

thanks @U1QJACBUM I’ve logged a ticket

thanks

it confirms, although not explains the behavior

The point of db as a value is that data retrieval from it is (mostly) repeatable. I’m not sure what the value of db equality would be because what kind of equality you want depends on what you are doing. Eg suppose dbs could compare, what would you use that for?

maybe caching - if you asked for this value/query on the same db value, don't ask datomic we already know

Dbs have a unique id, basis-t, ishistory, and optional as-of-t, since-t, and filter. You could say that if these are all equal, the dbs are equal, but that means some “equivalent” dbs are not equal (eg if one uses as-of to travel back to the same t as another unfiltered later db)

i mean yeah - equals in the more general sense should return equals if all observable properties of a thing are definitely equal

Maybe caching, but how likely is it that you have an equal but not identical db in this caching context?

its more or less accepted that if you don't know if they are equal or not - or don't want to guarantee it - they aren't

but if all those things are equal you would know for sure they are equal so its odd

> how likely is it that you have an equal but not identical db in this caching context? lets say they query like graphql.

But the basis-t in a production db is constantly advancing

user_as_of(year: 2020) {
   name,
   age
}

user_as_of(year: 2020) {
   name
   friends {
      name
   }
}

you could get db/as-of 2020 for each of these

big caveat is that i'm stupidly new to this so i'm not sure what basis-t is exactly

Basis-t is the latest T the db contains

okay so if a db contains that the utility might go down somewhat

A db must necessarily contain that

i dunno how likely it is to do 2 (d/db ...) calls before a write happens

but it could be likely enough to warrant caching - idk

Depends on your system, but I’d say not likely enough to make caching useful

okay - but i think you'll agree that there is at least hypothetically a use

and datomic isn't widely or openly developed on so its kinda hard to get a feel for probabilities

Well I’ll put it this way, I’ve used datomic on-prem in production for over 5 years on massive 10bil datom+ databases. I think this equality makes sense in an abstract way, but I haven’t encountered any practical use for it. In practice, you’ll have execution contexts share an identical db anyway

also if you read the clojure docs through as to "what is a value" and then see this "value" be not-a-value it would throw you for a loop

I think there’s the opposite problem too, which is they may assume too much of db inequality or equality

Eg that equal dbs produce equal projections always (they may not due to non-determinism in how the projections are made) or that unequal dbs produce unequal values

> non-determinism in how the projections are made Wait, what

Eg pull expressions limits, calling impure functions in your queries, hash changes among versions, that sort of thing

Equal db guarantees that the datom sets you would get are the same, but most of what we do (query and pull) is projecting out of datoms and not part of the db per se, and the guarantees get weaker for all the usual reasons that (f x)=>y may not produce the exact same y for all time in all places

btw graphql and d/as-of is exactly the right context for my question

The as-of of a “fresh” db (the result of (d/db conn) ) is always nil.

and the basis-t of the db advances with each write

are you using an as-of db with graphql resolvers? My expectation is that a d/db is established in the resolver context at the beginning of the request and is the same through the entire request

I do “changelog” that returns a series of items from different as-of dbs

so no, in my case db is not in a context

I haven’t implemented it yet, but I think it might be possible for me to ensure identical as-of dbs for different items sharing the same tx…

Could you explicitly use T as a cache key? note the “effective T” of a database is (or (d/as-of-t db) (d/basis-t db)) iff since-t, isHistory, and filter are all unset and as-of-t >= basis-t

In another repo, https://github.com/Datomic/mbrainz-importer/ I've found two dataset files that refer to the same terminology: https://github.com/Datomic/mbrainz-importer/blob/master/subsets/batches/super-enums.edn https://github.com/Datomic/mbrainz-importer/blob/master/subsets/batches/enums.edn Looking at these files, it looks like there are only two differences: • super enums are just more numerous then simple enums • simple enum hold only name attribute while super-enum holds more of them Is that correct distinction between these two?

Super enums are "global" entities referenced by several types. For instance, :artist/country and :label/country can point to the same country entity. Each regular enums is only referenced by a single type, for instance artist/gender is only referenced by an artist.

:find ?friend-name ?sibling-name will return a set of [friend-name sibling-name] tuples, is that what you’re looking for?

Hmm, thanks, that was simple. Maybe I’ve been experimenting with bad data here. So, I will get a tuple of two collections that I can then concat afterwards, right?

you’ll get an array of tuples: [[f1 s1], [f1 s2], [f2 s2]... and so on

But all in all it will be the names of all the people that are either a sibling, friend or both of the person identified by person-id?

This will return all the tuples [friend-name sibling-name]

It's is maybe not what you want?

It’s always tricky to use a contrived example to try to illustrate. What I need is to follow two or more refs from an entity, where both refs are cardinality many. From the entities those links/edges leads to I need to extract an attribute.

What if friend and sibling both point to the same person? Do you want a query that excludes this possibility?

No, exclusion is not needed.

The refs essentially represents different reasons for targeting entities. Adding another example might make it worse, but here goes: A PC-technician gets the task to fix some computers. The task entity has refs to computer entities. Refs can be eg. :broken-down or :user-complaint, but all the tech needs is to get :serial from all the referenced computers to know which ones to work on. In this case a broken down computer could have a user complaint, but both leads to the same serial.

Could you use an or clause?

(d/q '[:find [?id ...]
       :in $ ?person
       :where
       (or [?person :person/sibling ?friend-or-sibling]
           [?person :person/friend ?friend-or-sibling])
       [?friend-or-sibling :person/id ?id]]
     db (d/entid pro-db [:person/id "1"]))

This will find those entities which satisifes either of the clauses (or both), and returns a vector of their value for some attribute (in this case :person/id )

Thanks, that looks like exactly what I was after. Reads better as well