Fork me on GitHub
#datomic
<
2016-06-13
>
danielstockton09:06:37

is it right that attributes are idents and therefore are stored in memory in every peer? is this why the recommended limit is 1000000?

potetm13:06:43

@danielstockton: That is my understanding.

danielstockton13:06:16

Reading the docs a bit more, it's a bit more clear: Idents should be used for two purposes: to name schema entities and to implement enumerated tags.

danielstockton13:06:32

What I'm calling an attribute is actually an ident to a schema entity (attribute)

danielstockton13:06:51

I wonder if you can create schema entities which don't have an ident and refer to them by id

marshall13:06:05

@danielstockton: the ident is a required element of a schema entity (attrib definition): http://docs.datomic.com/schema.html#required-schema-attributes

danielstockton13:06:38

ah yes, so it is

danielstockton13:06:20

i'm trying to get my head around the point of installing an attribute and why this is useful i understand why a schema entity must have particular attributes but not this one: :db.install/_attribute :db.part/db

danielstockton13:06:27

can't the partition be inferred from the tempid anyway?

rauh13:06:19

@danielstockton: Not an expert, but there is also :db.alter/_attribute so it doesn't have to be :db.install/_attribute

danielstockton13:06:23

i guess that could also be inferred, if the ident already exists

danielstockton13:06:34

im just wondering if there is any leverage to be had from this or its just an implementation detail

curtosis14:06:33

@danielstockton: not an expert, but as I understand it the purpose of :db.install/_attribute is to hook the schema entity to the db entity. Otherwise the database doesn't know about it (and can't use/enforce it).

conan14:06:18

Does anybody know of good resources about running multiple datomic instances on the same backing storage? we're trying to work out what the best practice is around this sort of thing

curtosis14:06:15

The tempid is intended to be opaque, so no inference about partition should be made from it. Implementation-wise, the transactor could presumably infer the partition, but it can't infer that the entity you're creating is intended to be a schema entity that should be used to define legal attributes.

curtosis14:06:35

s/define/constrain/

curtosis14:06:55

(remember that :db.install/_attribute is an inverse, so the schema entity doesn't have the partition as an attribute; the db partition entity has the schema entity as an attribute.)

danielstockton14:06:50

that's another thing i don't understand, i thought refs were in both directions

danielstockton14:06:57

it just depends whether you use the eavt or vaet index

danielstockton14:06:19

i guess you have to state a direction so datomic knows which one to use actually

curtosis14:06:29

they're traversable in either direction b/c of the indexes, but you still have to pick which entity has the attribute.

danielstockton14:06:40

so when you want to transact a new attribute on to an entity in a certain partition, i guess datomic looks up whether this is a valid attribute for that partition

danielstockton14:06:07

still seems like it would be possible to infer this information throug the tempid and if :db/cardinality is present

curtosis14:06:08

not quite... it looks up whether there is a valid attribute for the db.

danielstockton14:06:34

hmm, why does an attribute need to be installed in a specific partition then

curtosis14:06:18

it's not being installed in the partition, it's installed as an attribute of the db partition, which is special.

curtosis14:06:55

(that may not be really accurate, implementation-wise, but the idea is roughly the same. you're telling the db to use this entity as a schema entity; it's very different from just transacting an entity into a partition.)

danielstockton14:06:49

yeah, thats why i wonder if it can't be inferred

danielstockton14:06:13

:db/cardinality is a reserved attribute, it could tell the transactor that 'this is a schema entity'

curtosis14:06:19

by very extremely inappropriately coarse analogy to SQL: DDL does different things than DML.

danielstockton14:06:21

or there could have been a different function transact-schema

danielstockton14:06:27

it makes me think there is some leverage to be had from it

danielstockton14:06:55

there has clearly been some effort to define everything using the same data model

curtosis14:06:41

well, yes. that's very much the point.

curtosis14:06:27

there's nothing stopping you from writing transact-schema that adds the :db.install attribute for you

danielstockton14:06:34

can i transact an entity that uses an attribute defined in a different partition from that entity?

curtosis14:06:46

you can't not

curtosis14:06:18

attributes are always and only defined in the db partition; user entities should never be in the db partition.

curtosis14:06:15

essentially, all your data should be in :db.part/user unless you need to split it up for index locality reasons.

marshall14:06:16

@danielstockton: partitions are strictly an indexing performance enhancement. there are no restrictions about attributes from multiple partitions being associated with a given entity

marshall14:06:36

sorry - not indexing performance, but index locality

marshall14:06:56

which may impact query performance, depending on usage patterns

danielstockton14:06:13

makes sense, so you want an entity to be in the same partition as its attributes (typically) so you get more cache hits around them

danielstockton14:06:50

but it isn't a hard rule, there might be reasons for not having them in the same partition

marshall14:06:40

keep in mind, an entity is not something that ‘exists’ in it’s own

curtosis14:06:40

the db partition is special (and small) so it really shouldn't be a factor

marshall14:06:17

the entity is simply an ID, which is the first element of a set of datoms, each of which is an attribute/value combination

curtosis14:06:39

oh, sorry - getting a little imprecise in this about "attributes" and "attribute definitions"

danielstockton14:06:10

yeah, think i was getting them muddled too

marshall14:06:19

the idea with partitions is if you have a set of attributes that are likely to be accessed together, then having them in the same partition can improve performance via index locality

danielstockton14:06:00

i get that, since its encoded in the entityid and therefore they're adjacent in the eavt index and you'll get more segment cache hits

marshall14:06:06

yep exactly

curtosis14:06:22

right. :db.install puts the schema "entity" (attribute definition) into the db partition.

danielstockton14:06:23

just trying to think why you might want to install a schema entity in a different partition to db.part/db

danielstockton14:06:39

why that's explicit in the transaction, rather than just inferred

marshall14:06:46

you cant install the schema definition anywhere but the db partition

curtosis14:06:14

I would say that it's explicit because it has to be explicit somewhere that you intend to create a schema definition.

danielstockton14:06:51

the entity id is in the db.part/db partition and it has a :db/cardinality attribute, those are two clues it would seem

danielstockton14:06:56

i don't know, it seems strangely verbose

curtosis14:06:05

it would be surprising for "enforce this as a schema definition" to be implicit behavior.

curtosis14:06:31

@marshall: huh? I've not seen that constraint before. Or at least not understood it.

marshall14:06:40

sorry, I mis spoke

danielstockton14:06:51

what else would you be transacting into #db.id[:db.part/db]?

curtosis14:06:02

partition entites

curtosis14:06:06

(for example)

danielstockton14:06:09

ah and functions maybe?

danielstockton14:06:30

"The attributes you create for your schema must live in the :db.part/db partition."

curtosis14:06:58

functions don't need to be in :db.part/db, IIRC

danielstockton14:06:03

am i right that nearly everyone uses db.part/user for their data, even though it says its for experimenting (like the default repl namespace)

marshall14:06:03

they do not

marshall14:06:08

functions go in user data

curtosis14:06:56

(aha! I learned something today as well! I've been playing in :db.part/user by default. 🙂 )

danielstockton14:06:51

i haven't ever seen any justification for that advice though

danielstockton15:06:12

it seems to me like you'd usually have a totally separate db for experiments

curtosis15:06:34

I think it's at least advisable for the same reasons that you shouldn't put your code in the :user namespace, or in Java in the default package, or in SQL in the "dbo"/"admin" schema/namespace/partition/whatever-your-SQL-calls-it.

danielstockton15:06:59

but why create a user namespace at all, it seems like most people have taken to using it as the main partition for their data

curtosis15:06:45

for the same reason there is a default package, etc. 😉

curtosis15:06:57

so that it works out of the box

danielstockton15:06:20

yeah, fair point

curtosis15:06:07

a disturbing number of production MSSQL databases are in the dbo schema.

danielstockton15:06:54

mhm, should probably be a bit louder in the docs and getting started type tutorials, i imagine its a pain to start building your application and then realise later all your entities are in the wrong partition (especially with datomic)

danielstockton15:06:46

with datomic, you kind of have to get things right from the beginning

sdegutis17:06:57

Is it possible to get a list of transactions? Something like (d/q '[:find ?tx :where [_ _ _ ?tx]] db)?

Lambda/Sierra17:06:49

@sdegutis: yes, with the Log API

sdegutis17:06:56

@stuartsierra: Perfect, thanks. It seemed a bit hidden, but it's exactly what I need.

sdegutis17:06:28

Hmm, is 16,000 transactions high?