Fork me on GitHub
#datomic
<
2016-08-12
>
mrmcc300:08:51

if you’ve got a fixed list of application types you could make an attribute for each

:application/A (valueType ref, cardinality one)
:application/B (valueType ref, cardinality one)
...

d._.b00:08:11

@mrmcc3: In general, I am confused about cardinality and uniqueness (identity vs value).

d._.b00:08:08

Say I have:

:company/name (valueType string, cardinality one, db.unique/value)
:company/applications (valueType ref, cardinality many)

:application/name (valueType string, cardinality one)
:application/cool (valueType boolean, cardinality one)
I would like to make it so there can never be a duplicate :application/name belonging to a particular :company.

d._.b00:08:24

So transacting something like the following will fail:

[{:db/id #db/id[db.part/user]
  :company/name "Foo"
  :company/applications
    [{:application/name "Duplicate"}
     {:application/name "Duplicate"}]]

d._.b00:08:06

(I realize that sort of makes it look like they're component entities, but I am not using a component here, using a ref)

marshall00:08:14

@d._.b: you might want to look at transaction functions http://docs.datomic.com/database-functions.html

d._.b00:08:36

🙂 I sort of had a feeling that might be coming.

d._.b00:08:58

Is what I'm asking crazy talk?

marshall00:08:16

Entity-level uniqueness enforcement is something you'd need to implement via something like transaction functions. Uniqueness by value can be enforced database-wide with :db.unique/value

marshall00:08:07

:db.unique/identity is for enforcing unique entity identities (ie your company name)

d._.b00:08:42

@marshall: Based on my reading, the same is true of specifying an attribute is "required", yes?

marshall00:08:56

Datomic is 'inherently sparse'. If you need to ensure the presence of certain attributes you can do that via transaction functions used as 'constructors'

d._.b00:08:59

So, I could conceivably achieve the same effect as what I was describing above by enforcing some set of attributes are present

d._.b00:08:17

(which are also unique)

marshall00:08:36

Yep. That logic can be put in the transaction function.

d._.b00:08:42

Whether that's a good idea or not is another matter 🙂

d._.b00:08:03

"depends on what you're trying to do" of course

marshall00:08:03

In general, I'd say that is the right approach for that use case. The major caveat is that transaction functions run on the transactor and can affect overall write throughput, but I'd argue that these cases (creating a new customer/user/etc) are infrequent and important, so i would tend to implement validation and enforcement that way

d._.b00:08:02

Yeah, when I can in relational land, I like to get out of the way and let the database do the work. Validation functions vs simply enforcing constraints

marshall00:08:15

In a sense, transaction functions are exactly that - letting you define the behavior the db enforces

marshall00:08:35

S/behavior/constraints

d._.b00:08:44

@marshall: Perhaps I've just missed a couple of places in day-of-datomic, but a suggestion: One file containing a schema with some many/ref attributes and uniqueness and backward references (e.g. :_foo), where all of the datoms are transacted in the same file, using a nested map, the list form, and finally, adding onto and retracting an item from the many/ref attribute.

d._.b00:08:38

That might be deeply specific, but I've read the Transaction, Schema, Identity and Uniquness, etc. docs several times over, and clicked around the day of datomic repo, and wanted to pass along the feedback.

d._.b00:08:01

Of course, I missed constructors 🙂

marshall00:08:16

I appreciate it. I'll look at what we have and see if I can put together some something along those lines

d._.b00:08:37

@marshall: In general, I think a fuller example of an app with some slightly-less-than trivial data model would be much appreciated. For instance, the Best Practices documentation mentions that Database updates often have a two-step structure: .... The examples I've seen don't do a whole lot of this.

d._.b01:08:32

Last suggestion is: the mbrainz example doesn't create a partition, and it seems to me it ought to.

d._.b01:08:30

Along those lines, two more questions: - Is :foo.bar a legal partition name? - "Your schema should create one or more partitions appropriate for your application." <- I seem to recall hearing during the Q&A at Datomic Conf something about when it's appropriate to mess with partitions, and I thought it was "not much, if ever". If I have :foo/name, :bar/name, :baz/qux, should I be creating partitions for :foo, :barr, and :baz?

marshall01:08:15

Partitions are an optimization for index locality only. If you know ahead of time that some set of data will often be accessed together, then it might make sense to put those data in a partition

marshall01:08:16

You are very unlikely to suffer from not having defined your partitions perfectly.

d._.b01:08:16

@marshall: I promise these are the last of my last questions for the night, and want to thank you for all of your help so far: I notice that the samples/seattle/getting-started.clj example shows creating a partition for :communities. It transacts a :community datom using a :communities tempid. It made me wonder a couple things: 1.) The partition was created after import of the seattle data. Can the creation of a partition after-the-fact change anything about locality of datoms that were already added? 2.) Along the same lines, is there any secret handshake w/r/t the partition's name :foo, and attributes which use the namespace :foo. (See: :community/name w/ a partition named :communities) There's not right? I assume that the only way to get the locality boost that partitions allow for, is to reference that partition as the :db/id when adding a datom.

d._.b01:08:35

And, I suppose finally -- if you were to realize at some point: "Wow, I really need better locality..." what would you do?

marshall01:08:44

Correct. The partition a datom is in can only be defined when it is transacted

marshall01:08:11

And the name namespaced keyword you use for the attribute id is not related to the partitin

marshall01:08:56

For your last question- I've never seen that happen, but if it did the approach would be the same as that for 'I need fulltext' or 'I need to change the data type of an attribute' - rename the existing attribute - create a new one with the old name in the desired partiton - migrate the data from the 'old' attribute to the new one

d._.b01:08:02

@marshall: I realize you get paid for it, but it's late, so I owe you a beer.

d._.b01:08:18

to help people out in this channel, I mean

d._.b01:08:40

Thanks a lot for all of the help; I really appreciate it.

d._.b01:08:44

Have a good night.

marshall01:08:47

No worries:) you too

val_waeselynck10:08:40

@robert-stuttaford: was a typo in my schema, my bad 😕 thanks for your help

jaret14:08:19

@yonatanel: Per your question from yesterday at 4:31 AM EST, I talked to Stu about the 2012 post you linked. We optimized query with predicates in 2013:

## Changed in 0.9.5130
* Performance Enhancement: The query engine will make better use of AVET
  indexes when range predicates are used in the query.
In terms of your second question, the API is correct and is essentially saying :vaet contains datoms for attributes of :db.type/ref and is the reverse index.

yonatanel18:08:46

@jaret: Do you know if only a single index is used in queries? I wonder if I should cram filtering logic into queries, or have a minimum of that in queries and the rest in regular clojure code. I have reasons for both

kenny18:08:18

How can I query to see if a :db.cardinality/many value is exactly equal to a passed in collection? For example, I pass in a collection ?coll and I want to find all entities whose :cardinality-many value is exactly equal to ?coll. So I write:

'[:find ?e .
  :in $ [?coll ...]
  :where
  [?e :cardinality-many ?coll]] 
But this returns all entities whose :cardinality-many value contains an entity in ?coll.

kenny18:08:27

Where the passed in collection is a set, so order does not matter in the equality check.

robert-stuttaford18:08:38

@kenny: you can just compare the set of coll to (:c-m (d/entity db your-e)) with =

kenny18:08:22

I am trying to find your-e though

robert-stuttaford18:08:45

afaik datalog doesn't support this. to express it in datalog terms, it'd be [:find ?e :in $ ?c1 ?c2 <and more> :where (and [?e :attr ?c1] [?e :attr ?c2] <and more>)]

robert-stuttaford18:08:44

you could write a function (defn has-exact-coll? [db e your-coll-as-set] (= your-coll-as-set (:attr (d/entity db e)))) and call it from within your datalog (d/q '[:find ?e :in $ ?your-coll-as-set :where [?e :attr] (your-ns/has-exact-coll? $ ?e ?your-coll-as-set)] db (set your-coll))

robert-stuttaford18:08:17

but you'll want to find some other way to first restrict which ?es you're looking at, because otherwise you're checking every entity this way 🙂

robert-stuttaford18:08:09

one simple way to do that is to include a clause that first checks for ?e with :attr, as i have done above

robert-stuttaford18:08:18

does that make sense?

kenny18:08:04

Yes. Thank you 🙂

flipmokid19:08:49

Hi all! I'm playing around with querying clojure data structures with datomic's query engine. I've covered off most of the things I wanted to try but I'm finding it difficult to express one particular thing. Say I have a two lists of lists, the first containing user information (account id, gender, zip code) and the second containing some replacements (account id and zip code). I want to get all users from the first list of lists and return the zipcode to be the one form the second list if the user is in it or the zipcode from the first list otherwise. I'm not sure the best way of expressing it, whether the data should be merged before I query it or whether I can do this within the query. So far I have ` (ns datalog-test.core (:use [datomic.api :only (db q pull) :as d])) (q '[:find ?accid ?gender ?zip :in $p $r :where (or (and [$p ?accid ?gender _] [$r ?accid ?zip]) [$p ?accid ?gender ?zip])] [[1 :m 22321] [2 :f 23343] [3 :m 32431] [4 :f 34958]] [[2 49884] [3 4857]]) ` but I get the error: ` :db.error/invalid-data-source Nil or missing data source. Did you forget to pass a database argument? {:input nil, :db/error :db.error/invalid-data-source} ` Can anyone offer guidance on how best to achieve the above?

vinnyataide20:08:29

hello, I've installed the datomic dep in my clojure project but don't know where to find the transactor

vinnyataide20:08:44

@flipmokid: the db connection is a obligatory property that should be passed as last parameter

vinnyataide20:08:01

aka

(def db (d/db conn)

flipmokid20:08:10

@vinnyataide Hi, I'm using this directly on Clojure data structures and not using a Datomic instance.

vinnyataide20:08:22

the problem is that you are using a q function that expects a db

vinnyataide20:08:38

the datomic api expects a data source, even an in memory one

bhagany20:08:29

passing data structures instead of db's is a thing you can do

bhagany20:08:51

not sure what's up with the error, though

vinnyataide20:08:59

I see 2 data structures right?

vinnyataide20:08:20

can you do that?

bhagany20:08:58

maybe not?

bhagany20:08:11

I also think that you don't need the :in clause

flipmokid20:08:24

@bhagany: Yes it's an odd one (unless I'm doing something silly), I find I'm only seeing the errors when using the or/and expressions @vinnyataide: Check out https://gist.github.com/stuarthalloway/2645453, I was surprised and happy that you could do datalog queries on clojure data directly

flipmokid20:08:35

In the gist he uses multiple collections too

Ben Kamphaus20:08:09

I suspect it’s related to the or clause and multiple data sources in there? and the in clause is definitely necessary when passing more than one data source.

bhagany20:08:12

I was looking at this: https://github.com/alandipert/intension, and noted the lack of :in… but now I realize that it's because it's implicit

bhagany20:08:32

ah, that's right… with or it has to be like ($ or …)

flipmokid20:08:58

I see, so I can only refer to one data source at a time in the or

flipmokid20:08:10

Hmm... I wonder how I could achieve what I want to with that restriction

Ben Kamphaus20:08:16

I believe or, not, and pull expressions may all have rough edges when it comes to handling multiple data sources.

bhagany20:08:35

fwiw, this returns results for me:

bhagany20:08:38

(d/q
'[:find ?accid ?gender ?zip
  :where (or (and [?accid ?gender]
                  [?accid ?zip])
             [?accid ?gender ?zip])]
[[1 :m 22321]
 [2 :f 23343]
 [3 :m 32431]
 [4 :f 34958]
 [2 49884 0]
 [3 4857 0]])

bhagany20:08:10

I added the 0's in the last two datoms to resolve an IndexOutOfBoundsException

flipmokid20:08:55

@bhagany: Thanks for trying, I'll give it a go now and see what results it gives

vinnyataide20:08:07

what about the transactor?

vinnyataide20:08:18

I can't find anything about the location in the documentation

vinnyataide20:08:36

it expects you to download the lib as a standalone service

bhagany20:08:20

are you trying to run a dev transactor?

bhagany20:08:46

there's a shell script to start it up in the package you download - bin/transactor

vinnyataide20:08:05

but I downloaded it as a dep in lein

vinnyataide20:08:16

idk where it is

bhagany20:08:17

there are two parts - the thing you downloaded via lein is the client library. the thing you download from http://my.datomic.com is needed as well, for the transactor

bhagany20:08:58

I'm using pro starter though, if you're using free, the process might be somewhat different

vinnyataide20:08:10

me too, I'm using pro starter

vinnyataide20:08:34

Yeah, since the transactor is only one per machine its kinda obvious

vinnyataide20:08:38

thanks for the help

bhagany20:08:58

okay, then just to put it all together, here's my whole install process: - download a zip from http://my.datomic.com and unzip it - bin/maven-install for the client library - modify the sample transactor.properties to fit my needs - bin/transactor + appropriate args to run the transactor

vinnyataide20:08:41

do I need the maven install if I did the project with credentials gpg?

bhagany20:08:00

I think that accomplishes the same thing, but I've never tried it

vinnyataide21:08:19

I'm gonna make a technical report about a system that I'm making in datomic with om next, so these details are really good to me 🙂

zane22:08:57

Are pull queries supposed to work with history databases? http://docs.datomic.com/best-practices.html#use-history

zane22:08:07

(d/q '[:find ?p (pull ?tx [:db/txInstant]) ?added
       :in $ ?userid
       :where [?u :user/purchase ?p ?tx ?added]
              [?u :user/id ?userid]]
     (d/history db) userid)

zane22:08:15

Throws an IllegalStateException for me.

marshall23:08:28

@zane: no, pull is only supported on current value of db

marshall23:08:56

Or rather, not on a history db. I believe it does work on asOf dbs