Fork me on GitHub
#datomic
<
2016-06-09
>
peterromfeld05:06:10

just a initial commit, and by far not finished. But would like some input: https://github.com/peterromfeldhk/datomic-schema

peterromfeld05:06:42

btw throw exceptions and lazyiness is a bit painfull

peterromfeld05:06:49

how to be able to still catch ex-data, without dropping lazyness? is it possible?

peterromfeld05:06:51

(i was working as sys admin and devops before, so this is my first lib since i entered developent! Im super thankful for any input and help!)

peterromfeld06:06:11

need to make it more clear what i mean without constraints compared to other libs with example, but too tired now. will do over weekend

peterromfeld06:06:43

its mostly about that a entity is not a table, and you might be open that certain attribute could belong to one or another entity

peterromfeld06:06:47

github example

peterromfeld06:06:54

a user can have repos

peterromfeld06:06:01

but also an org can have repos

peterromfeld06:06:32

but when you lookup you might dont want to care if its a user or org entity, but you still wanna know the associated repos

peterromfeld06:06:09

so you have a entid(user or org) and want to get repos for it, without caring if its a user or org entity

hans06:06:36

peterromfeld: we have such a library as well, seems like many people do

peterromfeld06:06:37

you could still use or, but why constrain yourself?

hans06:06:06

peterromfeld: one comment: why use refs for enums? we're moving over to keywords because they're much easier to work with.

peterromfeld06:06:24

a enum is just a ref to a ident

peterromfeld06:06:44

we had start problems with datomic because we used ADI at first

peterromfeld06:06:58

and almost all devs got missleaded how datomic schema works

hans06:06:14

peterromfeld: i know. yet, as there is no type checking anyway and a ref attribute can take any value, we don't see why one not want to use keywords instead.

hans06:06:27

peterromfeld: funny, we used and abandoned adi as well.

peterromfeld06:06:42

i dont know the difference of ref datoms to keyword datoms

peterromfeld06:06:05

but a ref to the same ident may be more effective then a new keyword for every entity

hans06:06:25

"effective" in what way?

peterromfeld06:06:26

i dont know whats the difference between a ref to same datom, and a new datom for every keyword for a entity

peterromfeld06:06:41

in the end it may be the same, you have a ref datom or a keyword datom

peterromfeld06:06:05

but i read that ident enums more effective then keyword datoms

hans06:06:16

not quite, because a keyword is stored as a string.

hans06:06:35

you probably mean efficient. i'd like to know more.

hans06:06:16

anyway, thank you for sharing your library. i hope that at some point, there will be a unified schema syntax for datomic

peterromfeld06:06:43

its not a lib yet, didnt published it yet 🙂

peterromfeld06:06:57

i would love that we could build it out a bit together

peterromfeld06:06:48

but i have to say, i have to deal much with afterdamage done from ADI

peterromfeld06:06:02

many still dont understand datomic

hans06:06:31

it is hard to understand, and there are many ways to use it.

peterromfeld06:06:32

and its was very bad discussion to use for initial learning

peterromfeld06:06:21

better go forward a bit slower, but at least you understand what you working with

peterromfeld06:06:46

now we need to refactor soo much..

peterromfeld06:06:15

anyway, i would love if we could build out my initial idea (which is most likly very similiar to many others)

hans06:06:43

we've kind of succumbed to not creating any abstraction layer over datomic and letting it bleed into our application code in various ways.

hans06:06:05

we cannot share our library for legal reasons. i don't like it, but that's how it is.

peterromfeld06:06:13

you can still have a ds namespace which takes care of it

peterromfeld06:06:35

and if you decide to change you only need to change there and in subns

peterromfeld06:06:21

always keep ds stuff in one place! you never know if you wanna change... easier to refactor in one ns + subns then all over the place

peterromfeld06:06:11

even with good IDE

peterromfeld06:06:05

you dont know, if/when you leave that next people wanna move forward

peterromfeld06:06:20

you always develop legacy of tomorrow

peterromfeld06:06:34

and needto make it easy to throw away

peterromfeld06:06:30

sry im a bit drunk and too talk active 😉

peterromfeld06:06:11

anyways if you have inputs, just put it into issues 🙂 i welcome every input

mtgred11:06:47

Is there any graphical data modeling tool that generates Datomic schema?

danielstockton12:06:15

why can some attributes be looked up and not others? (d/attribute (d/db conn) :db/add) => nil (d/attribute (d/db conn) :db/unique) => 42

stuartsierra13:06:35

@danielstockton: :db/add is not an attribute. It is a special form in transaction data.

danielstockton13:06:07

it looks like it is an attribute, at least there is an entity with this ident: 1 ([:db/ident :db/add] [:db/doc "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions and retractions of facts, e.g. [:db/add fred :age 42]."]),

danielstockton13:06:19

with entityid 1

danielstockton13:06:06

i also wonder what 'installing' an attribute really does and why it isn't allowed to just use anything with a :db/ident as an attribute

danielstockton13:06:24

(d/entid (d/db conn) :db/add) => 1 there seems to be something special that makes something an attribute but i don't really see what that is

jannis13:06:49

Are there any decent examples on querying tx data out there?

marshall13:06:40

This example also queries for the transaction entity to identify a user who changed a value: https://github.com/Datomic/day-of-datomic/blob/master/tutorial/provenance.clj

jannis14:06:29

Where you have the db after and the datoms from the transaction and want to query the db starting with one or more matching datoms in the tx data.

jannis14:06:23

Unfortunately, with the input datoms you don't seem to be able to use [?e :some/attr ?v2] to match attributes. You have to do something like [?a :db/ident :some/attr] to match the attribute separately. I guess I don't understand why that is yet.

marshall14:06:20

@jannis: if you look at the datoms that are returned, they are things like:

[#datom[13194139534313 50 #inst "2016-06-09T14:13:57.997-00:00" 13194139534313 true] 
where the attribute is a number (50 in this case) That is the entity ID of the attribute definition in the schema

jannis14:06:26

I suppose the reason is that the ?as are IDs of attribute entities in the db partition and not keywords.

marshall14:06:31

yep, there you go

jannis14:06:21

I wonder what the most elegant way to query just a particular transaction is if one already has the tx-data available, except in a slightly less useful form.

marshall14:06:23

so providing the database as a source allows you to look up the entity ID of the

?a
and resolve it to the
:db/ident
keyword

marshall14:06:06

You might want to look at the Log API if you’re looking for info about a specific transaction: http://docs.datomic.com/log.html

jannis14:06:25

Well, I already have the info by receiving the transaction data from the log (via onyx-datomic).

jannis14:06:09

When I do [?a :db/ident :command/data] then ?v will give me the ID of the entity that the :command/data attribute points to in the transaction. Now my only remaining problem is how to query information about that entity. [?v :data/some-attr ?v2] should give me - in ?v2 the value of :data/some-attr in the entity ?v, right?

jannis14:06:09

I'm not an experienced user of datomic terminology, so I hope this description isn't too weird.

marshall14:06:09

Yes I believe that’s right

marshall14:06:23

assuming I understand your question

jannis14:06:53

And that's what isn't working. It's almost as if it can't match the input datoms and the db in the same rule.

marshall14:06:53

How are you binding ?v

marshall14:06:11

to the entity ID you’re interested in

jannis14:06:22

?v comes from the tx datoms, just like in the example on http://docs.datomic.com/transactions.html#sec-5-3

marshall14:06:40

ah, ok. Then in that case it would only be an entity ID if the added datom was a reference

marshall14:06:10

so, in the example datom I posted above - the ?v there would be the inst

jannis14:06:56

QUERY [:find ?v :in $ [[?e ?a ?v ?tx ?added]] :where [?a :db/ident :command/data]]
> #object[java.util.HashSet 0x2071c184 [[285873023223181]]]

jannis14:06:45

So ?v is the entity ID of the entity refered to by the matching [?e :command/data ?v] datom in my tx data.

marshall14:06:45

Right, in that case you’re getting all the values of ?v that are passed in as input

jannis14:06:59

Mhm. So far so good. 🙂

jannis14:06:44

Now what I really want though is query an attribute of that entity, e.g. by adding a rule like [?v :data/some-attr ?v2] and than return ?v2.

danielstockton14:06:12

@jannis to your earlier question, you should be able to do [?e :some/attr ?v2] to match attributes but it doesn't bind it to anything. If you want to bind it, you need the ?a symbol and it gets bound to the entityid of the attribute so you then need to join that with :db/ident to the actual keyword value

jannis14:06:25

QUERY [:find ?v2 :in $ [[?e ?a ?v ?tx ?added]] :where [?a :db/ident :command/data] [?v :project.create/project ?v2]]
> #object[java.util.HashSet 0x7008a6b8 []]

jannis14:06:59

vs. the following query against the db directly (which works):

boot.user => (datomic.api/q '[:find ?v2 :in $ :where [?c :command/data ?d] [?d :project.create/project ?v2]] (datomic.api/db (:conn (:datomic system))))
#{[1]}

marshall14:06:24

try replacing [?v :project.create/project ?v2] with [?v ?a ?v2]

marshall14:06:40

oh, hang on, you’ve got two levels of reference here

marshall14:06:03

so, the same thing is happening with the project.create/project

marshall14:06:10

as with the command/data

marshall14:06:35

you need to resolve the keyword attribute ID to an entity ID if you’re going to query it that way

marshall14:06:20

alternatively, and possibly more simply, I’d do two separate queries

marshall14:06:49

since all query happens in process in Datomic there’s not the traditional client/server type of need to jam all the logic into a single query

marshall14:06:23

you can run the first query to get the entity IDs of the intermediate entites, then a second query to populate the values from those

jannis14:06:39

True. Run the first against the datoms and the second only against the db.

marshall14:06:02

especially if it’s data that was just transacted, that will all be in the peer’s local cache

jannis14:06:02

However, since it's such a simple chain of matching rules with only two levels of reference, this should easily be doable in a single query I'm thinking.

jannis14:06:14

Ah, that's certainly good to know 🙂

jannis15:06:46

Odd. A separate query against db-after, after extracing the entity ID of the first reference, fails:

VALUE FOR ?d 285873023223262
QUERY [:find (pull ?e [*]) . :in $ ?d :where [?d :project.create/project ?e]]
> nil
But running the query against the latest db of the connection separately works:
boot.user => (datomic.api/q '[:find (pull ?e [*]) . :in $ ?d :where [?d :project.create/project ?e]] (datomic.api/db (:conn (:datomic system))) 285873023223262)
{:db/id 1, :db/ident :db/add, :db/doc "Primitive assertion. All transactions eventually reduce to a collection of primitive assertions and retractions of facts, e.g. [:db/add fred :age 42].", :project/name "New project 1"}

jannis15:06:14

The onyx-datomic inject-db-calls lifecycle helper must be injecting an old version of the db into the function that runs the first query (basis-t = 1095 vs. 1503 in the second case).

jannis15:06:11

No wonder my query over db + tx datoms couldn't resolve references any further - they are probably missing from that old db.

jannis15:06:06

There we go. Using inject-conn-calls instead and then using the latest db for the queries makes it all work.

marshall15:06:59

Ah. well, that makes a lot more sense 🙂

jannis15:06:18

@marshall: I still can't get the single query to work but with the two separate queries you recommended it works.

jannis15:06:27

That'll do for now, thanks 🙂

marshall16:06:16

@jannis: I usually prefer the single query approach, but you could also do a nested query if necessary: https://groups.google.com/d/msg/datomic/5849yVrza2M/31--4xcdxOMJ

achan16:06:22

not sure if this is the place to ask for help but i am having some trouble with connecting to transactor from a remote peer

hans17:06:57

it can happen when you're specifying a host name in the transactor.properties file which is not resolvable by clients.

hans17:06:03

or if you use "localhost" there.

achan18:06:23

i can ssh by the ip address. Can i put the ip address in the transactor.properties?

marshall18:06:09

@achan: Yes, the hostname value in the properties file should be something that the peers can resolve to the transactor machine.

achan19:06:23

this is how i tried to connect:

achan19:06:28

(def uri "datomic:<sql://merckx?jdbc:postgresql://gmahg0ppav8v6d.cbv0gfpvcvqy.us-west-2.rds.amazonaws.com:5432/datomic?user=datomic&password=datomic0%22|sql://merckx?jdbc:postgresql://gmahg0ppav8v6d.cbv0gfpvcvqy.us-west-2.rds.amazonaws.com:5432/datomic?user=datomic&password=datomic0">)

achan19:06:33

(d/create-database uri)

achan19:06:48

got error: HornetQNotConnectedException HQ119007: Cannot connect to server(s). Tried with all available servers. org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:906)

achan19:06:01

i think i am missing something

achan19:06:41

how my peer know where the transactor is when in my uri for datomic specifies only RDS endpoint?

marshall19:06:49

The peer reads the transactor location from storage. http://docs.datomic.com/deployment.html#getting-connected

marshall19:06:49

Do you have a transactor up and running against that URI and does the machine you’re attempting to connect from have proper AWS credentials to access that RDS table?

marshall19:06:01

And to access the transactor machine

achan19:06:37

yes the transactor is up and running against that URI

marshall19:06:03

and, as @hans mentioned, the address of the transactor needs to be correct in the transactor properties file.

achan19:06:08

i tested with psql command to connect to RDS from my machine and passed

marshall19:06:24

Then, yes, it’s most likely the host value in the properties file

achan19:06:06

my properties file looks like this:

achan19:06:07

protocol=sql host=ip-10-16-25-24 port=4334

achan19:06:02

One thing is after i built the EC2. It couldn't resolve the hostname. So i had to add "ip-10-16-25-24" to /etc/hosts file

achan19:06:21

then Peer from the same EC2 can connected to it

marshall19:06:32

host needs to be something that your remote peer can use to connect to the transactor

achan19:06:18

i can connect to the EC2 from my machine.

achan19:06:24

4334 port is open

marshall19:06:34

also, if you’re using the provided CF template and you’re trying to connect from a non-EC2 instance, you need to set

aws-ingress-cidrs=0.0.0.0/0

achan19:06:12

no i didn't build it with CF template because of some permission error

achan19:06:29

so i built it by unzipping the file.

hans19:06:48

@achan: can you "telnet ip-10-16-25-24 4334" from your client machine?

achan19:06:09

ummm....no

marshall19:06:37

then you need to set the value of host to something you can reach from the client machine

marshall19:06:44

or add it to your /etc/hosts

achan19:06:46

is there a way for me to set aws-ingress-cidrs=0.0.0.0/0 without cloudformation?

achan19:06:32

ok, i will try that out. thank you so much.

marshall19:06:13

@achan: you should be able to set inbound rules from the EC2 instance security group settings: http://docs.datomic.com/aws.html#aws-management-console

achan19:06:18

great. that is what i need! thanks