Fork me on GitHub
#datomic
<
2015-10-27
>
raymcdermott00:10:02

@bkamphaus: could you paste in an example map … I can’t get it working

raymcdermott00:10:32

this is what I have

raymcdermott00:10:38

conn-map {:protocol :sql
                  :db-name "customer"
                  :sql-driver-params "ssl=true;sslfactory=org.postgresql.ssl.NonValidatingFactory"
                  :jdbc-url (env :jdbc-database-url) }
        conn (d/connect conn-map)
        db (d/db conn)

raymcdermott00:10:07

the JDBC URL is provided by Heroku

raymcdermott00:10:37

Exception in thread "main" java.lang.IllegalArgumentException: :db.error/invalid-sql-connection Must supply jdbc url in uri, or DataSource or Callable<Connection> in protocolObject arg to Peer.connect, compiling:(web.clj:57:9)

raymcdermott00:10:01

so I am a little lost how I should form the data

mitchelkuijpers10:10:10

Quick question will a datomic console count towards the peer process?

robert-stuttaford10:10:39

i believe it does

max17:10:11

Does the speed of a cut querye (i.e. [:find ?e . :where …]) depend on the number of possible results

max17:10:37

;; there are 1366 of these
user> (time (d/q '[:find ?e . :where [?e :vulnerability/title]] db))
"Elapsed time: 1.272812 msecs"
17592186931182

;; there are 676457 of these
user> (time (d/q '[:find ?e . :where [?e :artifact-version/number]] db))
"Elapsed time: 461.450483 msecs”
17592187088881

max17:10:38

I understand the problems with using time in a repl, I get similar results using criterium

max17:10:50

also both of those attributes have indexing turned on

stuartsierra17:10:27

@max: As a general rule, the time cost of a query is proportional to the number of Datoms matched by each clause.

stuartsierra17:10:21

The :find ?e . syntax is just a convenience for when you only expect one result.

domkm20:10:56

Is datomic.db.DbId an implementation detail or can I rely on tempid always returning an instance of DbId?

domkm20:10:50

I ask because it isn't in Datomic's javadoc

bkamphaus20:10:32

Not sure I follow. tempid returns a tempid, which is different from an entity id. A temp id has to be resolved to the actual id assigned in the db with resolve-tempid (in clojure) - http://docs.datomic.com/clojure/index.html#datomic.api/resolve-tempid

bkamphaus20:10:40

Oh you mean type

alexmiller20:10:44

I don't think you should consider that guaranteed

bkamphaus20:10:46

instance of, yeah.

domkm20:10:20

I want to be able to test if something is a tempid and then if that tempid is in the reserved space of -1..-1000000

domkm20:10:20

Which is (and (instance? datomic.db.DbId x) (<= -1 (:idx x) -1000000))), assuming that part of the API can be relied on.

domkm20:10:36

@alexmiller: If this were to change, would it be documented in the release notes?

alexmiller20:10:40

I'd defer to @bkamphaus for any official answer but my personal opinion is that you are relying on implementation details that may change without warning (and may or may not be documented when they do)

domkm20:10:00

Hmm, okay. Thanks @alexmiller. @bkamphaus, what do you think? Is there an official way to test if something is a tempid and to retrieve its index?

bkamphaus20:10:52

@domkm I’ll look into it. The standing answer for the mean time (and in general the default answer) consistent with @alexmiller ’s gut take is to consider undocumented specifics like types or members to be implementation details and not promised.

bkamphaus20:10:04

What’s the goal of the test?

bkamphaus20:10:17

Note that you can use with ( http://docs.datomic.com/clojure/#datomic.api/with ) to determine what will happen when you apply tx-data to a db without having to transact durably, which includes using resolve-tempid with :tempids in the returned map.

domkm20:10:43

@bkamphaus: I am writing a function that composes mutators (transactor functions) together using d/with and d/resolve-tempid. It's based on this (https://github.com/webnf/webnf/blob/master/datomic%2Fsrc%2Fwebnf%2Fdatomic.clj#L181-L244) function. There are problems with aliasing tempids if any mutators create sub-mutations with reserved tempids but no problems with unreserved ones since those are unique (right?). I want to be able to test if a tempid is in the reserved range in order to throw an error instead of transacting invalid data.

bkamphaus21:10:57

@domkm still parsing this. What’s the difference in unique behavior you’re discussing? Potential for collision with transactor functions also generating tempids? All unique tempids in a single transaction should resolve to unique entity ids, except in the event of upserts (dup values for unique identity attrs).

domkm21:10:59

@bkamphaus: In the linked function, if the same reserved negative tempid is used in multiple composed txs, it will result in aliased data. In other words, you can end up with an entity that is an invalid collection of facts (like an entity with both a :user/name and an :org/name). In terms of unique tempids, could using with in a transaction introduce duplicate tempids?

raymcdermott21:10:05

hi @bkamphaus … update on the connection saga

raymcdermott21:10:16

system properties set...

raymcdermott21:10:21

(System/setProperty "datomic.sqlUser" user-value)
    (System/setProperty "datomic.sqlPassword" password-value)
    (System/setProperty "datomic.sqlDriverParams"
                        "ssl=true;sslfactory=org.postgresql.ssl.NonValidatingFactory”)

raymcdermott21:10:43

checked it before connection and it’s in the System properties map

raymcdermott21:10:50

then I try to create the db

raymcdermott21:10:01

conn-map {:protocol          :sql
                  :db-name           "datomic"
                  :sql-driver-params "ssl=true;sslfactory=org.postgresql.ssl.NonValidatingFactory"
                  :sql-url           simple-jdbc}
        created! (d/create-database conn-map)

raymcdermott21:10:48

but unfortunately the connection does not play out

raymcdermott21:10:53

Exception in thread "main" java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: The server requested password-based authentication, but no password was provided., compiling:(web.clj:65:9)

raymcdermott21:10:19

so the good news is that the postgres server is being reached and SSL is working

raymcdermott21:10:34

odd news is that the password is somehow not being sent along

bkamphaus21:10:04

@raymcdermott: I don’t believe the peer looks for user and pw system properties (those are documented for txor only), it will require that username and password be set in the URI params or e.g. on a data source provided in the map.

raymcdermott21:10:44

ah, ok I will give that a try

bkamphaus21:10:41

with data source, last time I tested and built an example was in Java, but if you stick with the map route maybe this will be helpful.

bkamphaus21:10:33

Java example config (parameters elided, but described in comments).

Object driver = java.sql.DriverManager.getDriver(sqlUrl);
Object driverClass = driver.getClass();
String name = driverClass.getName();

setURL() //JDBC portion of Datomic URI
setValidationQuery() // Use provider default or one you use in transactor properties file
setValidationInterval() // this should match the transactor properties file heartbeat-interval-msec
setTestOnBorrow() // set to true
setInitialSize(2)
.setDriverClassName() // ^ from code block above
.setUsername // username
.setPassword // password
.setConnectionProperties // anything else, i.e. what would be in transactor properties optional sql params.

bkamphaus21:10:27

For my local toy config, though, I just URL as string arg to connect (though no SSL), e.g. datomic:

raymcdermott21:10:51

thanks - I’ll play around with that

raymcdermott21:10:51

what’s not obvious is the correct name of the parameters in the map

raymcdermott21:10:59

after setting username / password properties in the map … no joy

raymcdermott21:10:13

after adding the username to the driver params, message changed

raymcdermott21:10:34

it now seems to be picking up the user.name from the system properties

raymcdermott21:10:31

in the params I am using the ‘;’ separator convention … will try with ?&

raymcdermott21:10:12

and now I’m back to SSL off

raymcdermott21:10:08

Getting desperate, so I will try to set the user.name system property

bkamphaus21:10:14

@raymcdermott: have you tried everything hard-coded in one URI as string?

raymcdermott21:10:32

I’ll try that too

bkamphaus21:10:42

i.e. datomic:

raymcdermott22:10:36

another question… the transactor is on IP1 and the postgres is on IP2

raymcdermott22:10:50

both not localhost

bkamphaus22:10:04

the peer needs to connect to postgres IP

raymcdermott22:10:05

how do I encode that on the URL?

bkamphaus22:10:23

it will look up transactor endpoint from storage (transactor writes its location to storage as part of heartbeat)

raymcdermott22:10:11

I’m sure somebody told me that here the other day - I must have dropped a few packets

raymcdermott22:10:51

ha! I think this may have been working before… I think the error is coming from the data in postgres

raymcdermott22:10:12

Thinking about it, I think I need to update the hostname in the transactor properties file so that it no longer uses the default local host

raymcdermott22:10:39

I will need to mess around with a few files / deployments and get back to you

bkamphaus22:10:32

@raymcdermott: yeah, the transactor properties file hostname needs to be the address the peers can reach the transactor at.

raymcdermott22:10:25

I get it now… fingers crossed - now that I know what connections are going back and forth - I should get it started up

raymcdermott22:10:35

actually this will be a nice thing

raymcdermott22:10:57

Heroku have a VPC in beta to which I have access

raymcdermott22:10:13

this will demonstrate it working in the VPC

raymcdermott22:10:48

I will write a small blog once it’s all done (with appropriate attributions for support!!)

bkamphaus22:10:31

@raymcdermott: good luck simple_smile Feel free to drop any additional questions here as needed.

raymcdermott22:10:20

don’t worry, will do - great help so far - thanks a lot

domkm22:10:01

@bkamphaus: Could you clarify for me under what circumstances multiple invocations of (tempid :db.part/user) could possibly return duplicate tempids? There is no guarantee of uniqueness between transactor invocations and peer invocations, right? Inside a transactor function, can using tempid inside and outside of with potentially cause duplicates?

domkm22:10:07

Note that I am asking about the unary tempid function that returns a random-ish tempid. I understand the binary tempid function.

bkamphaus23:10:04

@domkm sorry, my comment above was the opposite side of the uniqueness constraint. If you specify (tempid :db.part/user) on multiple entities making assertions e.g. in a map with the same value for an attribute set as unique identity, then those will resolve as upserts on the same entity id.

bkamphaus23:10:53

Re: the collision issue re: user reserved (which you saw you understand, but just for clarification/verifying), I mean the issue described on group here: https://groups.google.com/forum/#!searchin/datomic/transactor$20function$20tempid/datomic/xRWXX0coMcI/wrBY-YMzbE8J

raymcdermott23:10:51

@bkamphaus: all is hooked up now… next to put some actual data and run a few queries

raymcdermott23:10:05

don’t worry - I’ll keep it simple!

raymcdermott23:10:21

I can sleep tonight simple_smile

domkm23:10:20

@bkamphaus: Thanks for the link. Magnar Sveen's last response answers my question about whether unreserved tempid conflicts can occur between a peer and the transactor and, as I suspected, they can. I don't think it answers the question about conflicts in a transactor function that uses with. I was thinking that with might cause the uniqueness-within-a-transaction guarantee of tempid to be violated because with is sort of like a transact (probably shares a lot of the same code). I haven't been able to clearly describe this question so let me go put together an example.