Fork me on GitHub

@bkamphaus: could you paste in an example map … I can’t get it working


this is what I have


conn-map {:protocol :sql
                  :db-name "customer"
                  :sql-driver-params "ssl=true;sslfactory=org.postgresql.ssl.NonValidatingFactory"
                  :jdbc-url (env :jdbc-database-url) }
        conn (d/connect conn-map)
        db (d/db conn)


the JDBC URL is provided by Heroku


Exception in thread "main" java.lang.IllegalArgumentException: :db.error/invalid-sql-connection Must supply jdbc url in uri, or DataSource or Callable<Connection> in protocolObject arg to Peer.connect, compiling:(web.clj:57:9)


so I am a little lost how I should form the data


Quick question will a datomic console count towards the peer process?


i believe it does


Does the speed of a cut querye (i.e. [:find ?e . :where …]) depend on the number of possible results


;; there are 1366 of these
user> (time (d/q '[:find ?e . :where [?e :vulnerability/title]] db))
"Elapsed time: 1.272812 msecs"

;; there are 676457 of these
user> (time (d/q '[:find ?e . :where [?e :artifact-version/number]] db))
"Elapsed time: 461.450483 msecs”


I understand the problems with using time in a repl, I get similar results using criterium


also both of those attributes have indexing turned on


@max: As a general rule, the time cost of a query is proportional to the number of Datoms matched by each clause.


The :find ?e . syntax is just a convenience for when you only expect one result.


Is datomic.db.DbId an implementation detail or can I rely on tempid always returning an instance of DbId?


I ask because it isn't in Datomic's javadoc

Ben Kamphaus20:10:32

Not sure I follow. tempid returns a tempid, which is different from an entity id. A temp id has to be resolved to the actual id assigned in the db with resolve-tempid (in clojure) -

Ben Kamphaus20:10:40

Oh you mean type

Alex Miller (Clojure team)20:10:44

I don't think you should consider that guaranteed

Ben Kamphaus20:10:46

instance of, yeah.


I want to be able to test if something is a tempid and then if that tempid is in the reserved space of -1..-1000000


Which is (and (instance? datomic.db.DbId x) (<= -1 (:idx x) -1000000))), assuming that part of the API can be relied on.


@alexmiller: If this were to change, would it be documented in the release notes?

Alex Miller (Clojure team)20:10:40

I'd defer to @bkamphaus for any official answer but my personal opinion is that you are relying on implementation details that may change without warning (and may or may not be documented when they do)


Hmm, okay. Thanks @alexmiller. @bkamphaus, what do you think? Is there an official way to test if something is a tempid and to retrieve its index?

Ben Kamphaus20:10:52

@domkm I’ll look into it. The standing answer for the mean time (and in general the default answer) consistent with @alexmiller ’s gut take is to consider undocumented specifics like types or members to be implementation details and not promised.

Ben Kamphaus20:10:04

What’s the goal of the test?

Ben Kamphaus20:10:17

Note that you can use with ( ) to determine what will happen when you apply tx-data to a db without having to transact durably, which includes using resolve-tempid with :tempids in the returned map.


@bkamphaus: I am writing a function that composes mutators (transactor functions) together using d/with and d/resolve-tempid. It's based on this ( function. There are problems with aliasing tempids if any mutators create sub-mutations with reserved tempids but no problems with unreserved ones since those are unique (right?). I want to be able to test if a tempid is in the reserved range in order to throw an error instead of transacting invalid data.

Ben Kamphaus21:10:57

@domkm still parsing this. What’s the difference in unique behavior you’re discussing? Potential for collision with transactor functions also generating tempids? All unique tempids in a single transaction should resolve to unique entity ids, except in the event of upserts (dup values for unique identity attrs).


@bkamphaus: In the linked function, if the same reserved negative tempid is used in multiple composed txs, it will result in aliased data. In other words, you can end up with an entity that is an invalid collection of facts (like an entity with both a :user/name and an :org/name). In terms of unique tempids, could using with in a transaction introduce duplicate tempids?


hi @bkamphaus … update on the connection saga


system properties set...


(System/setProperty "datomic.sqlUser" user-value)
    (System/setProperty "datomic.sqlPassword" password-value)
    (System/setProperty "datomic.sqlDriverParams"


checked it before connection and it’s in the System properties map


then I try to create the db


conn-map {:protocol          :sql
                  :db-name           "datomic"
                  :sql-driver-params "ssl=true;sslfactory=org.postgresql.ssl.NonValidatingFactory"
                  :sql-url           simple-jdbc}
        created! (d/create-database conn-map)


but unfortunately the connection does not play out


Exception in thread "main" java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: The server requested password-based authentication, but no password was provided., compiling:(web.clj:65:9)


so the good news is that the postgres server is being reached and SSL is working


odd news is that the password is somehow not being sent along

Ben Kamphaus21:10:04

@raymcdermott: I don’t believe the peer looks for user and pw system properties (those are documented for txor only), it will require that username and password be set in the URI params or e.g. on a data source provided in the map.


ah, ok I will give that a try

Ben Kamphaus21:10:41

with data source, last time I tested and built an example was in Java, but if you stick with the map route maybe this will be helpful.

Ben Kamphaus21:10:33

Java example config (parameters elided, but described in comments).

Object driver = java.sql.DriverManager.getDriver(sqlUrl);
Object driverClass = driver.getClass();
String name = driverClass.getName();

setURL() //JDBC portion of Datomic URI
setValidationQuery() // Use provider default or one you use in transactor properties file
setValidationInterval() // this should match the transactor properties file heartbeat-interval-msec
setTestOnBorrow() // set to true
.setDriverClassName() // ^ from code block above
.setUsername // username
.setPassword // password
.setConnectionProperties // anything else, i.e. what would be in transactor properties optional sql params.

Ben Kamphaus21:10:27

For my local toy config, though, I just URL as string arg to connect (though no SSL), e.g. datomic:


thanks - I’ll play around with that


what’s not obvious is the correct name of the parameters in the map


after setting username / password properties in the map … no joy


after adding the username to the driver params, message changed


it now seems to be picking up the from the system properties


in the params I am using the ‘;’ separator convention … will try with ?&


and now I’m back to SSL off


Getting desperate, so I will try to set the system property

Ben Kamphaus21:10:14

@raymcdermott: have you tried everything hard-coded in one URI as string?


I’ll try that too


another question… the transactor is on IP1 and the postgres is on IP2


both not localhost

Ben Kamphaus22:10:04

the peer needs to connect to postgres IP


how do I encode that on the URL?

Ben Kamphaus22:10:23

it will look up transactor endpoint from storage (transactor writes its location to storage as part of heartbeat)


I’m sure somebody told me that here the other day - I must have dropped a few packets


ha! I think this may have been working before… I think the error is coming from the data in postgres


Thinking about it, I think I need to update the hostname in the transactor properties file so that it no longer uses the default local host


I will need to mess around with a few files / deployments and get back to you

Ben Kamphaus22:10:32

@raymcdermott: yeah, the transactor properties file hostname needs to be the address the peers can reach the transactor at.


I get it now… fingers crossed - now that I know what connections are going back and forth - I should get it started up


actually this will be a nice thing


Heroku have a VPC in beta to which I have access


this will demonstrate it working in the VPC


I will write a small blog once it’s all done (with appropriate attributions for support!!)

Ben Kamphaus22:10:31

@raymcdermott: good luck simple_smile Feel free to drop any additional questions here as needed.


don’t worry, will do - great help so far - thanks a lot


@bkamphaus: Could you clarify for me under what circumstances multiple invocations of (tempid :db.part/user) could possibly return duplicate tempids? There is no guarantee of uniqueness between transactor invocations and peer invocations, right? Inside a transactor function, can using tempid inside and outside of with potentially cause duplicates?


Note that I am asking about the unary tempid function that returns a random-ish tempid. I understand the binary tempid function.

Ben Kamphaus23:10:04

@domkm sorry, my comment above was the opposite side of the uniqueness constraint. If you specify (tempid :db.part/user) on multiple entities making assertions e.g. in a map with the same value for an attribute set as unique identity, then those will resolve as upserts on the same entity id.

Ben Kamphaus23:10:53

Re: the collision issue re: user reserved (which you saw you understand, but just for clarification/verifying), I mean the issue described on group here:!searchin/datomic/transactor$20function$20tempid/datomic/xRWXX0coMcI/wrBY-YMzbE8J


@bkamphaus: all is hooked up now… next to put some actual data and run a few queries


don’t worry - I’ll keep it simple!


I can sleep tonight simple_smile


@bkamphaus: Thanks for the link. Magnar Sveen's last response answers my question about whether unreserved tempid conflicts can occur between a peer and the transactor and, as I suspected, they can. I don't think it answers the question about conflicts in a transactor function that uses with. I was thinking that with might cause the uniqueness-within-a-transaction guarantee of tempid to be violated because with is sort of like a transact (probably shares a lot of the same code). I haven't been able to clearly describe this question so let me go put together an example.