Question for the Cognitects out there: I noticed that there is a :db.install/function ident. Its docstring says that it is used to install functions but it is not documented on http://docs.datomic.com. Could you shed some light on :db.install/function?

Also, any hint about when we can use :db.install/valueType? 😉

another connection question for the Datomics

why does the client need to provide the JDBC url when making a connection to the transactor?

this seems like a leaky abstraction

of course the fact that the client needs to specify the storage at all is leaky

or am I missing somthing?

apart from an e in something

@raymcdermott The client get's the transactor url from the database, this means that if you for example start a new transactor or one crashes and another on comes up it get's the new transactor url from the database. This also means that you data fetching keeps working when the transactor is down. It is actually pretty nifty

@raymcdermott: it rocks that peers talk to storage directly. transactor doesn’t have to be involved for reads at all. means scaling reads is independent of scaling writes.

also failover process uses storage; so peers know when failover is happening even while primary transactor is dead

@raymcdermott: You can suppress the printing of the login information by passing:

-Ddatomic.printConnectionInfo=false 

http://docs.datomic.com/system-properties.html

hey @marshall!

Mornin’ @robert-stuttaford . Well, I guess not for you

hah, no

so when’s the Next Big Thing coming for Datomic, @marshall? i just can’t imagine that you guys are ‘done’

Nope, definitely not done, but I don’t have any visibility into when/what is coming; at least not until it’s essentially release-ready

ok, cool good to know there’s still stuff cooking!

thanks @robert-stuttaford and @mitchelkuijpers

oh and you too @marshall !!

Hi. Does anyone have a solution for partial backups of a datomic db?

The use case: I'f like to extract a set of "fixtures" from an existing production db. With mysqldump (for example) I'd just exclude ceratain tables from the dump.

But a normal Datomic dump will typically include all facts (whereas I'd like to exclude, say, all events or all payment information).

hah – is that you, @bkamphaus ?

pesterhazy: we wrote custom code for this that exports to a transit file, zips it, and puts it on S3

that sounds like a good solution

takes a database and a configuration of attrs to export, and exports all the schema (whether in the config or not), enums, and data for those attrs

our prod db backup is 9gb. this exports the ‘control’ information which comes to 12mb gzipped

or 55mb unzipped. it’s super useful

lemme see if i can gist it

that would be PERFECT

I tried https://groups.google.com/d/msg/datomic/eQQnnqYl67Y/z-Ib60NCJAAJ but it doesn't seem to cope well with 100s of MB of data

@marshall: I noticed that password data does get leaked out when things go wrong (it dumps the properties) … maybe something more complete is needed?

@pesterhazy: not a simple cut and paste job, i’m afraid. i’ve put a note on my list to get it done this week. are you on twitter?

sure

same handle

would be useful for lots of users, I think

totally. i’d share it as a public gist and tweet it, and cc you

cool thanks!

👍 was super fun to write with transducers

everything is better with some transducers thrown in

they’re the sriracha of clojure

this code actually builds transducers dynamically based on the config, and memoizes them

for teh speedz

sriracha +1

(had to look that up)

@raymcdermott: taking over from @marshall here - if you can email me — bkamphaus @ cognitect — we definitely don’t want the creds to leak anywhere if they’re printing has been disabled by the command line arg. If you can pass along your transactor launch command, logs or console output that shows the leaked creds (redact/replace the actual creds), and the type of failure required to repro (or steps to take to induce that failure), I’ll investigate as a bug.

thanks @bkamphaus - I will double check the issue on this side first

@raymcdermott: cool, feel free to ping me here whenever you send it along - also would be helpful to have the txor properties file (again w/any creds or sensitive info redacted from it).

I struggling with Postgres / SSL on Heroku at the moment

hey, does anyone have any experience with this error? Our transactor is currently unresponsive:

2015-10-26 16:35:07.170 INFO  default    datomic.update - {:tid 133, :pid 1947, :message "Update failed"}
java.lang.NullPointerException: null
	at datomic.db$next_valid_inst.invoke(db.clj:2412) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.db.ProcessExpander.getData(db.clj:2468) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180$fn__10181.invoke(update.clj:246) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.Atom.swap(Atom.java:37) ~[clojure-1.6.0.jar:na]
	at clojure.core$swap_BANG_.invoke(core.clj:2232) ~[clojure-1.6.0.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180.invoke(update.clj:240) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177.invoke(update.clj:238) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173.invoke(update.clj:235) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor.doInvoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:139) [clojure-1.6.0.jar:na]
	at clojure.core$apply.invoke(core.clj:626) [clojure-1.6.0.jar:na]
	at datomic.update$background$proc__10093.invoke(update.clj:58) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]

we're seeing timeouts every time we try to issue a transaction. the above error comes from the transactor log. we suspect the transactor is basically paused because of that error.

@kbaribeau: two quick questions (1) does the problem persist across transactor restart (2) do you have the logs from when the error first started occurring?

(1) yes

(2) there's a bunch of instances of that exception in our logs. double checking

we've got the logs, for sure, there's just a lot of data to search through

there's :txid in these logs entries, can we use that to find out what was in the transaction that was failing?

If you can get first log (just grep or something) that has “Update failed” in it w/the accompanying “next_valid_inst” in stack trace, can you email it to me at bkamphaus @ cognitect ?

@kbaribeau have you backed up your database? Two pieces (1) back up right away to be safe, maybe to a new location in file store/s3 or w/e you back up to if your latest db backup does/did not show this issue, (2) verify backup will complete successfully

wait a minute, I may be misreading the error

@kbaribeau: what’s the most recent instant in the database? Is it possible a transaction was incorrectly written future dated? w/explicit db/txInstant as specified here http://docs.datomic.com/transactions.html#reified-transactions

hmm, there's a datomic api call to get that right?

basis-t?

I doubt we've ever set a T value explicitly. d/basis-t tells me our most recent T value is 4053248. should that show up in the log?

d/next-t seems to work. but I suppose that's different from datomic.db$next_valid_inst

Datoms about transaction entity from last successful transaction, replace safely-before-inst with date prior to issue occurring:

(let [log (d/log conn)
      safely-before-inst #inst "2015-10-01" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

if you annotate transactions with domain specific info, verify you’re ok sharing prior to pasting, or pm me w/tx datoms from last successful transaction.

^ @kbaribeau

thanks, running it

just need a minute, lost my repl 😞

(let [log (d/log @connection/conn)
      safely-before-inst #inst "2015-09-25T00:00:00.000-00:00" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

; (#datom[13194143586558 50 #inst "2015-10-23T21:30:09.200-00:00" 13194143586558 true])

(d/touch (d/entity (d/db @connection/conn) 13194143586558))

; {:db/id 13194143586558, :db/txInstant #inst "2015-10-23T21:30:09.200-00:00"}

^ not quite the most recent stuff

@kbaribeau: are you able to upgrade to latest? 0.9.5327 - on transactor

we've got more than one database

@kbaribeau: preferably upgrade both transactor and peer lib dep to latest, 0.9.5327 - we think there’s a possibility it stems from an issue corrected in that version.

we can update. what's in the latest release that's interesting? we're on 5206

oh ok

we'll update

does upgrading from 5206 to 5327 require that we also upgrade the peer? seeing some errors that might be indicating that

@kbaribeau: yeah, push dep upgrade for peer apps as well

new error?

#object[datomic.promise$settable_future$reify__6305 0x262beb24 {:status :failed, :val #error {
 :cause "java.lang.NullPointerException"
 :via
 [{:type java.util.concurrent.ExecutionException
   :message "java.lang.NullPointerException: java.lang.NullPointerException"
   :at [datomic.promise$throw_executionexception_if_throwable invoke "promise.clj" 10]}
  {:type java.lang.NullPointerException
   :message "java.lang.NullPointerException"
   :at [clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]}]
 :trace
 [[clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]
  [datomic.error$deserialize_exception invoke "error.clj" 135]
  [datomic.peer.Connection notify_error "peer.clj" 401]
  [datomic.connector$fn__8480 invoke "connector.clj" 169]
  [clojure.lang.MultiFn invoke "MultiFn.java" 233]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490$fn__8491 invoke "connector.clj" 194]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490 invoke "connector.clj" 189]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487 invoke "connector.clj" 187]
  [clojure.core$binding_conveyor_fn$fn__4444 invoke "core.clj" 1916]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 266]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}}]

@kbaribeau: this is after upgrading both?

yes

this is on transactor or peer on startup?

this is on a d/transact call

transactor and peer startup fine

@kbaribeau: is there an exception being reported on the transactor for that transaction attempt also?

Modelling question:

I have “user” entities and “club” entities. Users are members of one or more clubs, and clubs have one or more users (as members).

I could go with :user/clubs or :club/users.

But I want that “membership” to be typed, as in, “president” “secretary” “member” and so on.

So, I can't just associate a user with a club via a cardinality/many because that association itself needs to be annotated.

Do I create a “membership” entity which has refs from users to clubs along with a role enum (pres, veep, etc)?

That works, but feels kinda RDBMSish.

Or do you go with :club/presidents :club/veeps :club/members?

(Of course, what I want is to do a Pull API thing in a user and get a list of all the clubs s/he belongs to, and what the membership role is.)

@zentrope: It depends

Yeah. I don’t have a lack of ideas, just enough lack of experience to choose between them. ;)

For simple case, I would probably use multiple attributes, :club/president :club/veep :club/member … etc. as card/many and define a rule that says a user is one of these. Of course the Pull API use case with this is less straight forward. I might also try to define the president of relation from the user if a cardinality one constraint should apply to that direction. You could reify the relation, also, as you mention.

Is the Pull API use case you mention above the most common use case you envision, or just something you want to make sure is included?

I’m using the Pull API to make pulling a user’s data kind of document-like.

Get me the whole state for that user so I can blob it up to a UI and render it.

Doing the pulls in stages works, of course, but I’m hoping I’m not missing something given my past thinking in terms of using Postgres.

I guess it depends on how far your use of this membership idea. You could go so far as to make each “membership” a component entity of a person entity that refers to the class of membership and the organization, start date, end date, billed amount, etc.

Yes, that actually seems to be a reasonable approach, keeping it open ended.

Using the Pull API, how do I reference the membership?

[:user/id :user/name …] ?

[:user/id :user/name {:membership/_user [{:membership/role [:db/ident]}]}] seems to be getting me somewhere.

http://docs.datomic.com/pull.html#component-defaults — component defaults in pull documentation

I can get from user -> membership -> club but can’t add -> membership -> users ; to get a list of all the members of that club.

that would be assuming a structure like:

{:user/id
 :user/name 
 :user/membership
 [{:membership/org …
   :membership/role ...
   :membership/start .. }]}

Hopefully it’s just syntax.

Yes. That’s working for me.

What I want, though, is [user] -> [clubs] -> [members in each club] but I can’t seem to make the club to a membership lookup to find users. Hm.

[:user/id
   :user/name
   :user/email
   {:membership/_user
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       :membership/_user
       ]}]}]

so from :membership/club -> (club entity id) -> :membership/_club (all membership entities that point to club) -> :user/_membership (back to user ids)

That last :membership/_user doesn’t appear.

I can map this stuff through a transform that then pulls in, say, a list of all the members of a specific club via separate queries. I’m just wondering if the Pull API can do all that for me.

Should user have a :user/membership (isComponent) and club have a :club/membership (isComponent)?

Right now, I have just a standalone membership set of attributes for an entity, with :one for club and :one for user.

Probably differences of opinion on this — my quick mental mock up would say have :user/membership as isComponent of a user, points to an organization of which the user is a member.

I’m not sure what you mean be “standalone membership set of attributes for an entity” precisely.

(attr :membership/role :ref :one "")
(attr :membership/user :ref :one "")
(attr :membership/club :ref :one "")

Well, that’s just my dsl. I mean there’s no :user/memberships attribute of any type. Nor :club/membership.

Just like a SQL join table, is what I’m doing (and thus feels wrongish).

yeah, I would make membership entities that are components of users and have similar attributes. So no :membership/_user backref, but still role (points to an enum?), :club (points to a club org), and you can go from club org by back ref to all membership that apply to that club, and from those membership to their user parent entities.

Okay. I totally understand what you’re saying. That seems quite reasonable.

I’m much more interested in being able to walk from a user through to all the information they can see vs starting with a club.

How do you “go from club org by back ref to all membership” in a Pull API starting with a user?

I think :user/memberships :many shouldn’t be a component such that you can have a :club/memberships :many also pointing to the same membership entity as the :user/memberships attribute does.

@zentrope: why :club/membership also pointing to it?

Because otherwise I can get a list of the members of the club via a Pull API pattern.

I saw :user/memberships card many pointing to a membership entity which has :membership/club as a cardinality one

:membership/_club being the reverse ref you would follow from club entity to get all memberships

and you can use :user/_memberships for all users with a membership to that club

Hm.

but I haven’t thrown in the toy schema/data yet

my reasoning being that it makes sense to constrain a membership to only be for one club

so there’s a direction of the relationship to which you could apply a cardinality/one constraint (have to try to enforce that via some other logic if you model the ref in the other direction)

Right. I think it’s the Pull API pattern I might be having trouble with.

hey guys … I am trying to connect to Datomic that is connected to a PostgresSQL DB

they require SSL and have some recommendations

for properties

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club '[*]}]}]}])

I can’t get from :membership/_club to a list of users.

ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory

but I’m still getting SSL off

Caused by: java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "52.22.208.29", user "dyno", database "dd7fmhk85j9m9d&ssl=true", SSL off

is there another way to enable SSL on the Postgres connection?

I think I figured it out.

it seems like it’s misinterpreting any additional parameters on the JDBC URL

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club
        [{:user/_memberships
          [:user/name {:user/memberships [{:membership/role [:db/ident]}]}]}]}]}]}])

Tag soup, but it works.

there is a sql-driver-params in the JDBC property file … I will give that a try

nope that did not work nor sqlDriverParams

@zentrope: it looks like you figured it out. I did end up doing a gist of a quick example of what I was talking about: https://gist.github.com/benkamphaus/4f991901b2fe8d00e20b

@raymcdermott: you’re following the Heroku PostgreSQL instructions here? http://docs.datomic.com/storage.html#sql-database

@bkamphaus: Thanks!

yes - the transactor is connecting OK

but the Peer is failing

@raymcdermott is it possible that there are issues with creds in the connection URL that need to be percent encoded?

Another option to try would be to use the map form for the URI described here http://docs.datomic.com/clojure/#datomic.api/connect

I have set the creds as system properties as mentuined in the docs

oooh so datomic:sql://{db-name}?{jdbc-url} is two maps?

I have just made them strings

Connects to the specified database, returing a Connection.
URI syntax ({} indicate place holders to fill in, [] indicate optional):

ah OK, read further now

still not clear how I set SSL

but maybe I can use the same terms as in the property file?

@bkamphaus: Another version of the many-to-many thing: https://gist.github.com/vaughnd/3705861