Fork me on GitHub
#datomic
<
2015-10-26
>
domkm06:10:12

Question for the Cognitects out there: I noticed that there is a :db.install/function ident. Its docstring says that it is used to install functions but it is not documented on http://docs.datomic.com. Could you shed some light on :db.install/function?

domkm06:10:20

Also, any hint about when we can use :db.install/valueType? 😉

raymcdermott12:10:43

another connection question for the Datomics

raymcdermott12:10:08

why does the client need to provide the JDBC url when making a connection to the transactor?

raymcdermott12:10:20

this seems like a leaky abstraction

raymcdermott12:10:59

of course the fact that the client needs to specify the storage at all is leaky

raymcdermott12:10:09

or am I missing somthing?

raymcdermott12:10:26

apart from an e in something

mitchelkuijpers12:10:34

@raymcdermott The client get's the transactor url from the database, this means that if you for example start a new transactor or one crashes and another on comes up it get's the new transactor url from the database. This also means that you data fetching keeps working when the transactor is down. It is actually pretty nifty

robert-stuttaford12:10:32

@raymcdermott: it rocks that peers talk to storage directly. transactor doesn’t have to be involved for reads at all. means scaling reads is independent of scaling writes.

robert-stuttaford13:10:17

also failover process uses storage; so peers know when failover is happening even while primary transactor is dead

marshall13:10:01

@raymcdermott: You can suppress the printing of the login information by passing:

-Ddatomic.printConnectionInfo=false 
as a parameter to the transactor launch

marshall13:10:07

Mornin’ @robert-stuttaford . Well, I guess not for you simple_smile

robert-stuttaford13:10:35

so when’s the Next Big Thing coming for Datomic, @marshall? i just can’t imagine that you guys are ‘done’ simple_smile

marshall13:10:57

Nope, definitely not done, but I don’t have any visibility into when/what is coming; at least not until it’s essentially release-ready

robert-stuttaford13:10:22

ok, cool simple_smile good to know there’s still stuff cooking!

pesterhazy13:10:11

Hi. Does anyone have a solution for partial backups of a datomic db?

pesterhazy13:10:15

The use case: I'f like to extract a set of "fixtures" from an existing production db. With mysqldump (for example) I'd just exclude ceratain tables from the dump.

pesterhazy13:10:44

But a normal Datomic dump will typically include all facts (whereas I'd like to exclude, say, all events or all payment information).

robert-stuttaford13:10:14

pesterhazy: we wrote custom code for this that exports to a transit file, zips it, and puts it on S3

pesterhazy13:10:48

that sounds like a good solution

robert-stuttaford13:10:55

takes a database and a configuration of attrs to export, and exports all the schema (whether in the config or not), enums, and data for those attrs

robert-stuttaford13:10:16

our prod db backup is 9gb. this exports the ‘control’ information which comes to 12mb gzipped

robert-stuttaford13:10:25

or 55mb unzipped. it’s super useful simple_smile

robert-stuttaford13:10:30

lemme see if i can gist it

pesterhazy13:10:39

that would be PERFECT simple_smile

pesterhazy13:10:59

I tried https://groups.google.com/d/msg/datomic/eQQnnqYl67Y/z-Ib60NCJAAJ but it doesn't seem to cope well with 100s of MB of data

raymcdermott13:10:49

@marshall: I noticed that password data does get leaked out when things go wrong (it dumps the properties) … maybe something more complete is needed?

robert-stuttaford13:10:05

@pesterhazy: not a simple cut and paste job, i’m afraid. i’ve put a note on my list to get it done this week. are you on twitter?

pesterhazy13:10:54

would be useful for lots of users, I think

robert-stuttaford13:10:16

totally. i’d share it as a public gist and tweet it, and cc you

pesterhazy13:10:24

cool thanks!

robert-stuttaford13:10:53

👍 was super fun to write with transducers

bkamphaus13:10:58

everything is better with some transducers thrown in

bkamphaus13:10:22

they’re the sriracha of clojure

robert-stuttaford13:10:49

this code actually builds transducers dynamically based on the config, and memoizes them

robert-stuttaford13:10:31

(had to look that up)

bkamphaus13:10:36

@raymcdermott: taking over from @marshall here - if you can email me — bkamphaus @ cognitect — we definitely don’t want the creds to leak anywhere if they’re printing has been disabled by the command line arg. If you can pass along your transactor launch command, logs or console output that shows the leaked creds (redact/replace the actual creds), and the type of failure required to repro (or steps to take to induce that failure), I’ll investigate as a bug.

raymcdermott14:10:23

thanks @bkamphaus - I will double check the issue on this side first

bkamphaus14:10:37

@raymcdermott: cool, feel free to ping me here whenever you send it along - also would be helpful to have the txor properties file (again w/any creds or sensitive info redacted from it).

raymcdermott14:10:24

I struggling with Postgres / SSL on Heroku at the moment

kbaribeau16:10:16

hey, does anyone have any experience with this error? Our transactor is currently unresponsive:

2015-10-26 16:35:07.170 INFO  default    datomic.update - {:tid 133, :pid 1947, :message "Update failed"}
java.lang.NullPointerException: null
	at datomic.db$next_valid_inst.invoke(db.clj:2412) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.db.ProcessExpander.getData(db.clj:2468) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180$fn__10181.invoke(update.clj:246) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.Atom.swap(Atom.java:37) ~[clojure-1.6.0.jar:na]
	at clojure.core$swap_BANG_.invoke(core.clj:2232) ~[clojure-1.6.0.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180.invoke(update.clj:240) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177.invoke(update.clj:238) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173.invoke(update.clj:235) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor.doInvoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:139) [clojure-1.6.0.jar:na]
	at clojure.core$apply.invoke(core.clj:626) [clojure-1.6.0.jar:na]
	at datomic.update$background$proc__10093.invoke(update.clj:58) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]

kbaribeau16:10:11

we're seeing timeouts every time we try to issue a transaction. the above error comes from the transactor log. we suspect the transactor is basically paused because of that error.

bkamphaus16:10:15

@kbaribeau: two quick questions (1) does the problem persist across transactor restart (2) do you have the logs from when the error first started occurring?

kbaribeau16:10:46

(2) there's a bunch of instances of that exception in our logs. double checking

kbaribeau16:10:53

we've got the logs, for sure, there's just a lot of data to search through

kbaribeau17:10:45

there's :txid in these logs entries, can we use that to find out what was in the transaction that was failing?

bkamphaus17:10:05

If you can get first log (just grep or something) that has “Update failed” in it w/the accompanying “next_valid_inst” in stack trace, can you email it to me at bkamphaus @ cognitect ?

bkamphaus17:10:30

@kbaribeau have you backed up your database? Two pieces (1) back up right away to be safe, maybe to a new location in file store/s3 or w/e you back up to if your latest db backup does/did not show this issue, (2) verify backup will complete successfully

bkamphaus17:10:47

wait a minute, I may be misreading the error

bkamphaus17:10:02

@kbaribeau: what’s the most recent instant in the database? Is it possible a transaction was incorrectly written future dated? w/explicit db/txInstant as specified here http://docs.datomic.com/transactions.html#reified-transactions

kbaribeau17:10:48

hmm, there's a datomic api call to get that right?

kbaribeau17:10:03

I doubt we've ever set a T value explicitly. d/basis-t tells me our most recent T value is 4053248. should that show up in the log?

kbaribeau17:10:19

d/next-t seems to work. but I suppose that's different from datomic.db$next_valid_inst

bkamphaus17:10:02

Datoms about transaction entity from last successful transaction, replace safely-before-inst with date prior to issue occurring:

(let [log (d/log conn)
      safely-before-inst #inst "2015-10-01" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

bkamphaus17:10:49

if you annotate transactions with domain specific info, verify you’re ok sharing prior to pasting, or pm me w/tx datoms from last successful transaction.

kbaribeau17:10:57

thanks, running it

kbaribeau17:10:09

just need a minute, lost my repl 😞

kbaribeau17:10:02

(let [log (d/log @connection/conn)
      safely-before-inst #inst "2015-09-25T00:00:00.000-00:00" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

; (#datom[13194143586558 50 #inst "2015-10-23T21:30:09.200-00:00" 13194143586558 true])

(d/touch (d/entity (d/db @connection/conn) 13194143586558))

; {:db/id 13194143586558, :db/txInstant #inst "2015-10-23T21:30:09.200-00:00"}

kbaribeau18:10:41

^ not quite the most recent stuff

bkamphaus18:10:43

@kbaribeau: are you able to upgrade to latest? 0.9.5327 - on transactor

kbaribeau18:10:45

we've got more than one database

bkamphaus18:10:23

@kbaribeau: preferably upgrade both transactor and peer lib dep to latest, 0.9.5327 - we think there’s a possibility it stems from an issue corrected in that version.

kbaribeau18:10:27

we can update. what's in the latest release that's interesting? we're on 5206

kbaribeau18:10:43

we'll update

kbaribeau18:10:26

does upgrading from 5206 to 5327 require that we also upgrade the peer? seeing some errors that might be indicating that

bkamphaus18:10:59

@kbaribeau: yeah, push dep upgrade for peer apps as well

kbaribeau18:10:47

new error?

#object[datomic.promise$settable_future$reify__6305 0x262beb24 {:status :failed, :val #error {
 :cause "java.lang.NullPointerException"
 :via
 [{:type java.util.concurrent.ExecutionException
   :message "java.lang.NullPointerException: java.lang.NullPointerException"
   :at [datomic.promise$throw_executionexception_if_throwable invoke "promise.clj" 10]}
  {:type java.lang.NullPointerException
   :message "java.lang.NullPointerException"
   :at [clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]}]
 :trace
 [[clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]
  [datomic.error$deserialize_exception invoke "error.clj" 135]
  [datomic.peer.Connection notify_error "peer.clj" 401]
  [datomic.connector$fn__8480 invoke "connector.clj" 169]
  [clojure.lang.MultiFn invoke "MultiFn.java" 233]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490$fn__8491 invoke "connector.clj" 194]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490 invoke "connector.clj" 189]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487 invoke "connector.clj" 187]
  [clojure.core$binding_conveyor_fn$fn__4444 invoke "core.clj" 1916]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 266]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}}]

bkamphaus18:10:10

@kbaribeau: this is after upgrading both?

bkamphaus18:10:58

this is on transactor or peer on startup?

kbaribeau18:10:09

this is on a d/transact call

kbaribeau18:10:16

transactor and peer startup fine

bkamphaus18:10:15

@kbaribeau: is there an exception being reported on the transactor for that transaction attempt also?

zentrope21:10:28

Modelling question:

zentrope21:10:01

I have “user” entities and “club” entities. Users are members of one or more clubs, and clubs have one or more users (as members).

zentrope21:10:33

I could go with :user/clubs or :club/users.

zentrope21:10:00

But I want that “membership” to be typed, as in, “president” “secretary” “member” and so on.

zentrope21:10:34

So, I can't just associate a user with a club via a cardinality/many because that association itself needs to be annotated.

zentrope21:10:05

Do I create a “membership” entity which has refs from users to clubs along with a role enum (pres, veep, etc)?

zentrope21:10:22

That works, but feels kinda RDBMSish.

zentrope21:10:02

Or do you go with :club/presidents :club/veeps :club/members?

zentrope21:10:25

(Of course, what I want is to do a Pull API thing in a user and get a list of all the clubs s/he belongs to, and what the membership role is.)

zentrope21:10:44

Yeah. I don’t have a lack of ideas, just enough lack of experience to choose between them. ;)

bkamphaus21:10:22

For simple case, I would probably use multiple attributes, :club/president :club/veep :club/member … etc. as card/many and define a rule that says a user is one of these. Of course the Pull API use case with this is less straight forward. I might also try to define the president of relation from the user if a cardinality one constraint should apply to that direction. You could reify the relation, also, as you mention.

bkamphaus21:10:00

Is the Pull API use case you mention above the most common use case you envision, or just something you want to make sure is included?

zentrope21:10:33

I’m using the Pull API to make pulling a user’s data kind of document-like.

zentrope21:10:53

Get me the whole state for that user so I can blob it up to a UI and render it.

zentrope21:10:59

Doing the pulls in stages works, of course, but I’m hoping I’m not missing something given my past thinking in terms of using Postgres.

bkamphaus21:10:56

I guess it depends on how far your use of this membership idea. You could go so far as to make each “membership” a component entity of a person entity that refers to the class of membership and the organization, start date, end date, billed amount, etc.

zentrope21:10:45

Yes, that actually seems to be a reasonable approach, keeping it open ended.

zentrope21:10:03

Using the Pull API, how do I reference the membership?

zentrope21:10:28

[:user/id :user/name …] ?

zentrope21:10:25

[:user/id :user/name {:membership/_user [{:membership/role [:db/ident]}]}] seems to be getting me somewhere.

zentrope22:10:21

I can get from user -> membership -> club but can’t add -> membership -> users ; to get a list of all the members of that club.

bkamphaus22:10:23

that would be assuming a structure like:

{:user/id
 :user/name 
 :user/membership
 [{:membership/org …
   :membership/role ...
   :membership/start .. }]}
Where a membership is isComponent true, not sure if that’s a model that sounds reasonable for your use case.

zentrope22:10:30

Hopefully it’s just syntax.

zentrope22:10:43

Yes. That’s working for me.

zentrope22:10:16

What I want, though, is [user] -> [clubs] -> [members in each club] but I can’t seem to make the club to a membership lookup to find users. Hm.

zentrope22:10:54

[:user/id
   :user/name
   :user/email
   {:membership/_user
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       :membership/_user
       ]}]}]

bkamphaus22:10:08

so from :membership/club -> (club entity id) -> :membership/_club (all membership entities that point to club) -> :user/_membership (back to user ids)

zentrope22:10:12

That last :membership/_user doesn’t appear.

zentrope22:10:28

I can map this stuff through a transform that then pulls in, say, a list of all the members of a specific club via separate queries. I’m just wondering if the Pull API can do all that for me.

zentrope22:10:01

Should user have a :user/membership (isComponent) and club have a :club/membership (isComponent)?

zentrope22:10:43

Right now, I have just a standalone membership set of attributes for an entity, with :one for club and :one for user.

bkamphaus22:10:33

Probably differences of opinion on this — my quick mental mock up would say have :user/membership as isComponent of a user, points to an organization of which the user is a member.

bkamphaus22:10:48

I’m not sure what you mean be “standalone membership set of attributes for an entity” precisely.

zentrope22:10:25

(attr :membership/role :ref :one "")
(attr :membership/user :ref :one "")
(attr :membership/club :ref :one "")

zentrope22:10:04

Well, that’s just my dsl. I mean there’s no :user/memberships attribute of any type. Nor :club/membership.

zentrope22:10:17

Just like a SQL join table, is what I’m doing (and thus feels wrongish).

bkamphaus22:10:09

yeah, I would make membership entities that are components of users and have similar attributes. So no :membership/_user backref, but still role (points to an enum?), :club (points to a club org), and you can go from club org by back ref to all membership that apply to that club, and from those membership to their user parent entities.

zentrope22:10:58

Okay. I totally understand what you’re saying. That seems quite reasonable.

zentrope22:10:37

I’m much more interested in being able to walk from a user through to all the information they can see vs starting with a club.

zentrope22:10:41

How do you “go from club org by back ref to all membership” in a Pull API starting with a user?

zentrope22:10:19

I think :user/memberships :many shouldn’t be a component such that you can have a :club/memberships :many also pointing to the same membership entity as the :user/memberships attribute does.

bkamphaus22:10:54

@zentrope: why :club/membership also pointing to it?

zentrope22:10:23

Because otherwise I can get a list of the members of the club via a Pull API pattern.

bkamphaus22:10:58

I saw :user/memberships card many pointing to a membership entity which has :membership/club as a cardinality one

bkamphaus22:10:22

:membership/_club being the reverse ref you would follow from club entity to get all memberships

bkamphaus22:10:45

and you can use :user/_memberships for all users with a membership to that club

bkamphaus22:10:55

but I haven’t thrown in the toy schema/data yet

bkamphaus22:10:05

my reasoning being that it makes sense to constrain a membership to only be for one club

bkamphaus22:10:30

so there’s a direction of the relationship to which you could apply a cardinality/one constraint (have to try to enforce that via some other logic if you model the ref in the other direction)

zentrope22:10:20

Right. I think it’s the Pull API pattern I might be having trouble with.

raymcdermott22:10:34

hey guys … I am trying to connect to Datomic that is connected to a PostgresSQL DB

raymcdermott22:10:50

they require SSL and have some recommendations

raymcdermott22:10:54

for properties

zentrope22:10:32

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club '[*]}]}]}])

zentrope22:10:52

I can’t get from :membership/_club to a list of users.

raymcdermott22:10:58

ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory

raymcdermott22:10:06

but I’m still getting SSL off

raymcdermott22:10:08

Caused by: java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "52.22.208.29", user "dyno", database "dd7fmhk85j9m9d&ssl=true", SSL off

raymcdermott22:10:36

is there another way to enable SSL on the Postgres connection?

zentrope22:10:33

I think I figured it out.

raymcdermott22:10:21

it seems like it’s misinterpreting any additional parameters on the JDBC URL

zentrope22:10:19

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club
        [{:user/_memberships
          [:user/name {:user/memberships [{:membership/role [:db/ident]}]}]}]}]}]}])

zentrope22:10:27

Tag soup, but it works.

raymcdermott22:10:36

there is a sql-driver-params in the JDBC property file … I will give that a try

raymcdermott23:10:12

nope that did not work nor sqlDriverParams

bkamphaus23:10:32

@zentrope: it looks like you figured it out. I did end up doing a gist of a quick example of what I was talking about: https://gist.github.com/benkamphaus/4f991901b2fe8d00e20b

bkamphaus23:10:40

@raymcdermott: you’re following the Heroku PostgreSQL instructions here? http://docs.datomic.com/storage.html#sql-database

raymcdermott23:10:47

yes - the transactor is connecting OK

raymcdermott23:10:58

but the Peer is failing

bkamphaus23:10:56

@raymcdermott is it possible that there are issues with creds in the connection URL that need to be percent encoded?

bkamphaus23:10:36

Another option to try would be to use the map form for the URI described here http://docs.datomic.com/clojure/#datomic.api/connect

raymcdermott23:10:42

I have set the creds as system properties as mentuined in the docs

raymcdermott23:10:35

oooh so datomic:sql://{db-name}?{jdbc-url} is two maps?

raymcdermott23:10:54

I have just made them strings

raymcdermott23:10:30

Connects to the specified database, returing a Connection.
URI syntax ({} indicate place holders to fill in, [] indicate optional):

raymcdermott23:10:27

ah OK, read further now

raymcdermott23:10:37

still not clear how I set SSL

raymcdermott23:10:24

but maybe I can use the same terms as in the property file?

zentrope23:10:16

@bkamphaus: Another version of the many-to-many thing: https://gist.github.com/vaughnd/3705861