Fork me on GitHub
#datomic
<
2015-10-26
>
domkm06:10:12

Question for the Cognitects out there: I noticed that there is a :db.install/function ident. Its docstring says that it is used to install functions but it is not documented on http://docs.datomic.com. Could you shed some light on :db.install/function?

domkm06:10:20

Also, any hint about when we can use :db.install/valueType? šŸ˜‰

genRaiy12:10:43

another connection question for the Datomics

genRaiy12:10:08

why does the client need to provide the JDBC url when making a connection to the transactor?

genRaiy12:10:20

this seems like a leaky abstraction

genRaiy12:10:59

of course the fact that the client needs to specify the storage at all is leaky

genRaiy12:10:09

or am I missing somthing?

genRaiy12:10:26

apart from an e in something

mitchelkuijpers12:10:34

@raymcdermott The client get's the transactor url from the database, this means that if you for example start a new transactor or one crashes and another on comes up it get's the new transactor url from the database. This also means that you data fetching keeps working when the transactor is down. It is actually pretty nifty

robert-stuttaford12:10:32

@raymcdermott: it rocks that peers talk to storage directly. transactor doesnā€™t have to be involved for reads at all. means scaling reads is independent of scaling writes.

robert-stuttaford13:10:17

also failover process uses storage; so peers know when failover is happening even while primary transactor is dead

marshall13:10:01

@raymcdermott: You can suppress the printing of the login information by passing:

-Ddatomic.printConnectionInfo=false 
as a parameter to the transactor launch

marshall13:10:07

Morninā€™ @robert-stuttaford . Well, I guess not for you simple_smile

robert-stuttaford13:10:35

so whenā€™s the Next Big Thing coming for Datomic, @marshall? i just canā€™t imagine that you guys are ā€˜doneā€™ simple_smile

marshall13:10:57

Nope, definitely not done, but I donā€™t have any visibility into when/what is coming; at least not until itā€™s essentially release-ready

robert-stuttaford13:10:22

ok, cool simple_smile good to know thereā€™s still stuff cooking!

genRaiy13:10:43

oh and you too @marshall !!

pesterhazy13:10:11

Hi. Does anyone have a solution for partial backups of a datomic db?

pesterhazy13:10:15

The use case: I'f like to extract a set of "fixtures" from an existing production db. With mysqldump (for example) I'd just exclude ceratain tables from the dump.

pesterhazy13:10:44

But a normal Datomic dump will typically include all facts (whereas I'd like to exclude, say, all events or all payment information).

robert-stuttaford13:10:46

hah ā€“ is that you, @bkamphaus ?

robert-stuttaford13:10:14

pesterhazy: we wrote custom code for this that exports to a transit file, zips it, and puts it on S3

pesterhazy13:10:48

that sounds like a good solution

robert-stuttaford13:10:55

takes a database and a configuration of attrs to export, and exports all the schema (whether in the config or not), enums, and data for those attrs

robert-stuttaford13:10:16

our prod db backup is 9gb. this exports the ā€˜controlā€™ information which comes to 12mb gzipped

robert-stuttaford13:10:25

or 55mb unzipped. itā€™s super useful simple_smile

robert-stuttaford13:10:30

lemme see if i can gist it

pesterhazy13:10:39

that would be PERFECT simple_smile

pesterhazy13:10:59

I tried https://groups.google.com/d/msg/datomic/eQQnnqYl67Y/z-Ib60NCJAAJ but it doesn't seem to cope well with 100s of MB of data

genRaiy13:10:49

@marshall: I noticed that password data does get leaked out when things go wrong (it dumps the properties) ā€¦ maybe something more complete is needed?

robert-stuttaford13:10:05

@pesterhazy: not a simple cut and paste job, iā€™m afraid. iā€™ve put a note on my list to get it done this week. are you on twitter?

pesterhazy13:10:54

would be useful for lots of users, I think

robert-stuttaford13:10:16

totally. iā€™d share it as a public gist and tweet it, and cc you

pesterhazy13:10:24

cool thanks!

robert-stuttaford13:10:53

šŸ‘ was super fun to write with transducers

Ben Kamphaus13:10:58

everything is better with some transducers thrown in

Ben Kamphaus13:10:22

theyā€™re the sriracha of clojure

robert-stuttaford13:10:49

this code actually builds transducers dynamically based on the config, and memoizes them

robert-stuttaford13:10:31

(had to look that up)

Ben Kamphaus13:10:36

@raymcdermott: taking over from @marshall here - if you can email me ā€” bkamphaus @ cognitect ā€” we definitely donā€™t want the creds to leak anywhere if theyā€™re printing has been disabled by the command line arg. If you can pass along your transactor launch command, logs or console output that shows the leaked creds (redact/replace the actual creds), and the type of failure required to repro (or steps to take to induce that failure), Iā€™ll investigate as a bug.

genRaiy14:10:23

thanks @bkamphaus - I will double check the issue on this side first

Ben Kamphaus14:10:37

@raymcdermott: cool, feel free to ping me here whenever you send it along - also would be helpful to have the txor properties file (again w/any creds or sensitive info redacted from it).

genRaiy14:10:24

I struggling with Postgres / SSL on Heroku at the moment

kbaribeau16:10:16

hey, does anyone have any experience with this error? Our transactor is currently unresponsive:

2015-10-26 16:35:07.170 INFO  default    datomic.update - {:tid 133, :pid 1947, :message "Update failed"}
java.lang.NullPointerException: null
	at datomic.db$next_valid_inst.invoke(db.clj:2412) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.db.ProcessExpander.getData(db.clj:2468) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180$fn__10181.invoke(update.clj:246) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.Atom.swap(Atom.java:37) ~[clojure-1.6.0.jar:na]
	at clojure.core$swap_BANG_.invoke(core.clj:2232) ~[clojure-1.6.0.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177$fn__10180.invoke(update.clj:240) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173$fn__10177.invoke(update.clj:238) ~[datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172$fn__10173.invoke(update.clj:235) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171$fn__10172.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor$fn__10171.invoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at datomic.update$processor.doInvoke(update.clj:216) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.RestFn.applyTo(RestFn.java:139) [clojure-1.6.0.jar:na]
	at clojure.core$apply.invoke(core.clj:626) [clojure-1.6.0.jar:na]
	at datomic.update$background$proc__10093.invoke(update.clj:58) [datomic-transactor-pro-0.9.5206.jar:na]
	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]

kbaribeau16:10:11

we're seeing timeouts every time we try to issue a transaction. the above error comes from the transactor log. we suspect the transactor is basically paused because of that error.

Ben Kamphaus16:10:15

@kbaribeau: two quick questions (1) does the problem persist across transactor restart (2) do you have the logs from when the error first started occurring?

kbaribeau16:10:46

(2) there's a bunch of instances of that exception in our logs. double checking

kbaribeau16:10:53

we've got the logs, for sure, there's just a lot of data to search through

kbaribeau17:10:45

there's :txid in these logs entries, can we use that to find out what was in the transaction that was failing?

Ben Kamphaus17:10:05

If you can get first log (just grep or something) that has ā€œUpdate failedā€ in it w/the accompanying ā€œnext_valid_instā€ in stack trace, can you email it to me at bkamphaus @ cognitect ?

Ben Kamphaus17:10:30

@kbaribeau have you backed up your database? Two pieces (1) back up right away to be safe, maybe to a new location in file store/s3 or w/e you back up to if your latest db backup does/did not show this issue, (2) verify backup will complete successfully

Ben Kamphaus17:10:47

wait a minute, I may be misreading the error

Ben Kamphaus17:10:02

@kbaribeau: whatā€™s the most recent instant in the database? Is it possible a transaction was incorrectly written future dated? w/explicit db/txInstant as specified here http://docs.datomic.com/transactions.html#reified-transactions

kbaribeau17:10:48

hmm, there's a datomic api call to get that right?

kbaribeau17:10:03

I doubt we've ever set a T value explicitly. d/basis-t tells me our most recent T value is 4053248. should that show up in the log?

kbaribeau17:10:19

d/next-t seems to work. but I suppose that's different from datomic.db$next_valid_inst

Ben Kamphaus17:10:02

Datoms about transaction entity from last successful transaction, replace safely-before-inst with date prior to issue occurring:

(let [log (d/log conn)
      safely-before-inst #inst "2015-10-01" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

Ben Kamphaus17:10:49

if you annotate transactions with domain specific info, verify youā€™re ok sharing prior to pasting, or pm me w/tx datoms from last successful transaction.

kbaribeau17:10:57

thanks, running it

kbaribeau17:10:09

just need a minute, lost my repl šŸ˜ž

kbaribeau17:10:02

(let [log (d/log @connection/conn)
      safely-before-inst #inst "2015-09-25T00:00:00.000-00:00" ;example date, replace
      txes (d/tx-range log safely-before-inst nil)
      last-tx-data (:data (last txes))]
  (filter #(= 3 (d/part (:e %))) last-tx-data))

; (#datom[13194143586558 50 #inst "2015-10-23T21:30:09.200-00:00" 13194143586558 true])

(d/touch (d/entity (d/db @connection/conn) 13194143586558))

; {:db/id 13194143586558, :db/txInstant #inst "2015-10-23T21:30:09.200-00:00"}

kbaribeau18:10:41

^ not quite the most recent stuff

Ben Kamphaus18:10:43

@kbaribeau: are you able to upgrade to latest? 0.9.5327 - on transactor

kbaribeau18:10:45

we've got more than one database

Ben Kamphaus18:10:23

@kbaribeau: preferably upgrade both transactor and peer lib dep to latest, 0.9.5327 - we think thereā€™s a possibility it stems from an issue corrected in that version.

kbaribeau18:10:27

we can update. what's in the latest release that's interesting? we're on 5206

kbaribeau18:10:43

we'll update

kbaribeau18:10:26

does upgrading from 5206 to 5327 require that we also upgrade the peer? seeing some errors that might be indicating that

Ben Kamphaus18:10:59

@kbaribeau: yeah, push dep upgrade for peer apps as well

kbaribeau18:10:47

new error?

#object[datomic.promise$settable_future$reify__6305 0x262beb24 {:status :failed, :val #error {
 :cause "java.lang.NullPointerException"
 :via
 [{:type java.util.concurrent.ExecutionException
   :message "java.lang.NullPointerException: java.lang.NullPointerException"
   :at [datomic.promise$throw_executionexception_if_throwable invoke "promise.clj" 10]}
  {:type java.lang.NullPointerException
   :message "java.lang.NullPointerException"
   :at [clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]}]
 :trace
 [[clojure.core$eval329$fn__330 invoke "NO_SOURCE_FILE" -1]
  [datomic.error$deserialize_exception invoke "error.clj" 135]
  [datomic.peer.Connection notify_error "peer.clj" 401]
  [datomic.connector$fn__8480 invoke "connector.clj" 169]
  [clojure.lang.MultiFn invoke "MultiFn.java" 233]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490$fn__8491 invoke "connector.clj" 194]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487$fn__8490 invoke "connector.clj" 189]
  [datomic.connector$create_hornet_notifier$fn__8486$fn__8487 invoke "connector.clj" 187]
  [clojure.core$binding_conveyor_fn$fn__4444 invoke "core.clj" 1916]
  [clojure.lang.AFn call "AFn.java" 18]
  [java.util.concurrent.FutureTask run "FutureTask.java" 266]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}}]

Ben Kamphaus18:10:10

@kbaribeau: this is after upgrading both?

Ben Kamphaus18:10:58

this is on transactor or peer on startup?

kbaribeau18:10:09

this is on a d/transact call

kbaribeau18:10:16

transactor and peer startup fine

Ben Kamphaus18:10:15

@kbaribeau: is there an exception being reported on the transactor for that transaction attempt also?

zentrope21:10:28

Modelling question:

zentrope21:10:01

I have ā€œuserā€ entities and ā€œclubā€ entities. Users are members of one or more clubs, and clubs have one or more users (as members).

zentrope21:10:33

I could go with :user/clubs or :club/users.

zentrope21:10:00

But I want that ā€œmembershipā€ to be typed, as in, ā€œpresidentā€ ā€œsecretaryā€ ā€œmemberā€ and so on.

zentrope21:10:34

So, I can't just associate a user with a club via a cardinality/many because that association itself needs to be annotated.

zentrope21:10:05

Do I create a ā€œmembershipā€ entity which has refs from users to clubs along with a role enum (pres, veep, etc)?

zentrope21:10:22

That works, but feels kinda RDBMSish.

zentrope21:10:02

Or do you go with :club/presidents :club/veeps :club/members?

zentrope21:10:25

(Of course, what I want is to do a Pull API thing in a user and get a list of all the clubs s/he belongs to, and what the membership role is.)

zentrope21:10:44

Yeah. I donā€™t have a lack of ideas, just enough lack of experience to choose between them. ;)

Ben Kamphaus21:10:22

For simple case, I would probably use multiple attributes, :club/president :club/veep :club/member ā€¦ etc. as card/many and define a rule that says a user is one of these. Of course the Pull API use case with this is less straight forward. I might also try to define the president of relation from the user if a cardinality one constraint should apply to that direction. You could reify the relation, also, as you mention.

Ben Kamphaus21:10:00

Is the Pull API use case you mention above the most common use case you envision, or just something you want to make sure is included?

zentrope21:10:33

Iā€™m using the Pull API to make pulling a userā€™s data kind of document-like.

zentrope21:10:53

Get me the whole state for that user so I can blob it up to a UI and render it.

zentrope21:10:59

Doing the pulls in stages works, of course, but Iā€™m hoping Iā€™m not missing something given my past thinking in terms of using Postgres.

Ben Kamphaus21:10:56

I guess it depends on how far your use of this membership idea. You could go so far as to make each ā€œmembershipā€ a component entity of a person entity that refers to the class of membership and the organization, start date, end date, billed amount, etc.

zentrope21:10:45

Yes, that actually seems to be a reasonable approach, keeping it open ended.

zentrope21:10:03

Using the Pull API, how do I reference the membership?

zentrope21:10:28

[:user/id :user/name ā€¦] ?

zentrope21:10:25

[:user/id :user/name {:membership/_user [{:membership/role [:db/ident]}]}] seems to be getting me somewhere.

Ben Kamphaus21:10:05

http://docs.datomic.com/pull.html#component-defaults ā€” component defaults in pull documentation

zentrope22:10:21

I can get from user -> membership -> club but canā€™t add -> membership -> users ; to get a list of all the members of that club.

Ben Kamphaus22:10:23

that would be assuming a structure like:

{:user/id
 :user/name 
 :user/membership
 [{:membership/org ā€¦
   :membership/role ...
   :membership/start .. }]}
Where a membership is isComponent true, not sure if thatā€™s a model that sounds reasonable for your use case.

zentrope22:10:30

Hopefully itā€™s just syntax.

zentrope22:10:43

Yes. Thatā€™s working for me.

zentrope22:10:16

What I want, though, is [user] -> [clubs] -> [members in each club] but I canā€™t seem to make the club to a membership lookup to find users. Hm.

zentrope22:10:54

[:user/id
   :user/name
   :user/email
   {:membership/_user
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       :membership/_user
       ]}]}]

Ben Kamphaus22:10:08

so from :membership/club -> (club entity id) -> :membership/_club (all membership entities that point to club) -> :user/_membership (back to user ids)

zentrope22:10:12

That last :membership/_user doesnā€™t appear.

zentrope22:10:28

I can map this stuff through a transform that then pulls in, say, a list of all the members of a specific club via separate queries. Iā€™m just wondering if the Pull API can do all that for me.

zentrope22:10:01

Should user have a :user/membership (isComponent) and club have a :club/membership (isComponent)?

zentrope22:10:43

Right now, I have just a standalone membership set of attributes for an entity, with :one for club and :one for user.

Ben Kamphaus22:10:33

Probably differences of opinion on this ā€” my quick mental mock up would say have :user/membership as isComponent of a user, points to an organization of which the user is a member.

Ben Kamphaus22:10:48

Iā€™m not sure what you mean be ā€œstandalone membership set of attributes for an entityā€ precisely.

zentrope22:10:25

(attr :membership/role :ref :one "")
(attr :membership/user :ref :one "")
(attr :membership/club :ref :one "")

zentrope22:10:04

Well, thatā€™s just my dsl. I mean thereā€™s no :user/memberships attribute of any type. Nor :club/membership.

zentrope22:10:17

Just like a SQL join table, is what Iā€™m doing (and thus feels wrongish).

Ben Kamphaus22:10:09

yeah, I would make membership entities that are components of users and have similar attributes. So no :membership/_user backref, but still role (points to an enum?), :club (points to a club org), and you can go from club org by back ref to all membership that apply to that club, and from those membership to their user parent entities.

zentrope22:10:58

Okay. I totally understand what youā€™re saying. That seems quite reasonable.

zentrope22:10:37

Iā€™m much more interested in being able to walk from a user through to all the information they can see vs starting with a club.

zentrope22:10:41

How do you ā€œgo from club org by back ref to all membershipā€ in a Pull API starting with a user?

zentrope22:10:19

I think :user/memberships :many shouldnā€™t be a component such that you can have a :club/memberships :many also pointing to the same membership entity as the :user/memberships attribute does.

Ben Kamphaus22:10:54

@zentrope: why :club/membership also pointing to it?

zentrope22:10:23

Because otherwise I can get a list of the members of the club via a Pull API pattern.

Ben Kamphaus22:10:58

I saw :user/memberships card many pointing to a membership entity which has :membership/club as a cardinality one

Ben Kamphaus22:10:22

:membership/_club being the reverse ref you would follow from club entity to get all memberships

Ben Kamphaus22:10:45

and you can use :user/_memberships for all users with a membership to that club

Ben Kamphaus22:10:55

but I havenā€™t thrown in the toy schema/data yet

Ben Kamphaus22:10:05

my reasoning being that it makes sense to constrain a membership to only be for one club

Ben Kamphaus22:10:30

so thereā€™s a direction of the relationship to which you could apply a cardinality/one constraint (have to try to enforce that via some other logic if you model the ref in the other direction)

zentrope22:10:20

Right. I think itā€™s the Pull API pattern I might be having trouble with.

genRaiy22:10:34

hey guys ā€¦ I am trying to connect to Datomic that is connected to a PostgresSQL DB

genRaiy22:10:50

they require SSL and have some recommendations

genRaiy22:10:54

for properties

zentrope22:10:32

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club '[*]}]}]}])

zentrope22:10:52

I canā€™t get from :membership/_club to a list of users.

genRaiy22:10:58

ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory

genRaiy22:10:06

but Iā€™m still getting SSL off

genRaiy22:10:08

Caused by: java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "52.22.208.29", user "dyno", database "dd7fmhk85j9m9d&ssl=true", SSL off

genRaiy22:10:36

is there another way to enable SSL on the Postgres connection?

zentrope22:10:33

I think I figured it out.

genRaiy22:10:21

it seems like itā€™s misinterpreting any additional parameters on the JDBC URL

zentrope22:10:19

(def club-pattern
  [:user/id
   :user/name
   :user/email
   {:user/memberships
    [{:membership/role [:db/ident]}
     {:membership/club
      [:club/id
       :club/name
       {:membership/_club
        [{:user/_memberships
          [:user/name {:user/memberships [{:membership/role [:db/ident]}]}]}]}]}]}])

zentrope22:10:27

Tag soup, but it works.

genRaiy22:10:36

there is a sql-driver-params in the JDBC property file ā€¦ I will give that a try

genRaiy23:10:12

nope that did not work nor sqlDriverParams

Ben Kamphaus23:10:32

@zentrope: it looks like you figured it out. I did end up doing a gist of a quick example of what I was talking about: https://gist.github.com/benkamphaus/4f991901b2fe8d00e20b

Ben Kamphaus23:10:40

@raymcdermott: youā€™re following the Heroku PostgreSQL instructions here? http://docs.datomic.com/storage.html#sql-database

genRaiy23:10:47

yes - the transactor is connecting OK

genRaiy23:10:58

but the Peer is failing

Ben Kamphaus23:10:56

@raymcdermott is it possible that there are issues with creds in the connection URL that need to be percent encoded?

Ben Kamphaus23:10:36

Another option to try would be to use the map form for the URI described here http://docs.datomic.com/clojure/#datomic.api/connect

genRaiy23:10:42

I have set the creds as system properties as mentuined in the docs

genRaiy23:10:35

oooh so datomic:sql://{db-name}?{jdbc-url} is two maps?

genRaiy23:10:54

I have just made them strings

genRaiy23:10:30

Connects to the specified database, returing a Connection.
URI syntax ({} indicate place holders to fill in, [] indicate optional):

genRaiy23:10:27

ah OK, read further now

genRaiy23:10:37

still not clear how I set SSL

genRaiy23:10:24

but maybe I can use the same terms as in the property file?

zentrope23:10:16

@bkamphaus: Another version of the many-to-many thing: https://gist.github.com/vaughnd/3705861