Fork me on GitHub
#datomic
<
2018-03-01
>
devn00:03:17

a little confused as it's been awhile since I played with datomic. I am trying pro starter. What I remember doing in the past was setting up license key, storage, adding the postgres driver, adding datomic-pro (not client) to my deps, running the transactor, requiring datomic.api :as d, specify URI, create-db on it, make connection

souenzzo00:03:51

What storage backend are you using? Besides the peer, you're also running the transactor, right?

devn00:03:35

postgres

devn00:03:48

the docs now refer to starting a peer server, does this start a transactor?

devn00:03:37

I tried it the old way with the latest datomic-pro library using the URI, but it failed to transact my schema. now trying it with the client library and it's complaining about "SSL doesn't have a valid keystore" when trying to make the client

souenzzo00:03:56

Transactor "alone" on a JVM. You need to download the jar and run

souenzzo00:03:51

the minimal production datomic setup you need 3 components: 1- Storage backend (SQL, in your case) 2- Your app, that requires and use datomic.api (we call it peer) 3- The transactor, that you download a jar and run in a independent JVM(looks like it's missing). --- Dataflow: peer READ from DB peer send writes to transactor transactor WRITE on DB

devn01:03:23

got it figured out. had some problems with conflicting dependencies that caused a few of the issues, and then there was some confusion over small diffs in the API, like the client using {:tx-data ...} when transacting vs the peer library.

Desmond05:03:23

i'm back with more performance questions. last time I found that my query was slow because of some naive mistakes. now the same query is slow because of a not-join across :db/id values.

Desmond05:03:32

This is the part of the query that's slowing it down:

(not-join [?c ?u]
  [?v :comment-viewing/user ?u]
  [?v :comment-viewing/comment ?c])

Desmond06:03:12

where ?c and ?u are ids

Desmond06:03:25

total in the database there are up to 114 ?u datoms, 1649 ?c datoms, and 6846 ?v datoms

Desmond06:03:44

the query takes over 5 seconds with the not-join, 500 ms without the not-join, 700 ms when I drop the [?v :comment-viewing/comment ?c] clause, and 99 ms when i drop the [?v :comment-viewing/user ?u] clause.

Desmond06:03:28

are :db.type/ref indexed by default?

Desmond06:03:02

could :db/noHistory help here?

Desmond07:03:02

solved! I just need to flip the clauses around like so:

(not-join [?c ?r]
                                 [?v :comment-viewing/comment ?c]
                                 [?v :comment-viewing/rapper ?r])
Takes the time from 5 seconds to 90 ms

Desmond07:03:45

Only figured this out by looking at day-of-datomic:

;; This query leads with a where clause that must consider *all* releases
;; in the database.  SLOW.
(dotimes [_ 5]
  (time
   (d/q '[:find [?name ...]
          :in $ ?artist
          :where [?release :release/name ?name]
                 [?release :release/artists ?artist]]
        db
        mccartney)))

;; The same query, but reordered with a more selective where clause first.
;; 50 times faster.
(dotimes [_ 5]
  (time
   (d/q '[:find [?name ...]
          :in $ ?artist
          :where [?release :release/artists ?artist]
                 [?release :release/name ?name]]
        db
        mccartney)))

laujensen12:03:16

@stuarthalloway / @marshall: We talked about Html and CSS blowing up the indexing service. Would prepending something like <!-- rev 0x0f32f3ffff --> fix that, if I remove all history on these entities?

marshall13:03:34

@laujensen prepending to the value with something that changes (i.e. the sha of the value itself) would help with the segment size issue Note that enabling noHistory will not remove existing history from an attribute

alexk15:03:43

When I get :restore/collision from datomic restore-db, is there any way to tell it to overwrite without having to manually delete the db?

alexk15:03:09

I’m upset that I have to delete it at all, because I don’t know what that would do to a datomic client/peer connected to it

alexk15:03:20

What am I missing?

marshall15:03:18

@alex438 Why are you restoring into a ‘live’ database? You must restart all peers and transactor after a restore: https://docs.datomic.com/on-prem/backup.html#sec-5

alexk16:03:23

and transactor? aww

alexk16:03:43

It’s more of a “migrate latest real db into a staging db” than “restore into live db”

marshall16:03:58

yeah, you’ll need to restart

alexk16:03:32

How to restart a transactor running with CloudFormation? There’s no restart/stop option - would I have to reboot the associated EC2 instance?

marshall16:03:04

the provided cloudformation makes an autoscaling group

marshall16:03:11

so you can terminate the instance and the ASG will replace it

marshall16:03:18

if you dont you should get a transactor failover the first time you try to write anyway

alexk16:03:06

What’s the meaning of “You do not need to do anything to a storage (e.g. deleting old files or tables) before or after restoring.“, which seems to be at odds with the :restore/collision error?

marshall16:03:09

what version of Datomic?

alexk16:03:35

datomic-pro-0.9.5561.62 more or less

marshall16:03:24

and you get that error on the peer that is running the restore job?

alexk16:03:53

the transactor is created by the ensure* commands

alexk16:03:23

and I’m calling bin/datomic restore-db file:/etc file:/Users/etc/Desktop/etc datomic:

marshall16:03:56

are you getting “is already in use by a different database” or “database already exists under the name”

alexk16:03:59

It worked if I gave it a db name that didn’t exist yet

alexk16:03:09

> already in use by a different database

marshall16:03:27

ok. yes you can’t ‘fork’ a DB within the same storage under two different names

marshall16:03:52

the basis from which that database was created (call it foo) has been restored into that storage before

alexk16:03:54

oh I read that but didn’t realize that’s the rule I broke, but yeah of course

alexk16:03:03

ok so I can restore, but not update

marshall16:03:28

so if you restore foo into storage then change some stuff in it (and in the original source of foo) then try to backup the original source again and restore into the same storage you’ll see that error

alexk16:03:04

is that something that can be fixed in a later release of datomic - to use new ids for restored data?

marshall16:03:05

b/c those two databases share some history (whatever was there before the original restore), but diverge later

marshall16:03:38

it’s something that could be suggested in our Feature request portal

alexk16:03:49

ok, just wondering if it’s design or implementation

alexk16:03:55

thanks marshall

mgrbyte17:03:46

where's the right place to report an issue with the peer api? I've found that tx-range doesn't work when supplied a tx id for start (but does when using (d/tx->t tx-id))

mgrbyte17:03:30

I'm using [com.datomic/datomic-pro "0.9.5561"]

Datomic Platonic21:03:14

I saw a video where the presenter mentioned tricking the transactor into inserting custom transaction timestamps. What is the best way to do that?

devn21:03:23

Fun fact: midtaco is an anagram for datomic

devn21:03:53

one of my prouder local repo naming moments, thought i'd share