Fork me on GitHub
#datomic
<
2019-01-25
>
codelimner04:01:26

Is there a way to turn off datomic.process-monitor info logs from the repl?

codelimner04:01:14

I get plenty of these:

codelimner04:01:25

INFO datomic.process-monitor - {:MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :AvailableMB 5970.0, :ObjectCacheCount 0, :event :metrics, :pid 18037, :tid 127}

marshall14:01:57

The settings for metrics reporting are all configured with your logback.xml file

notid05:01:03

Are there any updated recommendations on how to sorting/pagination on large datasets? Most of the posts I’ve seen about this are 5 years old by now. My approach has been to do two queries. The first applies filters, and pulls just the ids and sort field. I sort in-process, and then do another query to fetch the collection of desired entities by id. This is still pretty slow (500ms for 30k entities).

favila05:01:17

If there is a single pretty-selective attribute you can use, you can use it with index-range to pull subsets then feed into queries

favila05:01:45

Another alternative is to use a secondary database as an index, either polling or watching the tx queue to update

notid05:01:42

Interesting. What do you mean by pretty-selective attribute? Could you give an example?

favila07:01:10

It’s usually the first clause in your :where

favila07:01:35

Suppose you want all things posted on a certain day and also a bunch of other stuff

favila07:01:46

So your first :where clauses assert that the thing is in the date range (because that is most selective) then a bunch of other clauses check other stuff before finally deciding if it’s in the result set

favila07:01:40

But there’s 100000 things in that date range and you only need the first 10 that match

favila07:01:50

So you don’t want those first few clauses to look at everything just to get extra results you won’t use

favila07:01:19

So you can either divide up the range in the query itself, or you can use d/datoms or d/index-seek to lazily fetch a subset of that range and feed those entity ids to the query (to test the other stuff) as input

favila07:01:59

If you get less than your desired limit in results, advance the range then repeat

favila07:01:40

The important thing is that this attribute test/range must by itself ensure that a thing may or may not be in the result set. Otherwise you may advance the range and end up with repeated results

favila07:01:12

Ie the attribute test if subsetted must produce non-overlapping result sets

notid07:01:38

Thanks, that’s helpful. 🙂

tony.kay18:01:45

So, I’ve been working with on-prem Datomic for a while now, but now I have a client that is using the client API…with on-prem my testing is super easy since I can leverage datomock to make a db with sample data in it, transact/query/etc, and see the result. With client I have an external server dependency…how are others testing against that?

Lennart Buit19:01:42

I started using Datomic client memdb

mss19:01:09

hey all, playing around locally with datomic 0.9.5385 using the dev storage protocol. trying to transact a schema, which looks something like the following:

(def my-schema [{:db/ident       :user/id
                 :db/valueType   :db.type/uuid
                 :db/cardinality :db.cardinality/one}

                {:db/ident       :user/email
                 :db/valueType   :db.type/string
                 :db/cardinality :db.cardinality/one}

                {:db/ident       :user/first-name
                 :db/valueType   :db.type/string
                 :db/cardinality :db.cardinality/one}

                {:db/ident       :user/last-name
                 :db/valueType   :db.type/string
                 :db/cardinality :db.cardinality/one}])
when I call (d/transact my-conn my-schema) I get an error ":db.error/entity-missing-db-id Missing :db/id". I was under the impression that datomic assigned db/ids automatically once an entity was transacted. is that not the case?

Lennart Buit19:01:22

It suits me in sort of small-scoped integration testing

Lennart Buit19:01:05

Not sure how commonly used it is, or if there is anything better, but it at least works for me

marshall19:01:10

@mss what version of datomic?

mss19:01:33

datomic-pro 0.9.5385 w the dev storage protocol

marshall19:01:11

the implicit db/id was introduced in 5530

mss19:01:30

ah I see. well that explains that 😂

mss19:01:33

appreciate the help

idiomancy21:01:42

whats the difference between Datomic Cloud and Utility Bastion per https://aws.amazon.com/marketplace/pp/prodview-otb76awcrb7aa?ref=vdr_rf ? Or, more succintly, wth is Utility Bastion?

idiomancy21:01:34

I'm trying to make some purchasing decisions today lol

lilactown21:01:54

the bastion is a node that you can connect to to give you access to the Datomic VPC in order to do local development

lilactown21:01:08

normally, the only things that can talk to the Datomic DB have to be within the same VPC

lilactown21:01:11

mainly for security reason

lilactown21:01:24

the bastion is an EC2 instance that you can connect to from your computer so that you can access the Datomic Cloud system without pushing your code to AWS

👍 5
idiomancy21:01:47

Gotcha, that makes sense