Fork me on GitHub
#datomic
<
2017-09-06
>
pesterhazy13:09:58

why run as www-data??

uwo14:09:44

we have a common query that appears to be causing an out of memory on GC overhead. the query itself at the moment scans 3.3m records, which we then have to sort and page. Implementing a ranged query is top on my list, but are there any other recommendations?

uwo14:09:29

@marshall the query only has one clause [?e :attr]

marshall14:09:52

@uwo so that sounds like a very scan-like operation - “find me all the entities” - I’d just use the datoms API

uwo14:09:07

@marshall yeah, definitely I’ve considered that as well. Will the memory profile be identical though since I still have to sort before I take/drop?

marshall14:09:31

are you sorting on a single attribute?

uwo14:09:11

initially, yes. though we’re populating a table, and the user can sort by up to 3 columns if they click on enough column headers

marshall15:09:12

well, presumably you could get your initial sort from a d/datoms call on AVET

marshall15:09:21

which means if you have additional sub-sorts, you’d only need to iterate through AVET until you reach a new V (i.e. the first ‘chunk’ of datoms that have the same value when sorted by your primary attribute), which you then would need to sub-sort by the additional attribute(s)

marshall15:09:58

depending on your data size/distribution, that approach may be able to reduce your memory overhead

uwo15:09:35

thanks @marshall. I’ll experiment with that approach

uwo15:09:12

also, I realize this is app specific, but we’ve currently got 6GB on our app server. I’ve read the capacity planning documentation, but I’m still a little unsure how to figure out how large an object cache we should target,etc. any tips?

uwo15:09:29

we’re starting the peer with -Xmx4g -Xms4g atm

marshall15:09:43

it entirely depends on what your app is doing and what your db usage patterns are the default is half the heap

marshall15:09:25

if you have a 6gb box and you’re running into memory pressure, i’d probably use more for the heap unless you are running other things on the box

marshall15:09:54

you can look at your objectCache metrics to see how frequently you’re missing

marshall15:09:03

to help tune whether you need more or could get away with less

marshall15:09:36

also, if you can use memcached that will help alleviate a lot - you can potentially leave object cache alone (or even reduce it) while adding memcached to give your application more headroom

uwo15:09:18

thanks. make sense!

oc17:09:22

is there a datomic equivalent of LIMIT or FETCH FIRST N ROWs? Im running a query that returns thousands of entities and i only want the first 500

favila17:09:23

(take 500 query-result)

oc17:09:19

that takes the same amount of time to excute, so i'm assuming it's just filtering after the query has found all the thousands of results

oc17:09:56

(time (d/q '[:find (take 500 ?e) :in $ ?txt :where [(fulltext $ ::s/ACCT_NME_TXT ?txt) [[?e]]]] db "test")) "Elapsed time: 14039.683358 msecs"

favila17:09:02

I'm surprised that works at all

favila17:09:11

I mean take after you get the results

favila17:09:58

so, result order is undefined, and querys run in a map/reduce fashion, so it's eager--it doesn't terminate early

favila17:09:06

so limit is not especially useful

oc17:09:24

we're trying to compare apples to apples with oracle text search which lets you short circuit the query with a limit

oc17:09:46

datomic is searching 8 million records and return 10000 in 12 seconds, but oracle responds in 2 seconds because it's only return the first 500

oc17:09:39

wanted to see if we could get something closer to an equal comparsion

pesterhazy19:09:25

Unfortunately there is no Limit clause

arohner20:09:40

If you use datoms, you can build a query that uses take, but ofc at that point you’re not using the query planner

arohner20:09:51

oh nvm, text search

oc20:09:11

is there a way to stop the query and just keep the results so far? i know having the queries run in the peer is great because your'e not blocking the 'db' like in sql but i could imagine not wanting my web server to block a thread for a minute if the user finds a nasty query

pesterhazy21:09:36

I'd love to be proven wrong, but AFAIK you can either use Datalog or limit the results, but not both - it's a real limitation.

oc21:09:15

kind of makes me want to find out how quickly d/datoms + java.lang.String#contains can return 500 items

ddellacosta21:09:30

I’m having trouble finding this info online: is there a way to drop/delete a Datomic partition? I’m testing out some schema stuff in a dev environment with datomic running in a docker container, and I’m having trouble figuring out how to override a attribute spec which I’ve already foolishly inserted into and now want to add :db.unique/identity to. <- alternatively telling me how to do that via other means would work.

ddellacosta21:09:01

well, I’m getting an error trying to add the constraint

ddellacosta21:09:22

and I assumed it was because of > In order to add a unique constraint to an attribute, Datomic must already be maintaining an AVET index on the attribute, or the attribute must have never had any values asserted.

uwo21:09:43

@ddellacosta so you’ve tried this? 1) add :db/index 2) d/request-index 3) add :db/unique

ddellacosta22:09:40

@uwo sorry for the slow response. I get this when I try to add an index (or set it unique for that matter)

CompilerException java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: :db.error/invalid-install-attribute Error: {:db/error :db.error/incompatible-schema-install, :entity :my/id, :attribute :db/index, :was nil, :requested true}, compiling:(form-init2080569204125065060.clj:1:35)

ddellacosta22:09:09

the :was nil bit confuses me in particular

ddellacosta22:09:28

wonder if I’ve got something else wrong with this schema item

marshall23:09:11

@ddellacosta can you share the tx data you're submitting to add index to your attribute

ddellacosta23:09:59

@marshall

1│  [
  2│   {:db/id #db/id[:db.part/db]
  3│    :db/ident :my/id
  4│    :db/index true
  5│  ;;  :db/unique :db.unique/identity
  6│    :db/valueType :db.type/uuid
  7│    :db/cardinality :db.cardinality/one
  8│    :db/doc "UUID for the thing"
  9│    :db.install/_attribute :db.part/db}]

ddellacosta23:09:10

sorry for the slightly weird formatting

ddellacosta23:09:01

basically I had that previously without the index or unique, messed around and transacted on a few things, then realized I wanted that to be unique

ddellacosta23:09:33

based on my reading of the schema doc, it sounds like I’m simply not going to be able to add unique to it at this point because I’ve already transacted some data

ddellacosta23:09:41

so I figured the best thing to do was delete the partition

ddellacosta23:09:54

this is all dev work so I don’t care about this data so much

marshall23:09:13

You can add unique

marshall23:09:30

Use a slightly different syntax

marshall23:09:39

What version of datomic

ddellacosta23:09:12

0.9.5544 I believe is what you’re looking for

marshall23:09:47

{:db/ident :person/name :db/cardinality :db.cardinality/many ;; explicit alter :db.alter/_attribute :db.part/db}

marshall23:09:06

If you want to use the explicit alter attribute

marshall23:09:41

In your case it would be your ident of :my/id

marshall23:09:26

And instead of cardinality put in the :db.index true

ddellacosta23:09:46

so let me make sure I understand: I want this?

[{:db/ident :my/id
  :db/unique :db.unique/identity
  :db.alter/_attribute :db.part/db}]

marshall23:09:06

Alternatively, you can do something like [{:db/id :my/id :db/index true}]

marshall23:09:21

Yes just like that

ddellacosta23:09:40

got it--but where were you going with that? I’m curious

marshall23:09:26

Just that if you're using explicit syntax you need to use alter attribute, not install

ddellacosta23:09:26

or is that it, I should also be able to simply do

[{:db/id :my/id
  :db/unique :db.unique/identity}]
?

marshall23:09:38

Yep that should work too

ddellacosta23:09:13

gotcha. I’ll have to go read up on the distinction between explicit syntax and the alternative, as I don’t think I understand that

marshall23:09:18

The implicit install and alter syntax was introduced in 0.9.5530 (iirc)

ddellacosta23:09:26

nor have I ever seen that :db.alter/_attribute thing

ddellacosta23:09:00

@marshall, I’m going to go give this a shot. Thanks so much for your help! I learned a lot.

marshall23:09:41

@ddellacosta ^ talks about the implicit install and alter stuff