hi, I have been running into the problem where Datomic Entities are not maps....

and I have found this bit of code that https://gist.github.com/jtmarmon/0a644fbca15a1742964c seems to solve it (Almost)

but now I get NPE's

I added a (if (nil? e)... to my code, but then it looks like I get way too much data.

what is the best way of dealing this?

@thomas What are you trying to accomplish? If you want maps to start with, d/pull may be an easier API to use, since it does return normal Clojure maps.

As a side note, (-> e class .toString (clojure.string/split #" ") second (= "datomic.query.EntityMap")) is a complicated way to write instance?.

I am using an rather old version of datomic... so can't really use d/pull 😞

and yes... re the second item I think you are correct.

and I am trying to do a postwalk on the EntityMap. and that was causing problem.

then @dominicm pointed out that EntityMaps are not Clojure maps... there is a (into {} entity) in the code, but from what I understand that won't work on nested maps

So what is the best way of turning EntityMaps into Clojure maps? With (into {} entity) I can't do a postwalk and with the above mentioned function I am getting different results it seems and those result in failure higher up the stack.

(clojure.walk/postwalk (fn [x] (if (instance? datomic.query.EntityMap x) (into {} x) x)) input)

let me try

no, that gives me an java.lang.AbstractMethodError error again 😞

from what I understand a prewalk should do the trick....

let me try htat

that looks like it...

prewalk instead of postwalk... let me try that in our slightly bigger env

that looks ok!

prewalk it is...

Thank you @stuartsierra and @dominicm !!!

Now it is working as expected.

oh, of course. Makes sense.

most things makes sense after you have fixed them 😉

Guys for query [:find (max 5 ?e) (pull ?r [*]) :with ?player :where [?player :player/region ?r] [?player :player/attr.stats.wins ?e]]

I understand that in order to get all of the attributes for the entity layer:p I'll have to write your function aggregation?

what?

I don't understand the question

what is ?e here? why are you calling max on it?

@favila want to choose 5 entities which have a maximum value of an attribute :player/attr.stats.wins

query is works just me instead of one attribute (:player/attr.stats.wins) are interested in the whole essence of :player

what is the type of :player/attr.stats.wins?

long

cardinality-one?

yes

so you want (d/pull db ?player [* {:player/region [*]}]) for the 5 players with the most wins?

@favila i'm sorry)) (d/q '[:find (max 5 ?e) (pull ?r [:region/key]) :with ?player :where [?player :player/attr.stats.wins ?e] [?player :player/region ?r] (d/db conn)

top 5 players per region?

yes

[:find (max 5 ?n-wins) ?r (pull [*] ?player)
         :where [?player :player/attr.stats.wins ?n-wins]
                      [?player :player/region ?r]

sorry hit enter too soon

is that what you are thinking?

since you are aggregating two different ways I'm not sure you can do it in a single query

@favila the query above does not work( 'invalid expression pull'

did you remove the :with?

yes

I don't think that query gives you what you want anyway

oh, you have to reverse arg order

man I hate that so much, I always get that wrong

I actually agree on something to get :db/id for each :player

d/pull and pull reverse arg order

@favila I don't quite understand the use case, it is not difficult to show the minimum sample?

anything in :with is kept for result-set, but removed for aggregation in :find

so you can't both aggregate by a subset and also keep the full tuple

oh(

So you must either query twice, or do the aggregation yourself afterwards

@favila Thank you, but I still don't understand how two queries you can get at least :db/id for :player with the highest number of victories

In first query, you get [?region #{?max-five-wins}]

in second query, you find players whose regions and wins match

you may get more than 5 results per region if multiple players have the same number of wins

but you can sort+limit that yourself

I will write an example

(let [max-wins-per-region           (d/q '[:find ?r (max 5 ?n-wins)
                                           ;; No point to :with ?player
                                           :where
                                           [?player :player/attr.stats.wins ?n-wins]
                                           [?player :player/region ?r]] db)
      max-wins-per-region-relations (into []
                                      (mapcat (fn [[region-id n-wins]]
                                                (map #(do [region-id %]) n-wins)))
                                      max-wins-per-region)]
  (d/q '[:find ?r-key (pull [*] ?player)
         :in $ [[?r ?n-wins]]
         :where
         [?player :player/region ?r]
         [?r :region/key ?r-key]
         [?player :player/attr.stats.wins ?n-wins]]
    db
    max-wins-per-region-relations))

(I assume :player/region is cardinality-one? players have only one region?)

both question answer is yes

so here is another approach

@favila thanks) for a short time discovered many newlifer

You can always do the aggregation yourself, too, if you can hold the intermediate result set

query that returns [?player-id ?region-id n-wins], then do a reduce over it

in that case the player cutoff will be arbitrary. If 6 players in a region all have 10 wins, which player is not listed?

if that doesn't fit in memory, you can use (d/datoms :aevt :player/region), and reduce over it, grabbing player data and aggregating at the same time

Remember, queries run on the peers. You can do everything with code instead of queries if you want. The client is going to download every intermediate value either way.

There's no need to fit everything into queries, certainly not into one single query

@favila when in the base of the 305 000 players request to get top players totally leaves 2.2 minutes

thx)

@a.espolov if you have an index on :player/attr.stats.wins, it may be faster to put that clause first

(depends on the selectivity of the index)

@favila you can add an index to the entity attribute if already exist in the database instances are selected?

If :db/index = false, you can add an index

By selectivity I just mean how many records you will get for a given value

e.g., if number-of-wins clusters around a few common values, it may touch fewer datoms to find the region then the number of wins. But if number of wins tends to vary widely the index is more selective, so getting by number of wins, then the region will touch fewer datoms

@favila la index i can disable attribute have no problems with existing records in the database?

@a.espolov I don't understand?

:db/index = true => :db/index = false such an operation for data in database is executed without problems?

These are the permitted schema alterations: http://docs.datomic.com/schema.html#altering-schema-attributes

you can change :db/index by itself at will

But removing an index in your scenario will not help query performance. The reasons to remove an index are: less storage for the index, less indexing time.

I think it's usually better to index everything, then turn it off later if either of those becomes a problem

@favila thanks