Fork me on GitHub
#datomic
<
2016-09-09
>
magnars05:09:15

How can I query for the x most recently created entities (in my case chat messages) without pulling in all of them, sorting them in the client, and then do a take ?

robert-stuttaford06:09:52

it's a tough problem, because you can't traverse the transaction log in reverse order - you can go back some arbitrary period and walk forwards, and keep taking chunks like that until you've found PAGE_SIZE

robert-stuttaford06:09:06

if you're modelling a new system with an empty db, then you can save an ever-decrementing value with all the things you want to walk backwards in this way, because then you can take advantage of d/datoms to walk that index in sorted (incrementing) order -- giving you a naturally reversed index to traverse

robert-stuttaford06:09:17

happy to discuss in more detail, @magnars, because it's an interesting problem that i'm wide open to solving better 🙂

magnars06:09:16

ah, that's an interesting solution. I'll give that a shot. Thanks again, Robert. 🙂

robert-stuttaford06:09:37

i'd love to hear how it goes, if you're amenable!

magnars06:09:54

I'll let you know. 🙂

drankard09:09:14

@jared i now se logs in S3, but no transactor metrics in Cloudwatch. There must be missing a step or something in the documentation. The only thing i can think of is that the ensure-transactor is calling out to aws and setting something up ? Im not using ec2, i have 'on premis' cassandra storage and a transactor running with: aws-s3-log-bucket-id=ice-dev-transactor aws-cloudwatch-region=region=eu-west-1 aws-cloudwatch-dimension-value=ice-dev-transactor The Policies are set up as documentet (PutObject and PutMetricData,PutMetricDataBatch) in Security Credentials->Policies I se the logs in S3 and i se the S3 metrics in Cloudwatch (PutMetricData and BucketSizeBytes) but no transactor metrics.

jaret12:09:27

@drankard do you see a HeartbeatMsec metric in the transactor logs? or any of the metrics IN the transactor logs?

jaret12:09:19

To see your Transactor's logs: Go to the S3 console Select your log bucket (see the transactor properties file output by the bin/datomic ensure-transactor command, it contains the bucket name) Drill down in the directory hierarchy to find .zip'd log files

drankard12:09:26

nop no metrics

drankard12:09:16

im unable to run ensure-transactor but added the properties manually

drankard12:09:31

ensure-transactor cassandra-transactor.properties cassandra-transactor.properties cassandra-transactor.properties.ensured java.lang.IllegalArgumentException: No method in multimethod 'ensure-transactor*' for dispatch value: :cass at clojure.lang.MultiFn.getFn(MultiFn.java:156) at clojure.lang.MultiFn.invoke(MultiFn.java:229) ...

jaret13:09:06

If you do not have metrics in your transactor logs then you wont see any metrics in Cloudwatch. It indicates to me that your transactor is not up. Additionally wherever (environment) your transactor is running it needs AWS access keys. Those access keys have to be for the user you have granted permissions for in AWS.

magnars16:09:45

@robert-stuttaford Using an indexed attribute with a declining value to find the x last entities seems to be working fine. 🙂

(defn next-chat-event-id [db]
  (if-let [datom (first (d/datoms db :avet :chat-event/id))]
    (dec (nth datom 2))
    0))

(defn get-recent-events [db num]
  (->> (d/datoms db :avet :chat-event/id)
       (take num)
       (map (fn [[eid _ _ tx]]
              (get-event (d/entity db eid)
                         (:db/txInstant (d/entity db tx)))))))

magnars16:09:47

(d/datoms) is powerful stuff!

jdkealy16:09:19

does anyone else find converting dates to instants to be a little tedious? Coming from using ORM's, I'm used to passing a string value and having things be converted automatically. The case I have is a datepicker component from the frontend, and saving its value in datomic. I'm tempted to just store it as a timestamp and have the type be a long... Since I will be living with this decision for years to come, is this a bad idea ?

magnars16:09:43

Is your datepicker on the frontend sending longs, tho? Wouldn't you have to convert the string either way?

jdkealy16:09:02

well... converting is more convenient at the datepicker level than in some backend function that updates the database -- which may or may not have the date attribute passed to it

jdkealy16:09:57

so.. yes in javascript, before the the update API call is made, convert it to a long, and parse from long when setting its value

magnars16:09:23

I think I would go for the proper data type in the db over a little convenience.

jdkealy16:09:40

cool thanks for the advice. I'll go for the proper data type then 🙂

robert-stuttaford16:09:21

fantastic, @magnars 🙂 i'm working on a tx-by-tx rebuild of our prod database ( north of 40mil txes so far ), and i'm definitely going to include a decrementing index with the new data where necessary

magnars16:09:29

oh man! Are you worried about the 10 billion datom limit at all?

robert-stuttaford16:09:55

i am, a little. we need to make another ~165 copies of our current database to reach that

robert-stuttaford16:09:12

this is why i want a codebase that can rebuild the db, given rules for each transaction shape

robert-stuttaford16:09:18

so that we can shard things later on

magnars16:09:30

Aye, makes sense.

robert-stuttaford16:09:05

admittedly, a lot of the data in our db right now is trash. we made so many beginner mistakes in the first couple years 🙈

robert-stuttaford16:09:17

which is another reason for the rebuild

magnars16:09:25

Haha, I bet you're not alone in that.

robert-stuttaford16:09:42

indeed 🙂

jdkealy17:09:24

i've been storing cookies in my datomic DB.. i guess it's time to rethink that

pesterhazy17:09:52

a selective database rebuild would be incredibly useful, for almost every production user of datomic

jdkealy18:09:26

what's the preferred way to unparse the instants... i've been so confused about these different date classes, like joda vs clojure.instant... Is there a simple way to take the instant and return "yyyy-MM-dd"... clj-time seems to want a joda instance, so do you convert from instant to joda, then use clj-time on it ?

robert-stuttaford18:09:17

clj-time.coerce/from-date clj-time.coerce/to-date gets you 80% there

jdkealy18:09:03

perfect, exactly what i was looking for 🙂

domkm20:09:51

I ran into an error with fulltext search. (fulltext $ :recall/search-text ?search) errors when ?search is RAGE3:10". The " in the search criteria is the culprit. Is this a bug?