Fork me on GitHub
#datomic
<
2017-01-30
>
robert-stuttaford05:01:21

@ezmiller77 yes. define two attributes 🙂

chrisetheridge11:01:40

could someone help me with altering the :db/fulltext attribute, please?

chrisetheridge11:01:44

i have this tx:

chrisetheridge11:01:53

[{:db/id               :content-item/description
                  :db/fulltext         true
                  :db.alter/_attribute :db.part/db}]

chrisetheridge11:01:59

and datomic gives me an error saying:

chrisetheridge11:01:04

:db.error/invalid-alter-attribute Error: {:db/error
   :db.error/unsupported-alter-schema, :attribute :db/fulltext, :from
   :disabled, :to true}

chrisetheridge11:01:31

you cannot alter fulltext

jaret14:01:02

@nooga That is essentially correct. With a starter license you will get 1 year of maintenance and upgrades. After which, you will still be able to use your license/Datomic but will not be able to upgrade to any new releases.

jaret14:01:02

@biscuitpants :db/fulltext cannot be altered. So you cannot add full text search to an attribute after its creation. You will need to create a new attribute, pouring in or importing values from the old attribute. http://docs.datomic.com/schema.html

jaret14:01:13

Oh looks like you realized that

chrisetheridge14:01:53

yes, thank you @jaret! maybe the docs should be updated to give an example of how to do it?

chrisetheridge14:01:59

(its not a super straightforward process)

chrisetheridge14:01:14

or, it is, but its not completely evident 🙂

pesterhazy14:01:37

you do, however, have a disconcertingly robert-stuttaford-like avatar photo, @biscuitpants

chrisetheridge14:01:00

ha, we have a discussion about this often at work

pesterhazy14:01:30

even the position of the head relative to the frame

chrisetheridge14:01:45

possibility that we planned it

robert-stuttaford15:01:24

i ain’t dun nuffin

jaret15:01:11

I can't tell whose talking

robert-stuttaford16:01:40

i’m the handsome one 😁

bmaddy17:01:23

Does anyone know where I would find info on how to query for transaction annotations? My Google searching has yielded nothing (also, I’m a newbie at datomic). I’ve added an annotation to a transaction and want to write a test to ensure it added the annotation.

favila18:01:09

@bmaddy Transactions are normal entities. You need to get either a t or a tx somehow, then just access it as you would any entity

limist20:01:15

@shaun-mahood I was wondering similarly recently, how to handle geo data with/from Datomic. Besides PostGIS, I wonder if ElasticSearch could do spatial searches on properly structured Datomic data. Found anything so far?

bmaddy20:01:18

Hmm, I’ll try that out. Thanks @pesterhazy & @favila.

shaun-mahood20:01:36

@limist: I've done a very brief amount of research and it looks like it would be a lot of work to implement anything performantly - essentially, implementing r-trees and adding some sort of indexing that PostGIS and other GIS systems already have built in. I can't see it being anything easy or quick, so in all likelihood it makes more sense to keep the location info outside of datomic and figure out a good way to link things together. I've only just started looking into GIS stuff in general so I could be wildly off base on anything at this point though. 🙂

nooga20:01:53

I’m gathering data from a bunch of sensors, they send me timestamps with their readings. I can’t trust their clocks and I can’t trust they they will send their readings from in order. On the other hand, I wand these timestamps in my DB since reading times can significantly differ from acquisition times. How should I approach this?

favila20:01:40

@nooga As a normal data-modelling problem. Datomic's time/history features concern "meta" time (i.e. time of record) not data-model time. Think git commit or file-modified times.

favila20:01:19

@nooga Sometimes you can get these to line up and you can use txInstant to store all time info, but it sounds like you can't in your application

nooga20:01:35

I'd like my entities to reflect the sensor state at any given time, I can store the timestamp as an attribute there, but this doesn’t guard me from out of order scenarios

eoliphant20:01:50

hi, i need to model a ‘composite key’. Any best practices for that? I was thinking about using a transaction function to create the ‘real’ key from the associated fields

favila20:01:21

@nooga but that's exactly what you need, right? you need to store sensor data as a domain time so you don't confuse it with the time you recorded the sensor data

favila20:01:34

it may be wrong, out of order, corrected in the future, etc

eoliphant20:01:54

thanks ! @favila also, you helped me with a tree waking approach a while back over in google groups. I’m trying to generalize it to search ‘up and down’, but i’m having some issues, I realize what I did wrong in what I posted over there, but still not sure how to make it work lol

favila20:01:20

@nooga if you want to use datomic time-travel abilities to explore domain time, consider using two dbs: one for raw data, and a projected one where you transact sensor data in domain-time order (asserting txInstant on each tx)

favila20:01:07

@eoliphant the "ancestors" query is inefficient when working backwards, think about what var is bound when evaluating the recursive rule with your change and what the query eval would have to do to fix it

favila20:01:53

@eoliphant you can use the [?var] syntax to force a var to be bound, so you won't let the query engine try to eval rules in the "wrong" direction

eoliphant20:01:07

yeah, that’s that I’d kind of figured out. Will try tweaking

favila20:01:45

(in fact ancestors is badly named without it, since rules are inherently bidirectional)

eoliphant20:01:00

yeah that’s whhere I messed up initially, I just tried to be cute and switch the internal call, but the [?var] wasn’t switched

eoliphant20:01:08

so it went all over the place lol

nooga20:01:12

I guess I can ask datomic about values changed between T1 and T2 and then sort results by domain time?

eoliphant20:01:15

ok i think i know what I need to do

favila20:01:49

@eoliphant try to write a blood-relations rule with three impls

favila20:01:26

one that handles parent<->child (in any direction), one that follows ancestors, and one that follows progeny

favila20:01:45

use forced-binding on the latter two

favila20:01:46

@nooga depends on what you are after. If you just want to sort all sensor data by the time it reported, a simple indexed date field is enough

favila20:01:28

in what way does it matter that the sensor time is not trustworthy? you want to rely on transactor's time? you want to correct sensor time in a second pass?

favila20:01:13

Do you want to use datomic as-of/since to explore a reconstructed view of the world as seen by the sensors? or you don't care about that and have some application on top which does it? or you just want to store it, not process/query at all

nooga21:01:05

The sensors sometimes don’t have realtime clocks on board, somethimes they don’t have any NTP configured etc. I might want to prefer transactor time over sensor time in some cases.

nooga21:01:22

My idea was to simply update facts about sensors rather than treat each observation as a new entity related to the sensor.

nooga21:01:32

domain time is another reading, in some sense

nooga21:01:50

My transactions would look like this: [sensorid :reading/time 123456] [sensorid :reading/temp 19.5] [sensorid :reading/pressure 1012]

nooga21:01:41

this would make me use as-of/since to see the history

favila21:01:27

You could do that, but keep in mind the history you see is always the history sensor data was recorded, not time it happened in the world

favila21:01:11

This is why I suggested generating a second db as a projection, where :reading/time is corrected and used as the txInstant

favila21:01:53

without each observation as a separate entity, you can't correct old readings

favila21:01:08

if you don't care about that, it's fine to just use tx history

nooga21:01:15

this is mostly for acquisition

nooga21:01:54

reading will be less frequent and heavier with computation

favila21:01:30

keep in mind if you go this route, as-of/since is really just useful to debug your applications (e.g. whatever collects sensor data), not so much to scrub through old sensor readings

favila21:01:06

by confusing transaction time and reading-time you may make your life harder later (depending on what you plan to do)

favila21:01:42

but it is still possible to use the history database and pull out the individual observations if you need

favila21:01:32

also keep in mind that you may need to batch writes for performance, which will further confuse the sensor-time vs. record-time

favila21:01:54

(datomic is designed to scales reads, not writes)

nooga21:01:52

@favila then maybe I should reconsider using datomic for this task

favila21:01:15

@nooga Likely there are better sensor data db packages out there. It may still be useful to create a projection in datomic, as I said

favila21:01:31

what is the scale you are looking at?

nooga22:01:02

it’s an open source, community driven air quality aggregator

nooga22:01:44

so, depending on the success, we may have thousands of facts incoming per minute

nooga22:01:51

I was looking at datomic because its fact orientation and time travel abilities seem really nice for this case

nooga22:01:06

we’re collecting facts about the world, from multiple sources, without making any hard assumptions about their validity and reliability

nooga22:01:30

and then, we want to view the state of the world at any given time or time range

favila22:01:10

I'm not sure how you would do this at scale with a single storage technology

favila22:01:06

seems like you need a write silo (kafka queue, time-series db, etc) to store observations, and an aggregation silo for reads and exploration

nooga22:01:25

oh yes, I was placing datomic on the aggregation end

favila22:01:51

That would still work for near-real-time but I would use datomic to make throwaway projections

favila22:01:34

depends on how much processing occurs from writes to aggregation

favila22:01:38

I have more experience with heterogenous data (records, documents, human artifacts, etc) not homogenous data so I can't give much more advice

favila22:01:19

Datomic is a life-changer for that kind of data, not so sure about time-series data

favila22:01:10

there were many long discussions about people trying to use datomic for time-series data (financial mostly) on the google group. Perhaps that would help

nooga22:01:46

I used datomic for graph models coming from web scraping/mapping

nooga22:01:50

and it was awesome

wei23:01:39

using a heroku postgres backend, and my (d/create-database ..) call is taking a long time to run (on the order of 2-3 minutes). after connecting, querying is relatively fast. any suggestions for pinpointing the bottleneck for the initial connection?