Fork me on GitHub
#datomic
<
2017-04-27
>
dimovich09:04:54

hello fellow clojurians

dimovich09:04:03

what would be a good place to start learning about datomic?

dimovich09:04:10

some tutorial, or maybe a specific book?

deg14:04:26

What is the cheapest/smallest AWS EC2 instance type that can run Datomic? (This is for tiny toy apps; my data size is negligible).

nottmey14:04:13

@deg my dev instance runs just fine on a free tier ec2 instance with 1gb ram. Using bin/transactor -Xmx512m -Xms512m mytransactor.properties and I think I changed the following properties:

memory-index-threshold=32m
memory-index-max=256m
object-cache-max=128m
Runs for a few months now, so it should be ok. ¯\(ツ)

deg14:04:19

@nottmey Great, thanks. I had tried 0.5gb (the smallest lightsail tier) and EOM'd at startup. I then found https://github.com/WormBase/db-prototypes/wiki/Installing-Datomic, which suggested that much more was needed, and kindof freaked. But 1gb is not too bad. Means that I can't play for free, but still a pretty reasonable cost.

deg14:04:39

Btw, are you running with Docker, or bare on the machine? My current plan is to run with Docker, which I imagine will add a tiny bit more overhead too.

nottmey14:04:57

Not using docker for this one, but some other processes are also running. Since the DB process itself only takes 54% of the memory there is some room for other processes (e.g. the datomic console - java process - with 16%). Maybe scale down my numbers a bit and try the 0.5gb again.

deg14:04:42

Will do, thanks.

jeff.terrell14:04:24

@deg - I remember having the same experience at some point. But it's been a while since I tried. Maybe a more recent version of Datomic can work with lower memory requirements? I don't know.

deg14:04:06

Thanks. I'm playing with a bunch of moving pieces now (mostly https://github.com/pointslope/docker-datomic-example), and a pile of outside world interrupts. But, I'll report back when I figure out what is the smallest size that works.

deg14:04:46

Hmm. Is there any way to specify the memory usage in the .properties file, rather than on the cmdline?

deg14:04:45

The tooling that I'm trying (https://github.com/pointslope/docker-datomic) has the call to bin/transactor hardcoded with no parameters. Changing it will mean forking two repos and maybe adding images to the Docker cloud; actions I'd just as soon avoid for this simple playing around.

mgrbyte15:04:29

@deg You can set the mem options via environment variables (e.g XMX)

deg15:04:51

cool, thanks!

deg15:04:37

No, actually, not so cool. The environment is in the docker image, so that still requires forking the repos.

mgrbyte15:04:20

@deg the Dockerfile in the repo you link to above is quite small. There may be other things in there you want to change, perhaps rolling your own is not too bad?

deg15:04:36

Agreed. I almost certainly will do so soon. But, had wanted to play quickly before I had to relearn dockerfile syntax. Ah well; I'm just trying too hard to avoid work. 🙂

mgrbyte15:04:00

Note that in their Dockerfile, at the bottom, apparently you can augment the ENTRYPOINT command (Provide a CMD argument with the relative path to the # transactor.properties file it will supplement the ENTRYPOINT)

karol.adamiec15:04:57

@deg do not know your use case but fiddling with datomic inside docker seems like making it hard. You mentioned you want to play. is memory/fileseystem local thing not enough? and if you need to deploy is official way on AWS prohibitely expensive? it is a latte once a week no??

deg15:04:50

🙂 All good points. Mostly trying to use Docker out of philosophy (not polluting the machine; reproducibility of porting to new envs; etc.). Then, I descended down a yak-shaving path here, wanting to test if I could run on a smaller machine.

karol.adamiec15:04:57

i do have docker for my peers (jvm clients for datomic). Datomic transactor itself is vanilla one on AWS (not vanilla, terraformed one but still close to the metal)

karol.adamiec15:04:06

m3.medium works absolutely fine

karol.adamiec15:04:18

$0.073 per Hour

deg15:04:56

I have decided to let the yak roam free. Thank you for that wake-up call.

karol.adamiec15:04:08

just remember than when yak is satisfied and you have to scale production workloads it will be more messy to get help/diagnose etc when transactor is inside docker. i see all the time docker/transactor related stuff here 😄

slpssm18:04:11

Hi, I’m running into this error:

:db.error/not-an-entity Unable to resolve entity: {:part \"db.part/user\", :idx -1001669} in datom [{:part \"db.part/user\", :idx -1001669} :field/docId \"44d3a3b6670dfc39bef95993\"]
and I’m not sure how to fix it. Using version 0.9.5302 of datomic. (Yes, it’s a bit old)

slpssm19:04:10

The :db/id specification looks like: :db/id #db/id[:db.part/user] if that helps.

marshall19:04:38

try replacing the reader macro with a call to tempid: :db/id (d/tempid :db.part/user)

slpssm19:04:17

Same error. The :idx value is different.

marshall19:04:37

can you share the entire transaction

slpssm19:04:54

Sure:

[{:data/docId "44d3a3b6670dfc39bef95993"
                                                :data/state "disabled"
                                                :data/repo "1e9b"
                                                :data/dimensions "728x90"
                                                :data/name "name"
                                                :db/id (d/tempid :db.part/user)
                                                }]
I should mention that this is massaged a bit and expands like:
[({:data/docId "44d3a3b6670dfc39bef95993", :data/state "disabled", :data/repo "1e9b", :data/dimensions "728x90", :data/name "name", :db/id {:part "db.part/user", :idx -1001669}})]

marshall19:04:30

you do the expansion to the nested :db/id map?

slpssm19:04:07

I shouldn’t?

marshall19:04:35

what happens if you submit first thing as the tx data?

slpssm19:04:51

Easier said than done. 🙂

wei19:04:21

anyone have ideas for representing a spreadsheet (i.e. tabular data) in datomic? main functions I want to support are looking up rows by column value, and updating a spreadsheet entity with a new one (with rows potentially added and deleted)

slpssm19:04:59

@marshall That gives me a good clue. Let me go poke at this a bit more. Thanks!

bmaddy19:04:12

The Datomic docs say to “prefer a collection binding in a parameterized query over running multiple queries” (http://docs.datomic.com/best-practices.html#collections-as-inputs). Does anyone know how a collection binding compares to a logical or like this?

(d/q '[:find (pull ?a [:artist/name])
       :in $
       :where [?a :artist/country ?country]
              (or [?country :country/name "Canada"]
                  [?country :country/name "Japan"])]
     db)

bmaddy19:04:31

Also, how would those relate to using ground?

(d/q '[:find (pull ?a [:artist/name])
       :in $ [?c ...]
       :where [?a :artist/country ?country]
              [?country :country/name ?c]
              [(ground ["Canada" "Japan"])] [?c ...]]
     db)

favila20:04:53

@bmdaddy The point of that advice is that query plans can be reused if they don't change. If you inline dynamic parameters into the query, a query plan needs to be rebuilt each time. if you supply them as input to an unchanging query, the query plan is reused and the input changes

favila20:04:24

if your collection is truly a static part of the query itself, then inlining it is fine

favila20:04:53

I don't know if there's any measurable difference between ground and or in your example

Drew Verlee20:04:24

This might sound wanky, but are their any automation tools that leverage Datomic out there? I’m just starting to learn ansible and i feel like its super brittle. Like grouping things together and searching for things is a huge afterthought. I have the same feeling about Artifactory…

bmaddy20:04:26

Cool, thanks @favila.