Fork me on GitHub
#datomic
<
2017-01-22
>
dominicm11:01:07

There is a importer of git to datomic, by the datomic team, might be of interest

Ethan Miller11:01:40

@dominicm interesting. do you know where I can find some info about it?

Ethan Miller14:01:33

Wow. What are the use cases for this kind of thing?

witek16:01:33

Hi, I have a conceptual question. When I want to change an attribute, why do I have to db/retract and then db/add? Why isn't there a db/set?

pesterhazy17:01:10

you don't have to retract for :db.cardinality/one attribute

pesterhazy17:01:25

if you add a new value, the old value is automatically retracted

zalky18:01:37

Anyone have any idea what would be the most efficient way to return, given a cardinality many attribute, only those entities for which there is actually more than one value?

zalky18:01:22

The best I can think of is something like:

(d/q '[:find ?a ?b
       :where
       [?a :some/attribute ?b]
       [?a :some/attribute ?c]
       [(not= ?b ?c)]]
     db)
Is there anything better?

pesterhazy18:01:38

use (count) and filter out those a count less than two?

Ethan Miller19:01:22

When I try to run ensure-transactor with an aws ddb configuration like this datomic ensure-transactor transactor.properties ensured-transactor.properties, I get a bizarre error: > java.io.FileNotFoundException: transactor.properties (No such file or directory)

Ethan Miller19:01:47

I even opened permissions on the config file all the way up in case for some reason this was a permissions problem.

marshall19:01:42

@ezmiller77 you need to run it from the root of the Datomic distribution. So something like bin/datomic ensure-transactor config/my.properties

Ethan Miller19:01:45

Huh, interesting. That connects to a larger confusion i Have about where datomic is supposed to sit in my system.

marshall19:01:53

Where config/my.properties is the path to your actual properties file

Ethan Miller19:01:12

So when running datomic commands they need to be run from within the root directory of the unzipped package?

marshall19:01:22

For the most part anyway

Ethan Miller19:01:45

Does that include commands that run, say, a datomic transactor?

marshall19:01:52

Transactor is usually started via bin/transactor -yourJVMFlags path/to/properties.file

Ethan Miller19:01:42

Do most people place a unique instance of datomic then directly in their project path?

marshall19:01:57

None of this is strictly required, bit the included startup scripts that do things like set your classpath and such more or less assume that is what you're doing

marshall19:01:53

Not really. You only need the distribution for running the transactor. You use the peer or client dependency via maven or lein or whatever for your actual app

Ethan Miller19:01:10

Huh. Okay. I’m still trying to understand all these difference “services” if that’s the right word.

Ethan Miller19:01:01

I’m just trying to build out a little project that would serve “notes”, like research notes, from a datomic server. Can you suggest which service I should try to use?

Ethan Miller19:01:38

I was under the impression that in order to setup a peer server that used the file system locally, I need to configure a transactor, and then it seemed necessary to configure a storage service as well…

marshall19:01:20

If you want to use clients then yes you'll need a transactor and a peer server

marshall19:01:51

If you're running it all locally you can use the dev storage protocol for local disk storage

Ethan Miller19:01:14

“clients” here refers to what exactly?

marshall19:01:16

Dev isn't supported for production use, but is fine for running locally

marshall19:01:26

The Datomic Client library

marshall19:01:38

As opposed to the Datomic Peer library

Ethan Miller19:01:14

Right okay. Saw that doc.

val_waeselynck20:01:48

@ezmiller77 regarding your tech choice question: I would advise you not to expect too much of the 'time travel' features of Datomic as a tool for versioning. In my experience, you usually don't want to query history in your application code, because you have very little control over history (example: imagine you want to import several dated revisions of a document at once. You could not do that if the way you store revisions is using Datomic history).

val_waeselynck20:01:57

If I were you, I'd just store all the revisions on cheap storage like s3. If you really need to save space, you could store a base document along with a list of edits, and recombine them using a diffing library.

val_waeselynck20:01:39

In summary: there are tons of very good reasons for choosing Datomic (one of them being that it's fundamentally more sound than most alternatives) but your versioning use case isn't one IMHO.

val_waeselynck21:01:08

@zalky if you're after efficiency, you'll probably want to perform an AEVT traversal instead of a Datalog query

val_waeselynck21:01:16

(->> (d/datoms db :aevt :my/attribute) seq (partition-by :e) (filter #(> (count %) 1)) (map #(-> % first :e)))

val_waeselynck21:01:21

(Have not tried this code, but you get the idea)

Ethan Miller21:01:55

@val_waeselynck I less interested in loading multiple version of a document at once, than I am in having access to those versions, and perhaps also in the diff that I think datomic provides…

Ethan Miller21:01:02

I’m interested, though, in what you mean by one having “very little control over history”. Is it not possible to query the revision history of an entity?

Ethan Miller21:01:53

In any case, what’s proving most frustrating about datomic is the challenge of setup and the inscrutability of the documentation.

Ethan Miller21:01:48

I’ve been trying to get a setup working all day with very little progress. Quite frustrating.

val_waeselynck21:01:11

Datomic does not give you diffs. You provide writes, and it gives you an aggregated view of those writes. It's the opposite from Git.

Ethan Miller21:01:23

@val_waeselynck huh. Well I hope to have a sense of what you mean when I can get it running.

val_waeselynck21:01:14

@ezmiller77 what kind of setup are you aiming for exactly? Setting up Datomic for production can be a bit of a hassle, but setting up for development should not be too hard

val_waeselynck21:01:37

You can even start coding with just the in-memory storage

Ethan Miller22:01:52

Yeah, the only thing I’ve been able to get running is the in memory peer-server.

Ethan Miller22:01:37

I think the main thing that may be confusing me is the variety of services and combinations thereof, combined with a confusion about what a basic (even dev) project setup might look like in, say, leinigen with clojure (which is what I’m working with).

Ethan Miller22:01:22

What I think I understand is that one runs some sort of datomic database using a peer-server or a client, and that process runs outside the project. Then, depending on what kind of db process one runs, you connect to it/interface with it in different ways.

zalky22:01:23

@val_waeselynck , that's an interesting approach, I've never really tried traversing the index directly that way. I will give that a shot thanks!

Ethan Miller22:01:42

I tried to get a dynamo-db local service running by: 1) installing dynamodb-local, 2) running ensure-transactor on a properties file setup for ddb-local, and then 3) running a transactor on that properties file. But then I started getting an error “Unable to load license key”.

Ethan Miller22:01:25

That’s as far as I was able to get, and had to waste a bunch of time just to get there because it wasn’t apparent that the ensure-transactor needed to be run in the datomic root.

marshall22:01:09

@ezmiller77 for dev work and exploration I'd definitely suggest using a dev transactor. Put your starter license key in the dev transactor properties sample included under config/samples and start the transactor with something like bin/transactor config/samples/dev.properties

marshall22:01:13

Then you can connect via peer with a uri like: datomic:<dev://localhost:4334/mydbname>

marshall22:01:35

You could also start a peer server against the dev transactor with the approach here: http://docs.datomic.com/peer-server.html

marshall22:01:57

Details on starting dev transactor can be found here http://docs.datomic.com/run-transactor.html

Ethan Miller22:01:54

@marshall i will give it a shot. i guess i shied away from dev transactor for not knowing what that really means. what is it using as storage etc… but at this point if i can get up and running that’d be great.

marshall22:01:24

It uses an internal H2 db for local disk storage

marshall22:01:39

So it is persistent on your hard drive

marshall22:01:28

Not for production work, but totally appropriate for development work. You can also backup and restore to and from dev just like any other storage

Ethan Miller22:01:35

@marshall when i run with that file i also get this “Unable to load license key” message.

marshall22:01:15

Did you put your Starter license key in the file?

Ethan Miller22:01:30

I put the key there that I have from my http://my.datomic.com page.

marshall22:01:46

Hrm. Directly copied from the email?

marshall22:01:53

Should be like 6 lines

marshall22:01:13

Yeah the license key comes via email

marshall22:01:31

The key on the site is a download key for getting the distro via curl

Ethan Miller22:01:55

That must be it. Thanks!

marshall22:01:17

Nah, lots of keys and creds. You're definitely not the first to miss it

Ethan Miller22:01:47

Nevertheless, much appreciated.

Ethan Miller22:01:01

@marshall so I’ve run the transactor. the next step I think is to run a peer-server. i tried that using this: bin/run -m datomic.peer-server -p 8998 -a datemo,datemo -d datemo,datomic:. Now I get an error regarding the db itself I think: > Exception in thread "main" java.lang.RuntimeException: Could not find datemo in catalog

marshall22:01:58

Yeah, you need to create the db from a peer first

marshall22:01:11

Run bin/repl

Ethan Miller22:01:16

But isn’t this server the peer?

Ethan Miller22:01:23

I mean this peer-server?

marshall22:01:33

Heh. This is admittedly a bit confusing

marshall22:01:53

Peer server can only serve existing databases

marshall22:01:01

It doesn't create them

Ethan Miller22:01:03

That makes sense I guess.

marshall22:01:11

You should run bin/repl

Ethan Miller22:01:29

ahhh so the repl is kind of like running mysql...

Ethan Miller22:01:44

and the manually creating a db

marshall22:01:54

Then:

(require '[datomic.api :as d])
(d/create-database "datomic:)

marshall22:01:42

after that you should be able to close the repl and connect your peer server

Ethan Miller22:01:00

Much much thanks!

marshall22:01:34

I'll see about improving the docs around this stuff

Ethan Miller22:01:53

I might write up a blog post or something to sum up this process, maybe that can help fill in some of the blanks. I find it’s usually just missing steps especially where stuff isn’t intuitive. In this case, I think the potential for confusing is higher because there are so many different pieces (by design of course).

Ethan Miller22:01:03

I watched the day-of-datomic training videos and one had this picture with the db services all decomposed, and that looked cool on a high level, but I realized that that was also why I hadn’t gotten anything running yet.

marshall22:01:58

Well it also doesn't help that those videos were from before clients and peer server existed:)

Ethan Miller22:01:09

Yes, that too.

Ethan Miller22:01:11

Yeah, I partly got into this mess because I wanted to be able to play with the console to get a feel for the query language and the structure of the db visually. But discovered that that wouldnt work with the in memory db peer-server.

marshall22:01:08

Right, console requires a storage backed db

Ethan Miller23:01:05

@val_waeselynck is there an easy test that you can think of whereby I could see the limitation vv history to which you were referring?

val_waeselynck23:01:05

Try to edit history in any way :)

Ethan Miller23:01:45

I’m not sure what that means…

Ethan Miller23:01:47

I thought that in datomic you don’t edit history because you accumulate new information.

val_waeselynck23:01:41

Sure, but that's not really the same as having several versions of a document.

val_waeselynck23:01:54

But the fact that history cannot be edited is the reason that it should not be used to implement first class notions of your domain model

val_waeselynck23:01:30

If that's not clear, you should go ahead and try implementing versions using history, the limitations of that approach will soon become evident to you

Ethan Miller23:01:20

Well. That sounds ominous!

val_waeselynck23:01:44

One situation where querying pasr dbs is especially problematic is when you need to evolve your schema

Ethan Miller23:01:47

Admittedly, I can’t really follow the meaning of what you wrote. I think I’m not keyed into the lingo well enough yet. So that “first class notions of your domain model” sounds like a foreign language for the most part.

val_waeselynck23:01:45

There has to be a mailing list thread where someone has expressed this more clearly :)

Ethan Miller23:01:57

My assumption/hope was that I could simply provide text (which would just be markdown) and then be able to rewind somehow to see previous versions.

Ethan Miller23:01:42

And have the power of a db to be able to query at the same time. Git gives me much of what I need, but it’s not queryable really.

Ethan Miller23:01:15

My understanding was that with datomic you can go back in time, and also query a history.

Ethan Miller23:01:47

Something perhaps like this example in the tutorial code in the docs:

(require '[clojure.pprint :as pp])
(def db-hist (client/history db))
(->> (<!! (client/q
           conn
           {:query '[:find ?tx ?sku ?val ?op
                     :where
                     [?inv :inv/count ?val ?tx ?op]
                     [?inv :inv/sku ?sku]]
            :args [db-hist]}))
     (sort-by first)
     (pp/pprint))
=> ([13194139534399 "SKU-21" 7 true]
    [13194139534399 "SKU-42" 100 true]
    [13194139534399 "SKU-22" 7 true]
    [13194139534400 "SKU-22" 7 false]
    [13194139534402 "SKU-42" 1000 true]
    [13194139534402 "SKU-42" 100 false])

val_waeselynck23:01:38

The way to do that would be to simply store all the versions as different entities with a timestamp attribute