Fork me on GitHub
#datomic
<
2021-08-04
>
zendevil.eth11:08:10

what is the classpath of datomic.api peer library in the datomic full distribution? I’m running using clj a file that requires datomic.api inside the datomic distribution but it isn’t picking up datomic.api, while bin/repl does pick up datomic.api

favila15:08:33

run bin/classpath It’s basically lib/* and the two datomic-pro*.jars in the root

zendevil.eth15:08:19

this is my deps.edn

zendevil.eth15:08:30

1   {:paths ["./"
  1          "lib/*"
  2          "target/classes"
  3          "build/src"
  4          "bin"
  5          "src/clj"
  6          "test/src"
  7          "samples/clj"]}

zendevil.eth15:08:11

and when I run clj create_db.clj in the same directory as the deps.edn which is the root of the datomic distribution, it says:

Could not locate datomic/api__init.class, datomic/api.clj or datomic/api.cljc on classpath.

favila15:08:02

The jvm doesn’t include a jar on the classpath unless explicitly referenced or you use “*”

favila15:08:20

"./" is not actually including the datomic jars

favila15:08:25

$ bin/classpath
resources:datomic-transactor-pro-1.0.6269.jar:lib/*:samples/clj:bin:

favila15:08:11

I take back what I said, I thought this included the datomic-pro jar

favila15:08:51

you need datomic-pro-1.0.<whatever>.jar and every jar in lib, and whatever storage driver

favila15:08:21

and these are probably not deps.edn paths. Have you considered just invoking with an explicit classpath?

zendevil.eth15:08:57

how do you do that?

favila15:08:23

clojure -Scp ...

favila16:08:32

or just java -cp … clojure.main the-script.clj to avoid clj completely

zendevil.eth16:08:26

actually using “*” in deps works splendidly

uwo20:08:13

Quick Q: If you retract a datom with a noHistory attribute, will that data eventually be tossed? (I understand that noHistory is not intended to make semantic guarantees.) Put another way, if an attribute is not "high-churn", but it is eventually retracted, will noHistory eventually clean that up? (Tangentially, also I understand that Datomic is not the place to put operational data.)

favila12:08:57

I don’t think already-indexed datoms are ever removed except by excision

favila12:08:51

I don’t have inside knowledge here, but I suspect what noHistory means is literally “when updating the index with unindexed datoms from the tx log, don’t write any of these datoms to the history index, just the current (assertion-only) index.”

uwo19:08:39

Thanks Favila!

uwo19:08:22

so to be explicit, the guess here is that a retraction would not be enough to trigger the behavior of noHistory?

favila19:08:21

I wouldn’t count on a retraction retroactively rewriting (i.e. removing entry from) history indexes for an attribute. Maybe it does, maybe it only updates those portions of the index that are “dirty”, maybe it ignores the history index in code and doesn’t rewrite old indexes, donno.

uwo19:08:16

Gotcha. Thanks again, Favila.

favila19:08:36

I will say, if an attribute has always been nohistory, then obviously no history index is going to contain those datoms. I’m thinking mostly of cases where you might be turning nohistory on and off over time.

favila19:08:19

In that case, any history you see via d/datoms (or whatever query) for that attribute is coming from the memlog, not from the indexes.

favila19:08:42

and it will “disappear” once there’s a reindex

uwo20:08:20

so are we hedging toward noHistory+retraction eventually reclaiming space then?

favila20:08:22

In the case of “always been noHistory”? Not really. It’s not reclaiming space, it’s just not writing history (i.e. the space was never used).

uwo20:08:46

This is coming up because my team is considering writing high-volume metadata that is accrete-only, i.e. the individual values are never updated. Their hope was that if the values are eventually retracted then it would be less of a capacity planning issue. (They are aware that datomic is not a good fit for operational data, but there is time-pressure.)

uwo20:08:00

I guess my question there is, is noHistory+retraction really doing anything for us, since these values are only written once. My suspicion is no, but I thought I'd ask.

favila21:08:36

No, that would help in this way: when a value is retracted, the total datom count of the database will eventually (after the next index) reduce by one instead of increase by one.

favila21:08:16

I’m wondering if you don’t have a mental model of what’s actually written to indexes and how nohistory influences that. Maybe this will help? https://tonsky.me/blog/unofficial-guide-to-datomic-internals/

favila21:08:44

“noHistory” means “write datoms to the ‘current (now, effective-assertions-only)’ sub-index, not to the history (retractions and old no-longer-in-effect assertions) sub-index”. Normally datoms get written to both, and a complete history is reconstructed by merging them together. Nohistory only writes assertions to the “current” sub-index.

favila21:08:32

So when retracting the assertion and retraction datom in effect “disappear” because they never get written to the history index

uwo22:08:22

Groovy. Thanks a ton Favila

ghadi22:08:06

noHistory != update-in-place

uwo03:08:59

@U050ECB92 absolutely. I don't think we were thinking of it that way.

naomarik23:08:14

There a way to use pull syntax on an entity if it's a ref many within a hetero tuple? Instead of returning something like this: :space/included-addons [[17592186049388 0 1]] the entity would be resolved out the the fields I'd like to select. Result of this {:space/included-addons [*]}

1. Unhandled datomic.impl.Exceptions$IllegalArgumentExceptionInfo
   :db.error/invalid-lookup-ref Invalid list form: [17592186049388 0 1]
   {:cognitect.anomalies/category :cognitect.anomalies/incorrect,
    :cognitect.anomalies/message "Invalid list form: [17592186049388 0 1]",
    :db/error :db.error/invalid-lookup-ref}