Fork me on GitHub
#xtdb
<
2021-09-23
>
metasoarous05:09:34

Hi folks! I've been looking through the docs but haven't been able to figure out whether it's possible to query against the valid time (or transaction time, for that matter). To be clear, I'm not talking about querying against a snapshot of the db at a particular tx/valid time, but rather against it, like you can with tx times in Datomic. Is this something xtdb can do?

metasoarous05:09:59

For a bit of added context, what I'm interested in is putting together timeseries data, using valid time. In Datomic, if I was okay with using transaction time, I could query against the tx value in the datoms, like [:find ?t ?v :where [?e :some/attr ?v ?t]]. That would give me all of the past values that :some/attr had taken on over time. While this sort of works, it's really begging for the valid time treatment.

tatut08:09:58

afaict, the tx and valid time data isn't exposed to queries

alexdavis09:09:46

Yes this isn't possible atm but is actively being worked on and something along these lines will be possible in a future release

metasoarous16:09:37

@U7KPK060K Wonderful! That's great to hear. Thanks for the response.

refset12:09:33

Hi @U05100J3V - sorry for the very delayed response(!) > afaict, the tx and valid time data isn't exposed to queries This is essentially correct, currently. However we will be exposing both possibilities in the next-gen indexes coming soon, discussed lightly in the blog https://xtdb.com/blog/dev-diary-aug-21.html#_future As a workaround in the meantime, you can access the data efficiently in userspace, e.g. https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/bench/src/xtdb/bench/ts_devices.clj#L215-L240

metasoarous18:09:44

No worries! Thanks for circling back around to it!

metasoarous18:09:37

That's great to hear that this is being worked on. Is it safe to take "over the next six months" as a rough ETA? 🙂

metasoarous18:09:28

As for the workaround, seems like xt/open-entity-history is the key there, yes? Which if I understand correctly returns a iterator of entity maps that were transacted into the db? I think this will work for now for the project I'm working on, but I'm looking forward to being able to query against this!

✔️ 1
metasoarous18:09:30

Thanks again!

🙂 1
refset18:09:40

> Is it safe to take "over the next six months" as a rough ETA? I can't promise anything, but we have lots of working code (in a private repo, for now) and good momentum at the moment so don't see why we won't hit that target 🤞

metasoarous18:09:00

Awesome! Please let me know if you need any beta testers on that ;-)

🙌 1
❤️ 1
Kris C08:09:31

I am running juxt/xtdb-in-memory:1.19.0-beta1 and have problem executing clojure code in where clause (posting via ) This works: [(str ?birth-date) ?date] but this doesn't: [(.getTime ?birth-date) ?date] Error msg: "Clause refers to unknown variable: .getTime {:pred {:pred-fn .getTime, :args [?birth-date]}, :return [:scalar ?date]}" Any clues?

Kris C09:09:18

This also doesn't work: [(java.util.Date.) ?today] Error msg: "Clause refers to unknown variable: java.util.Date. {:pred {:pred-fn java.util.Date.}, :return [:scalar ?today]}"}

tatut09:09:49

I think it may need a clojure fn, not interop

tatut09:09:39

so you can (defn time-of [d] (.getTime d)) and use it like [(some.ns/time-of ?birth-date) ?date]

Kris C09:09:00

how do I define a function in a post request?

tatut09:09:37

ah, you don't... missed that part of the context

Kris C10:09:07

Okay, I have a simple question. I am running xtdb via docker: docker run -p 3000:3000 juxt/xtdb-in-memory:1.19.0-beta1. I am invoking a query via POST on endpoint. Why doesn't this work in a where clause: [(java.util.Date.) ?today] Error msg: "Clause refers to unknown variable: java.util.Date. {:pred {:pred-fn java.util.Date.}, :return [:scalar ?today]}"}

refset13:09:38

Hi @U013100GJ14 sorry for the very delayed response(!) As per @U11SJ6Q0K's reply on the earlier thread, you can't do interop like this within Datalog (nor can use evaluate raw Clojure), and instead you will need to define a Clojure function to do the work, and it must be made available to the XT node via the classpath. This consequently means that you would need to build your own Docker container using the build module, as per https://xtdb.com/blog/xtdb-build.html - I think you could register your custom functions as a local dependency using deps.edn but I've not tried that before (and would be happy to help!)

refset13:09:55

To help during development time however, there is a small trick you can use to evaluate arbitrary Clojure: https://gist.github.com/refset/a7f86c8a3ad34aea5d20eac0b5720a28 (you would still need to add this inside your Docker container though....but perhaps we can thinking about exposing it by default in future :thinking_face: - subject to the allow-list, of course)

Kris C13:09:32

Hmm, raw clojure like =,<,str etc did work (but not nested funcitons)

Kris C13:09:22

Anyway, I have stopped doing stuff via HTTP module and am now into doing stuff in code.

👍 1
Kris C13:09:14

What bothers me here is that they are deprecating the Map<Keyword, ?> submitTx(List<List<?>> transaction); method

👀 1
refset13:09:14

yep, you can already use most of clojure.core/* but constructing completely arbitrary nested expressions is hard work / impossible

Kris C13:09:51

it bothers me, because I want an initial dataset which is the easiest and most readable to do in EDN format

Kris C13:09:23

but that is harder and cumbersome to do via the non-deprecated api...

refset13:09:28

So, you are using the Java API?

Kris C13:09:36

From Kotlin.

refset13:09:50

🙂 okay understood, I'll take a quick look at that now

Kris C13:09:58

(unfortunately, I would rather code in Clojure, but it's a no go)

Kris C13:09:44

Also, as a side note regarding the HTTP API, this example does not work with EDN: https://xtdb.com/reference/1.19.0-beta1/http.html#submit-tx

refset13:09:46

Have you tried calling the xtdb.api/submit-tx Clojure API via Java interop?

refset13:09:27

Ah, yeah sorry about that one, we have fixed it on master but not created a new release yet to make the changes obvious yet, but the new version is viewable here https://xtdb.com/reference/http.html#submit-tx

👍 1
Kris C13:09:16

yes, ::keyword is not allowed in EDN...

💯 1
Kris C13:09:57

regarding

Have you tried calling the xtdb.api/submit-tx Clojure API via Java interop?
What exactly do you mean?

Kris C13:09:15

I am now importing EDN dataset the following way:

val data = File(dataResource.toURI()).bufferedReader().readText()
        val tr = xtdbNode.submitTx(toClojure(data) as MutableList<MutableList<*>>)

Kris C13:09:40

fun toClojure(edn: String?): Any? {
        return try {
            val read = Clojure.read(edn) ?: throw IllegalArgumentException("EDN String did not produce a value!")
            read
        } catch (e: Exception) {
            logger.error("Exception while reading Clojure edn value: $edn", e)
            null
        }
    }

Kris C13:09:54

Aha, I see what you mean

Kris C13:09:14

Yeah, maybe I will do it that way.

refset13:09:55

thanks for sharing your code, my Java is not very good, so I daren't try to suggest what the solution looks like exactly but it seems to be possible in theory 🙂

Kris C13:09:18

this is kotlin code, but similar to java yes

👍 1
refset13:09:28

please let me know if it works, or not, maybe it's a terrible idea and I should avoid suggesting it in future 😅

Kris C13:09:46

no problem, thanks a lot for your input!

refset13:09:51

my pleasure. And I'm very curious about what you're up to with Kotlin! I look forward to hearing more

Kris C13:09:23

I have seen this

👍 1
Kris C13:09:34

But not sure I like it...

😄 1
Kris C13:09:59

I much prefer the queries/transactions in the EDN format, without abstractions

Kris C13:09:07

So, native...

refset13:09:34

totally fair. I also prefer the edn, though I've never written a line of Kotlin so can't really judge

Kris C13:09:03

you can check the tests and judge fast... 😉

🙂 1
Kris C13:09:49

One question regarding keywords vs strings for attributes/entity id: is it better (more performant?) to use keywords for attribute names and entity ids?

Kris C13:09:15

A quick look at java api, the XtdbDocument.put seems to favor (support) Strings only for keys?!

refset13:09:39

we recognise that keywords are mainly a Clojure thing and want to appear more JSON-oriented. There should be no meaningful performance difference between the two though, in either storage or lookup costs

Kris C13:09:02

aha, thanks

👌 1
refset13:09:15

during queries, all the joins & comparisons are based on byte encodings

Kris C13:09:31

how are you going to do this:

:xt/tx-time #inst "2020-10-16T14:24:17.025-00:00"
in JSON? 😉

Kris C13:09:05

no need to reply, just joking...

Kris C13:09:15

thanks for your help and see ya later

☺️ 1
refset14:09:14

oh gosh, I know you said you're joking...but is there a real API call like that?! I believe (and hope) that either ISO 8601 or Unix Longs can be applied to any edge cases that there may be