afaict, the tx and valid time data isn't exposed to queries

Yes this isn't possible atm but is actively being worked on and something along these lines will be possible in a future release

@U7KPK060K Wonderful! That's great to hear. Thanks for the response.

Hi @U05100J3V - sorry for the very delayed response(!) > afaict, the tx and valid time data isn't exposed to queries This is essentially correct, currently. However we will be exposing both possibilities in the next-gen indexes coming soon, discussed lightly in the blog https://xtdb.com/blog/dev-diary-aug-21.html#_future As a workaround in the meantime, you can access the data efficiently in userspace, e.g. https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/bench/src/xtdb/bench/ts_devices.clj#L215-L240

No worries! Thanks for circling back around to it!

That's great to hear that this is being worked on. Is it safe to take "over the next six months" as a rough ETA? 🙂

As for the workaround, seems like xt/open-entity-history is the key there, yes? Which if I understand correctly returns a iterator of entity maps that were transacted into the db? I think this will work for now for the project I'm working on, but I'm looking forward to being able to query against this!

Thanks again!

> Is it safe to take "over the next six months" as a rough ETA? I can't promise anything, but we have lots of working code (in a private repo, for now) and good momentum at the moment so don't see why we won't hit that target 🤞

Awesome! Please let me know if you need any beta testers on that ;-)

This also doesn't work: [(java.util.Date.) ?today] Error msg: "Clause refers to unknown variable: java.util.Date. {:pred {:pred-fn java.util.Date.}, :return [:scalar ?today]}"}

I think it may need a clojure fn, not interop

so you can (defn time-of [d] (.getTime d)) and use it like [(some.ns/time-of ?birth-date) ?date]

how do I define a function in a post request?

ah, you don't... missed that part of the context

Hi @U013100GJ14 sorry for the very delayed response(!) As per @U11SJ6Q0K's reply on the earlier thread, you can't do interop like this within Datalog (nor can use evaluate raw Clojure), and instead you will need to define a Clojure function to do the work, and it must be made available to the XT node via the classpath. This consequently means that you would need to build your own Docker container using the build module, as per https://xtdb.com/blog/xtdb-build.html - I think you could register your custom functions as a local dependency using deps.edn but I've not tried that before (and would be happy to help!)

To help during development time however, there is a small trick you can use to evaluate arbitrary Clojure: https://gist.github.com/refset/a7f86c8a3ad34aea5d20eac0b5720a28 (you would still need to add this inside your Docker container though....but perhaps we can thinking about exposing it by default in future :thinking_face: - subject to the allow-list, of course)

Hmm, raw clojure like =,<,str etc did work (but not nested funcitons)

Anyway, I have stopped doing stuff via HTTP module and am now into doing stuff in code.

What bothers me here is that they are deprecating the Map<Keyword, ?> submitTx(List<List<?>> transaction); method

yep, you can already use most of clojure.core/* but constructing completely arbitrary nested expressions is hard work / impossible

it bothers me, because I want an initial dataset which is the easiest and most readable to do in EDN format

but that is harder and cumbersome to do via the non-deprecated api...

So, you are using the Java API?

Yes.

From Kotlin.

🙂 okay understood, I'll take a quick look at that now

(unfortunately, I would rather code in Clojure, but it's a no go)

Also, as a side note regarding the HTTP API, this example does not work with EDN: https://xtdb.com/reference/1.19.0-beta1/http.html#submit-tx

Have you tried calling the xtdb.api/submit-tx Clojure API via Java interop?

Ah, yeah sorry about that one, we have fixed it on master but not created a new release yet to make the changes obvious yet, but the new version is viewable here https://xtdb.com/reference/http.html#submit-tx

yes, ::keyword is not allowed in EDN...

regarding

Have you tried calling the xtdb.api/submit-tx Clojure API via Java interop?

https://stackoverflow.com/a/23555959/1953640

I am now importing EDN dataset the following way:

val data = File(dataResource.toURI()).bufferedReader().readText()
        val tr = xtdbNode.submitTx(toClojure(data) as MutableList<MutableList<*>>)

fun toClojure(edn: String?): Any? {
        return try {
            val read = Clojure.read(edn) ?: throw IllegalArgumentException("EDN String did not produce a value!")
            read
        } catch (e: Exception) {
            logger.error("Exception while reading Clojure edn value: $edn", e)
            null
        }
    }

Aha, I see what you mean

Yeah, maybe I will do it that way.

thanks for sharing your code, my Java is not very good, so I daren't try to suggest what the solution looks like exactly but it seems to be possible in theory 🙂

this is kotlin code, but similar to java yes

please let me know if it works, or not, maybe it's a terrible idea and I should avoid suggesting it in future 😅

no problem, thanks a lot for your input!

my pleasure. And I'm very curious about what you're up to with Kotlin! I look forward to hearing more

Did you already see https://github.com/AlistairONeill/crux-kotlin-dsl ?

I have seen this

But not sure I like it...

I much prefer the queries/transactions in the EDN format, without abstractions

So, native...

totally fair. I also prefer the edn, though I've never written a line of Kotlin so can't really judge

you can check the tests and judge fast... 😉

One question regarding keywords vs strings for attributes/entity id: is it better (more performant?) to use keywords for attribute names and entity ids?

A quick look at java api, the XtdbDocument.put seems to favor (support) Strings only for keys?!

we recognise that keywords are mainly a Clojure thing and want to appear more JSON-oriented. There should be no meaningful performance difference between the two though, in either storage or lookup costs

aha, thanks

during queries, all the joins & comparisons are based on byte encodings

how are you going to do this:

:xt/tx-time #inst "2020-10-16T14:24:17.025-00:00"

no need to reply, just joking...

thanks for your help and see ya later

oh gosh, I know you said you're joking...but is there a real API call like that?! I believe (and hope) that either ISO 8601 or Unix Longs can be applied to any edge cases that there may be