how would the uuid thing even work in the current setup ? it’s pretty much impossible to define ranges in the same way we do for the non-uuid case, so all i can think about is something similar to what used to be in place

Uuids are evenly distributed, so the main factor will be the chunk size. Otherwise I think we can do the same thing. I could be wrong though

hmmm right, but that would still include the awkward count(*) to figure out the total size, so we can calculate the # chunks based on that

if we want to keep it the most simple, it would just ask for the # of chunks, rather than the chunk size

then you can determine the ranges solely on that

and don’t need an extra step

That would be simplest

@lucasbradstreet well, apart form some escaping issues for the jsonb format, the proof of concept is working! i’ll need to tidy this up big time though, but expect an incoming patch somewhere next week

@lmergen Nice! 🙂

might need some help with optimizing this somewhere down the road, my java-fu is not that great, and i think i’m doing too much copying

anyway, java cpu is the new bottleneck, so i claim victory 🙂

@lmergen Hah, good place to be. Yeah we can help with that for sure.

hi, i have a “101” question. I’m trying to figure out the best way to build/debug workflows. For instance, i’m pulling from datomic with read-log. I’d like to dump/inspect the segments that are read to help me craft the functions for the subsequent stages

@eoliphant I dont know if its the best answer but i make an spy function. like this:

{:onyx/name               :read-log
    :onyx/plugin             :onyx.plugin.datomic/read-log
    :onyx/type               :input
    :onyx/medium             :datomic
    :datomic/uri             (env :database-url)
    :datomic/log-start-tx    (:datomic/log-start-tx (env :config-onyx))
    :datomic/log-end-tx      nil
    :checkpoint/key          :t
    :checkpoint/force-reset? false
    :onyx/max-peers          1
    :onyx/fn                        ::spy
    :onyx/batch-size         10
    :onyx/doc                "Reads a sequence of datoms from the d/datoms API"}

(defn spy [segmento] ... )

ah cool, I was doing something like [:read-log :spy] in my workflow, but didn’t realize I could do it on the entry itself

thanks @lellis