I have no idea of the actual answer, but would like to invite you to #ions-aws. We set it up because there’s a lot to figure out with AWS/Ions, and we’re kind of saturating the #datomic channel. 🙂

Yep, it blew up ^_^ I’m considering writing a protocol that just rips out anything that’s nil and wrap all transactions in it. Can you think of any reason why this would be a bad idea?

Result: https://gist.github.com/eneroth/81fe5acf0aab82c355889f28887e08ca

Actually, blows up on UUIDs.

I promise you that you’ll be happier 6 months from now if you spend the time to avoid making nil attribute values in the first place

then blowing up on nils is a feature, not a problem

I’m not sure I get that choice. I’m converting wildly varying XML blobs from scholarly publishers and sticking them in Datomic. Sometimes stuff is missing. Sometimes VITAL stuff is missing. I only get the choice of checking attribute by attribute, or the entire thing at the same time.

if stuff is missing, just don’t make an attribute?

you may need to rework some ingest code, but I promise you from experience this is a path that will result in less code and less pain in the long run

Yeah, maybe my pipeline is wonky. So I do, 1. XML -> EDN (generic) 2. EDN -> EDN (for Datomic) 3. Transact. So step two, can look like this, for example:

(defn prepare-data [{:keys [external-ids issn website title description]}]
  (let [online-issn (:online issn)
        print-issn (:print issn)]
    [{:journal/id (java.util.UUID/randomUUID)
      :journal/external-ids (mapv prepare-external-id external-ids)
      :journal/issn-print {:identity/issn print-issn}
      :journal/issn-online {:identity/issn online-issn}
      :journal/website {:internet/URL website}
      :journal/title title
      :journal/description description}]))

Essentially, take a bunch of stuff and stick them in a template.

So, is it preferable to wrap each attribute in a conditional?

Or a list of conditional assocs?

cond-> tends to be very helpful in stuff like this

(cond-> {:journal/id (java.util.UUID/randomUUID)}    ;; etc, the part that's always there
  ;; check each optional thing and assoc if needed
  print-issn (assoc :journal/issn-print {:identity/issn print-issn})   
  online-issn (assoc :journal/issn-online {:identity/issn online-issn}})

which you can read as: start with an init map if print-issn exists, assoc it into the map (otherwise pass along) if online-issn exists, assoc it into the map (otherwise pass along)

the starting object threads into the first arg of assoc

once you’ve seen this form a couple times, it becomes very easy to read

ive also been using this before transacting data (into {} (remove #(nil? (second %)) some-map))

could probably be converted to work with nested data

as I said above, I promise you the better long-term strategy across your app is to avoid ever creating or passing around nil attributes in the first place

it requires a little more up-front care but Clojure and Datomic both come from an aesthetic where that is preferred

Agreed about just not making nils. I made the mistake initially and have regretted it in record time. Luckily I may still have time to fix it.

I would love to, but to do that I would have to make my case with those third parties 🙂

Why is cond-> preferable to just wiping out the nils in one go? Is it more obvious what’s going on?

inevitably you need to handle nested nils too - then you’re doing recursive walks and modification of your data

instead making something, then removing parts of it, it is better to just make what you want in the first place

if you’re taking data from external sources, then that’s not an option of course

Thank you for the advice @U064X3EF3 🙂

Don't quote me on this but I am pretty sure clauses of a query are evaluated in parallel already

unless you can get parts of the query running on different machines I don't think there's any advantage to splitting

make sure the parameter you destructure as [?entid ...] is a true vector

not a set or seq or something that doesn't allow direct index access

> evaluated in parallel that is in line with my early exploration. increasing the parallelism on the client side does not seem to improve overall query performance.

> make sure the parameter you destructure as [?entid ...] is a true vector interesting. why does direct index access matter? i would have thought simply being a seq would be sufficient

under the hood is the clojure.core.reducers/CollFold protocol, which has an efficient parallel implementation for vectors but not for seqs

the parallelism in query is most likely implemented with reducers

in the psql console i see this

mydatabase=> \l
                                     List of databases
    Name    |   Owner    | Encoding |   Collate   |    Ctype    |     Access privileges     
------------+------------+----------+-------------+-------------+---------------------------
 datomic    | currentoor | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 mydatabase | currentoor | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 rdsadmin   | rdsadmin   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | rdsadmin=CTc/rdsadmin
 template0  | rdsadmin   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/rdsadmin              +
            |            |          |             |             | rdsadmin=CTc/rdsadmin
 template1  | currentoor | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/currentoor            +
            |            |          |             |             | currentoor=CTc/currentoor
(5 rows)

mydatabase=> \d
             List of relations
 Schema |    Name     | Type  |   Owner    
--------+-------------+-------+------------
 public | datomic_kvs | table | currentoor
(1 row)

so datomic_kvs definitely exists

oh looks like i might have figured out my mistake

i was creating the relation datomic_kvs in mydatabase, i needed to make it in the datomic database

Not yet