Fork me on GitHub

yeah, I felt tension speccing pull-patterns vs pulled data too, but I think pull-pattern should not be specced with s/keys, which kinda resolves the situation


In my case, we only do two layers of transformation before passing the data out. Input is big, mangled and flaky. Given all that it felt the domain - if there is one indeed - was rather thin, and thus there were few benefits to be gained in namespacing everything. But then we're using it for data engineering over Kafka, so we're basically talking about transit transformations, rather than domain manipulations.


> spec 2 select just says that the notion of required/optional keys is contextual ... but there is no explicit syntax of marking some keys as optional in select? Rather implicit "If the key if not part of the select, it's optional", right?


there's no equivalent for the opt declaration in spec 1? "if present, it has to be this shape" thing

Alex Miller (Clojure team)12:10:40

Schemas in spec 2 are all optional

Alex Miller (Clojure team)12:10:17

Select specifies what’s required in a particular context


I have a question that I think is related to the conversation above. If I have the following datomic-style schema:

{:db/ident       :user/email
    :db/unique      :db.unique/identity
    :db/valueType   :db.type/string
    :db/cardinality :db.cardinality/many}
How would one spec :user/email? The difficulty stems from the fact that I can transact {:user/email "myemail"} or {:user/email ["myemail"]}, but when I query the db, I would always receive {:user/email ["myemail"]}. Perhaps it’s recommended to define contextual specs:
(s/def :email/address ::non-empty-string)
(s/def :user/email (s/coll-of :email/address))    ;; matches the schema in name and shape
(s/def :user.input/email (s/or :one :email/address :many :user/email))  ;; matches what the boundaries of the system would accept
I’m especially interested in the use-case of integrating specs with datomic’s :db/ensure feature in order to validate entities before they enter the db.

Alex Miller (Clojure team)21:10:56

with specs it's always best to try to "state the truth" so I'd go with something that took either


I think the truth is that these are semantically different uses of the same key. One is a DSL, and one is data. Requiring all my functions to handle a collection or singular because spec doesn't allow me to distinguish the contexts makes pointless additional work for me. You could move the spec to the aggregate, applying additional constraints. I'm not sure how well that would work on a conventional spec 2 system.

Alex Miller (Clojure team)21:10:21

for the last question, might be best to ask that in #datomic, I'm not well-versed enough to have a good answer


That makes sense. The truth here being that, indeed, the input is a different thing than what comes out of the database?


I suppose I could restate the question as: if I want my specs to “follow” the schema, should I spec what comes out, or what goes in?


By “follow” I mean “possess the same semantics”


I think I may have answered my own question…I’m realizing that spec’ing input is much more useful than having a truthful “spec+schema” (the latter of which is likely redundant)


In other words, spec’ing what comes out of the database is pointless…at that point it’s too late. Spec efforts should be focused at the input boundaries of the system.


Interestingly tho this makes spec and the :db/ensure feature of datomic feel a bit incompatible, because predicate functions given to :db/ensure have the signature (defn my-pred [db eid])…and so they can only validate what comes out


I think it’s better to think of the transaction as a DSL. You can spec a dsl’s grammar, but not its meaning


so you can’t spec :user/email specifically as input to d/transact, only as an opaque key in a transaction map


you should either spec a level higher, e.g. some function that takes well-formed input and produces a set of transaction assertions/retractions/maps that do what you want; or a level lower with data-aware assertions, e.g. perhaps the DSL produces an ast which is easier to analyze, or in the case of datomic :db/ensure to guarantee your invariants.


it’s a little less obvious how to leverage spec in the “level lower” cases


combining spec with datomic, I’ve had more luck specing what you would want to see out of d/pull, because that’s what’s flowing through the functions in an application


that feels like where spec shines the most


and I’ve just accepted there’s no silver bullet for the cases where the data approaches the edges, e.g. to serialize/deserialize or to transact to the db