Fork me on GitHub

you could try writing your own with gen/recursive-gen

gfredericks00:09:30 this a DAG?


or I guess it's a proper tree?


I did try with recursive-gen and got the same error. seems like it should be a proper tree yeah, which I guess is a type of DAG


yeah I think recursive-gen would work


can you share the recursive-gen code?


this is what I tried:

boot.user=> (spec/def ::posts
       #_=>   (spec/with-gen
       #_=>     (spec/map-of ::post-id ::post)
       #_=>     #(tc-gen/recursive-gen (fn [post-gen] (tc-gen/map (spec/gen ::post-id) post-gen)) (spec/gen ::post))))


I think the problem there is the (spec/gen ::post)


you can't use that generator as part of what you're building, since it's your original problem


if you had a ::post that had everything but the ::posts key, so that it didn't recurse, then you could use that




but a post without the ::posts key is not valid, the valid value for the leaf would be {}


Yeah I meant a version where the key always has {}


@gfredericks out of curiosity, could you try: (s/with-gen ::post #(s/gen (s/or :post ::post :empty #{{}})))


no, because it will still try to generate the default generater for a ::post, which can overflow the stack or whatever it was


what would be the best way to spec a collection of tuples, e.g.

[[:k1 :v1] [:k2 :v2]]
i'm thinking of just doing an into {} ... before speccing and speccing it as if it were a map, but is there perhaps a more elegant way to do this ?


the alternative i came up with was (s/* (s/or (s/cat ...) (s/cat ....))), but that's also ugly


(as a bit of background, the reason i'm not using a map is that the ordering of the elements is important)


Maybe this?

(s/coll-of (s/tuple keyword? keyword?))


yeah that's similar to the cat use case... the problem is that for a specific key, i need a specific value so then you would end up with

(s/or :k1 (s/tuple #(= :k1 %) ::v1)
           :k2 (s/tuple #(= :k2 %) ::v2)


that is, if you want to have a similar effect as

(s/keys :opt-un [:k1 :k2])


Ok. I also don’t know a better way than your s/or way


ok. guess i'll wrap this in a macro or something to make it pleasant.


maybe a multi spec is the answer though...


yes, i definitely think that can work


you can shorten it a bit with (s/tuple #{:k1} ::v1)


otherwise maybe keys* can be used


basically (s/keys* :req-un [(or ::k1 ::k2)])


conforming it will return a map tho


(s/valid? (s/keys* :req-un [(or ::foo ::bar)]) [:foo 1]) -> ok (::foo is #{1} here for the example)


but it will happily take "longer" tuples so not sure it's the best thing tbh


seems like keys* exists (among other things) to validate unspliced kw args


@mpenet thanks! that looks like what i want


:data #:clojure.spec.alpha{:problems [{:path [:args], :reason Extra input, :pred (clojure.spec.alpha/cat :docstring (clojure.spec.alpha/? clojure.core/string?) :attr-map (clojure.spec.alpha/? clojure.core/map?) :clauses :clojure.core.specs.alpha/ns-clauses), :val ((:require-macros [respo.macros :refer [defcomp div span input <> cursor->]]) (:require [clojure.string :as string] [hsl.core :refer [hsl]] [ :refer [comp-task]] [respo.core :refer [create-comp]] [ :refer [=<]] [respo.comp.inspect :refer [comp-inspect]] [ :refer [comp-zero]] [ :refer [comp-wrap]] [polyfill.core :refer [text-width* io-get-time* set-timeout*]] [ :as widget])), :via [], :in [1]}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__1200 0x60330e39 clojure.spec.alpha$regex_spec_impl$reify__1200@60330e39], :value ( (:require-macros [respo.macros :refer [defcomp div span input <> cursor->]]) (:require [clojure.string :as string] [hsl.core :refer [hsl]] [ :refer [comp-task]] [respo.core :refer [create-comp]] [ :refer [=<]] [respo.comp.inspect :refer [comp-inspect]] [ :refer [comp-zero]] [ :refer [comp-wrap]] [polyfill.core :refer [text-width* io-get-time* set-timeout*]] [ :as widget])), :args ( (:require-macros [respo.macros :refer [defcomp div span input <> cursor->]]) (:require [clojure.string :as string] [hsl.core :refer [hsl]] [ :refer [comp-task]] [respo.core :refer [create-comp]] [ :refer [=<]] [respo.comp.inspect :refer [comp-inspect]] [ :refer [comp-zero]] [ :refer [comp-wrap]] [polyfill.core :refer [text-width* io-get-time* set-timeout*]] [ :as widget]))}


how to read this error please?


There is a good sample of it in Original :problems was totally unreadable (recursive specs). Would like it just to report the deepest error, but still it's awesome.


@jiyinyiyong According to the :args and :in [1], it's not expecting that :require-macros form -- so I'm thinking you tried to run this in Clojure, rather than ClojureScript? (I think :require-macros is ClojureScript-only?)


it turned out shadow-cljs compiler did not handle it correct 🙂


(but, yes, some of ns spec output is a bit overwhelming)


Extra input generally means a sequence (`s/cat`) ran into a form that was not recognized by the syntax specified in the spec.


:value ( (:require-macros [... is the value that failed and :in [1] gives you the position within that sequence.

Kevin Lynagh18:09:05

Is validating data with runtime context in the purview of spec? For example, instead of just saying that :foo/id is an integer, can I specify that "it's an integer that corresponds to an entity in the provided database?"


not recommended

Kevin Lynagh18:09:55

From what I can tell, even though specs are being checked at runtime, there's no idiomatic way (short of dynamic var based hacks) to pass additional context into the validation logic.

Kevin Lynagh18:09:43

@ghadi Is there any kind of rationale or discussion on this topic somewhere that you know of? I haven't been able to find anything in my research more specific than "don't use spec as a runtime transformation system"

Kevin Lynagh18:09:51

which is sort-of-related, but not quite.


if you look up in a DB your predicates / specs might change their contract


the meaning of a spec might change based on state, which loses a whole bunch of value (inverting a spec into a generator)


I think specs should not depend on any ambient state

Kevin Lynagh19:09:08

besides the implicit spec database


e.g. just check that the email address is valid looking, don't try to open up an SMTP connection

Kevin Lynagh19:09:20

Yeah, but that's something that can't be technically enforced by the API of spec

Kevin Lynagh19:09:27

given that it accepts arbitrary predicate fns


i get that, it's just a terrible idea to curry in state into your predicate


nothing I can do to prevent it ¯\(ツ)

Kevin Lynagh19:09:28

Right, I don't mean to challenge the underlying design decision

Kevin Lynagh19:09:46

as a consumer, it just wasn't clear to me whether what I wanted to do was in the design space at all

Kevin Lynagh19:09:09

I thought it might be, since conceptually it's just further validation


i think you'd use spec to check data at the door, then do extrinsic checks afterwards


i don't think it's a design goal to be a total system

Kevin Lynagh19:09:59

At that point, it's easier to do the entire validation / generation myself, rather than using spec.

Kevin Lynagh19:09:10

since in my domain pretty much all of the incoming data is just ids


there's so many more benefits you'd be dropping

Kevin Lynagh19:09:21

so I need to realize those IDs into entities, then check properties of those entities.

Kevin Lynagh19:09:53

ditto on the generation front, just getting integers won't help. I need to be able to generate integers that correspond to entities of type X


don't you need to check the entities?

Kevin Lynagh19:09:33

unless I'm missing something where spec might still be able to help in here?


you're right the IDs are trivial, but I doubt that's the extent of the applicability of spec for your use case


I'd certainly want something to describe one of the "entities" and get a free generator


A non-trivial generator

Kevin Lynagh19:09:29

I can walk you through a concrete example if you have 5 minutes


go for it; I might not be able to help, but there are many 👀 in here

Kevin Lynagh19:09:28

I'm using datascript as the db, so it's all in-memory.

Kevin Lynagh19:09:45

the application architecture is event sourced

Kevin Lynagh19:09:09

basically, I have (step db event) ;;=> new-db

Kevin Lynagh19:09:45

basically, create a new database from the old database, given an event

Kevin Lynagh19:09:51

The events implement a few protocol methods like (valid? [this, db]) which says whether or not the event is valid for a given database.

Kevin Lynagh19:09:06

So, for example, #events.AddChild{:parent-id 5} is a record that represents the event "add a child to node 5". The valid? function needs to look and see if node 5 exists in the db and, if it does, make sure it's the type of node that is allowed to have children

Kevin Lynagh19:09:19

All this make sense so far?

Kevin Lynagh19:09:14

Anyway, I don't understand how I can implement valid?w/ spec, since everything depends on having the runtime db value. Likewise, I cannot generate these events with spec, because it'd need to only generate integer IDs that correspond to nodes that can have children.

Kevin Lynagh19:09:24

If wrote some specs to define what it means for a node to be allowed to have children, then I could turn the ID into a datascript entity at runtime and validate the entity using spec.

Kevin Lynagh19:09:48

but it doesn't seem like those specs would help me generate events to test against the system


your scenario makes sense. I think the fact that you and spec named something the same thing (valid?) but has a different contract is confusing the situation

Kevin Lynagh19:09:30

Ah, I only did that once I gave up on spec =P


You might make a spec to check a JWT. (s/def ....) Doesn't mean that the token is good enough to get through authentication. Spec will check that it is structurally sound


The spec might be pretty involved, but you can't have something that is valid? one minute and not the next


spec predicates should not be hooked up to state. a point of the specs is to have enduring, sharable meaning

Kevin Lynagh19:09:58

Right, that makes sense


I wish I had a more succinct rationale. Maybe @alexmiller will chime in later

Kevin Lynagh19:09:33

I didn't realize that it was a deliberate decision to prioritize that sharable meaning by disallowing runtime context.

Kevin Lynagh19:09:06

yeah, I hope I don't sound cranky about spec at all --- I think it's a really interesting idea and solves some hard problems.


it doesn't disallow it, it's the runtime context (AKA mutable state) that will work against spec's goals

Kevin Lynagh19:09:20

that's why I'm trying to use it in this context

Kevin Lynagh19:09:51

I don't want to rely on implicit runtime context or mutable state

Kevin Lynagh19:09:00

I'm just trying to pass that in

Kevin Lynagh19:09:10

Anyway, thanks for the chat and all your work on this stuff

Kevin Lynagh19:09:30

There are some other places where I've very happily used spec for validation/generation = )

Alex Miller (Clojure team)20:09:37

I don’t have a lot to add to Ghadi’s comments which I mostly agree with. Some people are doing runtime generation and registration of specs (by loading values from a db for example). That seems fine to me (it does get a little trickier with s/keys).

Alex Miller (Clojure team)20:09:00

in constrast, specs that use stateful stuff (dynvars, atoms, etc) seem bad to me

Alex Miller (Clojure team)20:09:24

what is spec buying you in this case vs just writing code to assert whatever predicate you have?

Kevin Lynagh21:09:49

@alexmiller I have the same kinds of problems that spec solves very well: composing predicates and giving back specific errors about why things failed validation. E.g., the AddChild event I mentioned earlier actually needs to take multiple parent-ids, and it would be great to get the detailed validation errors from spec.

Kevin Lynagh21:09:29

"Cannot add child because the third item in the list is of type X, but you can only add children to types U, V, W"


you can easily express that in spec if you write a predicate


it's the db extrinsic stuff that is a anti-recommendation

Kevin Lynagh21:09:11

@ghadi The predicate can only run against reified entities from the DB. Are you suggesting something like (->> parent-ids #(d/entity db %) #(s/valid? (s/coll-of ::parent))?

Kevin Lynagh21:09:03

Where ::parent is a spec that has the relevant predicates?


sure, that's better -- fetch the data before asking spec


(I was suggesting just building plain, offline predicates)


if there are different types of nodes, some that have parent-ids and some that don't, you should be able to represent that in the data directly, without querying something external

Kevin Lynagh21:09:32

Just to clarify, all of this is happening in an in-memory database, so there is no network fetching or anything like that.


Right but if that thing can change, YMMV

Kevin Lynagh21:09:55

it's immutable.


ok as long as a payload that is valid? right now, is valid? in the future, that's good.


If that can flip, bad kitty


i'm assuming you wouldn't call it a database if the data could never change


this will probably get me in trouble: specs should only check stuff intrinsic in the data.


(s/def :my.expiration/date #(re-matches #"\d{4}-\d{2}-\d{2}" %)
Spec can tell you if something conforms to :my.expiration/date... but not if it's expired


(aside: are regexes valid specs?)


Spec will generate you a bunch of :my.expiration/date's, some might be expired, some not

Kevin Lynagh21:09:11

Another way to phrase that question: Should spec allow you to validate {:time #inst"..." :expr #inst"..."}

Kevin Lynagh21:09:29

part of the spec is that both values should be dates

Kevin Lynagh21:09:54

but should you check in spec whether the :expr date is before or after the :time date.


it's a bad idea to call (Clock/now) while validating the expiration


because that's extrinsic

Kevin Lynagh21:09:04

Yeah, I understand that part

Kevin Lynagh21:09:13

I've been drinking the clojure kool aid for a while, don't worry = )


Then you should have no problem extrapolating (Clock/now) to a database 😃

Kevin Lynagh21:09:18

calling (d/entity some-db some-id) essentially turns my integer id into an immutable view into the whole database

Kevin Lynagh21:09:30

so reifying the IDs into entities, then calling spec makes sense.


it's weird though, when you check {:time #inst"..." :expr #inst"..."} it's a totally different entity that you're checking. Something like a token-request


we're in sync


i think originally I was thinking the db lookup was happening within the spec definition

Kevin Lynagh21:09:05

well, that was my question, if it'd be possible to do that.

Kevin Lynagh21:09:11

but you can't without dynamic var tomfoolery

Kevin Lynagh21:09:18

which I don't want to do.

Kevin Lynagh21:09:50

It's still awkward, though, since I can't really spec out my events, since I can't spec their integer-id arguments

Kevin Lynagh21:09:10

The best I can do is write my own validation fn that hydrates the ids into immutable entities, then calls spec on those entities.

Kevin Lynagh21:09:32

ditto for generation, I can't use spec to generate anything.

Kevin Lynagh21:09:04

The best I could do is, for a given DB, enumerate all entities, see which ones are valid? against a specific spec, then return their ids

Kevin Lynagh21:09:49

Anyway, I'll give that a shot and see how tidy I can make things. I may come back in a week or two with a follow up report + gist example

Kevin Lynagh21:09:55

Thanks again for the conversation!