Fork me on GitHub
#clojure-spec
<
2021-04-05
>
dvingo18:04:54

How do you deal with creating specs for maps whose keys require differing semantics based upon the context in which they are used? Use case, of two contexts for modeling a 'task': 1. UI input form of a domain entity. 2. DB version of a domain entity. The description must be a certain length when persisting to the DB: (s/def :task/id uuid?) (s/def :task/description (s/and string? #(> (count %) 100))) In a UI form we allow the description to be empty: (s/def :task/id uuid?) (s/def :task/description string?) (s/def ::task (s/keys ::req [:task/id :task/description])) These are obviously in conflict, but given the use case they are both valid in differing contexts. How does one design a solution using spec (alpha) for this use case?

localshred18:04:41

Usually we'll just specify ns specific s/def's, since the context is different for the data. ::task-id instead of :task/id

localshred18:04:24

since the registry is global there's not many other options we could come up with that preserved the exact same namespaced key

localshred18:04:52

you can use (s/or ... and tag the different cases, then when conforming verify it's the case you're expecting

localshred18:04:25

(s/def :task/description
  (s/or
    ::db (s/and string? #(> (count %) 100))
    ::ui  string?)

dvingo18:04:46

I landed on something similar - saying the description could always be empty and then checking the DB requirements in the map spec - which knowingly goes against the design of spec in that the entity map is now aware of the semantics of its keyset

localshred18:04:15

right, it gets a little muddy

localshred18:04:05

this has a different set of tradeoffs since you're now joining db and UI validation into a single spec in some shared ns... ultimately trade-offs you need to decide on

dvingo18:04:38

thanks for the quick reply - so in the ::task-id case your map data would not have :task/id instead it would have ns/task-id? and then at some point you transform to :task/id format before persisting?

localshred18:04:54

for us we've kept the UI specs in ui namespaces, and the db specs correlating to the datomic keys we ultimately store them under

localshred18:04:27

so the translation from front to backend keys occurs at the http layer

dvingo20:04:29

Ok thanks for the info, good (err bad) to hear that others have the same problems

em20:04:40

@danvingo What about multi-specs? https://clojure.org/guides/spec#_multi_spec Separating your task specs into various domains is entirely okay, but if you want to keep the same structure but have different contexts, just reifying that context with a keyword tag (or however else you want to do it, multimethods are open) keeps things fairly simple and open.

dvingo20:04:18

I believe the core of the problem is that I want to use the kw :task/description and have its spec be different based on some context - I don't think mutli-specs would alleviate that problem

em20:04:35

It's one of the only solutions for your exact problem statement - keeping everything else about your spec the same, but having it be treated differently depending on context. You would inject context in the map wherever you needed it, and update that context at the boundary (when submitting from frontend to DB, for example). Injecting is literally an assoc with an extra keyword, and the multi-spec just checks that context.

dvingo21:04:38

(s/def :task/type keyword?)
(s/def :task/description string?)

(defmulti task-type :task/type)

;; in this "context" I only care that :task/description is a string.
(defmethod task-type :ui/task [_]
  (s/keys :req [:task/type :task/description]))
  
;; in this "context" I want :task/description to be a non-empty string of a certain required length.
(defmethod task-type :db/task [_]
  (s/keys :req [:task/type :task/description]))
I don't see how multi-methods make this tractable. I think the answer above is the point - I'm misusing the tool fundamentally, as it is designed.

em21:04:58

Yeah, on further thought you're totally right, my bad. Multi-specs are one of the only dynamic dispatch mechanisms for specs, but only really in the context of maps. My envisioned solution is too clumsy, as you'd need to change :task/description to be a map, and not a raw value. It'd work, but for every "context-variable" attribute you'd need to do a lot of extra work injecting context, as you couldn't do it at the top level. Technically, in accordance to the design and intention of spec, you'd want the information determining the specification of an attribute at the level of that attribute, so it makes sense why multi-specs work pretty much only on maps, as they can contain such extra information.

dvingo20:04:56

it seems like spec's answer to this is that you should create two specs under two different names, which: 1. feels like a bad idea - naming things are hard enough already. 2. If I have existing code and want to add specs for it, this requires me to change the names of my existing codebase, also tough decision to justify

Alex Miller (Clojure team)20:04:09

You are indeed at cross purposes to the principles guiding spec

Alex Miller (Clojure team)20:04:38

But this is why spec requires qualified names - the qualifier exists to disambiguate context

dvingo20:04:39

Questions: 1. What is its purpose then? 2. What is the recommended approach to solve this problem. (I can acknowledge my data design skills are subpar and I should have thought about better names up front, but one of the things I like about clojure is the iterative design it enables when discovering a domain. And well, it's too late to go back now and rewrite that name all over a large system).

Aron08:04:36

i have a similar problem to yours on the frontend, when i want to write a form generator, I have to keep 3 different specs for the same entity 1. the remote state specs that the backend API expects to get when I put or post or update the resource 2. the ui state specs which contains invalid data that was inputted by the user and is fully controlled by the user, but it still needs to be validated and used 3. my local display state specs about the inputted resource that holds stuff like a list of strings to shown in a predictive input while the user is typing their stuff it's all in the same Entity! the only solution is to accept that actually no, these are separate entities and require separate specs with separate names. similarly in your case, if your context is different then you need a new spec

dvingo20:04:19

thanks, I've read it a few times. This is a real world problem that I don't have a good answer to at the moment and seems like other people do as well.

Alex Miller (Clojure team)20:04:51

Given that you are at cross purposes with intents, there are no easy answers (well, don’t use spec is easy)

Alex Miller (Clojure team)20:04:36

But given that qualified names have global meaning, you need to spec the union of possible global shapes

dvingo20:04:57

Cross purpose how? I think that's what I'm struggling to understand

Alex Miller (Clojure team)20:04:27

Spec wants you to assign a spec with global meaning to qualified names

clojure-spin 1
Alex Miller (Clojure team)20:04:57

That is, specs are not contextual

dvingo20:04:27

I see this makes more sense now

Alex Miller (Clojure team)20:04:31

Unqualified names are assumed to be contextual and can be somewhat handled in s/keys with :req-un and :opt-un with contextual specs

clojure-spin 1
dvingo20:04:25

perhaps the wording here should change: > These maps represent various sets, subsets, intersections and unions of the same keys, and in general ought to have the same semantic for the same key wherever it is used From "in general ought to" to "will always"

em20:04:35

It's one of the only solutions for your exact problem statement - keeping everything else about your spec the same, but having it be treated differently depending on context. You would inject context in the map wherever you needed it, and update that context at the boundary (when submitting from frontend to DB, for example). Injecting is literally an assoc with an extra keyword, and the multi-spec just checks that context.

dvingo20:04:53

could you provide an example?

dvingo21:04:43

I have come to see the problem with my :task/description example. task is not a useful namespace and should be updated to be globally unique. Perhaps it would be useful to add to the guide some suggestions on use and data design in Clojure - as well as some anti-patterns. I think for small apps (which large apps usually start off as) it is common to use namespaces from the domain, but which are not globally unique, such as task instead of com.myorg.myapp.task and then as the app grows you get into problems such as the above ui/db distinction but now you have that too-simple namespace all over your code and probably persisted in storage and you have a tough decision to make on how to refactor it. The guide actually includes these simple names in some places (`:animal`, :event) which I think helps encourage the notion that this is a good idea, even though it is directly against the rationale: https://clojure.org/about/spec#_global_namespaced_names_are_more_important In my specific case I am thinking, if I wish to continue using spec, the design that is aligned with spec's design would be to have: :com.myorg.myapp.task.ui/description and :com.myorg.myapp.task.db/description or similar names. Thanks for the feedback and discussion. Things are making more sense now.

marciol18:04:20

Seems that this is the correct way of use. If you think about :task/description and handle it differently depending on the context you can somewhat get in trouble because the context sometimes is not devised clearly. So when you start to specify directly these keys each in their namespace, you are giving them a clear and unambiguous definition, so that you can even if needed to combine these two keys in the same map (keyset). The only extra burden is to handle the encoding of data to an external medium when all this meaningful context will be lost, but for it, I'd expect some convention established by the team in charge of such system.

marciol18:04:44

And following the accretion paradigm, you can create a new namespace when your system grows, and eventually remove the old namespace, without any conflict or problems with contextualization, etc.

Alex Miller (Clojure team)21:04:04

the guide is intended to teach spec concepts so aims for easier names, but I agree this could probably be better to align with intent. issue welcome at https://github.com/clojure/clojure-site/issues

dvingo21:04:22

agreed - when starting out the simple namespaces are great - but then there's no "advanced usage" or "scaling issues" or similar for perusing when you're in the intermediate stages of spec use. Thanks - I'll add an issue

marciol18:04:20
replied to a thread:I have come to see the problem with my `:task/description` example. `task` is not a useful namespace and should be updated to be globally unique. Perhaps it would be useful to add to the guide some suggestions on use and data design in Clojure - as well as some anti-patterns. I think for small apps (which large apps usually start off as) it is common to use namespaces from the domain, but which are not globally unique, such as `task` instead of `com.myorg.myapp.task` and then as the app grows you get into problems such as the above ui/db distinction but now you have that too-simple namespace all over your code and probably persisted in storage and you have a tough decision to make on how to refactor it. The guide actually includes these simple names in some places (`:animal`, `:event`) which I think helps encourage the notion that this is a good idea, even though it is directly against the rationale: https://clojure.org/about/spec#_global_namespaced_names_are_more_important In my specific case I am thinking, if I wish to continue using spec, the design that is aligned with spec's design would be to have: `:com.myorg.myapp.task.ui/description` and `:com.myorg.myapp.task.db/description` or similar names. Thanks for the feedback and discussion. Things are making more sense now.

Seems that this is the correct way of use. If you think about :task/description and handle it differently depending on the context you can somewhat get in trouble because the context sometimes is not devised clearly. So when you start to specify directly these keys each in their namespace, you are giving them a clear and unambiguous definition, so that you can even if needed to combine these two keys in the same map (keyset). The only extra burden is to handle the encoding of data to an external medium when all this meaningful context will be lost, but for it, I'd expect some convention established by the team in charge of such system.

marciol18:04:44
replied to a thread:I have come to see the problem with my `:task/description` example. `task` is not a useful namespace and should be updated to be globally unique. Perhaps it would be useful to add to the guide some suggestions on use and data design in Clojure - as well as some anti-patterns. I think for small apps (which large apps usually start off as) it is common to use namespaces from the domain, but which are not globally unique, such as `task` instead of `com.myorg.myapp.task` and then as the app grows you get into problems such as the above ui/db distinction but now you have that too-simple namespace all over your code and probably persisted in storage and you have a tough decision to make on how to refactor it. The guide actually includes these simple names in some places (`:animal`, `:event`) which I think helps encourage the notion that this is a good idea, even though it is directly against the rationale: https://clojure.org/about/spec#_global_namespaced_names_are_more_important In my specific case I am thinking, if I wish to continue using spec, the design that is aligned with spec's design would be to have: `:com.myorg.myapp.task.ui/description` and `:com.myorg.myapp.task.db/description` or similar names. Thanks for the feedback and discussion. Things are making more sense now.

And following the accretion paradigm, you can create a new namespace when your system grows, and eventually remove the old namespace, without any conflict or problems with contextualization, etc.