Shaping? How so?

A Dispatcher is a mini router.

Any communication between an external queue to a channel, and vice versa, typically goes through the same set of validations, rules, policies that is otherwise required in a scenario where a router is required.

I don’t have a right answer yet, but I believe that discussing the shape of the message can help better supervisors.

“Shaping the channels” is essentially using a higher-level abstraction to bind multiple async/pipelines.

Hey @mss, I use (s/protocol impl/Channel) to identify channels w/ Schema, but I go one step further.

within my core.async utils I have a

(def Channel (s/protocol impl/Channel))

(s/defn chan :- (s/protocol s/Schema)
  "Channel schema, val defines the schema of of successful takes (does not include Throwable, which is implicit)"
  [val :- s/Any]
  Channel)

then I can either:

(s/defn write-batch :- es.async/Channel e.g. write-batch will return a channel

or

(s/defn read-catalog :- (es.async/chan Event) e.g. read-catalog will return a channel of Event

both resolve to the same thing (the fn is basically a no-op), but one gives me more information about the schema of vals I expect to be returned on the channel (it has no impact on validation, it's purely for documentation)

the comment about Throwable relates to the standard model of exception handling that I apply to channel processing.

@d-t-w: could you do validation with a transducer?

I'm not sure it would be a good idea to enforce schema on the elements of a channel, and with my exception handling it's always possible that the value may be an ex-info. You could have a transducer that validates each element, but you would need to apply that xform to the channel at channel creation time, and that may be outside the scope of the fn you're adding the schema validation to.

just the documentation tends to be enough for me

@pri: Sounds a little bit like a pipeline feeding a pubsub?

@d-t-w: I tend to agree that in production validating channel contents is probably not a great idea, but I'm into the idea of having a transducer that validates in dev.

However, if you turn on all validation in dev you kind of get that for free when the functions that consume from channels choke.

Or the producing functions break schema.

All of your transforming functions could be schemafied, the only trick is figuring out how to handle exceptions that come out of schema validation.

My general feeling so far is that we should test functions, not channels. Channels should dictate very little about behaviour.

And pipelines can be tested piecemeal by testing each of the transforming functions (or the composed ones)

@d-t-w: we pass on a tuple of [events-chan, finalize-chan, error-chan] at the start and cascade them from our schemafied functions. A global error channel can admix them and monitor for error flows and metrics. You’re spot on with testing functions, not channels. Channels should dictate operations.

Oops, I think that was for @angusiguess.

hmmmmmm

that's pretty interesting!