Fork me on GitHub
#duct
<
2017-11-09
>
rickmoynihan14:11:24

@weavejester thanks for yesterday… still working to resolve it in our project (as upgrading duct is causing us to shave a few yaks, so will get back to you on that). FYI though, I think we’ve noticed a small bug in ^:replace which is that you can’t seem to replace with an empty vector… e.g.

:duct.middleware.web/defaults
 {:static {:resources ^:replace []}}
doesn’t seem to work, but your example above with a string inside the vector does.

rickmoynihan14:11:20

assuming somewhere (seq []) is producing a nil and losing the IMeta’s

rickmoynihan14:11:41

but that’s just a guess

weavejester14:11:39

@rickmoynihan Thanks for the bug report. If you have time, could you raise the issue on the duct-framework/core repository?

rickmoynihan15:11:13

oh actually scratch that, can confirm the above is not a bug; it’s an issue in our app

rickmoynihan15:11:33

@weavejester: A question, what do you think the best way to do a multi-tenanted duct app is? Basically I’d like to have a cascade that goes --[local]----[env]--->[customer]---->[base config] but I want to be able to swap customer out dynamically. Obviously we could use :duct.core/include for this, but it will mean creating 2 more boiler plate files per customer. We avoided this with a read-config that does:

(->> (concat (main/common-config-resources site)
                    ["dev.edn" "local.edn"])
            (map (comp #(duct/read-config % main/zib-readers) io/resource))
            (apply duct/merge-configs))
However this has caused us some grief, because the (apply duct/merge-configs) loses ^:replace metadata by applying the replace, before we’ve prep’d; which was the cause of our recent wrap-resource problem.

rickmoynihan15:11:06

I’m thinking :duct.core/include is the way to go, even if it means a few more config files

rickmoynihan15:11:00

sometimes I do wish it was possible to compose aero/duct; so we could do things like :duct.core/include [#aero/join ["foo/bar" #aero/env CUSTOMER_NAME ".edn"]]

rickmoynihan15:11:17

I have combined aero/integrant successfully; however I’m not sure doing so would work with :duct.core/include as I suspect the readers will be lost.

weavejester16:11:18

@rickmoynihan If you think #aero/join would solve your problem, why not add that reader into main/zib-readers?

weavejester16:11:45

Alternatively, add a pre-processor that adds in an include, or the configuration required?

rickmoynihan16:11:40

yes, I supose that would be the plan… I’m assuming the issue with readers not being applied to :duct.core/include’s is fixed then…

weavejester16:11:13

I haven’t done much work with multi-tenanted apps in some time, so if you have any suggestions or ideas around them, please feel free to send me a message 🙂

rickmoynihan16:11:44

cool will do… we need to tidy up what we have a bit but some ideas might fall out of that.

rickmoynihan16:11:17

FWIW I really love integrant, it’s pretty close to perfect. And duct is definitely the best module system I’ve seen for clojure… but I find it hard to love the complexities of modules, whether they honour cascades in the right way, and the power they have over the whole config map. Then if a module does the wrong thing, sometimes your only choice is to copy/paste it’s code into your app and tweak. It’s a hard problem, and I don’t really have any better ideas btw… but wondered if you had any thoughts on what makes modules bad; and how those issues could be resolved?

rickmoynihan16:11:50

I’ve not really groked the new hierarchy stuff, but that looks like a nice addition

rickmoynihan16:11:24

Also I should probably add that I’m totally willing to accept that perhaps I don’t understand it (and the problems it solves enough) to comment yet.

lovuikeng16:11:17

as much as I admired duct for it's elegant data-driven design, I still find it hard to start with even though the blog post seems to suggest otherwise. Still, I have high hope it's going to get better with extensive docs and samples

rickmoynihan16:11:59

One other thing is that we’re fortunate that almost all of the modules have been written by @weavejester, who’s carefully avoided most of the problems… but if we ever start consuming modules from 3rd parties I have the feeling more problems will arise due to their complexity/power.

lovuikeng17:11:03

not to mention dozens other projects under the same care... what a prolific and diligent clojure contributor 🙂

weavejester17:11:48

@rickmoynihan The module system is something I’m not 100% happy with, which is why it isn’t in Integrant. I think it’s better to have modules than not; there needs to be some mechanism to make broad changes to the configuration.

weavejester17:11:46

I guess potentially we could limit modules to certain keys, or make them all merges or something.

weavejester17:11:00

I think it’s a hard balance to get right.

rickmoynihan17:11:27

Yes, I definitely get the sense that you either have too much power or too little.

rickmoynihan17:11:10

Does there really need to be a way to make broad changes to the config? I’m doubtless missing something, but Is inheriting defaults from resources not enough?

weavejester17:11:50

There are some modules like Ataraxy’s that are more complex

weavejester17:11:12

And I’m not 100% happy with meta-merging, either.

rickmoynihan17:11:16

yeah but isn’t it essentially just sugar?

rickmoynihan17:11:31

agree meta-merging can get pretty complex

weavejester17:11:48

I feel like some parts of the Leiningen-inspired meta-merging can be opaque. Not as opaque as a pure function, though.

rickmoynihan17:11:53

(not sure what would be better though) but the pragma’s can be confusing

weavejester17:11:12

Ataraxy is kinda sugar, but then all modules are to a degree.

rickmoynihan17:11:32

I think this is my point… are modules worth the cost for a bit of sugar?

rickmoynihan17:11:07

I don’t know the answer to that question… but I’d like to ask it 🙂

weavejester17:11:23

So you’re saying that merging in defaults is good, but the extra sugar might not be.

weavejester17:11:01

I’m not sure. I feel like there should be some way to get that power if you need it, but maybe that’s just a pre-processor.

rickmoynihan17:11:06

I’m suggesting that might be the case… not sure though.

weavejester17:11:39

Another possible solution is to represent the changes as data.

weavejester17:11:04

[[:assoc-in [::foo :bar] 1]]

rickmoynihan17:11:38

isn’t that a slippery slope to all functions being integrantified?

weavejester17:11:45

[[:conj-in [::middleware] [:foo]]

rickmoynihan17:11:50

recreating sexpressions in edn

weavejester17:11:34

Right. But there’s a middle ground, I think. All DSLs are (or should be) limited to a subset of functions/capabilities.

weavejester17:11:12

Otherwise how do you add a new middleware ref to a vector.

weavejester17:11:32

Meta-merge does assoc/conj implicitly.

weavejester17:11:58

Could make it similar to meta-merge, but at the end of each value there’s a piece of data describing how to add it in.

weavejester17:11:02

(merge
 {:a {:b [1]}}
 {:a {:b [:conj 2]}})
=> {:a {:b [1 2]}}

rickmoynihan17:11:18

ahhh ok I get you

weavejester17:11:31

(merge
 {:a {:b 1}}
 {:a {:b [:set 2]}})
=> {:a {:b 2}}

rickmoynihan17:11:20

how do you distinguish between the operator and the data? Presumably with some kind of ^:operator metadata

weavejester17:11:21

Maybe vectors are always the operator. I’m not sure. It needs to be a language that’s predictable and simple, but allows for 90% of what you need.

rickmoynihan17:11:38

assuming :conj etc are integrant keys to (defmethod ig/init-key :conj [_ _] conj) or something.

rickmoynihan17:11:10

or are you wanting something more constrained?

weavejester17:11:36

Oh, I didn’t mean Integrant. I meant some other DSL. Maybe extensible with multimethods. I’m not sure - as you say, it might be a slippery slope.

rickmoynihan17:11:03

yes, again balancing power and predictability is key

weavejester17:11:12

It’s almost like destructuring in reverse.

rickmoynihan17:11:40

interesting… that makes me think of spec conform/unform duality

rickmoynihan17:11:46

isn’t it also essentially a meta-merge-with?

rickmoynihan17:11:01

except the data supplies the function

weavejester17:11:58

Yes, though maybe a little more explicit.

weavejester18:11:25

There’s a lot of query languages, but not a lot of building languages, or languages designed to build some data structure.

rickmoynihan18:11:37

I think when you get into “building” you’ll hit turing completeness very quickly.

rickmoynihan18:11:59

query languages can avoid it more easily because they’re working with something finite and reducing it

weavejester18:11:45

I’m not sure that’s true; Turing completeness requires loops, which I don’t think are necessary when building most data structures, particularly immutable ones.

weavejester18:11:56

However, there is a question of complexity, I agree.

rickmoynihan18:11:55

agree on the loops front… but I think combining loops and reduction is better by definition than combining loops and construction — if you want to avoid being TC

weavejester18:11:32

Maybe. I don’t think it’ll be hard to avoid loops in construction, though.

rickmoynihan18:11:41

that’s what I meant by query languages have it easier at avoiding TC

rickmoynihan18:11:40

I think historically that’s been shown not to be true. Many languages accidentally become turing complete… e.g. java type system.

rickmoynihan18:11:56

often it’s not really a problem I concede, outside academic papers

weavejester18:11:36

I think in those cases you often have recursion, though. If there’s no variables it’s hard to accidentally add in recursion or loops.

rickmoynihan18:11:13

does :duct.core/include detect cycles?

weavejester18:11:48

I don’t believe so.

rickmoynihan18:11:52

I think many configuration languages are turing complete by accident, because of things like that.

rickmoynihan18:11:09

ok got to catch my train… interesting chatting.

weavejester18:11:12

Though :duct.core/include is something else I want to replace.

weavejester18:11:26

Interesting to talk to you, too!

rickmoynihan22:11:59

Just to finish this line of chat. I think the scope of the conversation confused me; in the context of a DSL are we talking about just meta-merge or modules, or both? I suspect :duct.core/include plus the metadata pragmas ^:replace/^:displace/^:promote/^:demote etc could come close to giving you turing completeness.

rickmoynihan22:11:00

anyway not meaning to pick holes, Turing Completeness doesn’t necessarily concern me, so long as it doesn’t tend to happen in normal use.

rickmoynihan22:11:23

by language for “building datastructures”, do you more mean language for merging? Merging seems by definition seems to naturally reduce the scope to finite values from EDN files therefore less likely to be TC by accident.

weavejester17:11:18

@lovuikeng What part do you find hard to start with?

lovuikeng17:11:42

Sorry, @weavejester I know you're busy, I'd asked the same before about a simple To-Do sample should be good to start with

weavejester17:11:03

Ah, I see. I’m planning to present a todo-like example at ClojureX in December, so I can add the code to GitHub. It will be API-only though.

lovuikeng17:11:38

I do understand that duct is more for advanced clojure user

lovuikeng17:11:23

cool 🙂 To be recorded too?

weavejester17:11:31

@lovuikeng Perhaps… I’d like it to be more generally applicable, though.

weavejester17:11:03

@lovuikeng Yep, Skillsmatter are good at getting up the recorded versions within a few days of the event.

lovuikeng17:11:34

seeing metosin makes use of integrant in some projects too 🙂

weavejester18:11:53

@rickmoynihan Another thought I had was to tailor the configurations themselves around a normal merge. Currently that’s difficult because of things like the handler middleware, which is represented as a vector. However, if instead it was represented as something like:

{:foo {:fn wrap-foo, :before :bar}
 :bar {:fn wrap-bar, :after ALL}}

rickmoynihan22:11:55

Hmm not sure I like this approach, it’s basically recreating a linked list in EDN (with some extra flexibility admittedly). Also don’t really like it because vectors are just a fact of life. I think whatever approach we choose has to work with the full richness of EDN, not a subset of it.

rickmoynihan22:11:33

Though the middleware case has me thinking that perhaps they can be injected like with behaviour like integrant. Think about it, the biggest problem with middleware stacks is that they don’t declare their dependencies explicitly. i.e. middlewares implicitly depend on things before them; but you never know what. Injecting middlewares by just normal integrant config would almost solve this, though thinking about it some more integrant might make a hash of it, because middlewares might be applied multiple times etc… I guess the thing doing to job of ss/dependency would need some deduping logic or something to handle it.

weavejester23:11:42

I think vectors have to be supported for other things, like custom routing frameworks and so forth.

weavejester23:11:16

I like the idea of supporting combining key values via multimethods, as then custom behaviour can be defined that depends on the key.

weavejester23:11:41

I’ve also been thinking about profiles for Integrant, which again implies some sort of merging.

weavejester23:11:26

I don’t think we can get away from merging, and either we try to get a meta-merge syntax that does everything, or… maybe we choose different merge strategies for different keys.

weavejester23:11:17

Profiles can also be used to replace :duct.core/include

weavejester23:11:46

{:duct.profile/dev #duct/include "dev"}

weavejester23:11:24

Then to start it:

(-> config
    (ig/profile [:duct.profile/dev])
    (ig/init))

rickmoynihan10:11:51

> I think vectors have to be supported for other things, like custom routing frameworks and so forth. Agreed, vectors are just a fact of EDN that need to be handled.

weavejester18:11:49

So essentially we shunt complexity around to make the general merging simpler.

ikitommi18:11:17

interesting discussion. Have been thiing about the merging too. Our new routing lib is happy with using meta-merge: :middleware, :interceptors and :controllers are presented in vectors, meta-merge is awesome for those.

ikitommi18:11:20

but, with muuntaja, the formats are presented in a map with format-strings as keys and thus, merging defaults by default would mean that one can’t remove any formats.

rickmoynihan22:11:20

You can currently ^:replace the whole map; but yes I don’t think there’s a way to dissoc a single key, and reuse all the other defaults.

weavejester18:11:49

@ikitommi Supporting third-party libraries would be a sound use-case for a more complex merge (like meta-merge).

weavejester18:11:11

I wonder if merging should be key-specific, handled by a multimethod.

rickmoynihan22:11:10

Yes, this seems to have a lot of potential! It would restrict the scope to just the RHS, and again reuse concepts from integrant; if not integrant directly. Like @ikitommi I think presenting alternative options is worth doing though

ikitommi18:11:50

I think there could be different options in a gist to see which one is best? all options seem non-perfect.

ikitommi18:11:30

btw, CodeMesh just ended, off to 🍺 @weavejester?

weavejester18:11:16

@ikitommi I’m afraid I’m not at CodeMesh

ikitommi18:11:52

ok. do you live in London btw?

weavejester18:11:16

@ikitommi I live just outside the M25. Takes about an hour to get in.

ikitommi18:11:08

oh, forgot that London is bigger than Tampere where everything is close 😉 But I’ll poke you here some day, have some questions & ideas for middleware (as data).

weavejester18:11:26

No problem 🙂

weavejester18:11:18

I don’t think I’m that far geographically from the centre of London, but getting in is a little bit of a trek!