Clojurians Log v2

Finally got around to a spike on porting things to Clojure https://github.com/johnmn3/ti-yong/blob/main/src/ti_yong/alpha/dyna_map.clj

There's probably bugs. Tried to have some decent test coverage but trying to get them to act exactly like maps took some banging

I wouldn't be opposed to separating out a lib for just dyna-map (maybe better named proxy-map) and give it a simpler API. If anyone wants more interfaces or protocols added to the default dyna-map (with just an UnsupportedOperation exception if a method isn't provided) just let me know

Might be useful to add a bunch of core protocols that were defined before :extend-via-metadata was around, so you can choose from any of the protocols/interfaces. Some master object, with all the interfaces stubbed out. Would that be bad performance-wise? Having a hundred or two hundred stubbed out methods?

@lidorcg I initially didn't agree with you about the expression problem, thinking it didn't have to do with this. But yeah, re-reading the official definition, I guess it's similar to existing takes on that

But in terms of extending types globally, these aren't updating any global dispatch table. But I'm not seeing that the expression problem implies updating some global dispatch mechanisms

I kinda feel Clojure solved the expression problem by using a few core data structures and lisp/macros. You can do 99.99% of what you need to do without touching a deftype. That's expressibility by their def imo

So yeah, not as much a problem these days, I thought. But apparently people are still debating about it

Function composition can be just as efficient for expressing the same things as transformers. Especially if you don't need to maintain multiple similar versions of the same code. Transformers make it easier to maintain many slightly different versions of the same function, when that becomes a pattern in your code. It's not more efficient in all ways though.

In terms of expressibility, vs plain function composition

A todo is to get transients impl'd. I had some issues with that so just shipped without. But for sure, nothing about that data deftype has been optimized performance wise

"what you need to do" is extremely subjective. The space of expressions is less subjective. The space of useful expressions is somewhere in between (I'm guessing closer to the later than the former, given our very weak perception of "what everyone needs to do"). And I acknowledge that single dispatch (on the type of the first arg) extensibility (using protocols and types) from anywhere (and not just the definition) is more powerful than the average facilities you get. But extending on multiple dispatch (on a combination of all the args) is more powerful. And arbitrary predicate dispatch is even more powerful. At this point you should shout: "but clojure can do that too!!" and you'd be correct. But where? Only if the creator of the function bothered to make it a protocol or a defmulti (!!). This is a really huge point: the expression problem in essence tells us: please allow me to further extend/specify the meaning down the line without having to contact the creator to make changes. That means that the amazing (99.99%) of what I need is again depends on the foresight of the creator 😕 . (And I of course personally faced cases in which the creator, even in clj.core, has not left me a hook in which to express my meaning) At this point I want to give a counter example: Julia. (disclaimer: I'm familiar with Julia only from reading and listening and never created anything real with it). In Julia (IIUC) every function is multiple-dispatched (not to be confused with arbitrary dispatch), you cannot define otherwise, which means that in an hierarchical computation definition (i.e nested functions calling each other) I can hook into virtually any point and further specify my meaning for specific cases without the cooperation nor foresight of the creator (my impression: far extends clojure"expressiveness"). In clojure this will be very costly, Julia designed the entire language around this mechanism and managed to make it extremely efficient, probably at some cost that I'm not yet familiar with. But the point here is not the entire choice of abstractions to embrace and endorse (i.e Clojure vs Julia) but just these specific abstractions and facilities to tackle the expression problem. I would very much like to deeply experiment in such a system and find out what it does to reusability (and even the fear to approach others' code).

Have you seen my predicate dispatch thingy? https://github.com/johnmn3/dispacio

That's also not a performance oriented lib. Just a reference design

I like to call it a "predicate stack dispatch" system, so as not to be confused with the dispatch work with logic and pattern matching, which have their notions of "arbitrariness" idk

But if you want mult-fns, but with arbitrary dispatch, that'll do it. And you can plug in core logic or match or whatever

What you'll find there is that it's a much bigger footgun vs multi methods

Because multi-fn dispatch fns outputs a static value, which creates an exclusive set, defmethods don't shadow each other. Once you allow every defmethod to have its own dispatch function, they can start shadowing each other and you have to deal with that complexity.

and the order in which you define them matters, if you want to use the shadowing constructively. Basically each predicate in the stack is tried until there's a winner. Super simple

Bro, have you noticed that every time you ask for a lib in Clojure, I happen to have made a lib for that 🙂

Yes, I am familiar with dispacio 😁. After I realized we were working on the same abstractions basically (even from the same reasons I reckon 😁) I went through your github starting from affect and found dispacio and perc 😁. I also noticed the more general close similarities in our ideas and approach to problems and solutions, that's why I went all blazing in to this discussion (only realizing later I'm a few months late) to "defend' that position as it is similar to mine and I feel like I received a similar reaction (which is basically "don't do that" in the disguise of "why would you do that"). I realize I can't expect everyone to encounter the same problems which in turn shape their approach to problem solving, but I am expecting them to respect others' strive for ingenuity and instead of tearing it down because it doesn't fit their paradigm, admit the incompatibility and encourage exploration.

Idea: right now, :tf-pre runs on every change to the map (because it creates a new map, which runs :tf-pre). What if it only ran on updates to the :id key? That way you can transform a transformer from one state to another with invalid states (not passing specs, for instance) in between. Every time you add a new id to the :id vector, we consider that the instantiation point? Otherwise, you'd never be allowed to transition through invalid states during a transformation.

Which might be okay, but seems pretty restrictive

There's anonymous use-cases for transformers though I think, where you add behavior but don't give the thing a new id/name, and I think I want to continue supporting that use case, which benefits from running :tf-pre on every change

Mmm, maybe I'll have both

Add a :tf-pre that implements :tf-id that only runs on id change, then hang spec checks on that

Then you can add specs by either :specs or :id-specs

Changed the name to ti-yong https://github.com/johnmn3/ti-yong

I like how the function/data non-dualism (homoiconicity) in lisp relates to philosophical concepts of subject/object dichotomies and especially non-dual philosophies/substance monism/etc, and the word ti-yong is an ancient East concept of the idea, also present in the Advaita branch of hindusim and some strains in Christianity, Sufism and Judaism - a very pervasive debate seemingly present in most religions and cultures throughout time

IMO, the yong is teleological and the ti is ontological, and the key takeaway here, about non-duality, the way to understand it, is that ti is a superset of yong. "Not all water is an ocean wave, but all ocean waves are water," they say in this Eastern philosophy. One is a subset of the other - the behavior of one contains the behavior of the other, like data contains the behaviors of functions (among other things). So we're going back to describing the teleological, abstract function in the more ontological terms of data, while still maintaining a higher level of abstraction than the code tokens themselves, like we have with macros. Again, macros do tame these "dualist dragons" but only at a syntactic level, while transformers try to tame them at a higher semantic level, ideally providing a "middle way" between the two extremes.

step-up was supposed to be a pun on electricity transformers but it's a young project so nbd

Hmm, so apparently malli has https://cljdoc.org/d/metosin/reitit/0.7.0/doc/http/transforming-interceptor-chain

This idea of spec transformers aligns with the idea of transformers here I think. It's a transformation of one data structure to another. Here we're transforming data that represents a function into some other data that represents a function, which is actually closer to the category theory definition of the term

Yeah, by Rich's definition in that talk, objects are inherently stateful

I see your point about Haskell. That being said, we could put state in a transformer in Clojure, whereas in Haskell I assume that won't even compile. For some things, we'll want to put state in a transformer. Multifns can pretty easily be built on top of a stateful transformer, for instance. Would that suddenly turn the transformer into an object? Doesn't feel like it - feels like the difference between a function and a function with state. Still feels like a function.

And in java we can make objects that are essentially stateless and immutable, right? Are those suddenly not objects?

I'm arguing that transformers won't lead to the spaghetti-state that object systems traditionally lead to because by default they are immutable, but I personally don't care if transformers are considered objects or not. I just think they scratch some of the same itches that objects scratch, just in a more clojurey way

I wouldn't consider a Java objects without state as an object really. Then it's just a module.

I mean, you'd definitely call it an object still in Java. But conceptually it becomes like a module.

One issue with object inheritance is that the hierarchy becomes inflexible. Over time it feels like you got it wrong all the time. For example if you want to have a Car and then a WashableCar which is a Car that also contains state about how dirty/clean it is, and therefore can be "washed".

Car -> WashableCar

Car -> WashableCar -> Truck

Car -> WashableCar
    -> Truck

Car -> WashableCar
    -> Truck -> WashableTruck

Mixins are nicer because it doesn't have this issue. If washable is a Mixin, of you say Car has mixin Washable it means that "dirt-factor" is added as a state to Car. And similarly if you say Truck has mixin Washable, it'll add "dirt-factor" added. And now methods can be written for any type that has the washable mixin. But they have another issue. If you actually wanted to add a new field to a bunch or related types all at once. With inheritance you can add it to a parent type. With mixin there's no such derivation. You have to go and add it to all types you want it on, or make it a new mixin and add that mixin to all those types. It also makes it hard if you want a method over more things, like maybe a method about washing a car needs some state of the car and about the washable, and that differs for a truck. So it's not just about being washable, it also matters if the washable is a Car or a Truck. Inheritance can do that as well, because it looks at the whole derived hierarchy. With mixin you can't. Or maybe you can say this method works for any type that has Car and Washable.

If I understand transformers as you described you're doing. It sounds like the inheritance hierarchy is dynamic. So you could like fork the hierarchy itself and reconfigure it at any point?

Well, I think transformers are providing both here - single inheritance and mixins

I've never analyzed any other mixin implementations, so I could be wrong, but the way I think about it is that mixins just provide a very controlled form of multiple inheritance

And you tend to define them minimally, so they can be added to many things, rather than a complex object with lots of features predefined

You're more likely to have a large transformer with many mixins mixed in, and then that provides the parentage for a bunch of other similarly complex functions

But its true that transformers branch/fork, just like Clojure persistent data structures do, because that's what they are

I'm not sure what you mean by "reconfigure" though

With respect to contrasting single inheritance vs mixins, and aspects of reconfiguring those, I'm not sure if I caught your meaning. I agree with your analysis of the problems with class hierarchies and how they're a round peg for many square holes. And I do think that transformers provide the best of both worlds there, wrt single inheritance vs mixins

(fyi, I pinged you via dm about the README I'm working on for this repo)

If by "reconfigure it at any point" you mean change any point in the hierarchy, but as a branched version of that entire hierarchy, without affecting existing callers of the existing hierarchy, then yes, exactly like that.

And it's all pretty efficient because of persistent data structures, by branching on shared structures underneath

About as efficient as you could hope for from an object hierarchy, at least

Yes that's what I mean.

Though you're more focused on behavior. So I think it's easier in some way.

Yeah, just lean into your existing intuitions around data

You're just changing data

So like say I have a window that draws itself, then draws a frame around it, then let's you click inside it and it changes color where you click. If that's an inheritance chain, and now I want that same window but without the frame around it.

Right, it requires only one change in the transformation

Shorter path between where you are and where you want to be

Ya, and that's what you can't do in Java.

So... I was thinking about this recently... Can OO people say, "Actually, we've been data oriented from the beginning - We just program a tree, explicitly!"

And there's no reason an object system shouldn't be able to provide that shorter path described above, as long as all three parts of the impl are still exposed as a first class tree of things

But yeah, I think Java is more locked down in a lot of ways... Not sure if the above is impossible though

Ya, I think it's more about that the hierarchy is static. It's true that maybe in a prototype object system, not a class based one, you might be able to do something similar. I think you basically clone things in prototype object systems. So maybe you can clone the top object, and merge it with the leaf object, and you get an object that is the top merged with the leaf without the middle object. Not sure, I'm also not too familiar.

That makes sense yeah

I do think OOPs langs have a relationship with performance. How methods are looked up efficiently and all that mattered. I've never used say Smalltalk. And even newer more dynamic OOP langs I think just copied the existing more static ones. But rethinking an object system within a more dynamic context probably allows a lot more flexibility. CLOS object system for example is pretty flexible. I also have never really worked with it, just read about it a bit.

Another thing that is going to make transformers easier to work with than objects, I think, is that they follow Clojure function pattern following the single responsibility principle for building things. Again, like an object of one method. You want a different behavior, make a different function! Whereas objects are like entire namespaces of behaviors.

Yeah, I'm just now reading up on CLOS the last few weeks, to understand their difference from transformers. Pretty wild how similar CLOS/MOP is to the idea.

I certainly never cared to investigate CLOS, I just needed to scratch an itch with overly complex UI widget libraries that needed to evolve faster

Also cool read is this: https://en.wikipedia.org/wiki/Circle%E2%80%93ellipse_problem

I read that recently and didn't grok it. I impled square/rect in the user.cljs of the repo with no problem.

Ya, I feel, Objects kind of more conceptually are about state modeling and changes to it. And like this ellipsis-circle problem is related to state. A circle is-a ellipsis, but it requires less state, and valid state changes on an ellipsis break what a circle is. But your transformer seems to be more about code reuse. A function is about data transformation, not state changes, even though if the data being transformed is fetched from some state handling place and the result is updated back then it can also be used to change state.

To me this is a big difference. A function takes data, not state. And it operates over this data within the confine of the function, and then it returns some data as well, which often has a relation to the input data, but not necessarily.

The transformer seems similar. Where the value is more about reusing data manipulation logic, so you can create new functions out of existing code chunks.

Rich Hickey I think calls it place-oriented programming. Objects create a place for data to live, and that I call "state". A function doesn't do that, a closure can hold some state, but a normal function doesn't give you a place to put data, manipulate the data you put there, and retrieve it back.

Right, like, in the user.cljs, we show how we enforce 2d rectangularness by imposing a unit of 1 to the 3rd dimension of a box here:

(def rect
  (-> box
      (update :id conj ::rect)
      (assoc :ln 1)))

I just don't see the problem with the circle-ellipses thing - I just think a circle always is-a ellipses but an ellipses isn't always is-a circle. That just seems obvious, if you want to share impls, you'll want to start with the ellipses and constrain it to a circle.

Oh, and with square and rect:

(def square
  (-> rect
      (update :id conj ::square)
      (update :methods assoc #% %:size #%(* %:size %:size))))

So in OOP, you would say that a Circle is-a Ellipses. That implies that all methods of Ellipses are usable on a Circle. And a Circle should have additional methods/fields. So if you create a Circle, and then you change it's width, it turns the circle into an Ellipses, so it's no longer a circle, but it's still of type circle, so it's broken.

Circle circle = new Circle(20); // Sets the radius to 20
circle.setWidth(40); // This is an inherited method from Ellipses. It doesn't make sense for a Circle, because now width is 40 and height is 20, so it's no longer a circle, it's an ellispes again.

Right, so what's the alternative? Just to say it out loud... If Ellipses is-a circle. Then circle has no .setWidth impl yet.

So can you then reuse any of the impl for circle (`.setRadius`) for the impl of ellipses?

Wait, is ellipses like rect in that it doesn't take a radius like circle does?

I actually forgot the formula for an ellipses lol

Ya, ellipses doesn't take a radius.

I think ellipses needs height/width. Circle could also use height/width if you make them the same. Or it could instead use radius on its own.

Not too sure about the alternatives haha. I know one option is to make them immutable. So you can't change the shape of the ellipsis, and now if it's a circle, you also can't change the shape of it.

It says you can make it so setWidth() when called on the circle would throw, or return false (meaning it did not change it). You have to override the Ellipsis methods in those case.

You can also have it that circle.setWidth(...) returns an Ellipsis if the new width is different from the curent height. So allow the type to change back to an Ellipsis.

I mean, this seems like a problem inherent to the liskov principle alone

Not necessarily objects in general

Or you can have setWidth also change the height to be the same. So both setWidth and setHeight of Circle act like setRadius

Right, so, you could have circle be the parent of ellipses, but either way, you have one shadowing the behavior of the other, right?

Well, the issue with objects is that they don't let you reuse the parts of Ellipses you want. Like the truth here is that, a Circle is-a Ellipses, in a mathematically constrained way, not in the typical OOP subtyping way. And so many things you might want to model won't be 1:1 with the OOP inheritance way, and eventually your hierarchies will all have these flaws.

The other way can also be wrong. Because not all Ellipses are Circles, so some methods will also be wrong.

Solely because of the liskov principle, right?

Stating that a child should reproduce all of the behaviors of their parent in full, right?

I mean, yes and no. The issue is that in OOP, a child can be used in-place of its parent. If you don't make the child semantically correct, then it's only a source of bugs, or you could even argue, why have it be a child.

Like they say on wikipedia: > it requires the client code to test the return value for support of the stretch function, which in effect is like testing if the referenced object is either a circle or an ellipse. Another way to look at this is that it is like putting in the contract that the contract may or may not be fulfilled depending on the object implementing the interface. Eventually, it is only a clever way to bypass the Liskov constraint by stating up-front that the post condition may or may not be valid.

Men may have women as parents - that doesn't mean they'll reproduce all women behavior. That seems bonkers to me...

Yes, that's why people say OOP is not good enough to model real-life problems.

IIUC, that rule is only the L in SOLID, standing for the Liskov principle, and without it, you don't have that constraint. But if you don't have that constraint, is it no longer OO?

Like that's a great example. Do you have Woman derive from Man or the other way around? Or maybe, neither direction can really model the true Man/Woman similarities and differences because they're not strict supersets of each other in any direction.

The Liskov principle seems ridiculous

No, if you don't have that constraint, you lose the benefit, and are at risk of bugs. So it just moves the downside of OOP. Now it can be used to model the thing, but you'll have more bugs, and need to be more careful when you use the objects that the method you are using is valid to the concrete type.

For example, in Clojure, your method my just do: (s/valid input circle) And if the width and height are not valid, it will throw invalid. But you don't pretend that the map of circle is a subtype of the map of ellipses.

Nope, just data

I'd like to get a better intuition around how the liskov principle avoids bugs

You certainly could program transformers with the Liskov constraint, but it'd be way too constraining IMO

I think you kinda nailed in with the sense that transformers take the place orientation out of OO

It kinda captures and summarizes a lot of the discourse around the pros and cons all at once

Right, these aren't types, they're not classes, they're just data. So there's no subtypes. They're two different values. The fact that the path between Function A and Function B happened to be a single assoc doesn't make one sub of the other in any way other than the data shared between them.

Right exactly. The immutability helps too, but so does not having types in a way. Like say I have: (def circle {:width 20, height 20}) Maybe I have a method, and it can change a map of width and height, so I can use it. (def ellipses (assoc circle :width 30)) But that did not break my circle, it just returned an ellipses. And so if my circle methods validate for circle, and the spec checks that width and height are same, I will not accidentally use an ellipses with a function that works only when width and height are same. So functions can just validate that the invariants of the data they get is what they expect to successfully return their output. If an unequal width and height they know would break their logic, they can throw.

Yeah, that's true

Okay, so I added the readme here https://github.com/johnmn3/step-up/ I added most of the folks in the discussion here to the thank you section (Thanks!) Please give it a looksee and lemme know if there's an glaring omissions / things that I missed in the overview.

I just skimmed this thread (haven't read the other one) but the idea of "inheritance" or (in less emotion provoking words) reusing some functionality while being able to adjust it precisely without rewriting / copying the entire thing is a very much established problem known as the https://en.m.wikipedia.org/wiki/Expression_problem. It is/was tackled by many (quite smart) people through many "mechanisms" or "tricks". Inheritance + dynamic dispatch (single or multiple) is just one example. In clojure there are types but you can't override existing behavior for them without copying their code. You can write your own over the existing tools like I did (using multi methods and ad-hoc hierarchies) or (what I think is) @john's attempt but it'll never be able to reach as deep like for example in CLOS (read metaobject protocol). This can be labeled "simple" or "good" or "better" but it is , in fact, a non-solution for anyone who faces some instance of the expression problem that requires him to override just a part of the behavior of some existing functionality (even just for a restricted scope). The only argument I can see left to defend this decision (or non-attempt) to solve this problem is: "this isn't really a problem" or "this isn't a real problem" or "I don't see a problem". I think we need to learn how to say: "as amazing as clojure is, it didn't solve this specific problem yet, it's a very hard and important problem, there are many attempts out there, but go for it champ you might be the one we were waiting for" (@john 😉)

PS @john About a year ago I needed exactly what you're describing. potemkin wasn't good enough for exactly the reasons you mentioned and I ended up just copying the entire deftypes of map and vector and reimplementing what I needed and the rest was just calling the same protocol function on the internal map/vector. The tool you're describing was exactly what I was looking for and hoping to find: just overriding what I needed, no more.

So I read a lot of the back and forth, but your initial description sounds a lot like a map prototype

As in javascript’s prototype based inheritance

@andrew303 how you figure? I'm not too familiar

I was curious about that, as they have the everything-is-an-map-even-arrays thing, which are their objects, which you can update via the attrs on the map

I suppose js is almost, "objects are just maps"

But they're still mutable, same ish

What you were describing about a base impl is available that you can override is exactly the JavaScript object behavior

There’s a prototype chain all the way up to object with methods and attributes

Right

Some say a function is an object of one method

JS objects can have many methods like most object systems. What's decidedly different here is that we're still keeping one method and the map is all in support of that function and for modeling the lifecycle and behaviors of methods. And it's just a map.

So it’s a function map sort of?

With data in it that happens to model a function, and whatever else you want to put into that "environment"

And that model allows you to manipulate it at runtime

A bit like it’s exposing its closure to you?

Yeah, turning it inside out

Or, abstracting the idea of a function into data

So in that way it's probably different than your average object system (not to mention the default immutability) but apparently CLOS's generic functions do provide similar lifecycle hooks and the Meta Object Protocol system models the object in data like we're doing similarly here

Never heard of "transformers" beyond LLMs, what's that? Also, are you just trying to implement some kind of class support?

It's a new way to build functions using data, where you transform one function into another, using regular data slicing tools, to build your functions. No classes. Just maps. It's like using a macro to transform one kind of function form into another - here we're using regular Clojure data manipulation functions to transform one function map into another function map.

In Category Theory, a Transformer is a thing that transforms on functor into another functor. A functor is a mapping of inputs and outputs. So this is a function transformer in the category theory sense of the term

So it's only used if I want to create a new map type?

dyna-map can be used to create a new map type. root or transformer can be used to make a new function who's implementation is backed by a map

it's like a "structured function"

root uses dynamap to add the IFn behaviors

I don't really get it 😛

transformer builds more opinionated mixin and spec checking capabilities on top of root

Is the idea, I want to have a map that stores User data, like email, and then I want to add a method for "send-email" on it?

Or the idea is that like, I want to create an S3 based map, where all the actual key/vals are stored in an S3 docs persisted, that is usable as an IMap in CLojure ?

The idea is to not close over impl details, so that you can later evolve fns by simply doing data manipulation, without having to "unwrap" the implementation you would have wrapped in a closure

like here:

(def filter-anchor
  (-> comp/a
      (update :id conj ::filter-anchor)
      (update :with conj a/selected? a/void-todo styled/filter-anchor)
      (assoc-in [:props :on-selected]
                #(-> % (assoc-in [:style :border-color]
                                 "rgba(175, 47, 47, 0.2)")))))

(def filter-all
  (-> filter-anchor
      (update :id conj :all)
      (update :children conj "All")))

(def filter-active
  (-> filter-anchor
      (update :id conj :active)
      (update :children conj "Active")))

(def filter-done
  (-> filter-anchor
      (update :id conj :done)
      (update :children conj "Completed")))

Those are functions. Components we put in a re-frame app. But they're also data

they're both functions and data

What's data about it? (except that clojure code is data)

so now, filter-all, filter-active or filter-done can create custom versions filter-anchor for just themselves, without clobbering each other

The data is about the model of a function

and you fill in that model

with pieces of the function, so that you can later swap those pieces out at runtime

The map is about the implementation of the function, and not the data the function operates on

What makes it a function though?

(defn blah [some & stuff]
  (let [a 1
        b 2]
    (do-some some stuff a b)))

It has a name. It takes parameters, it makes some local state to do some local trasformations, it might apply some operator on that transformed data, then it returns something, right?

That's what makes a function

in general

Ok, so you're saying, if you wanted to create an alternate "blah" but where "do-some" did something different?

You could swap that out with a macro, right?

But if you store the fn as data, you can just swap it out with functions, as if they're macros

So it's like giving yourself macro powers, where you hadn't closed over things

Right, I get it.

It's a meta programming trick thing

So like, if "do-some" was indirect, you could pick the implementation for it. Are you generating functions in the end, or the whole machinery is subverted?

They stay as functions the whole time

And start off as an identity function that you mold into something else

I mean, if I understood. Say I have:

(defn foo [a b]
  (+ a b))

(def foo
  (-> transformer
      (update :id conj ::foo)
      (assoc :op +))))
#_(foo 1 2 3) ;=> 6

(def foo2
  (-> foo
      (update :id conj ::foo2)
      (update :tf conj
              ::foo2
              (fn [{:as env :keys [args]}]
                (if-not (sting? (first args))
                  env
                  (assoc env :op str))))))
#_ (foo "1" 2 3) ;=> "123"

I'm assuming you mean, "Check something about the param and do the str thing in that case" otherwise you'd just rewrite it with str

:tf is a vector of "transforms"

interposed by qualified keywords representing the id of the transformer defining the transform

These transforms can be mixed in from other transformers, so preceding them by their id helps keeps things organized

the with mixin deduplicates all transforms based on those tags as well, as it collects all the transforms from all the mixins

now, ostensibly, foo2 above is a lot more complicated than simply wrapping foo like (defn foo2 [a b] (if-not (string? a) (foo a b) (str a b))), but by wrapping foo, we make things much more complicated for ourselves down the road, for certain use-cases of foo

Ok, I think I'm seeing it. Not sure when I would need this. But it's interesting

You as the author have the choice of granularity for your modeling of the function, depending on what behaviors you don't want closed over

It's only for situations where you have large hierarchies of hand built, self-similar functions, that need to evolve quickly over time. 95% of Clojure problems are just not like that.

Especially useful for building custom crafted widget toolkits that need to evolve fast under quickly changing business requirements

And why is it a map?

Because we don't use objects here 😉

lol. The thing is, it really doesn't seem to semantically be a map anymore. And the APIs are like subverted a bit. Seems it be a good use-case for making it a protocol no? With it's own methods

Why is it not a map?

Like, sure it's "callable", but it does not do key lookup

Other than the one and two arity invokable

on maps

get?

or just call by keyword

But you're not providing a map interface to your user, that's not the point

the map interface is used for manipulating the function's implementation

I mean that: (foo 1 2 3) when foo is a map, I expect it would return the associated values of those keys. Not perform some computation on 1,2 and 3

The applicative of foo is not a map and those consumers aren't supposed to be consuming it that way

They don't have to know about the assoc/dissoc on the "backend"

Isn't foo a map here though?

on the "backend" it is

its implementation is in a backend map

Ok, but it leaks a bit, because I'm using assoc, update, etc.

leaks? On the backend, you're dealing with an actual map, inside the transformer type. All map protocols and interfaces are passed through transparently to the internal map

I guess I just expect all implementation of map to be compatible with one another, like you can substitute one for another. It's not a big deal. But because of that, I feel it would make sense for it to be a custom type and you could call assoc, update, etc. in a more appropriate to what it's doing names

dyna-map is compatible with hash-map. transformer built on dyna-map has keys in it like :args and :in and :out, etc, that are always in a transformer map, which is the backend of a function implementation

If you don't override IFn, transformers go back to just being a dyna-map, which can act just like a hash-map. You can just override any of dyna-maps methods easily

So transformers aren't 'equivalent to maps', sotospeak, because they're a convention of data in a map, which has an IFn attached that uses the data in the map

It uses the data in the map to store the implementation for the IFn interface

Right, so can't transformers be their own type, that uses a dynamap under the hood. So now it does not look like a map

Oh, you mean it leaks the associative api to the downstream consumer of the fn. Yeah, true. You can build transforms that "freeze" a transformer into a plain function that can't be modified, if you really need that.

Ya, like I said, it's not a huge deal. Cause probably you'd just refer to the documentation anyways when using it.

dyna-map adds and removes methods with assoc-method/`dissoc-method` and if you dissoc those methods then you'll freeze the type and it won't be changeble from what you molded it into

It's basically two different developer interfaces - a higher level one and a lower level one. But, the lower level one lets you use Clojure's high level associative and sequence manipulation tools to build a map that contains the implementation of the higher level api of the transformer. The consumer of the higher level api simply uses the invoke/apply interface and composes it into their program using normal function composition methods and that's the only interface they need to know about, until they learn how to work on the backend datas structure. The "backend" is "low level" but it's way higher level than working with deftype for instance. It turns function/type implementation into more-so a data problem, which we're all a lot more familiar with than typical, low level type programming.

And how cool is this, where you can spec instrument the data itself?

#_(dissoc a :a) ;=> :repl/exception!
; Execution error (Error) at (<cljs repl>:1).
; :step-up.alpha.transformer/a-spec

such interesting meta programming techniques can be pretty easy like that

Hum... that's actually a pretty sweet idea. Validating maps. Could be a good way to use maps for entity modeling. Today you have a seperate spec and do something like: is this map a valid user, if yes, then I can proceed. But you could have a map with restricted assoc, dissoc, etc. That would error if you try to change it so it makes it an invalid User.

right, we spec function boundaries. With this, the spec can follow the data through unspeced functions

But... did we just create an Object? That's the phylosophical debate 😛. It still doesn't have methods, but it also doesn't let you fully operate on it like a map, since it restricts certain updates.

It's just a map!

🙂

I would say no. This would be quite convenient. There's no methods attached. The map just validates it's own entity invariants as it is being manipulated and error if broken.

Lots of super convenient things come out of organizing things this way

Not that you should use it on more than these 5% scenarios, but it can come in pretty clutch when you really do want it

Ya, but now that I think of it though. This particular use-case is pretty sweet. Maybe it's it's own lib. It doesn't need as much machinery. Just a SpeccedMap that let's you attach a spec that gets validated each time the map is modified and throws if the changes to the map no longer conform.

yea true, you could make it its own thing with deftype

I think you could make it with transformers and then just freeze it. Might be an easier impl

I also have a mocker example in there that exercises a function against real inputs and outputs of the function, as well as for downstream modifications of the transformer function, to ensure invariants hold for ancestors' modifications - on map instantiation time

So there's all kinds of ways of ensuring bad implementation maps aren't implemented

> "And why is it a map?" Another answer: > "The interesting part about the metaobject protocol is that it provides reflection to CLOS. Reflection is the ability of a program to manipulate as data something representing the state of the program during its own execution. Reflection can be introspective or intercessory, or both. Introspection is the ability to observe and analyze one’s own state; intercession is the ability to modify one’s own execution state or to alter one’s own meaning. The metaobject protocol is both introspective and intercessory." > https://dreamsongs.com/Files/amop-review.pdf Their sense of a "metaobject" in MOP is kinda like "metafunction" in our sense of the thing here. And we hold the "state" of the function not just as forms in code and vars in Clojure's runtime, but also as a map that we can reflect against at run time, turning meta programming techniques you normally get out of CLOS/MOP into a plain 'ol clojure data problem. So the reason it's a map is because maps are easy for us and by making meta programming a map problem (similar to how CLOS makes meta objects a data problem) we're making meta programming easier for us.

And CLOS's generic functions do in fact have :before and :after hooks like :in and :out and :around hooks like :tf-pre, :tf and :tf-end that take a whole environment, so there's some similarities there. But only to the extend we're modeling a function so that we can hook into it at different places.

No, sorry, their :around essentially let's you wrap their version of our :op with extra semantics. But again, different ways to model a function so you can hook into it

You can do all this transformation stuff at run time. Slice and dice plain old Clojure data into a transformer and invoke it anonymously right there in some pipeline without giving it a name if you want.

You could separate a separate associative interface just for method/IFn impls out from some user facing one, reserved for -assoc interop with the core fns like assoc, like assoc-method and the like does. Working on the impls and sharing them would be way less ergonomic, but then you could give your user a better-map-like thing while still storing the impl as data

But transformers are specifically about delivering an IFn interface as your main consumer interface - it's for making functions - while giving a luxurious data interface for storing and sharing the implementation details

But if all you're doing is providing some fancy map like that that will remain that one thing for a very long time, and not some thing that can evolve reliably like transformers, then you might as well not use this "luxury data interface" and just replace assoc/dissoc methods and ship that, easy. That's what dyna-map is for.

Haha nice. Yeah, sounds like you were in some front end trenches :)

Yeah, sometimes there's also issues with developer inertia, where even good ideas are avoided because it costs too much to move everybody off the old thing. I do it too - as much as I like some libs, it's rarely worth imposing that package debt and education debt on all your downstream users. So I tend to not even use my own libs in my own other libs 😆 so that a user of one lib doesn't have to also consume extra libs they don't care about. And taking this dispacio lib as an example - I wouldn't use it in a place where a defmethod could be used because then I'd just be adding more cognitive burden on all future maintainers. So I've only ever used it on some non public projects, with complex dispatch scenarios for exploratory programming involving an allocator. If I made that allocator public and the code that I end up with doesn't really need polymethods, I'd just use vanilla defmethods and I'd remove dispacio altogether.

Just so as not to impose the dep on my users

I've lost track of what's being discussed here. I feel examples might help. Right now, it sounds like wanting the ability to monkey patch every single expression from anywhere arbitrarily? I feel at some point, there has to be boundaries, because too much reuse creates another set of reasoning challenge, and grows the cascading effect of breaking changes. With tong-yi it seems that the body of functions is the boundary, you can't reach in and modify those. But you can add or remove composition of functions. I'm not totally tracking the benefits still over default Clojure mechanisms though to be honest.

For example, in the readme it creates add and add-then-inc functions. But I don't see why it's more powerful or more tenable with tong-yi then doing:

(defn add [a b] (+ a b))
(defn add-then-inc [a b] (-> (add a b) (inc)))

Well, wrt the discussion with Lidor, he's more interested in the dynamic map thing, which also has its own monkey patch semantics, you could say. But that's not to be confused with the method being used for the function transformers. At the function transformer level, what's being offered is a way to define functions as data so that their construction isn't closed over by a function boundary. And the only reason that really helps you is if you have the type of program that has lots of functions that are very self-similar. And that can either be an object system or a GUI system, or it could be a code base that is very very sensitive and you don't want to change anything, you just want to make new versions of everything as you go

It seems the only benefit is if you want to fork a long chain of composition in the middle. Instead of writing the chain again from the start, you can take the transformer from the midpoint and build a new chain off of it ? But you can also do that if you make a composed function at every step. Instead of one function composing a big chain. No?

Yes, wrt to the benefit. I'm not sure what you mean by " composed function of each step," that inherently closes over implementation details if I understand correctly

Ah now I get the conversation that @lidorcg was talking about. Not sure it's a fair take here though. Because the implementation of Persistent Map in Clojure is done in Java. As I understand that's one major limitation. If it had been done in Clojure, I think the problem would not exist no? There'd be a Protocol for it, or a bunch of multi-methods, or just some namespace with functions.

I don't think he was saying that difficulty is a problem with Clojure expressivity. He just really needed what dynamap provided. I think his point about the expression problem was moreso about my function transformer concept, orthogonal to his contention with deftyping a map, iiuc

I mean something like:

(def foo (comp inc +))
(def foo2 (comp #(* 2 %) foo))
(def foo3 (comp dec foo2))

(def foo4 (comp inc foo2))

(def foo3 (comp dec #(* 2 %) inc +))

If your functions are so small that they don't need to close over any complicated implementation details, like that, then for sure you're probably better off sticking with atomically small functions

But most of our functions have these let bindings with complicated implementations

I'm also saying this to try and understand the benefits of transformers. I take it that what's made into data is basically the composition of functions together. And because it's data, you can just add a function in-between two functions currently calling each other by associng into the list. And you can fork the list if you need to build off from the middle of the composition, etc.

But the grain is still the functions you have no?

If you have a too big function that does too much inline within its body. You're still stuck

The closing over part I didn't get. Does transformer combine state and behavior together? Like objects or closures?

Lol like a closure-object

A clobject

When you say "new behavior". You just mean, if you want to do something to its output or input? It's still limited to the boundary correct? Or can you like replace a chunk of its behavior?

It's a type that implements both associative and IFn interfaces

You couldn't replace a chunk of the :ops behavior, right? But the new transformer you derive from the original transformer with the original op, by adding the new behavior via data instead of via closure, then, yes, you can slip behind the boundary and change for a third time. What happens, while saving your self some implementation pain

I mean more, would you model say a Point along with the operations to move it around?

{:state {:x 0 :y 0}
 :op move}
 
 
(move 3 2)
;;> {:state {:x 3 :y 2} :op move}

Nah. These aren't for storing application data

The fact that they're maps are an implementation detail that only the fn writer needs to know about, not the fn consumer

Ok. So maybe then I'm confused what you mean by: > By adding the new behavior via data instead of closures > Can you show what you mean by "via closures" ? I mean an example, what would the non transformer approach look like?

It's more so a question of implementation details between move and move-faster, not the data those functions operate on. And the fact that we're using data to bridge between move and move-faster is just an implementation strategy - not about making existing data more functional or whathaveyou

I think maybe I misunderstood your use of the word closure. Cause I think of a closure as like:

(let [a [1 2]]
  (defn foo [b]
    (conj a b)))

(defn add-and-inc [a b]
  (inc (add a b)))

Both. You could attach just data to a function transformer, as you would when closing over data with a fn - but you'll be able to change that data in downstream derivations, if you choose. Also, you can change the add and inc later. And yeah, you could be storing mutable state in a fn. We do that with stateful components in Reagent, and there's a time and place for that, with caution, but in general we don't do that and you wouldn't normally do that with function transformers either

I know your example wasn't stateful - was just closing over data - but either way, same deal

Just imagine instead you did this:

[defn add-and-inc [a b]
  [inc [add a b]]]

and then you had some fn that just consumed all of your "function vectors" at run time, letting you treat your code like data, but without the macro compile step

Hum.... Do you mean that if I have a transformer that does something like add then inc then double then dec. I could tell the transformer to swap the implementation of inc to something else? Or I could remove the double from the chain? Or I could add a println in-between each step. And so on?

Yes

You can also just change data that would otherwise be closed over. World's your oyster

Ok, I didn't realize the transformer also added a dispatch table.

no

How do you use it with just data ?

It's not a dispatch table

Oh, you mean it's not that the implementation of inc would change, you'd just swap inc for something else?

Any manner in which there's a chaining of inc, dec, double, etc. is entirely dependent on how the user structures their transformer function. Going in and changing something in a function transformer is entirely dependent on how the author structured the data within the function transformers. I'm giving a little structure, but the author gives the rest. Swapping out double, for instance, depends entirely on where you put it

You might have made double an effect that takes place on the inputs. Or the outputs. Or just before the op. Or just after the op

The author has decide where things should go, for the function, in the data structure

And they have some responsibility to not make a mess too

But in my testing, it's all been pretty obvious and self explanatory

Right, but the key piece is that you can modify this definition by just assoc/update into the transformer map

Or in-effect, you can reuse this definition, and only change a small part of it to create a new behavior.

https://github.com/johnmn3/ti-yong/blob/main/anatomy_of_a_function.png

There's a picture of a function. Just imagine each part gets hung under a different key in a map. Now you can edit the function like you would a map.

Just trying to see if I understood. So say I have: :op + So I add some numbers. Now I want to inc and then double the output. And on the input I want to throw if negative. And I want to print after adding, incrementing and doubling. I can do this with transformers right? And then say I wanted inc to actually increment by 5, not by 1. So I'd update inc for #(+ 5 %) in the transformer map correct? And now I'd get the same function as before except it increments by 5 instead of 1.

Yes exactly. And all you had to do was update the inc part of the data structure, without having to talk about doubling and the other stuff all over again.

Ok, I see I see.

It's a very interesting idea.

It's weird but it has potential

Something about it feels clojurey to me though. Such a dataish way of managing functions

Like, transformer fns also share persistent data structures together too, as persistent hash maps

And you could manipulate them at runtime like macros over functions

I suspect the downside is that the programmer cannot see what each transformer are doing. Though you could inspect it in the REPL. I'm curious if, instead of using assoc/update and so on. You just copy/pasted a statically written transformer map. If the usability would improve? Cause I feel that's going to be the confusing part. Answering: "What is this doing here?".

I'd argue the same exists for functions

You've gotta follow your defs

And if you did that as well, you might be able to also compile them all, and make it more efficient.

Ya, but defs can be followed statically.

But, if you don't have the sources in whatever dep, you're screwed. With fn transformers, any downstream consumer can still inspect the whole thing

With transformers you have to mentally evaluate and think about what the resulting transformer is after all the update/assoc operations took place.

How is that different than functions?

You have to know all of the semantics of all of the fn your composing in your fn, or whatever fn/behavior your fn is wrapping

Jump to definition?

But also I mean the composition is shown.

In a normal function

If the source is accessible

But with transformers, the composition is dynamic. So what you see is the code that is constructing the composition.

same goes for fns forms, right?

You can say less with transformers. But all the impl details are still there

And the structure is even there for those who don't have the source

Hum... Not really.

(defn foo [a b]
  (when (or (neg? a) (neg? b))
    (throw (ex-info "Can't foo negative numbers" {:a a :b b}))
  (let [result (add a b)
        _ (println result)
        result (inc result)
        _ (println result)
        result (* 2 result)
        _ (println result)
        result (dec result)
        _ (println result)]
    result))

That's my example of before. Maybe what would be transformer version of this?

There'd be lots of ways you could structure that as a transformer. Do you want to keep the 4 let-bindings coupled to one action or would you like to decouple them?

You can still make them depend sequentially on one another, will keeping them as separate data

Well, so what I mean is. The first transformer might be statically defined:

(def foo
 {:op add
  :before #(when negative input throw)
  :out [+ println inc println #(* 2 %) println dec println]})

Now in the function version. When I go to make my foo2 which does all the same things, but instead of inc it should do #(+ 5 %). What you'll have to do is copy/paste it. Or you need to break what comes before and after as their own function, and then refactor foo and foo2 to call those before and after functions and change what's happening in the middle. Whereas in the transformer version, you would do: (def foo2 (update foo :out assoc 2 #(+ 5 %))) Which is super cool. But also, now the readability, I have to mentally evaluate the changes made to the foo transformer map and imagine what it looks like.

So, not tested, this just spit balling, but this is how you might start structuring that:

(def foo1
  (-> transformer
      (assoc :op identity)))

(def foo-add
  (-> foo1
      (update :tf conj
              ::foo-add
              #(merge % (let [result (apply add (% :args))]
                          (println result)
                          {:args result})))))

(def foo-inc
  (-> foo-add
      (update :tf conj
              ::foo-inc
              #(merge % (let [result (apply inc (% :args))]
                          (println result)
                          {:args result})))))

You could make the add your op, and layer the rest in as :tfs (transforms)

A transform gives you access to the whole env, which has the args

But you can also add those in the :in hook, which can change the inputs before tfs run

And there's an :out hook. You could spread these different steps in the let binding out across different hooks in the fn

There's also a hook that runs when a new transformer is defined. There's a lot of ways you can structure you functions. It's kinda like function transformers are to functions what interceptors are to ring

Ya, my concern I think is the readability. Now if I look at that code, it doesn't show what foo does. It shows a bunch of modifications to a transformer map. If I could see the resulting transformer it would alleviate that I think. But in a way, it's the cost of reuse versus copy/pasting.

And really, I need to do an impl of ring/pedestal on top of function transformers, to really stress test the idea

foo-add and foo-inc, as functions, don't say what foo does either. In my version there, it doesn't do anything, but I could have made add the operator and called the base transformer foo-add.

What I was wondering. What if you rewrite this, and don't use merge or assoc or conj, etc. Just statically write out the transformer, and copy paste it and make the edits manually when you want a different variant of it. The readability would be a lot better I think.

Ya, better names could help. But in my code example, the function is also just called "foo", but you look at its code and it's very clear what it does, it's all laid bare.

That defeats the purpose here of not having to maintain duplicative implementations

Well, if you want all four of those values to be coupled together in one let binding, you can do that in a transformer too, and then everything will be visible in one place. I was just showing how you could decouple those from one another. Here's another way:

Haha, right. I think that's the trade off ya? You lose readability for reuse.

That would imply that readability implies having maximal context in front of you - which could mean having giant functions, and not having small functions that depend on a tree of other functions that aren't clearly in front of you

But sometimes readability is not having all that in front of you

So I'm not sure having duplicative versions of code in a code base, and all those downsides, outweighs the upside of being able to see everything about an impl right in front of me in one place

But if it means risking downstream consumers that you can't easily test, then yeah, just copy and paste the old code to a new fn for the new consumers and let the old ones be bygones

But with transformers, you don't have to do that copy and paste. Just branch it. And if there's a security vulnerability in the parent one, the branched one will benefit, without having to chase down the 50 versions of an impl that might have that security vuln

I think what you lose in readability is familiarity. But with small functions, your just as likely to not have most of the semantics your using right in front of you

The second point is another downside of reuse. Like having control over when you want upstream changes to reflect and when you don't. But I was talking the first point. You're right, readability needs a balance. You don't want to see too many details, or too little. I take it transformers allow for finer grain of reuse than normal functions. Which would result in seeing less, and having to mentally piece together more parts to understand the full picture. It begs the question, what's the better balance for readability? Is transformer going too extreme and it will hurt readability, or maybe it won't and might even improve. And probably it depends on the exact problem you use it for.

Have you heard of ravioli code? https://wiki.c2.com/?RavioliWiki

I think vanilla fns and fn-transformers are going to have similar LOC and visual complexity for the average case. Once you start building an object system, or a UI system, or again a codebase where nothing should ever be deleted, that's when you'll start to see significantly less lines of code with fn-transformers. And that's just really not the average case. In those cases though, you really, really don't want all that duplicative stuff. It becomes a a technical debt and a just a cognitive burden on whatever project... mmm don't think so

Oups, this is the correct wiki: https://wiki.c2.com/?RavioliCode

So there's some objections to OO that some might wonder, if they apply to these transformers.

And one thing I'd say is that it doesn't push you to object-ify everything. Creating a new transformer in Clojure only happens because your application and business requirements needs it, not because some object system wants you to follow a bunch of ceremony. I don't think you'll see transformers look much different than fns. There'd just be a lot less of them in object-like systems in clojure

Ya, it's normally in the context of OO. But the essence is, when you break up the logic into too many small pieces spread across the code base. Now you can't see the full picture. Each piece is too small to understand what role it plays. And it can become a different source of confusion.

It's made worse when those pieces are also being glued together in implicit ways. Understanding how they're inherited, injected, annotated, etc.

Here's a version of fn-transformers (in a different impl where making a new one involved passing a special map in, but that was mostly just for demonstration purposes) https://github.com/johnmn3/comp.el/blob/main/ex/src/todomvc/views/comps.cljs

As you can see, you can structure fn transformers in almost exactly the same way you'd structure fns. You'd have roughly the same amount, in a lot of your namespaces. You can keep mostly the same logic. It'll just look a little different, as data. The real difference is when you want to go and make another component that derives from one of those components. That's way easier than the traditional way to do it

At the bottom of that file, you see this example:

(def filter-anchor
  (comp/a
   {:as ::a :with [styled/filter-anchor a/selected?]
    :props {:on-selected #(update % :style
                                  assoc :border-color
                                  "rgba(175, 47, 47, 0.2)")}}))

(def filter-all
  (filter-anchor
   {:as :all :with a/void-todo}
   "All"))

(def filter-active
  (filter-anchor
   {:as :active :with a/void-todo}
   "Active"))

(def filter-done
  (filter-anchor
   {:as :done :with a/void-todo}
   "Completed"))

There, each of the last three defs are deriving their data from filter-anchor. But are they much different than how they'd look if each filter just wrapped filter-anchor, as functions wrapping it as a function? No. Mostly the same number of characters. But here, we're leaving things open, which leaves open a lot more possibilities for growing code in a more strategic way

In the new version, fn-transformers are just top level maps, so it's even simpler than the above. In some ways, simpler than fn definition

There's a bug there too. I should have added the a/void-todo to the filter-anchor's :with vector and not repeated it for each child transformer.

@didibus here's another possible impl of foo that keeps all for let bindings in one definition, but splits each across 4 different transforms, allowing you to see things together but also changing any one of the steps, after the fact, as a downstream consumer:

(def foo
  (-> transformer
      (assoc :op identity)
      (update :tf conj
              ::foo-add
              #(let [result (apply add (% :args))]
                 (println result)
                 (assoc % :args [result]))
              ::foo-inc
              #(let [result (apply inc (% :args))]
                 (println result)
                 (assoc % :args [result]))
              ::foo-times-2
              #(let [result (apply (partial * 2) (% :args))]
                 (println result)
                 (assoc % :args [result]))
              ::foo-dec
              #(let [result (apply dec (% :args))]
                 (println result)
                 (assoc % :args [result])))))

Or something like that. I could probably come up with something prettier. There's lots of ways to do it. You have an environment being passed to each tf, and those'll be passed sequentially as the updates happen, so you can do (assoc % :res1 result) and in the next tf you'll be able to (:res1 %), so you can structure things however you want.

But by splitting the steps into data, any downstream consumer can modify those values, without having to entirely reimpl their whole ancestry to change that one, third let binding, or whathaveyou

I definitely think there's potential with the idea. Even though I wonder about some of the tradeoffs it makes. But ya, I think it's pretty neat.

Ya, I gotta port pedestal to transformers to see how it scales

As a calling convention, it's gonna have a performance tax vs plain fns, as it leans heavily into varargs to make things easy to impl

So it might not be very optimized, but I just want to see how DX feels at that scale

There's been some chatter about trying to improve vararg performance across the board in clojure though, which might make things like these fn transformers even more practical

changed the map constructor name to dyna-map and added some tests. The api is still a little in flux but I'm going to start writing docs soon.

Interesting talk on "https://youtu.be/ipceTuJlw-M?si=lJ-LakN524u1uiH6," which we're well familiar with in Clojure. He argues that with object oriented programming, your call graph goes in all directions. With functional pipeline programming, all the arrows of causality point in a single direction. So in a way function composition pipeline programming forces us into a discipline that ends up creating a DAG, which makes reasoning about causality in code much easier. I'd argue that transformers bring in some of the benefits of objects while still maintaining Clojure's pipeline orientation and unidirectional mental model of causality, because the user facing API of the transformer is invoke/apply of a plain 'ol function Lego block for easy functional composition.

And of course when composing transformers as data, with the "backend" API, those are all immutable data transformations that do not change prior definitions for existing users of those transformers - still very unidirectional as a mental model of usage

Interesting... In Rich Hickey's talk, "Are We There Yet," he ended the talk by asking, > "I’ll leave this an open question for everybody else – is there a way to reconcile this with Object Orientation? Could we separate perception of an object from its identity enough so that we’d still get the benefits of objects but we don’t get a mess later?" > I'd argue that this concept of transformers may be taking a crack at answering that question. If you were asked to come up with a clojurey object system, building functions out of maps, vectors, lists and sets seems like a pretty obvious way to do it.

Specifically though, in the strictly Rich Hickeyan sense of the word "object," transformers are immutable entities that do not smear their identities across time so they are not objects. But what benefits do objects get you that immutable transformers don't? And you could drop a transformer in an atom (or put atoms in the map) and now the function has persistent identity semantics, but you probably don't need that for 90% of the things we want from objects, like horizontal implementation reuse. Maps are all we need for most things (obviously)

I feel objects are more of a state thing no? They can act as a kind of module and module composition/inheritance as well, but their main purpose feels more about state management.

For example, in the sense how in Haskell, typeclasses and types are not really called OOP, even if, if you took away state from Objects, you'd be left with basically something almost the same as Haskell types and typeclasses.

my experience with smart-maps in pathom was that it just drastically changed the programming experience away from vanilla clojure to something else. I think pathom overlaps a lot with the forward chaining rules algorithms, but it also tries to incorporate in validation. If you liked the idea of smart maps, it might worth reading about some forward chaining systems. Odoylu-rules is a pretty minimal implementation that i have played with. It can be quite frustrating to think in the forward chaing fashion because it forces a certain amount of "if this then that" attitude, as i think most of us are far more used to backwards chaining aka graph queries like you would see in datascript that are more akin to "i need this to do that"

Yeah, this thing isn't really about providing smart map functionality. Though you could maybe build smart map functionality on top of this. This is the "dynamic map" impl I ended up going with: https://github.com/johnmn3/step-up/blob/main/src/step_up/alpha/dyna_map.cljs

And out of the box that should act mostly like a map

(minus bugs)

It just makes it easy for you to swap out methods

What are "forward references?" side effects that run on assoc/dissoc?

I don't really have a firm understanding of the literature to use the right terms. I was saying that forward chaining is a way of organizing information so that when it's gathered, a function fires. This is very similar to what re-frame does. It keeps a lot of the information in memory, in such a way that new information will optimal trickily to the subscribing sets of information. It's another way of saying "it keeps indexes".

ah

With their resolvers, right?

Which depend on the "smartness" of the maps, iirc, it's been a while since I played with it

I like that forward chaining pattern. I think https://github.com/chr15m/sitefox brings that idea to the backend with good effect, iirc

Or not... I played with some re-frame event system on top of sitefox at some point

I also haven't used pathom for a while, i don't consider myself an expert. Also Words usually fail to do valid comparisons. But in the exmaples in the https://pathom3.wsscode.com/docs/smart-maps/, it's easy to draw comparisons to a rules engine. Both a set of data (e.g first-name, last-name) and when that set is completed, it will likely cause the corresponding function to run (str first-name " " last-name). Simiarly in odoyle rules:

{::full-name
     [:what
      [::global ::first-name first-name]
      [::global ::last-name last-name]
      :then
      (str first-name " " last-name)]}

the "then" could insert back into the database and so be used to trigger more "what" rules.

Yeah, transformers are different than that - whole different purpose. My understanding is that pathom solves a graphql kind of problem moreso. There's similarity though with how smart maps carry an https://pathom3.wsscode.com/docs/environment/.

transformers are immutable by default

I think pathom/smart-maps are about caching/indexing over distributed systems, right? there's inherently a network thing involved, right?

Though you can use it locally, as a replacement for re-frame, but it's also used on the backend, right?

transformers are "structured functions" or "map driven functions" that you can use to solve problems involving highly self-similar component systems or any problem you'd normally use an object system to solve, but just using maps and clojure idioms

Where the data structure becomes the programming api of the function and the IFn interface is the consumer api, allowing you to do some powerful runtime meta programming stuff

But these dyna-map things, underneath transformers, are supposed to just be supersets of vanilla clojure maps, for easily overriding the default behavior of normal maps

I'm not sold on the "dyna-map" name but it has a ring to it

Theoretically, you could add in stub implementations for all protocol/interface methods across clojure core libs and have some generalized proxy-object that you can build any datatype out of just by passing in kv impls at runtime. Is that something like what you were suggesting @smith.adriane?

Have you checked to see if there's any prior art?

I'm aware of potemkin and smart maps

And proxy for some ad hoc type impls

and reify

Is this similar to what potemkin offers? Is this different?

Potemkin's def-map-type doesn't expose all of hash-map's implementations for reimplementation - only a few that it thinks would be useful in an API that is higher level than existing type def tools

This pre-defines some impls, but lets you override any of them

Well, it predefines all of them, and lets you over ride any on a one-off basis

So this is moreso about making a stub type that has a data interface where impls can be attached and detached at runtime

I think potemkin is moreso about being a higher-level deftype for maps

So, this allows you to get lower-level than potemkin and provides an associative, clojure data interface to attaching and detaching behaviors at runtime, which I'm not sure potemkin does, but also provides a high level interface like potemkin

And none of this stuff is macros. Doesn't have to be

I'm a little confused about what's happening. • You keep saying "hash-map's interface", but I think you mean just map? I know that sounds pedantic, but it's conceptually important that you can use a map and not care if it's an array map or hash map. • There's already an extension mechanism that works at runtime called deftype. I'm not sure why this other stuff is needed. • methods like conj already have implementations that satisfy conj. I'm not sure I want conj+ that is similar to conj that does extra stuff. If you're deftyping something that adheres to the map interface, then it doesn't need a new name. It's not a "new" type. It's a map. If it's an implementation detail, that's fine and it doesn't matter what you want to call it. Hopefully, it's not something a user of your library has to think about.

"Just use maps" is a reaction to the OO world where "new" types are created that are just goofy maps.

Yeah, I just mean the map interface

And I'm thinking about adding another associative protocol assoc-method/dissoc-method, so you can provide a type to a user that customizes those interfaces. But right now, the associative interfaces are reserved for programming the type, so as to make it easier to construct a type as data at runtime or whenever

And the types I'm constructing here mostly are functions

So the consumer API of transformers are their IFn and the backend programming interface of transformers is the IAssociative interface

There's also the option of making Transformer be a toggle-type kinda thing, like transients versus persistent. Calling transformer! switches the object to the backend data API, then calling persistent! switches the associative APIs back to the user facing ones you made in the transforming state 🤔

But that's really not necessary, I think. The backend data api should be doable with just different method names like assoc-method/dissoc-method. They'll only ever be used a few times in the life of the app, to construct the custom types.

So I guess I can't really call it a tool for making custom maps until I give better separation between the type-user api and the type-author api, so the user API can't accidentally make a backend change. Right now I'm just catching keys I've tagged as method implementation keys on the user facing assoc interface, which isn't a clean separation, just a temporary fix

Well, you can let the user override your assoc/dissoc, but then you've disabled your access to the easy data manipulation interface of the type's implementation, essentially freezing your types implementation into place

There's too many hard problems for my brain to handle at once: • new runtime extension mechanism • combining maps+IFn • some mixin inheritance mechanism on top of the above? • some UI component framework on top of the above?? Unfortunately, I don' think I'll be much help unless there's a narrower problem with a good description.

Personally, I would dramatically narrow the scope and try to sort out some of the smaller problems before trying to create general solutions for all the problems involved.

lol yeah sorry, there's a lot of moving pieces here now. But this question/poll is specifically about the lower level... Well, sure, I might split the map-proxy-thing into a separate lib. I'm just trying to brainstorm a good name for it.

And I tagged you here because this proxymap thing reminded me of the idea you were talking about... wasn't exactly sure what you were getting at, but this runtime extension thing is kinda what I was thinking you were getting at

and defrecords weren't going to cut it, unfortunately, because I can't override the IAssociative interface on records in clj

So I needed a new type name, for a thing that could be used to create transformers, and which is essentially a facade type that passes through calls to an inner map, allowing you to overlay new implementations for individual methods at runtime via an associative data manipulation interface

lots of talk about mechanism, but not much about problem

It's a hard problem to characterize. We need a better name for it. It's like a "diamond problem" for hierarchically composed functions, where it becomes harder and harder to add features to deeper functions in the hierarchy without risk of breaking too many downstream callers, and the fact that your functions have closed over so many implementation details, your downstream callers need a whole new upstream implementation to get that new feature. Every function in the chain back up to that feature providing function might have to get touched. This often isn't a "problem" because we're not often hand building hierarchies of functions. Custom front end UI libraries are some of the only instances where I've seen it so pervasively. In those situations, being able to modify the behavior of an upstream function, without actually having to rewrite that function, is a solution to that problem. But yeah, you're not going to see the problem often, I think, especially in Clojure, where we avoid the need for implementation reuse mostly by convention, simplicity and good practices.

And all this deftype mechanism stuff isn't strictly necessary. My first impl of this idea did it all in pure functions. The data api is just better though. Simpler to understand

But that's the rationale for transformers. This underlying map type is also a cat that can be skinned multiple ways, and this method I'm using of stubbing out all the protocol/interface methods, passing through calls to an underlying map unless the method is overridden, is one one way of skinning it. Might actually be a bad idea. Jury is still out. Publishing this code would make that assessment easier, but I can't think of a name! lol

naming/releasing is such a chicken/egg thing

If you're not sure if it's a good idea, put it in an impl namespace and call it define-map or defmap or whatever.

Thinking about just calling them flex-map and calling the library weird-flex

this is going to sound harsh because it's slack, but I haven't heard a problem yet

I'm trying not to argue from the point of experience that every time I've interacted with an application that uses tricked-out maps, the authors have regretted it

merging data and behavior is a choice, and the virtues of plain data have been extolled already

No, I think your challenge is reasonable. But just above, in this channel, there's a thread where @smith.adriane and I went back and forth on this idea for a week and he challenged me on exactly this point for a few days. I'm not sure if I've fully convinced him that it's a good idea yet, but that thread is like 500 messages long, so I don't think you'll want us to rehash it all here

I'm still not sure there's a problem statement anywhere. I still think it would be a good idea.

If you can't explain the problem succinctly, then it's going to be really hard to solve.

That "diamond problem" reference above is my best attempt at a summary version of the problem

It's going to be even harder for anyone to give you any meaningful advice.

I'd really have to build a large scale app, for you to see the utility of it

small apps like todomvc just don't necessitate the amount of component composition complexity to do it justice

I remain skeptical that the best way to explain the problem is to build a large scale app. I think describing a large problem succinctly is very hard, but I do think it's a valuable skill and will improve the design. Even a honest attempt is worth it. Especially, for larger problems.

In the past 2 weeks, I've seen the authors of 2 different UI frameworks explain how they created systems where their components can live as code data until a final render step, so as to ameliorate various composition problems they're having. By not closing over those impl details, and letting them live as data as long as possible before final render, we get lots of benefits. Here, we get those benefits too, but without the downsides of having to transform the whole world all at once in some series of inscrutable world transformation steps.

I've also seen apps in the wild built entirely in data

Lot's op powerful stuff you can do when you don't close over the details of your component's composition

I'm not asserting that there isn't a problem or that your intended method doesn't solve it. I don't know.

I mean, at a high level, it's the problem of data hiding behind closures

It's that general

It seems like this is a large enough problem that scattered slack messages is not a good way to explain the idea. I think if you want people to engage with the idea, it requires a longer format like a blog post or readme.

"Ever wanted to get at some of the data or implementation details out from inside the scope of a closure? Sure, you can have your cake and eat it too"

Yeah

Too general like that almost doesn't help. It's too abstract. Experienced devs have a hard time remembering wanting to cross those closure boundaries too

it's usually a newbie desire

"Hey, why can't I just have that data? right there? it's right there!" "Because you just can't, young padowan"

But what if you can?

You probably shouldn't, most of the time

It's like macros

This is macros over functions as data using functions, and they should probably be leveraged that way as sparingly as macros

But we can cross that closure boundary when we actually should be doing that

And we act like it's not possible, but it totally is

And without having to store the state in an external registry

Which is obviously another way to not close over data and/or impls

But that way is prone to memory leaks

@smith.adriane: It seems like this is a large enough problem that scattered slack messages is not a good way to explain the idea. I think if you want people to engage with the idea, it requires a longer format like a blog post or readme.

I would actually recommend explaining the idea first, before implementing it.

One issue is that my approach would largely go about it in mostly the opposite order: 1. define problem 2. explore possible solutions 3. implement solution It seems like you're doing it the other way around. It's not that there's only one way to do it and it's my way. It just means that I'm unable to offer meaningful help if you're approaching the problem from the opposite direction.

Well, I started with a problem, but the code base was proprietary 😕 So I no longer have access to the example that necessitated the solution. I'll continue to think about how to better explain the problem space though.

Okay, so I still have to work on the docs, but here's a user namespace that goes over some of the features: https://github.com/johnmn3/step-up/blob/main/src/step_up/alpha/user.cljs

And still need to make a clj version. But this should be good enough of an example to talk about it.

And I need to port the comp.el stuff to this new api, and then one day I may make a large enough example app on top of that, to really show the utility

Hmm, yeah, this has similarities to prototype based object systems, I'm reading. In prototype based object oriented programming, objects aren't constructed from a schema but instead use existing objects

Object systems also allow for extreme late binding, which we're leaning into so as not to close over impls