Zach talks about Principled code have an overall structure and being heirarchical. Where as Adaptable code doesn't have that and is like a graph. What does that even mean in clojure code?

Zach mentions Principled components having layers, however I think layers are abstractions hiding complexity - else what purpose does layering serve? Then how does principled code achieve high cohesiveness, be small and fast?

> Anyone has examples/code of systems/components where it is a bit more clearer what is principled and what is adaptable? When and where do we use principled components? He talks about the overall structure of a program or system. You use both. There's no cookie cutter rule as to how much of something is a separate component, module or w/e. This is the fundamental challenge of software design. It might get clearer when you think in terms of interfaces and implementations in a general sense. This can be applied to components, modules, namespaces, or even just functions: A function signature is its interface, its body is its implementation. As a user, you don't care about what black magic trickery goes on inside of the function. You care about its simple interface and what it does for you. In fact many of the every day clojure.core functions have hairy implementations that you don't necessarily need to worry about. You can just say (map ...) and it just works. Think of interfaces as the adaptive system of a particular layer in your program. In Clojure, they are often merely functions that take data and return data. That's an adaptive system. But inside they are do all of this specific, deep stuff. Recommended reading on this: A Philosophy of Software Design. It talks about how implementations should be deep and interfaces small in order to give you as much leverage as possible. Note that this doesn't conflict with Clojure's (Rich Hickey's) notion of simplicity. Simplification doesn't imply a large interface surface. It merely implies that each part stands on its own without an explicit or implicit dependency of another. > Zach talks about Principled code have an overall structure and being heirarchical. Where as Adaptable code doesn't have that and is like a graph. What does that even mean in clojure code? Hierarchical is a metaphor here, but it can be understood with a real world example. Databases often have internal hierarchical data structures, trees, in order to provide an efficient implementation of a graph or relational interface. An SQL database is principled and rigid inside, but the way we get to talk about data is via relations, which is highly adaptive, expressive and simpler. In the Clojure world you may look at XTDB as an example. The API namespace is small, simple and very adaptive. It mostly just talks in Clojure data structures, datalog syntax and so on. But inside, you also have parts written in Java, others in C/C++ (native LMDB/Rockdb etc. modules). There's all sorts of principled, highly specific stuff in there. > Zach mentions Principled components having layers, however I think layers are abstractions hiding complexity - else what purpose does layering serve? My memory is fuzzy as to what the book talks about here. But generally you want to write your program in a language that is optimised for your problem. You want to talk in terms of the domain that you're tackling. A program has natural layers of problem spaces, or domains. So you tend to separate your program into those layers, each having their own set of interfaces, or their own language so to speak. Clojure makes this very easy of course. In fact a lot of popular libraries talk via data driven DSLs. Think hiccup, malli, reitit, garden, honeysql, datalog dbs and many others. They all have their own little data languages that are optimized for their specific domain. When you write a program that uses these libraries, you create another layer on top of them. That layer exposes yet another interface that is fit for your domain. Again, it might be principled inside but expose a nice way of talking about things in terms of your problem space. It becomes a translation layer that translates your language (so to speak) to the little languages that you depend on (hiccup, honeysql etc.).

Another way of thinking about it is this: The adaptiveness of a system is an emergent property of its interfaces. The principledness of your components is based on their idiosyncratic implementation.

Thank you Denis for taking the time to respond in-depth. I think I am able to understand adaptive systems somewhat. The principled bits though, I can't wrap my head around. The properties of a principled system • Make everything predictably structured. We make a change and we know exactly what will be affected. • All the pieces are tightly connected because we are making strong assumptions. Creates a strong interdependence between everything. This makes it brittle. • Smaller and faster, because of the assumptions we only cater to specific cases, letting us be optimal, use the most efficient approach. A function is a like an interface - hiding the messy details, how does one write a principled component using such functions? A principled component will need to call other functions, and then those functions will abstract away the messy details. The only way I see to not do that, is to have the functions be very specific and not abstract away the messy details. And this seems to go against everything we learn. ... How would the functions inside a principled component be written, such that they achieve the goals of a principled component?

An example that isn't databases: Are you doing frontend? Think of how say a re-frame/reagent application is structured. Your reagent components are principled there. They are idiosyncratic trees that render objects that are very specific to your application. The system around it, is adaptive. You can wire them up freely with effects and subscriptions. If a component doesn't serve its purpose anymore, you can throw it away as a whole. You can add another that does something new. But inside they are quite rigid, specific and quasi optimised for certain use cases.

You describe the EDN analog of JSON Lines https://jsonlines.org/

Analogous to ndjson, I proposed Ned

Very useful for json or edn, especially with larger data or streaming use cases

Yes I do have a streaming use case in mind indeed. Thanks for the terminology, very useful!

lambda island has a lib for this

https://github.com/lambdaisland/edn-lines

Oh this is very useful! First time I encounter it

Glad I asked instead of doing the usual thing of overthinking things in the shower 😂

Although I'm a tad surprised that the focus is on lines and not forms.

There are a ton of existing tools that work well with newline-delimited data. Delimiting forms with newlines gives you a lot of leverage.

I think it's a very practical decision.

@U2FRKM4TW I’m guessing it would be slower if it was based on forms. I’ve done an implementation based on forms, but it was for a different use case

It's not really mentioned here, but writing readable edn is not very straightforward. There are at least 6 dynamic bindings that can affect how pr behaves. Most libraries set only few or none at all which usually is the wrong behavior (eg. the lambdaisland lib). Relying on context can be a footgun. Unfortunately, it's not unlikely that new dynamic bindings will be added that might break things. Here's my current version:

(defn write-edn [w obj]
  (binding [*print-length* nil
            *print-level* nil
            *print-dup* false
            *print-meta* false
            *print-readably* true

            ;; namespaced maps not part of edn spec
            *print-namespace-maps* false

            *out* w]
    (pr obj)))

I guess it would really help if there was a clojure.edn/print rather that relying on regular print

Sometimes you get corrupted edn because some lazy map calls println for debug output. Easy for a tenured clojurist to debug but a footgun for a novice

yeah, an “official” way to produce EDN would be good, given the many different options

Has this topic been raised at http://ask.clojure.org?

related to this, you can put raw unicode into clojure source right? so if one wanted to use something other than a newline in order to get both of (supporting multiline strings, parsing this data format without needing to balance parens), you'd need to pick what, a non-utf8 byte?

happy to post my best current attempt, currently juggling kids so tough to type code. I feel like there must be something elegant and fast here!

There is no solution better than linear 😞

probing my deepest intentions, I guess I just want it to LOOK pretty… and loop/recur and index checking on a vector always makes me think I’m missing some higher order thing

reduce over the rseq of the vector and cons up the result

recur seems to be the right thing, because it does early termination. reduce is more suited for complete traversal no?

You can break out with reduced, but rseq and cons get me out of vector land

This is very much a place oriented problem, so a loop fits here

Hi I'm learning clojure and took this problem to solve, to learn more, And I can't make it more simpler than (code below) this with my current knowledge but I am sad I used atom (mutability) to solve this, because my head still couldn't wrap how to make it immutably? Isn't cons/conj-ing too to an accumulator is same as mutating (indirectly) into a new vector? I also tried loop/recur version but failed miserably because there also, I couldn't avoid mutablity (using assoc). So, my question is, how to think in clojure for problems like this immutably and if immutablity is inevitable in such cases, then, is atom based solution which I used as flag to set/unset is right/efficient enough approach compared to others?

(defn decrement [number]
  (let [done? (atom false)
        digits (vec (map #(Character/getNumericValue %) (str number)))]
    
  (->>
     (rseq digits)
     (map (fn [digit]
          (if-not (zero? digit) 
            (do (let [updated-digit (if-not @done? (dec digit) digit)]
                (reset! done? true)
                updated-digit)) 9)))
     
     (reverse))))

Also, why? 1. I need to vec the result into digits inside let, without it rseq isn't working. 2. I can't rseq in the end the map output, is using reverse and vec two times traverse the whole vector twice?

Reframing a bit How did you end up with this problem? Could another representation work better?

I am trying to understand basics of language by trying simple problems like these, so I don't know what is the actual problem here about, but I saw it as simple vector manipulation problem and tried out as beginner.

Question for meant for Sam

Because maybe it could be approached like in hardware by representing negative numbers in a complement form then just doing addition

It’s a slightly simplified version of a portion of Knuth’s algorithm M for generating all partitions of a multiset, implemented in clojure.math.combinatorics

@U04FLJWMEPP nice work, I can add some comments here!

@UK0810AQ2 so a multiset would be a set with duplicates, like [1 1 1 2 3 4 4 4 4] and the algorithm represents this as a vector of “multiplicities” of each unique element. So this would be [3 1 1 4]

and a partition is some way of cutting the vector into pieces that, if you were to concat them together, give you the items in the original list. The way he does it is by generating every partition in non-increasing lexicographic order, so each partition read left-to-right should have its blocks in non-increasing order, and partitions should compare the same way.

clojure.math.combinatorics> (partitions-M [1 1 2 2])
(([1 1 2 2])
 ([1 1 2] [2])
 ([1 1] [2 2])
 ([1 1] [2] [2])
 ([1 2 2] [1])
 ([1 2] [1 2])
 ([1 2] [1] [2])
 ([1] [1] [2 2])
 ([1] [1] [2] [2]))

so you start with, in my example above,

[[3 1 1 4]]

[[1 0 0 0] [1 0 0 0] [1 0 0 0] [0 1 0 0] [0 0 1 0] [0 0 0 1] [0 0 0 1] [0 0 0 1] [0 0 0 1]]

@U04FLJWMEPP here was a first try using the indices to loop backward:

(defn decrement [xs]
  (let [n (count xs)]
    (loop [i (dec n)]
      (if (neg? i)
        xs
        (let [v (xs i)]
          (if (zero? v)
            (recur (dec i))
            (let [tail (repeat (dec (- n i)) 9)]
              (-> (update xs i dec)
                  (subvec 0 (inc i))
                  (into tail))))) ))))

(defn decrement [xs]
  (loop [stack xs]
    (if (empty? stack)
      xs
      (let [v      (peek stack)
            stack' (pop stack)]
        (if (zero? v)
          (recur stack')
          (let [n-tail (- (count xs) (count stack))]
            (into (conj stack' (dec v))
                  (repeat n-tail 9))))) )))

that second one is probably good for a closer look about how to do all of the immutably

@U04FLJWMEPP here is your code, modified a touch to remove the atom. I totally see why you reached for it and it makes sense. When I need to carry around extra state like that I’ll use reduce like so:

(defn decrement [number]
  (let [digits (vec (map #(Character/getNumericValue %) (str number)))
        rf     (fn [[acc done?] digit]
                 (if (zero? digit)
                   [(cons 9 acc) done?]
                   (let [updated-digit (if done?
                                         digit
                                         (dec digit))]
                     [(cons updated-digit acc) true])))
        [xs]   (reduce rf [[] false] (rseq digits))]
    (into [] xs)))

I switched your if-not conditions to if ; I also removed the redundant do around your let block.

I reduced across the reversed sequence, cons-ing onto the beginning of the accumulator and calling (into [] …) at the end

Here is the “transducer” version of digits:

(into []
      (map #(Character/getNumericValue %))
      (str number))

I also use destructuring to only take the first item from the pair that we’ve been using for reducer state, throwing away done?

@U017QJZ9M7W I'll go through the partition M comments and more about it on online. Also, your first solution for decrement using loop/recur resembles mine loop/recur where I used assoc (I guess same bad as using atom), but I really liked the second non-index tracking solution, I'm still digesting it all, I'll put my noob loop/recur version here for your insights on it, and will try to learn more about difference in two approaches, and huge ++ for improving my noob map version using atoms, I really need to learn this non-atom accumulator cons-ing version to apply on such problems.

Just for the record, here's mine loop/recur decrement version I tried using assoc 😅

(defn decrement [digits]
  (loop [i (dec (count digits)) digits digits]
    (if (< i 0)
      digits
      (let [new-digit (dec (nth digits i))]
        (if (>= new-digit 0)
          (assoc digits i new-digit)
          (recur (dec i) (assoc digits i 9))))))) 

No, I think the assoc is fine! I think it was faster than my subvec thing when I tested it

The partitions-M code confused me for literally probably 10 hours

I know my solutions are really beginner versions but I'm currently first implementing an imperative solution to practice problems in which I'm expert 😂 and then trying to implement them using higher order functions clojure provide. So still getting to understand that functional thought process to solve problems, I know this isn't right way but do suggest me some right ways to start approaching problems functionally inside clojure.

I like writing out these algorithms in different styles to see how they feel and then how they perform. Also that sounds like a fast track to getting good at Clojure, what a great approach

why not just work with numbers and convert them to their bit representation? then decrement is trivial

@UK0810AQ2 in the “real algorithm” every “digit” rolls over to a different number

Kept in a different vector

@UK0810AQ2 maybe there is a clever way to do your idea and still make it all come out!

Damn, well these two loop/recur and atom version took me 3 hours, so I'll try not to dive deep into partitions-M problem as I just started swimming 😂

It’s all good practice!

you can also loop backwards with a carry binding initialized to -1 and stop when it's 0

I assumed this decrement problem will be part of implementing some huge number arthimetic operations, where inputs are in string which are converted in vector form as no datatype can contain such huge numbers (yes there is bigInt, but I thought it might be someone's itch to re-implement it, so I just jumped on it, as it seemed really simple problem from my imperative eyes), but it turned out to be totally something else 😂 Had great fun thanks @U017QJZ9M7W for putting such itches.

I've done my fair share of hardware engineering 😅

here is a python version of this algo: https://github.com/sympy/sympy/blob/e56ec3aa582b05d306b973ba7290e3e6260df471/sympy/utilities/enumerative.py#L1 and here is the decrease step that we are playing with: https://github.com/sympy/sympy/blob/e56ec3aa582b05d306b973ba7290e3e6260df471/sympy/utilities/enumerative.py#L279-L296

here is a more confusing version of the decrement step in Clojure, but basically a “functional” port of the pseudocode, so very similar https://github.com/clojure/math.combinatorics/blob/master/src/main/clojure/clojure/math/combinatorics.cljc#L864-L901

the data structures are more confusing, since Knuth uses a sparse representation of the vector we were decrementing… I’m planning on doing a writeup of this at some point, but the description at the top of the python block is the best writeup, I’d say

here is the confusing knuth algo https://archive.org/details/B-001-001-251/page/428/mode/2up

pages 428-430

It's look like I'ma dive into this 😂, thanks man for these sources.

Ping @U06MDAPTP who's probably the clojurian most fluent in partitioning algorithms. He probably has a good solution

@U017QJZ9M7W I've implemented some of those algorithms here's the work I've done on partitions: https://github.com/noprompt/meander/blob/zeta/src/meander/algorithms/zeta.cljc#L1-L377

These functions grew organically over time so the API has some inconsistencies in places.

But specifically set and map partitioning is done using the techniques for this sort of thing described in the book.

So if you want a n buckets and you have m elements, the algorithm will make a vector of m integers of with modulus $n$ and then create a lazy seq of the bucketed elements by mapping an "increment" operation that does all the carrying etc.

Partitions in that file have been defined for all collections and positive integers.

There's also an implementation of heap's algorithm and other stuff dealing with permutations and combinations.

Awesome, more good material for studying!

Feel free to copy and modify the code to your needs.

Also, if you make any performance improvements, please share them with me. Partitions are a key part of something I've been working on.

Low hanging fruit would be to use protocols.

There's a lot of cool stuff in that volume of AoCP. I really like combinatorics and I often find myself getting sucked into that book every 6 months or so. 🙂

I submitted a patch that speeds up the partition-M algorithm in math.combinatorics, would be a good thing to check with a benchmark (you may have all the easy wins)

That code also has some nice improvements that make a big difference if you want to generate all partitions with a number of blocks above some min count or below a max

Ah, yeah. I did something like that recently with integer partitions where I wanted to create a lazy sequence of all the pairs of numbers that some to n where n is between some arbitrary inclusive bound [a, b].

I'll keep that patch in mind. I had considered contributing to the combinatorics library at one point but I didn't bother with it.

I think there were some issues with permutations blowing up when I was using it and I just decided that it would be better to learn and implement the algorithms myself.

Heap's algorithm is pretty solid if you have a smallish collection.

Something maybe worth doing would be to figure out where the cut off for Heap's is and switch to a lazy algorithm for a larger number of permutations.

What are you using partitions for?

No use case really. A cryptic bug came up here, and the author of math.combinatorics was the one who got me into clojure in 2010, so I figured I’d put some effort into figuring it out

And then that sent me off on another leg of my crusade against undocumented algorithm ports… that particular one feels like a nice subject for a talk on how to convert an imperative algorithm to functional style

Cool. There was an article I came across several years ago that discussed some techniques for converting eager/mutable algorithms to lazy/immutable ones but I don't have it bookmarked any longer. IIRC its not always clear cut but there are some tricks that work for things like dynamic programming.

You might try searching around the web to see with you find, use phind, or ChatGPT for some leads. 😄