I think (combo/partitions a) doesn't guarantee order and so (= a (flatten c)) is false. Hence (order-preserving-combinations [7 4 2 1 3 2 6 3 4]) is empty

thanks indy

Imo it becomes painful when the functions you use don’t have the same conventions: The data map sometimes in 1st, sometimes in 2nd position. The options being passed sometimes as a map, sometimes as positional arguments. Etc. Your example has a convention: data in first arg, optional arguments in a map as 2nd argument. It looks fine to me.

Merging both maps in one has additional problems: How to separate the actual data (that might need to be sent back in an http response, or stored in a db) and the optional arguments.

Separation of actual data from opts is fairly simple, just selecting a few keys I need. I think though passing the whole context map (combined map-at-hand with opts), with all its content, to one of the sub-functions (e.g. some funs used from add-totals) smells bad. It would break Interface Segregation Principle. Functions would know too much. At least if we talk about OPP. But we are not OPPing here. What do you think about ISP principle in the context of Clojure and that example?

> Separation of actual data from opts is fairly simple, just selecting a few keys I need. Having to select at the end which keys are relevant data and which are not is actually a smell for me 😅

Hi @U023TQF5FM3, just reading about OOP vs FP and trying to "unlearn" years of OOP ;) https://learning.oreilly.com/library/view/object-oriented-vs-functional/9781492048138/ (I have an O'Reilly subscription, the author is Richard Burton). You can find one of his talks here: https://www.youtube.com/watch?v=NHN_mqyjOCg.

Functions don’t really know too much, unless they are looking through the whole opts object, but why would they? They can destructure it at the signature level (`(fn ... [_ {:keys [a b]}`) and keep only the values that are relevant to them. This is actually another argument in favor of keeping it with two separate maps: In your original example, opts is effectively immutable. It gets passed through multiple functions, but it cannot be mutated by those functions. If you mix all of them in a single map, each function could mutate the opts values (since the map would be threaded through them)

@U027H4FHMBK I wouldn't say that all the OOP is crap and needed to be unlearn 🙂 Some of the guidelines are not that bad. I mean, there are some stuff that definitely you should not be replicated writing in FP - I wrote some OOP in Clojure in the past, I learned my lesson 😅😂 Anyway, I will definitely watch that talk. Thanks for posting it 😉 @U7S5E44DB > Functions don’t really know too much Yeah, I had the same thought, but just I didn't know yet why mixing it smells bad 🙂 > It gets passed through multiple functions, but it cannot be mutated by those functions. (...) If you mix all of them in a single map, each function could mutate the `opts` values Yes, the mutation. It makes sense to keep opts impossible to mutate. Although I'm not applying the same rule for the map-at-hand (first arg). > Having to select at the end which keys are relevant data and which are not is actually a smell for me Yeah, I know, but I'm doing it already. Some of the steps create keys that are only needed temporarily - within the -> thread for some of the other steps. Without that, I would end up with a complicated let. Such a let would show what each step takes and produce, but in overall the process function wouldn't be so readable (it wouldn't be so clear what the process does), so I moved that interface of what is needed and returned by each step to the step functions. At the moment the process fn doesn't contain the selecting keys step, but it will be there. I thought that as long as it is within the boundaries of the function it is alright. At the end, if it was the big let, I wouldn't return temporary data. Same here.

what does "txs" stand for? is it data related to ecommerce?

it refers to purchases, sales, withdrawals, deposits, actually all operations that affect balance on the trading account. Why you ask?

just reading the code and wondering if txs was a Clojure idiom

txs is transactions

ahh, I see... the function is doing currency conversions based on British pound?

@U023TQF5FM3 depending on requirements, you may want to use an interceptor-style framework to execute all the steps in the pipeline

you may want to terminate the flow early, or report on individual steps generically

@U050ECB92 can you share me some link on this approach? Googling doesn't give me anything specific

http://pedestal.io/reference/interceptors https://github.com/metosin/sieppari

you can use pedestal's interceptor library divorced from HTTP

(I use pedestal interceptors to handle some things for Kafka and SQS processing. There's one interceptor to deserialize, one to validate the message, one to handle, etc.)

Ok, so this is something similar to what I'm doing. I'll have a look at it. Thanks a lot @U050ECB92

At this point it might be maybe a little overkill, but I will add it in comments in case of refactoring when it expands to a more complicated pipeline

yeah -- same end result, but the mechanism is slightly different, a bit more ceremony

@U01F83MR4KV it is a current default. Most probably it should be refactored out of the scope of this method. Solving problems when they are problems 😄

yea, later, you can remove the hardcoded "GBP" and rename a couple things to explain the code... after you decide on the best design pattern.

@U050ECB92 would an interceptor pattern be more applicable than a transducer (morphing collections into other collections of data, without knowing about data details) ? is a transducer more abstract than an interceptor?

@U01F83MR4KV I think transducers fits better collections' transformations. In my case, and in the case of interceptors, we talk about transformation of a single map (if I understand interceptor pattern correctly). Of course you can use transducers here (functions returning transformation function), and you can combine them, but there is no benefits of using them in this case.

ahhh. transducers would be actually changing the collection keys... not just changing some data

transducers would be over kill for this

The rule of thumb I use is if I see ->> thread, it is time to (not necessary apply but at least) consider transducers.

I've never used rails. Do you mean some kind of a keystore stored within the project?

AWS has a couple tools for "secret" management.

https://aws.amazon.com/secrets-manager/

this is the most secure option because there is no chance your secrets are leaked out

many times, I see projects keeping AWS keys in their secrets... which is a big security risk. You should use AWS IAM and AWS Secrets Manager.

One time, I saw part of a code base leaked out to the public... with EC2 keys... and the team had a $25k AWS server bill the next morning from crypto miners hijacking their account... but AWS was kind of enough to waive the fees.

@U3JH98J4R are you referring to encrypted secrets introduced in Rails 5.1?

rails just has a command that spits out a secret you can put on aws or in your config or whatever

like for signing a jwt or session store or whatever

https://edgeguides.rubyonrails.org/security.html - ctrl+f for rails secret

@U3JH98J4R Ok, so you do refer to a kind of a keystore within the app. I don't think there is any ready-to-go solution for that. If you want to ship your app with secrets embodied into your binary, you probably need to use some based on Java keystores. As an alternative, you can always pass the secrets by env vars, or using AWS Secrets Manager as @U01F83MR4KV already mentioned.

yeah no, currently doing both env vars and parameter store

just generating the things without being on full secret manager is annoying

i literally just use the rails command, but i'm starting to feel silly

let me check if aws cli has a command to generate secrets... I like my cli too...

awesome, aws cli to the rescue :)

https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html

that sounds like it is for storing it in secrets manager

yes

i just want to generate a string i can copy paste into my .env or parameter tore

where do you need to store your secrets? in an exe? or on a server?

I'd be careful using .env because if it gets added into the repo... then leaked

make sure your code stays private... Bitbucket vs Github just as example

.env is not in repo directory even and thats for local dev

we inject secrets via secure strings in parameter store

just for your local dev

you might want to add a git post hook... that refuses to push code if .env file is added to the repo

?

that part is already handled

its just not even in the repo directory and we have a command to load in the stuff

but its also not where we store prod. stuff

ok... just giving you my tips based on how we handled this at a cyber security company... we didn't allow .env secrets

it's not an "either or"... just my opinion on picking the safer route

you should be ok... the biggest risk with .env secrets is GitHub leaks

example... an .env file gets added to a repo before .gitignore is applied

what's the goal here - produce less garbage, consume less memory, spend less time creating the result?

OK, I have 3 points here… 1. The API doesn’t provide this. Yes, it’s theoretically possible to delve into the transient structure, to duplicate how the persistent version creates a seq, but that would need a LOT of code, and almost certainly lead to bugs. You’re basically writing your own implementation at that point. The best way to do it would be to track on the side, like you’re doing. 2. Why use a java.util object? You’re scoped to a let block, so a Clojure volatile would work.

(defn group-by-transient [f coll]
  (let [ks (volatile! #{})
        grouped (reduce
                  (fn [ret x]
                    (let [k (f x)]
                      (vswap! ks conj k)
                      (assoc! ret k (conj! (or (get ret k) (transient [])) x))))
                  (transient {}) coll)]
    (doseq [k @ks] (assoc! grouped k (persistent! (get grouped k))))
    (persistent! grouped)))

Purely curiosity. But yeah, produce less garbage was my intent. Wanted to measure it with few different kinds of data to see what happens. edit: Just playing around with transients. I would absolutely not do something like this in a real-life situation.

@U051N6TTC if the goal is to do less work (fewer allocations for example), putting an immutable set in a volatile doesn't really help

and I suspect that a version of this code that simply used mutable objects would be less complex, and much better performing

yup, but a transient can go there just as easily. And jumping to that seemed like more than I needed to say

I've never seen transients make code simpler

You’ll see that I responded before the goal was stated. But even so, once the choice to use Clojure is made it seems like avoiding the Clojure constructs for it seems counter intuitive. I agree that the HashSet. will be faster, do less work, and have less code in this instance. I balk at it though, since I mostly write in .cljc files

Also, there’s already a commitment to transients here, so why not go all the way? 🙂

(defn group-by-transient [f coll]
  (let [ks (volatile! (transient #{}))
        grouped (reduce
                  (fn [ret x]
                    (let [k (f x)]
                      (vswap! ks conj! k)
                      (assoc! ret k (conj! (or (get ret k) (transient [])) x))))
                  (transient {}) coll)]
    (doseq [k (persistent! @ks)] (assoc! grouped k (persistent! (get grouped k))))
    (persistent! grouped)))

If I were after raw performance, then yes, I’d probably use Java/JavaScript objects. (I don’t target .net)

right, and the first question to answer is "is the performance gain worth the effort"? too often in the real world ™️ things like this are motivated by curiosity rather than need and make application code worse

That was my point 3. I needed nested maps, and while it was faster to build everything with transients, at the end I had to walk the tree to make everything persistent, and I lost all the performance benefits. This use case is simpler, but it reminded me of what I saw. After all, this code is not doing lots of insertions into the ends of the same vectors over and over. It’s jumping back and forth between vectors, and adding one at a time. It’s most likely faster to execute the reduce operation, but not necessarily by very much. Then the final step of moving to persistence will take time.

It’s an interesting exercise, and I encourage this pursuit for the sake of curiosity.

What it taught me was persistent structures work better than I expected 🙂

haha I guess I get a little prickly because this sort of curiosity has led to bad code (at my hands and by collaborators) but I think we are in agreement here

btw here's a first stab at using mutability under the hood (function is still pure - no escape of mutable impl details):

(import (java.util HashMap ArrayList))

(defn dirty-group-by
  [f coll]
  (let [grouped (HashMap.)]
    (doseq [element coll]
      (let [k (f element)
            existing (.get grouped k)
            entry (or existing
                      (ArrayList.))]
        (.add entry element)
        (.put grouped k entry)))
    (into {}
          (map (fn [entry]
                 [(.getKey entry) (vec (.getValue entry))]))
          grouped)))

Well, my curiosity about this was piqued when I saw https://gist.github.com/ericnormand/33d040d1568d1f0253707e42a1cdba85 in Eric Normand's newsletter this week, and I was https://gist.github.com/ericnormand/33d040d1568d1f0253707e42a1cdba85#gistcomment-3804146 it faster. ^ That was the goal 😅

haha I didn't realize this was a challenge, might be time to do some benchmarking

OK I got around to testing it and the perf is hilariously bad

user=> (def input (doall (range 10000)))
#'user/input
(cmd)user=> (time (dotimes [_ 10000] (group-by even? input)))
"Elapsed time: 7657.273975 msecs"
nil
(ins)user=> (time (dotimes [_ 10000] (dirty-group-by even? input)))
"Elapsed time: 121684.700624 msecs"
nil

Well, there's reflective code in there. Here's an impl with typehints to get rid of reflection:

(import (java.util HashMap ArrayList HashMap$Node))

(defn dirty-group-by
  [f coll]
  (let [grouped (HashMap.)]
    (doseq [element coll]
      (let [k (f element)
            entry (or (.get grouped k) (ArrayList.))]
        (.add ^ArrayList entry element)
        (.put grouped k entry)))
    (into {}
          (map (fn [^HashMap$Node entry]
                 [(.getKey entry) (vec (.getValue entry))]))
          grouped)))

user=> (def input (doall (range 10000)))
#'user/input
user=> (time (dotimes [_ 10000] (group-by even? input)))
"Elapsed time: 8568.308124 msecs"
nil
user=> (time (dotimes [_ 10000] (dirty-group-by even? input)))
"Elapsed time: 2955.335518 msecs"
nil

Happy to try to help here if you want to take it to #sql and ask a specific question -- I'm not sure what your actual question is in the above.

Oh sorry, ok. I might have figured it out, but I'll be sure to move to #sql if not. Thanks!

https://github.com/darkleaf/effect you could try ping him via email

I recognize it because I've caught myself more than once doing it myself

but I also recognize that these abstractions can be valuable - I just get fuzzy on when they are worth the effort to sort out vs. when they will waste my time and lead to code that is worse than the naiive solution

I am doing it intentionally though, mostly to see how far I can take the concept without trying to reinvent a type system

It's an experiment rather than a production system

these deep dives/navel gazing moments are also some of the most fun to be had in programming, i’ve found. generally don’t have a lot that’s directly applicable after i’m done but i’ve learned a lot about the problem domain and sated the curiosity

https://fabricate-site.github.io/fabricate/finite-schema-machines.html here it is, my experience report on "redefin[ing] a program as a special case of data structure" with the help of malli schemas