This will get you the Cartesian product, but I'm sure there's a tighter solution.

(let [l '(:a :b :c :d)]
 (for [x l
       y l]
  [x y]))

https://github.com/clojure/math.combinatorics/ is as close to a built in as there is I think

Thank you, will take a look

Make a set

(contains? (set '([:a :b] [:c :d] [:e :f])) [:a :b])
;;=> true

;; this works in a linear time
(some #(= % [:a :b]) '([:a :b] [:c :d] [:e :f]) )

(contains? [[:a :b] [:c :d] [:e :f]] [:a :b])
;;=> false

(contains? [[:a :b] [:c :d] [:e :f]] 0)
;;=> true

One more:

(.contains [[:a :b] [:c :d] [:e :f]] [:a :b])
=> true

@U01RL1YV4P7 is this java method?

Oh, yeah, java interop. Good point 🙂

Maybe this one:

(remove #(every? nil? ((juxt :x :y) %)) 
        coll)

You want to remove all items where both :x and :y are nil? Should be equivalent for filtering away nil xs, then filtering away nil ys.

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}]
     (filter :x)
     (filter :y))

;; => ({:x 1, :y 2})

@U3X7174KS I think your solution doesn't work correctly:

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}]
     (remove (every-pred #(nil? (:x %))
                         #(nil? (:y %)))))

=> ({:x 1} {:y "abc"} {:x 1, :y 2})

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}]
     (remove #(every? nil? ((juxt :x :y) %))))
     
=> ({:x 1} {:y "abc"} {:x 1, :y 2})

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}]
     (filter :x)
     (filter :y))
=> ({:x 1, :y 2})

if you're doing simple validation (filter #(and (:x %) (:y %)) ...)

@U050ECB92 I'm not sure what is OP's use case, but your code behaves differently:

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}
      {:x false :y false}]
     (remove (every-pred #(nil? (:x %))
                         #(nil? (:y %)))))

=> ({:x 1} {:y "abc"} {:x 1, :y 2} {:x false, :y false})

(->> [{:x 1}
      {:hello :there}
      {:y "abc"}
      {:x 1 :y 2}
      {:x false :y false}]
     (filter #(and (:x %) (:y %))))

=> ({:x 1, :y 2})

My use case is a collection of maps and I want to remove all the ones that have neither :x nor :y values. Sometimes the value is a nil (rather than just the absence of the key), and I want to remove those as well.

I wrote more about in #code-reviews here: https://clojurians.slack.com/archives/C053PTJE6/p1658785569248699

I've only been able to get my simple solution down to like 8.5 seconds which makes it one of the slowest solutions in the whole bunch. I used slurp which was technically against the rules as you are supposed to read in the data (make sure you use the 10x text, not just the text which I see many accidentally do in the HN discussions) line by line. That's where I get confused with having to do that and keep the running frequencies (I tried using a reduce function with the whole line-seq rdr thing but keep coming up short).

At this point, I'm just curious to see what other folks come up with and will probably ask you to guide me through your thought process. I would love to see a solution with transducers because I've been trying to wrap my head around those.

for the buffered input and line-seq:

(defn word-freqs [text]
  (with-open [rdr ( text)]                                                                      
    (->> (line-seq rdr)
         (mapcat #(clojure.string/split % #"\W+"))                                                   
         (map #(clojure.string/lower-case %))                                                    
         (frequencies)                                                                           
         (sort-by val >))))

This is less elegant, but also works with the buffered stream: https://gist.github.com/coyotesqrl/c9e57fee6cf61cafd7128648e83123f7

mapcat ! One tiny little tweak changes everything. I could not figure that out from the error messages I was getting.

I'm getting about 5.6 seconds on that one. Still quite a bit slower than the python, ruby, lua, etc solutions which I still find surprising but much faster already than what I had.

I also notice I get a huge amount of variance when running on my machine. My own solution will be 8.5 seconds most times but sometimes over 20 seconds.

The variance seems to be bigger when actually printing out the results (part of the requirement) too but I'm still learning how to use profilers to inspect it all.

@U01GXCWSRMW Thanks for providing your solution. I'm not seeing it being more performant but it really helps me see how I could keep that running tally of the frequencies going line by line.

Yeah in running it multiple times actually printing out the values I'm not seeing much difference between using the line-seq method and using my original slurp method. I was curious if there would be much of a difference there.

Yeah. Mine's creating too much garbage - a new map for each line. Good place for a transient, but then we'd need to create our own merge-with!. Or drop down to Java map manipulation directly, but that feels like cheating.

i'm surprised that it's that much slower than python, is the python string split based on the regex or a simple space ?

for the transducer part, unfortunately frequencies does not have it, but if you simply augment frequencies with a transducing arity you may get better performances

(defn frequencies2
  [xform coll]
  (persistent!
   (transduce xform
              (completing (fn [counts x]
                            (assoc! counts x (inc (get counts x 0)))))
           (transient {}) coll)))

(defn word-freqs [text]
  (with-open [rdr ( text)]
    (->> (frequencies2 (comp (mapcat #(clojure.string/split % #" "))
                             (map #(clojure.string/lower-case %)))
                       (line-seq rdr))
         (sort-by val >))))

for printing, maybe the IDE is slowing it down, you could print out to a file

Yeah I did change to just splitting on a single space (well a single space and \n because I was seeing different behavior without that). I didn't notice much difference. I am just running in the terminal instead of the editor. It cuts down on some of the variance but I'm still surprised I see so much. My "simple" Rust solution I wrote does consistently in about 2 seconds or under. I knew clojure would be much slower than that but not slower than things like ruby.

yes but if the requirement is only splitting on space and newline, then you can drop the regex and there is most likely a much faster solution (nothing comes to mind in idiomatic clojure though)

Haskell was also way slower than everything else. It might just be lazy, functional languages aren't the best fit for this specific problem.

i tried with a java scanner to avoid the creation of unnecessary datastructures but performances were worse 🙃

I warned ya it was a nerd snipe. hahaha

Java, itself, does great on it.

Reminds me of this: https://wiki.haskell.org/Why_Haskell_matters#The_speed_of_Haskell retort, and why these types of language 'benchmarks' are often a waste of time

Oh definitely, I see the merit in that and can agree these things are mostly silly unless maybe used as an opportunity to learn some low hanging fruit optimizations in the language you are using. But even knowing that, there is still something in human nature that draws you into these things. And even if it's silly, with the speed of modern computers it just physically feels slow watching something take over 10 seconds when you see it complete in under 2 seconds in almost every other implementation.

haha very true (and I myself am also a sucker for these silly wastes of time 😂 )

I haven't tried to implement this problem yet, just btw: I found this video talking about a similar problem: https://youtu.be/R_yX0XjdSBY?t=89 They seem to be optimizing it quite a bit.

Can you write some use cases/ problems for functions like this? (or add a link to some Github repo where you needed this functionality)

inserting a new item in the middle of a vector is not very efficient, it can't actually be done with the built in vectors without completely rebuilding the vector.

vectors are a java.util.List, so they have methods like indexOf (https://docs.oracle.com/javase/8/docs/api/java/util/List.html#indexOf-java.lang.Object-)

@U01RL1YV4P7 in general I have a vec of datastructures representing some thing (like a file, game entity, sound effect, etc.) and I want to keep track of the thing the user has selected last and do things like those I mentioned above [{:id 123 :name "thing" :property 42} {:id 456 :name "other" :property 1}...].

I would use a map, or multiple maps

@U0NCTKEV8 I don't care about efficiency. These are vectors of less than 100 items modified through user interaction. Rebuilding the vector is fine, and that's how I generally do it already. I am using ClojureScript, not Clojure. I know how to find the position of and item x already, I just want a library that does all of these things for me so I don't have to re-implement it every time.

I don't use a map because I care about order.

I'm looking for a library which makes it simple to perform the operations I listed above on a vec.

How about a sorted map?

this kind of thing comes up with people writing TODO apps a fair bit

the thing with vectors is the order is implied by the vector structure, and vectors are less flexible, but maps don't have an implied order, but are more flexible

there is no reason you can't have map with an explicit order defined in some way, either each entry have a number (this can make inserting in the middle tricky) or some kind of before/after relation

I understand the desire for a library that makes your exact approach less burdensome, but I would be surprised to come across a well supported one, because the general advice is to do this a different way

I've tried to do it with a sorted map before but maybe I need to revisit that. Thanks for your input.

> the general advice is to do this a different way What way? What about for your example of a TODO list where items can be reordered?

for example a todo list would be a map (maybe multiple maps depending) of id to item, and each item would have a key :order which is a number

the initial item in the list gets an :order of 1

I guess removing an item would require a re-indexing.

and when you put an item at the top of the list, it gets an order of half whatever the previous top item's order was

when you put an item at the bottom it gets order of twice whatever was previously the bottom

and when you insert an item between two items, you give it an order halfway between their orders

you might want to periodically re-normalize the order to avoid your floating point numbers getting out of hand, but they might be fine for a good long while (in clojure there is a ratio type, which would be great for this, but harder to store in say a database if you would want that)

when it comes time to render the list in order, you sort-by :order

Interesting thanks. This sounds quite a bit more complicated than using vec's implicit order and just manipulating it as I have been.

alternatively you do some kind of linked list (every item has a :next which has the id of the next item, or both a :next and :previous) which might be faster, if you keep a note of whatever the head's id is, and generate the list in order by walking the links (might be faster than sorting)

Yeah a linked list is interesting. :thinking_face:

I have to say the existence of clojure.zip/vector-zip tells me that people do want to do stuff like this with vec . The only problem is that clojure.zip doesn't cover all of the use-cases I have. Likewise Stackoverflow questions like this: https://stackoverflow.com/questions/4830900/how-do-i-find-the-index-of-an-item-in-a-vector Maybe I just need to write a small library that does what I want so I don't keep re-implementing it. :thinking_face:

Many people still use buddy or even friend

I think a lot of people use Cognito, although that's also proprietary

I mean, we used Auth0 before for integrating with GitHub Oauth but we were also using buddy to manage cookie-based sessions

Oh neat, I hadn’t even heard of friend. I’ll look a little deeper into Auth0 and Cognito and see if I can find something that fits my shoestring budget, too.

Unlike in, say JavaScript, a library in Clojure does not become immediately useless if doesn't get daily updates to keep pace with v8. And popular mature libraries are commonly passed on to new owners after the creator backs out. Buddy is still in use by Luminus, which is well maintained. And the author (@U050CBXUZ) has already shown some willingness to take over stale projects if necessary.

I tend to use Buddy, so I would potentially be open to taking it over if it was being phased out. It’s a fairly small and simply library, so I definitely see why it doesn’t need a lot of updates.

Authentication with Buddy video: https://lambdaisland.com/episodes/buddy-authentication

> if it was being phased out @U050CBXUZ I'm pretty sure the claim of being phased out refers to this statement on the buddy-auth readme > NOTE: this project is in maintencance mode, and looking for a new maintainer. which does not appear on any of the other Buddy modules. There are only 20 open issues across the entire suite, but you will find a PR on buddy-hashers written in April 2020 with no conflicts, that (even after a new offer for amendment this January) has still not even received comment as to whether it might be considered for merge. ... So 🤷:skin-tone-3:

... unless @U03DKTAF52P Are you talking about the deprecation warning on https://github.com/funcool/buddy ? If so, read closer. Buddy used to be a single lib, that was split into 4. Links to all 4 appear on that page.

@U90R0EPHA Yeah, sorry, should’ve been more specific—I did mean the “maintenance mode” note on buddy-auth. And I appreciate the tip about those libraries still being fine to use—I’m coming from a React Native background so I’ve got an irrational fear of breaking changes 😅