Fork me on GitHub

removed my message for a more relevant channel


I have the following question posted in clj-http: Your help is much appreciated!


if that's real data you might have a bit of a headache


@dpsutton what do you mean by real data? It’s key value pairs where values are strings


He means that :basic-auth string I think...


That auth seems real (although the customer ID seems invalid)


You might want to delete that message @ps although at this point it's probably been archived to the logging site -- not sure if it respects deletes. The Zulip bot does respect deletes so it will at least remove it from there.


The #clj-http channel at least is not logged/archived as far as I can tell so at least it's only in this channel -- and Zulip doesn't copy the body of shared messages. Not sure if logbot does.


@ps You could try :body "customer= cus_IvNWjfw1q2clxS" in the post request. It’s equivalent to --data-raw option in curl IIRC.


Hello i am testing a maven project,where Clojure code is also inside the maven. It works fine , but i want to use a class from Clojure code gen-class , and it said it cant find it


in lein, i did :aot but in maven i dont know what to do to find it


normal call clojure from Java works IFn etc , but i want a class

Alex Miller (Clojure team)16:02:47

it hooks into the compile phase to compile clojure code


it does it automatically? or i have to set settings

Alex Miller (Clojure team)16:02:01

you'll have to configure your pom to make that happen


thank you i will try it now 🙂

Alex Miller (Clojure team)16:02:31

the readme has more info


anyone know of a good transducer that kinda sorts? Ideally holds onto the largest N items by some metric?


and not sorts but just retains the largest values, not necessarily sorted


(transduce (biggest 5 identity) conj [] (range 30) would build up a collection no more than five at any point in time as it reduced the collection


@dpsutton I think you can look at a transducer like distinct and then adapt it


good point. thanks @borkdude


yeah i'm not familiar with his excellent library so i was hoping it would ring a bell as something already done with a proper immutable min heap


by which i mean thanks for pointing me towards window!

Alex Miller (Clojure team)17:02:00

lots of other useful things there too


yeah. i've never delved in and its probably time. thanks


I need a TCP client library. I just want to send bytes and receive bytes. Aleph has exactly the kind of interface I'm looking for:

(def client @(aleph.tcp/client {:host host :port port}))
(! client (msg->bytes mx))
(bytes->str @(! client))
but I'm wondering if there are any lighter weight libraries out there (or even a gist?)


sockets appear to form part of the abstraction. Not very familiar with it though

Alex Miller (Clojure team)18:02:36

the java socket stuff is pretty easy to use via interop, you don't really need a library for it

👍 8
💯 4
☝️ 4
Alex Miller (Clojure team)18:02:22

on one side you mostly just create a ServerSocket and .accept on it, which gives you a connection, which has an input/output stream on the other, create a new socket and use its input/output stream


cool I'll give it a shot with sockets. I've used them a bit before but this time I got hung up on some (java) example code using Nio socketchannels and was trying to do it in that way which maybe has a bit more of a learning curve.

Alex Miller (Clojure team)18:02:55

that's a couple more steps (but opens some new options)


I'm writing a clj wrapper for (cef) and am trying to figure out the best way to make it consumable as a library. The cef framework by itself is 80MB compressed and ~220MB uncompressed which seems too large to include in a library jar. Currently, I'm putting everything except the cef framework in the library jar and including a function that will download/extract the framework if it doesn't exist on the local file system. Is there a better way to handle this?


@smith.adriane I think that's a good tradeoff

👍 4

This will maybe also allow your tool to upgrade CEF itself without upgrading the jar, might add some flexibility.


Or if users already have it, they could maybe set some env variable to indicate where it is


supporting env variables would make a lot of sense. Currently, I only allow the consumer to specify the target dir, but supporting env variables at some point would also be good.


was asking about a biggest-by transducer earlier but realized i needed a reducing function for this actually. Does anything about this stand out as problematic?

(fn queue-accumulator
  ([] (PriorityQueue. 30 cmp)) ;; custom comparator
  ([^PriorityQueue q]
   (loop [acc []]
     (if-let [x (.poll q)]
       (recur (conj acc x))
  ([^PriorityQueue q item]
   (if (>= (.size q) threshold)
     (let [smallest (.peek q)]
       (if (pos? (.compare cmp item smallest)) ;; custom comparator
         (doto q
           (.offer item))
     (doto q
       (.offer item)))))


What are you trying to achieve exactly? Find the biggest item in a collection by some definition?


Keeping only the biggest n items in a reducible collection. Imagine it’s a million rows from a db each with a score. Want to keep the largest scored values without requiring the entire result set and sorting to get the largest items


So are you OK witn O(N) performance? If that’s the case, perhaps a simple (filter …) transducer ?


i don't know how i could filter. given that i don't know a priori what some threshold will be


just that i want the largest N items. i don't have any fore knowledge of how to determine those N items until i pass a function over them


i certainly can't beat O(N) in the general case. i think i'll hit O(n log(n)) in the worst case scenario where its essentially just heap sort but i'm discarding values so i have much smaller memory footprint


but that's the point of using a heap here. i don't want to track order or sorting, just a collection of items with constant time access to the min so i can throw that one away if necessary


ah I see, largest N as opposed to a pre-determined threshold, hmm


correct. that's why it needs to be in the reducing function to manipulate the concrete collection rather than in a transducer


Maybe just (assoc …) all the items into a sorted map and (take N …)


(sorted-map-by …)


that realizes the whole collection. if there are a million my server falls over


or continually resorts a collection. you're essentially using it as a heap whereas i'm using the proper heap datastructure here. constant access to smallest item, no sorting of the rest of the collection


Is there a way to get the data sorted out of the database? Or that’s not a good option?


(assuming it’s coming from a database)


no. scoring things in code based on the rows from the db. far too complicated to implement that in sql


(just looking for a more idiomatic solution as opposed to using the Java option)


yeah i looked at that. i forget what it lacked


ah, it was ordering. it maintains the priorities to the items but doesn't expose a way to quickly get the lowest priority item


also, no good way to evict the lowest priority. just add new ones in.


i mean, just abstractly a heap is all that i want with nothing that i don't want


(require '[net.cgrand.xforms :as x])

(let [data (repeatedly 10000 (fn [] {:score (rand-int 1000)}))]
      (x/sort-by :score >)
      (take 3))
One attempt using the xforms library… I am not 100% sure if x/sort-by realizes the whole collection…. It might.


yeah it does. it just keeps all of your items in an array list and on the completing single arity of a transducer it calls (java.util.Collections/sort cmp)


(defn sort
  ([] (sort compare))
    (fn [rf]
      (let [buf #?(:clj (java.util.ArrayList.) :cljs #js [])]
          ([] (rf))
          ([acc] (rf (core/reduce rf acc (doto buf #?(:clj (java.util.Collections/sort cmp) :cljs (.sort cmp))))))
          ([acc x] (#?(:clj .add :cljs .push) buf x) acc))))))


Yes, I give up 🙂


Depending on the database you’re using, I would still try to get all the items in a sorted order first… That would actually prevent you from having to go through the 1 million items in the first place (even if you’re not keeping them all in memory).


can't. its for searching and there are a few different strategies to rank them


need rows and then run functions over the different columns using different ways to rank them


I see… you need all the items for other purposes.


i agree if we could ask the db engine to rank them and then just take that


well doing things like longest common subsequence between search term and the columns doesn't sound fun in sql


but i appreciate you going through this with me. for sure i'd prefer a native clojure immutable heap or something similar

👍 3

but those java util collections are rock soid


I am just suggesting asking the DB to do the sorting… I 100% agree about not doing most things in SQL, it would be a mess.


but mutable 😞


the values they sort by are computed dynamically according to search term. no way to ask the db to sort it


@dpsutton did you find a solution you were happy with for this? ... maybe something like this?

(defn keep-group [f init]
  (fn [rf]
    (let [g (volatile! init)]
         (rf (reduce rf r @g)))
        ([r i]
         (vswap! g f i)

(defn largest-n [n]
  #(let [group (conj %1 %2)] (if (< n (count group)) (disj group (first group)) group)))


  (into [] (keep-group (largest-n 5) (sorted-set)) (range 10))



I realise it's pretty similar to what you had with the mutable collection ...


yeah. and it doesn't make much sense to do so much accumulation in the transducer. it clogs the whole pipeline until you're done which feels a bit presumptuous of a transducer


also there's no comparisons in yours. so it would just be a huge volatile in a transducer chain


it's in the largest-n fn ... which limits the number of things in the volatile to n things ... right??


it's doing the comparison by putting the data into a sorted set (which I guess could be a sorted-set-by ... and it's dropping the first one from the set if the count is larger than the n we give to largest-n ... so there's only 5 things in the volatile ... but in the same way you can't return the first result till you've reached the end when sorting, you can't do that here ... right?


oh i missed that part. i somehow only saw the keep group bit