Fork me on GitHub
#clojure
<
2021-02-11
>
allenj1204:02:58

removed my message for a more relevant channel

zendevil.eth04:02:03

I have the following question posted in clj-http: https://clojurians.slack.com/archives/C8860D6BS/p1613018229002500 Your help is much appreciated!

dpsutton04:02:44

if that's real data you might have a bit of a headache

zendevil.eth04:02:09

@dpsutton what do you mean by real data? It’s key value pairs where values are strings

seancorfield05:02:21

He means that :basic-auth string I think...

seancorfield05:02:29

That auth seems real (although the customer ID seems invalid)

seancorfield05:02:15

You might want to delete that message @ps although at this point it's probably been archived to the logging site -- not sure if it respects deletes. The Zulip bot does respect deletes so it will at least remove it from there.

seancorfield05:02:28

The #clj-http channel at least is not logged/archived as far as I can tell so at least it's only in this channel -- and Zulip doesn't copy the body of shared messages. Not sure if logbot does.

KJO13:02:47

@ps You could try :body "customer= cus_IvNWjfw1q2clxS" in the post request. It’s equivalent to --data-raw option in curl IIRC.

Takis_16:02:21

Hello i am testing a maven project,where Clojure code is also inside the maven. It works fine , but i want to use a class from Clojure code gen-class , and it said it cant find it

Takis_16:02:37

in lein, i did :aot but in maven i dont know what to do to find it

Takis_16:02:25

normal call clojure from Java works IFn etc , but i want a class

Alex Miller (Clojure team)16:02:47

it hooks into the compile phase to compile clojure code

Takis_16:02:42

it does it automatically? or i have to set settings

Alex Miller (Clojure team)16:02:01

you'll have to configure your pom to make that happen

Takis_16:02:18

thank you i will try it now 🙂

Alex Miller (Clojure team)16:02:31

the readme has more info

dpsutton17:02:16

anyone know of a good transducer that kinda sorts? Ideally holds onto the largest N items by some metric?

dpsutton17:02:54

and not sorts but just retains the largest values, not necessarily sorted

dpsutton17:02:40

(transduce (biggest 5 identity) conj [] (range 30) would build up a collection no more than five at any point in time as it reduced the collection

borkdude17:02:35

@dpsutton I think you can look at a transducer like distinct and then adapt it

dpsutton17:02:01

good point. thanks @borkdude

dpsutton17:02:37

yeah i'm not familiar with his excellent library so i was hoping it would ring a bell as something already done with a proper immutable min heap

dpsutton17:02:46

by which i mean thanks for pointing me towards window!

Alex Miller (Clojure team)17:02:00

lots of other useful things there too

dpsutton17:02:58

yeah. i've never delved in and its probably time. thanks

jjttjj18:02:08

I need a TCP client library. I just want to send bytes and receive bytes. Aleph has exactly the kind of interface I'm looking for:

(def client @(aleph.tcp/client {:host host :port port}))
(manifold.stream/put! client (msg->bytes mx))
(bytes->str @(manifold.stream/take! client))
but I'm wondering if there are any lighter weight libraries out there (or even a gist?)

vemv18:02:28

sockets appear to form part of the https://github.com/clj-commons/byte-streams abstraction. Not very familiar with it though

Alex Miller (Clojure team)18:02:36

the java socket stuff is pretty easy to use via interop, you don't really need a library for it

👍 8
💯 4
☝️ 4
Alex Miller (Clojure team)18:02:22

on one side you mostly just create a ServerSocket and .accept on it, which gives you a connection, which has an input/output stream on the other, create a new socket and use its input/output stream

jjttjj18:02:05

cool I'll give it a shot with sockets. I've used them a bit before but this time I got hung up on some (java) example code using Nio socketchannels and was trying to do it in that way which maybe has a bit more of a learning curve.

Alex Miller (Clojure team)18:02:55

that's a couple more steps (but opens some new options)

phronmophobic18:02:13

I'm writing a clj wrapper for https://bitbucket.org/chromiumembedded/cef/ (cef) and am trying to figure out the best way to make it consumable as a library. The cef framework by itself is 80MB compressed and ~220MB uncompressed which seems too large to include in a library jar. Currently, I'm putting everything except the cef framework in the library jar and including a function that will download/extract the framework if it doesn't exist on the local file system. Is there a better way to handle this?

borkdude19:02:04

@smith.adriane I think that's a good tradeoff

👍 4
borkdude19:02:47

This will maybe also allow your tool to upgrade CEF itself without upgrading the jar, might add some flexibility.

borkdude19:02:25

Or if users already have it, they could maybe set some env variable to indicate where it is

phronmophobic19:02:03

supporting env variables would make a lot of sense. Currently, I only allow the consumer to specify the target dir, but supporting env variables at some point would also be good.

dpsutton21:02:17

was asking about a biggest-by transducer earlier but realized i needed a reducing function for this actually. Does anything about this stand out as problematic?

(fn queue-accumulator
  ([] (PriorityQueue. 30 cmp)) ;; custom comparator
  ([^PriorityQueue q]
   (loop [acc []]
     (if-let [x (.poll q)]
       (recur (conj acc x))
       acc)))
  ([^PriorityQueue q item]
   (if (>= (.size q) threshold)
     (let [smallest (.peek q)]
       (if (pos? (.compare cmp item smallest)) ;; custom comparator
         (doto q
           (.poll)
           (.offer item))
         q))
     (doto q
       (.offer item)))))

raspasov04:02:17

What are you trying to achieve exactly? Find the biggest item in a collection by some definition?

dpsutton04:02:43

Keeping only the biggest n items in a reducible collection. Imagine it’s a million rows from a db each with a score. Want to keep the largest scored values without requiring the entire result set and sorting to get the largest items

raspasov05:02:36

So are you OK witn O(N) performance? If that’s the case, perhaps a simple (filter …) transducer ?

dpsutton05:02:54

i don't know how i could filter. given that i don't know a priori what some threshold will be

dpsutton05:02:17

just that i want the largest N items. i don't have any fore knowledge of how to determine those N items until i pass a function over them

dpsutton05:02:27

i certainly can't beat O(N) in the general case. i think i'll hit O(n log(n)) in the worst case scenario where its essentially just heap sort but i'm discarding values so i have much smaller memory footprint

dpsutton05:02:18

but that's the point of using a heap here. i don't want to track order or sorting, just a collection of items with constant time access to the min so i can throw that one away if necessary

raspasov06:02:09

ah I see, largest N as opposed to a pre-determined threshold, hmm

dpsutton06:02:56

correct. that's why it needs to be in the reducing function to manipulate the concrete collection rather than in a transducer

raspasov06:02:54

Maybe just (assoc …) all the items into a sorted map and (take N …)

raspasov06:02:19

(sorted-map-by …)

dpsutton06:02:24

that realizes the whole collection. if there are a million my server falls over

dpsutton06:02:27

or continually resorts a collection. you're essentially using it as a heap whereas i'm using the proper heap datastructure here. constant access to smallest item, no sorting of the rest of the collection

raspasov06:02:51

Is there a way to get the data sorted out of the database? Or that’s not a good option?

raspasov06:02:06

(assuming it’s coming from a database)

dpsutton06:02:59

no. scoring things in code based on the rows from the db. far too complicated to implement that in sql

raspasov06:02:40

(just looking for a more idiomatic solution as opposed to using the Java option)

dpsutton06:02:59

yeah i looked at that. i forget what it lacked

dpsutton06:02:30

ah, it was ordering. it maintains the priorities to the items but doesn't expose a way to quickly get the lowest priority item

dpsutton06:02:55

also, no good way to evict the lowest priority. just add new ones in.

dpsutton06:02:09

i mean, just abstractly a heap is all that i want with nothing that i don't want

raspasov07:02:23

(require '[net.cgrand.xforms :as x])

(let [data (repeatedly 10000 (fn [] {:score (rand-int 1000)}))]
  (sequence
    (comp
      (x/sort-by :score >)
      (take 3))
    data))
One attempt using the xforms library… I am not 100% sure if x/sort-by realizes the whole collection…. It might.

dpsutton07:02:27

yeah it does. it just keeps all of your items in an array list and on the completing single arity of a transducer it calls (java.util.Collections/sort cmp)

dpsutton07:02:42

(defn sort
  ([] (sort compare))
  ([cmp]
    (fn [rf]
      (let [buf #?(:clj (java.util.ArrayList.) :cljs #js [])]
        (fn
          ([] (rf))
          ([acc] (rf (core/reduce rf acc (doto buf #?(:clj (java.util.Collections/sort cmp) :cljs (.sort cmp))))))
          ([acc x] (#?(:clj .add :cljs .push) buf x) acc))))))

raspasov07:02:41

Yes, I give up 🙂

raspasov07:02:08

Depending on the database you’re using, I would still try to get all the items in a sorted order first… That would actually prevent you from having to go through the 1 million items in the first place (even if you’re not keeping them all in memory).

dpsutton07:02:29

can't. its for searching and there are a few different strategies to rank them

dpsutton07:02:46

need rows and then run functions over the different columns using different ways to rank them

raspasov07:02:01

I see… you need all the items for other purposes.

dpsutton07:02:06

i agree if we could ask the db engine to rank them and then just take that

dpsutton07:02:29

well doing things like longest common subsequence between search term and the columns doesn't sound fun in sql

dpsutton07:02:50

but i appreciate you going through this with me. for sure i'd prefer a native clojure immutable heap or something similar

👍 3
dpsutton07:02:56

but those java util collections are rock soid

raspasov07:02:03

I am just suggesting asking the DB to do the sorting… I 100% agree about not doing most things in SQL, it would be a mess.

dpsutton07:02:04

but mutable 😞

dpsutton07:02:24

the values they sort by are computed dynamically according to search term. no way to ask the db to sort it

Ed14:02:31

@dpsutton did you find a solution you were happy with for this? ... maybe something like this?

(defn keep-group [f init]
  (fn [rf]
    (let [g (volatile! init)]
      (fn
        ([]
         (rf))
        ([r]
         (rf (reduce rf r @g)))
        ([r i]
         (vswap! g f i)
         r)))))

(defn largest-n [n]
  #(let [group (conj %1 %2)] (if (< n (count group)) (disj group (first group)) group)))

(comment

  (into [] (keep-group (largest-n 5) (sorted-set)) (range 10))

  )

Ed14:02:51

I realise it's pretty similar to what you had with the mutable collection ...

dpsutton15:02:50

yeah. and it doesn't make much sense to do so much accumulation in the transducer. it clogs the whole pipeline until you're done which feels a bit presumptuous of a transducer

dpsutton15:02:09

also there's no comparisons in yours. so it would just be a huge volatile in a transducer chain

Ed15:02:51

it's in the largest-n fn ... which limits the number of things in the volatile to n things ... right??

Ed15:02:22

it's doing the comparison by putting the data into a sorted set (which I guess could be a sorted-set-by ... and it's dropping the first one from the set if the count is larger than the n we give to largest-n ... so there's only 5 things in the volatile ... but in the same way you can't return the first result till you've reached the end when sorting, you can't do that here ... right?

dpsutton15:02:04

oh i missed that part. i somehow only saw the keep group bit