Thrift outside of FB isn’t exactly super well supported in general. What is your use case?

and my first guess would be that the best approach would be to use the java tools for this and just get data out of results with the appropriate methods

maybe in a prep required local dep if you are in the deps.edn world

Shameless plug but have you watched my transducers workshop from last month?

Tldr transducers transform reducing functions by wrapping them

The outermost wrap is the one you'll go through first

It isn't that the functions aren't applied from right to left, they are, but those functions aren't the pipeline, but transformations to be resolved when supplied a reducing function

It's like building up a stack. Last in, first out

I didn't see the workshop, no

link?

https://github.com/bsless/2021-nov-transducers-workshop -- not sure if there's a video too?

thanks, sean

I can't find a video link for it.

that's alright, I can look through the presentation

Maybe the workshop videos haven't been made public yet

Rich Hickey's talk introducing transducers also touches on this (linked at the proper timestamp): https://youtu.be/6mTbuzafcII?t=1530

The key thing to understand is that composition works the same way you expect, and the first transducer you list is the outermost wrapper and sits on top of the call stack – but values traverse the transducer stack from the bottom, being generated by the step function and then successively transformed by each transducer as they get returned up the stack.

Here’s the leaderboard, btw. https://plummerssoftwarellc.github.io/PrimeView/report?id=davepl-1639302015.json&hi=False&hf=False&hp=True&fi=&fp=&fa=&ff=&fb=&tp=True&sc=ps&sd=True

One super puzzling thing there is with the top three solutions. They are in the range of 10,000 X faster than the rest of the pack. The fastest solution is in Common Lisp, and it is 5000 times faster than the fastest Rust solution. (But I’m derailing my own question, please ignore this tangent 😄).

but... you're not actually parallelizing the sieving? just the extraction of the primes from the array?

I bet what is dominating the runtime is constructing the result vector & the intermediate mapcat list in

(into [2]
                  (mapcat deref)
                  futures)

shouldn't you just return the number of primes from sieve?

There isn't an intermediate mapcat list

oh right yeah sorry

the rules seem to say > • Your benchmarked code returns either a list of primes or the is_prime array, containing the result of the sieve. so why not just return the array then?

I don't think there's anything to be gained by clojure-specific stuff here. The optimum on JVM will be a tight loop over an array.

maybe I just can't read the results, but I don't see clojure solution #3 in the results, only #2

Yeah, the array building(not parallel) is all the work.of the sieve, all that faffing about with futures is nonsensical

The benchmark was run some days ago.

Right, so when I was experimenting with this getting the values out of the sieve was actually taking up a significant part of the time. It depended a bit on which storage I had. But, yes, it is still a very little work per access, so maybe that’s why the futures gain me so little?

yeah, you're effectively parallelizing the agets, which are cheap, while not parallelizing the construction of the result vector and adding some intermediate transients

but the fastest way to do something is to not do it at all, so I'd just skip the result vector creation

I'm probably sounding pretty harsh, sorry for that, I'm grumpy today for some reason

No worries!

The reason the common lisp one is at the top is it does the whole thing at read or macro expansion time

Which isn't counted in the execution time

Which is just silly

of course, those wily lispers

The fastest c++ does constant folding, similar kind of deal

So if I moved my sieve to a macro I could achieve something similar?

if you don't care about the rules, sure

I am not considering doing it. 😃

I’m just here to learn.

yeah the benchmark competition feels pretty pointless, but the journey can be educational

I dunno, I mean, the common lisp impl does it and sits at the top of the board, it may be fine with the rules

The readme for cl program even explains it

My solution was not written for this “competition”. It was just something I spent time on some year ago. But I will test with returning the array and see where that lands me.

the rules seem pretty ripe for rule lawyering

https://github.com/PlummersSoftwareLLC/Primes/blob/drag-race/CONTRIBUTING.md#faithfulness

it's kinda implying you need to actually do the computation at runtime, but not really stating it

That is usually how these things go

> So, to summarize: mayerrobert-cl-hashdot does not use compile-time evaluation, the evaluation happens before compile-time.

heh

I don’t think it makes sense contributing a solution that just moves the work to when the clock is not running.

at least they tag it as unfaithful

They tag it as both. 😃

yeah they run it in both modes

The faithful version ranks pretty decently, actually.

I haven’t run the benchmark but I think my solution would rank somewhere around 34-35 in this list https://plummerssoftwarellc.github.io/PrimeView/report?id=davepl-1639302015.json&hi=False&hf=False&hp=True&fi=&fp=&fa=&ff=uf&fb=&tp=True&sc=ps&sd=True

And Clojure solution 2 is not realizing the lazy sequences, so I don’t think we can really count that one.

it's realizing them in :valid? (= (count primes) ...)

but yeah a doall in the sieve function would be more honest, but probably wouldn't affect the timing that much

It’s only realizing the last pass.

If you compare to the runner in solution 3 you’ll see the difference (not that it is needed in my solution, but anyway).

If I just return primes, it runs 2-3X slower.

(defn sieve [^long n]
  (let [primes (boolean-array (inc n) true)
        sqrt-n (int (Math/ceil (Math/sqrt n)))]
    (if (< n 2)
      '()
      (loop [p 3]
        (if (< sqrt-n p)
          primes
          (do
            (when (aget primes p)
              (loop [i (* p p)]
                (when (<= i n)
                  (aset primes i false)
                  (recur (+ i p p)))))
            (recur  (+ p 2))))))))

You mean what you just pasted is slower than what's in the first message of this thread? I wonder if counting the primes could be the bottleneck then. But that's outside the sieve function of course, so I'm wondering if you even really need to do it... :)

I guess the counting could be parallelized if needed. Is your implementation now loop-aget-inc?

Whatever, it's a silly game with ill-defined rules. The tight array loop will be the same speed in pretty much all languages, but the varying ceremonies around it might cost a lot or nothing at all.

see also https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html (edit: fixed link)

Clojure used to be published in the computer language benchmarks game, and I and Alex Miller and a few others spent some time tweaking Clojure programs for it to get it close to Java. Beating the Java solution is pretty unlikely, unless the Java solution did something inefficiently, but there are enough players that the Java solutions were not leaving spare cycles sitting around.

And yes, there were discussions about whether the solutions for some problems were following the rules or not, although I don't recall any that precomputed the answer at compile time 🙂

@US1LTFF6D Those measurements where with criterium

(quick-bench (doall (sieve 1000000)))

clj꞉sieve꞉> 
#'sieve/sieve
clj꞉sieve꞉> 
Evaluation count : 54 in 6 samples of 9 calls.
             Execution time mean : 13.230384 ms
    Execution time std-deviation : 559.934025 µs
   Execution time lower quantile : 12.437373 ms ( 2.5%)
   Execution time upper quantile : 13.805512 ms (97.5%)
                   Overhead used : 6.866165 ns
nil
clj꞉sieve꞉> 
#'sieve/sieve
clj꞉sieve꞉> 
Evaluation count : 168 in 6 samples of 28 calls.
             Execution time mean : 4.281483 ms
    Execution time std-deviation : 218.415891 µs
   Execution time lower quantile : 4.055185 ms ( 2.5%)
   Execution time upper quantile : 4.511662 ms (97.5%)
                   Overhead used : 6.866165 ns
nil
clj꞉sieve꞉> 

that's really weird, since the first one should be doing strictly less work, right?

ah it's the doall

@U0CMVHBL2 thanks for sharing! This particular leaderboard is pretty young. It is a followup on this video: https://www.youtube.com/watch?v=D3h62rgewZM Which hasn’t been around for very long. Mar 25 2021.

no need for that since you're returning a vector or an array, so there's no laziness

but of course the doall for the boolean array is slower because it has more elements to walk through

Thanks! I didn’t realize it would traverse the array.

yeah I guess doall could just short-circuit on non-LazySeq arguments, but it doesn't

Just returning the array:
             Execution time mean : 1.119677 ms
Returning the sieve
             Execution time mean : 3.050549 ms

👍

There is a lot to learn from fighting with the various ceremonies around things. 😃

If I recall correctly I got faster results when updating a BitSet, but it was slower to collect the results. So just maybe I can shave off some little more within the rules of being allowed to skip collecting.

Do you have the option of bundling ::a and ::b into a composite value and making that optional?

no, porting a legacy system and need to be api compatible. that’s definitely the way to go if this was not the case. pretty happy with what I ended up with though. thanks for the input!

What did you end up with?

See separate message in main channel

there’s a #cursive channel. I know those are configurable key commands so just need to figure out where that is

Cool… I’ll ask there. 🙂

No real value add and we try to keep dependencies small/focused. That's also what has led us to move from Cheshire to data.json and from clj-http to http-kit (initially) and then to Hato (a wrapper for Java's HttpClient).

We've made several library switches over the years in a quest to prune back dependencies (and reduce deployment artifact size).

We switched from Redisson to Jedis (and wrote our own connection pool code) to get away from the mass of dependencies (and bloat) that Redisson brought in.

okay so its just the goal of minimizing deps

i dont think our org is there yet, and i’ll have to look at its full tree to make a call

> clojure -X:deps list :aliases '[:dev :everything]' | fgrep -v /Developer | wc -l
     232

Adding test deps is another 30 libs. Adding build is another 55. Adding poly is another 50. But the 232 above is just production code.

Bruno's about on slack, so you could ping him to see if he responds?

If environment values need to be logged, then the code that calls the logger should be getting those vars and supplying them to the logger. It’s a better approach from a composability point of view, and any security issues are the responsibility of the local system. Most people never need to include ${} in their logs. I believe that putting this “convenience” in there for everyone was a mistake in general, but the latest security nightmare would back me up on this point.

also, people put way too much trust in environment variables - there are simple accessible ways to access the list of all key value pairs in the environment for every mainstream language (I think it's a cargo cult thing that stems from the data at rest / data in motion distinction which is also overrated...)

I have such a library @U11BV7MTK . Hoping to open source it

that would be amazing @U050ECB92.

mulog and timbre are the only ones on my radar

But they satisfy only the first 3 items, right? Or do they also include logged data viewers?

i don’t think any logging library does that - and the tools that do that seem to like looking at lines of json

Well, I'm not speaking strictly of libraries. It can be a SaaS or something else. But yeah, I haven't seen one either, and JSON is... lackluster, to say the least.

When I've used mulog, the UI was cloudwatch insights and it was amazing

Thanks! I'll check it out.

@U065JNAN8 What was your “production” mulog setup, if i can ask?

It was pretty simple. Just included mulog and mulog-cloudwatch in the project.clj and configured as described here https://github.com/BrunoBonacci/mulog/blob/master/doc/publishers/cloudwatch-logs-publisher.md

My connections look like this New connection from #object[java.net.Socket 0x2973e18 Socket[addr=/0:0:0:0:0:0:0:1,port=37194,localport=8383]]

that creates a server socket, accepts one connection, prints out, then closes the connection and server socket

so the only way you can get multiple prints from that is if you have a loop somewhere else that restarts the server socket after you close it

Thanks!

I would fine a nice guide to socket programming somewhere

it almost doesn't matter in what language because the structure of socket programs is basically always the same

create a server socket, accept connections, hand off the connection to something that does something with it (often another thread) loop back to accepting connections

only accepting a single connection, then closing everything is not what you want

https://docs.oracle.com/javase/tutorial/networking/sockets/clientServer.html might be a good read

be sure not to skip the bit at the very end about multiple clients

@hiredman Sorry if I misunderstood the reading - is this fine? I'm not sure if threads are necessary as the socket closes (defn listener [port] (println "Listening on port" port) (with-open [server-socket (ServerSocket. port) socket (.accept server-socket)] (while true (println "New connection from" socket) (doseq [msg (receive socket)] (println msg)))))

receive is just (line-seq (io/reader ))

that will loop forever reading from the the one connection (until an exception is thrown) and cannot handle multiple connections

I guess you just misplaced the call to accept?

I ended up using future , I think this version should work

(defn listener [port] (let [running (atom true)] (println "Listening on port" port) (future (with-open [server-socket (ServerSocket. port)] (while (deref running) (with-open [socket (.accept server-socket)] (println "New connection from" socket) (doseq [msg (receive socket)] (println msg))))))))

Nevermind, apparently it isn't

you keep missing the loop around accept

1. accept connection 2. handle connection 3. goto 1

and if handle connection doesn't hand the connection off to a thread (if you aren't using nio) then you will only ever handle one connection at a time

I'm using telnet

I got this, hopefully it does what you described (defn listener [port] (let [running (atom true)] (println "Listening on port" port) (with-open [server-socket (ServerSocket. port)] (while (deref running) (future (with-open [socket (.accept server-socket)] (println "New connection from" socket) (doseq [msg (receive socket)] (println msg))))))))

this will attempt to spin up an infinite number of futures

@hiredman I think that using future again could work? Let me show you

(defn listener [port] (let [running (atom true)] (println "Listening on port" port) (future (with-open [server-socket (ServerSocket. port)] (while (deref running) (future (with-open [socket (.accept server-socket)] (println "New connection from" (.. socket getInetAddress getHostAddress)) (doseq [msg (receive socket)] (println msg))))))))) That's what I thought, doing so makes it stop listening to the port, for some reason

If you're on jdk 11+ there is built-in jdk support. I've heard good things about https://github.com/gnarroway/hato as a wrapper

I've heard people talk about Sente (haven't used it myself...that may change soon)

Using sente a little bit. I'm impressed.

about jdk11+ http client, be aware it has some nasty quirks

like handling of GOAWAY with http2

as long as you don't use http2 it's fine

I am not sure what's the status on that one in particular on more recent jdks, but we got bit by it a few times before we had to disable http2 on the client.

I still prefer jetty http client personally for "advanced stuff" and good old clj-http for the rest

also out of the box the jdk http client is missing a few important things, like timeout on response body read, you need to use something like mizosoft/methanol (or bake your own) to add support for basic stuff like this