Fork me on GitHub
#clojure
<
2022-04-24
>
raymcdermott15:04:10

regarding the io-prepl ... does anyone have a way to make sure that it emits :err tags (see https://clojure.atlassian.net/browse/CLJ-2645 for prepl which shows that flush is needed on *err*)

flowthing19:04:45

I mean you could do something awful like this after connecting:

(defn ^java.io.PrintWriter ^:private auto-flushing-print-writer
  "Like clojure.core.PrintWriter-on, but with auto-flushing enabled."
  [flush-fn close-fn]
  (let [sb (StringBuilder.)]
    (-> (proxy [java.io.Writer] []
          (flush []
            (when (pos? (.length sb))
              (flush-fn (.toString sb)))
            (.setLength sb 0))
          (close []
            (.flush ^java.io.Writer this)
            (when close-fn (close-fn))
            nil)
          (write [str-cbuf off len]
            (when (pos? len)
              (if (instance? String str-cbuf)
                (.append sb ^String str-cbuf ^int off ^int len)
                (.append sb ^chars str-cbuf ^int off ^int len)))))
      java.io.BufferedWriter.
      (java.io.PrintWriter. true))))

(def original-err *err*)
(def my-err (auto-flushing-print-writer #(doto original-err (.write ^java.io.Writer %) .flush) nil))
(set! *err* my-err)
(set! *warn-on-reflection* true)
(.toString (identity "foo"))

😬 1
flowthing19:04:38

Or if you control the server, you could just make your own version of io-prepl.

raymcdermott19:04:50

I do control the server 🙂

raymcdermott19:04:30

Thanks for putting the time into that answer!

raymcdermott21:04:46

looking at io-prepl I'm struggling to see how to affect the *err* flush operations in the prepl. What am I missing?

raymcdermott08:04:11

and I have tried setting *err* after starting the socket server as you suggest and I'm still not getting back any :err tags

flowthing08:04:20

Sorry, yeah, you'd actually need your own version of clojure.core.server/prepl, not io-prepl.

flowthing08:04:45

That's odd. I tried the above using nc and I did get the reflection warning back.

flowthing09:04:36

Or actually both io-prepl and prepl. 🙂

flowthing09:04:48

I didn't clean that up at all, but given that and then clojure -X clojure.core.server/start-server :name prepl :port 5556 :accept foo.core/io-prepl :server-daemon false, then nc localhost 5556, I can get the reflection warning to show up.

raymcdermott10:04:10

yes, I figured later that you must have meant prepl - thanks for getting back 🙂

raymcdermott10:04:50

kinda tedious to have to fork it but :man-shrugging::skin-tone-3:

flowthing10:04:55

Indeed, it's not great.

raymcdermott10:04:03

Thanks for all of your time and advice.

flowthing10:04:05

Sure thing, not sure how useful any of it was, though. 🙂 I just happened to have wrestled with the same thing (flushing err to make reflection warnings show up) in my own work recently.

raymcdermott10:04:02

It's useful: we arrived at the agreement that it's a simple correction but it needs to be fixed in core or we have to fork / patch it which is 😢

👍 1
raymcdermott19:04:59

Given that the fix is a one liner, is it even worth making a patch? @U064X3EF3??

Alex Miller (Clojure team)19:04:06

Can you put all this on the ask Clojure question for it? I get mentioned on dozens of threads a day, in multiple slacks. Ask Clojure is the place to create public durable conversations about requests for Clojure in a votable, trackable form

Alex Miller (Clojure team)19:04:18

Of particular interest are impact and need to fork/copy to work around

Alex Miller (Clojure team)19:04:49

I don't understand the question you asked of me, are you asking if it's worth fixing or worth doing the work of making the patch? (In both cases, yes)

Alex Miller (Clojure team)19:04:10

but I'd want to make sure we have isolated what the problem is here. is it that the error stream is flushed on delay? or that the delay is too long? or that warnings should flush? or something else.

raymcdermott20:04:06

ok thanks - I'll add some comments to the Ask Clojure posting and maybe @U4ZDX466T can weigh in too

raymcdermott20:04:52

I have added a patch for PrintWriter-on and prepl to that Ask Clojure question

Alex Miller (Clojure team)21:04:07

do you want to add them to the jira?

raymcdermott21:04:35

sure but I don’t have a login AFAIK

raymcdermott22:04:03

Thanks - I’ll post it tomorrow, getting late here now.

Alex Miller (Clojure team)21:04:08

Thanks, not going to look at it imminently, but I've added it to our 1.12 consideration list.

1
Ben Sless15:04:27

Is there a way to invoke a java method without going through Method. invoke()? What's the overhead, besides allocating the array and paying the price of invokevirtual?

ghadi15:04:31

MethodHandles

💯 1
Ben Sless15:04:44

Thanks, looks like what I was looking for

Ben Sless17:04:14

I think I'm doing something wrong. If I want to invoke a method I should lookup in an object with a method type which includes only the arguments?

Ben Sless17:04:45

When you find time, I'd appreciate a simple example. Perhaps with a StringBuilder?

ghadi17:04:44

Show what you tried, also search this slack for “signature polymorphism”

Ben Sless18:04:02

I actually don't need signature polymorphism for my specific use case, just trying to pull teeth working with gRPC

Ben Sless18:04:08

I'll send it over tomorrow, falling off my feet

Ben Sless06:04:34

Okay, I managed something like:

(let [lu (.in (MethodHandles/lookup) String)] (.invokeWithArguments (.findVirtual lu String "toUpperCase" (MethodType/methoType String)) ["abc"]))
Now the challenge is doing it without allocating the j.u.List (or array)

Kevin19:04:36

i'm not super familiar with java, but i'm trying to figure out how to convert a async/chan to a InputStreamReader. . The reason is that the cheshire/core library I'm using to parse json takes an InputStreamReader. Any ideas on how to do this? The only way I've come up with is:

(->> body-chan
   chan-to-lazy-seq!!
   (map (fn [chunk] (-> chunk .getBytes ByteArrayInputStream.)))
   Collections/enumeration
   SequenceInputStream.
   InputStreamReader.
   cheshire/parsed-seq)
this feels wrong though as i'm converting a async/chan to a lazy sequence. Ideally i want to park on the chan and write to the input stream reader some how

1
Kevin19:04:37

if there a clean way to parse json from a channel that would be amazing. The channel is made up of many raw strings that may be made up of json fragments (e.g. messages 1-3 can make up 1 json blob)

Kevin19:04:23

so channel messages can look like:

temp (async/chan 10)
(async/>! temp "{ \"a\":")
(async/>! temp "5 }")
and the resulting chan should provide the json blobs
result-ch (async/chan 10)
(println (async/<! result-ch))
; prints {:a 5}

ghadi21:04:35

I have a utility that does this that I can scrounge up @U01BK5E5ESY . Can I ask what the application is?

Kevin23:04:51

@U050ECB92 that would be super useful

Kevin23:04:46

the specific application is for reading streamed responses from an api thats sending json events

ghadi23:04:51

@U01BK5E5ESY this differs from what you want in one way: it is adapting a channel of byte[]s to an InputStream, whereas you are looking for a channel of Strings or char[] adapted into a Reader

ghadi23:04:19

both InputStream and Reader are blocking I/O sources

ghadi23:04:32

this is one of the wickedest fns I've written.

Kevin23:04:25

interesting:

{:inputstream (proxy [InputStream] []
                     (close [] (error! nil))
                     (read
                       ([] (read1))
                       ([b] (readN b 0 (count b)))
                       ([b off len] (readN b off len))))
so you are overwriting the methods here for the InputStream

Kevin23:04:23

What is the downside of this approach here:

body-chan
   chan-to-lazy-seq!!
   (map (fn [chunk] (-> chunk .getBytes ByteArrayInputStream.)))
   Collections/enumeration
   SequenceInputStream.
   InputStreamReader.

ghadi23:04:32

does it work?

ghadi23:04:43

I don't know what chan-to-lazy-seq!! is

Kevin23:04:04

(defn chan-to-seq!!
  "Takes a channel and returns a lazy sequence of channel messages"
  [c]
  (lazy-seq
    (when-some [v (async/<!! c)]
      (cons v (chan-to-seq!! c)))))

ghadi23:04:21

seems ok, I haven't thought too deeply about it

1
didibus17:04:32

(proxy [java.io.Reader] []
  (read [chars offset length] ...) ;; Here you take from channel and put into chars
  (close [] ...) ;; Here you close the channel

didibus17:04:27

chars is an array, offset is the array index you need to start at for inserting the chars into, and length is the max number of chars you can insert for a call to read

didibus17:04:03

So basically, take length number of char from the channel and aset them into the chars array starting at index offset

didibus17:04:28

And when close is called, just close the channel.

didibus17:04:05

You can now use this proxy with parse-seq on cheschire I believe. Or if not, wrap it in a BufferedReader and that should work I'm pretty sure.

Kevin18:04:32

> chars is an array, offset is the array index you need to start at for inserting the chars into, and length is the max number of chars you can insert for a call to read does that mean i need to store the entire total string in memory as I pull from the chan (async/<! ch) (i assume so because we need to support the various offset values)?

didibus23:04:18

No I don't believe so. I think what happens is: The reader provides an array, and ask for say 10 char to be read at offset 0, so it calls read with that. The reader then read the array from 0 to 10, if the reader feels they don't have enough... say the JSON is not complete like you only got: {"hey" : " and this can't be parsed yet. So the reader will call read again, with the same array, but will ask for another 10 char to be read from offset 10. Now maybe the reader gets: {"hey" : "bob"} and this is enough to parse, so the reader can parse that and be done, now it will call close.

didibus23:04:07

But, the reader could also choose to continue reading... and the next time it calls read, it can choose to pass a new char array and set the offset back to 0. Allowing the prior char array to be garbage collected.

didibus23:04:04

I think this is what say a LineReader will do. The LineReader will read from another Reader say 10 char at a time into a first array for the first line. When it sees a line ending, it will call read with a new array for the second line, Or it can even choose the same array but overwrite it, by just having the offset be 0 again.

didibus23:04:46

So its up to the user of the Reader to decide like if it cares to remember everything read till now or not, by choosing how to call read

Kevin23:04:23

okay I'll play around with it and report back later

didibus23:04:48

Keep in mind that read should block until some input is availaible.

didibus23:04:41

If read is called and the channel closes in the read, you can return -1 to indicate to the reader that the stream ended.

didibus00:04:43

I couldn't resist:

(defn chan->reader
  [c]
  (proxy [java.io.Reader] []
    (read [chars offset length]
      (let [first-char (async/<!! c)]
        (if (nil? first-char)
          -1
          (do
            (aset chars offset first-char)
            (reduce
             (fn[acc e]
               (if-let [char (async/poll! c)]
                 (do (aset chars (+ 1 offset e) char)
                     (inc acc))
                 (reduced acc)))
             1
             (range (dec length)))))))
    (close [] (async/close! c))))

(def char-chan (doto (async/chan)
                 (async/onto-chan! [\a \b \c \d \e \f \g \h \i \j])))

(def char-reader (chan->reader char-chan))

(slurp char-reader)
"abcdefghij"
(slurp char-reader)
""

didibus00:04:19

I think that should work. Can try it with cheshire, and maybe need to wrap it in a BufferedReader as well, since I saw cheshire parse-seq has a BufferedReader type hint

(def char-reader-buff (java.io.BufferedReader. (chan->reader char-chan)))

(slurp char-reader-buff)
"abcdefghij"
(slurp char-reader-buff)
java.io.IOException: Stream closed

grounded_sage21:04:39

Does anyone know if it's possible to dynamically update CORS policy?

p-himik21:04:15

Unless you mean some specific server, I don't think browsers cache CORS-related headers.

grounded_sage22:04:40

I mean server side.

grounded_sage22:04:43

I'm wanting to initially be open to requests but then lock down those requests after a period of time once I know the origins

p-himik22:04:07

You're in full control of the server side. Maybe some particular sever would not have an easy way of implementing what you want, but in general it should be possible.

emccue00:04:12

Write some custom ring middleware that conditionally applies the logic of https://cljdoc.org/d/ring-cors/ring-cors/0.1.13/api/ring.middleware.cors#wrap-cors

grounded_sage03:04:39

Yea I've tried that. I think Reitit might be doing something special. I even tried doing a namespace reload but it doesn't work when I package it up.

emccue03:04:11

Can you share a bit of that code? My guess is a #' can be slapped somewhere and your problem goes poof