Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2021-02-23

Channels

#clojure
<
2021-02-23
>
Nazral04:02:05

Is it safe to have a watcher send a message to the agent it is watching ?

hiredman04:02:32

It is as safe as recursion, you need a base case to avoid an infinite loop

👍 3
hiredman05:02:31

What people try to do with that sort of think is often not safe though

hiredman05:02:15

The action your watcher sends will get interleaved with other actions being concurrently sent

hiredman05:02:32

So you can easily make a race condition

Nazral05:02:19

It's my worry. I have a bunch of futures that all send to the agent to append to a list. When the watcher detects that the list is longer than a certain size, I want it to process that list and reset the agent to the empty list

Nazral05:02:33

If I am not mistaken, the messages received by the agent are handled sequentially, so there shouldn't be any race condition in this scenario, right? Can there be loss of data though?

Nazral05:02:12

Would a better approach be to just drop the part of the vector contained in the agent that I used rather than resetting the agent?

noisesmith17:02:24

this sounds like a better fit for a queue, with writers giving it items, and a single reader that builds up a buffer and periodically processes the whole buffer

noisesmith17:02:35

unless the senders need to see the pending buffer

Nazral06:02:45

It is what I ended up doing, you are right it is a much better (and simpler) solution

javahippie08:02:37

I was wondering about the best approach for logging in a library I am building. I already saw this great post: https://lambdaisland.com/blog/2020-06-12-logging-in-clojure-making-sense-of-the-mess For libraries, my thoughts are that clojure.tools.logging would be the best approach, as consumers of the library could change the clojure.tools.logging.factory flag to decide if the library should log to slf4j, apache commons, log4j, log4j2 or java.util.logging, so I won’t force them to pull some bridge-libraries into their deps. What are your thoughts on this?

mpenet08:02:13

it's one of those things were I personally prefer just relying on raw logback

mpenet08:02:20

+ tools logging

mpenet08:02:45

sure it's an xml file to deal with, but at least there's no indirection and it's quite straightforward

dharrigan08:02:33

Yeah, I like clojure tools logging too

dharrigan08:02:35

simple to use

👍 3
mpenet08:02:08

the only reason to use a third party lib would be if you want compat clj/cljs I would guess

mpenet08:02:30

but I am not expert on cljs so I don't know the details

javahippie08:02:39

Good point, but it won’t be targeted at cljs

javahippie08:02:43

Regarding logback: I wouldn’t ship it with the library, as it is a logging backend, I believe

Adam Helins09:02:13

I get an illegal reflective access warning when setting access on the private address field in java.nio.Buffer (java 11). Usually it goes away with type hinting but here, no matter what I hint, it doesn't. 

(def ^:no-doc ^java.lang.reflect.Field -field-address

  (doto (.getDeclaredField Buffer
                           "address")
    (.setAccessible true)))
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by helins.binf.native$fn__1426 to field java.nio.Buffer.address

vemv10:02:38

You are doing explicit reflection (`.getDeclaredField`) and getting a warning accordingly

Adam Helins13:02:42

Is it forbidden in Java now? From what I read, people still do that and nobody seem to complain. I thought I might be doing something wrong or maybe for some reasons it could be somehow Clojure related (didn't quite follow how Clojure blends with modules).

vemv14:02:16

People tend to try supressing warnings without trying to understand those first :) I don't have the authoritative source, but it's quite clear that newer JDKs want to enforce the Java privacy mechanism, forbidding things like setAccessible. Surely other reflection APIs will remain perfectly fine to use

Adam Helins17:02:23

Yes but since accessing private fields is so common in plenty of big Java libs... Are they really going to fully remove picking at private fields?!

vemv19:02:16

I'm not really qualified to do an assessment on that, but personally it SGTM. Privacy is there for a reason and can be solved in alternative ways (e.g. forking a dependency)

Adam Helins15:02:58

Well in a way, you are right. In another way... come on, let's make everything public and let people hack in peace 😛

🙃 3
rickmoynihan12:02:18

When you’re proxying a class with clojure proxy is it possible to typehint protected methods on this?

rickmoynihan12:02:41

I’ve wrapped (let [^MyProxyClass this this ret (.protectedMethod this) ,,,) over it and it still warns

Jamie Rumbelow13:02:02

Has anybody ever used something like redis as a substrate for a core.async channel? I'd like to be able to use the same semantics as channels across multiple processes / machines

hiredman16:02:08

tcp includes backpressure, but it can be non-obvious, gets weird when you aren't communicating directly, and lots of apis do a bad job of exposing it

Alex Miller (Clojure team)13:02:21

backpressure is a hard thing to implement over the network

telekid18:02:27

this makes some intuitive sense, but I’d love to understand why more formally – do you know of any good reads on the topic?

roklenarcic19:02:28

A server using async connector can theoretically handle a million connections, not read from them and slowly process them, stalling clients that are waiting for results, which is backpressure.

Jamie Rumbelow14:02:04

yes... I was afraid there'd be some theoretical reason why not!

roklenarcic19:02:12

Has anyone tried to implement a sandboxed Clojure interpreter? I’ve looked at SCI and it’s a nice interpreter, but I’m afraid of two things: • infinitely long execution (I can solve this by timing the execution and calling Thread/stop on the executing thread from a controller thread) • infinite memory allocation (I haven’t found a way to prevent this)

jumar04:02:22

I remembered that I saw something about measuring thread allocations. This could be a good enough approximation. It uses ThreadMXBean: https://stackoverflow.com/questions/61539760/benchmarking-jvm-memory-consumption-similarly-to-how-it-is-done-by-the-android-o Here's an example: https://github.com/jumarko/clojure-experiments/blob/master/src/clojure_experiments/performance/memory.clj#L166-L187 

(defn thread-allocated-bytes [t]
  (let [thread-mbean (java.lang.management.ManagementFactory/getThreadMXBean)
        thread-id (.getId t)]
    (.getThreadAllocatedBytes thread-mbean thread-id)))

(defn allocated-bytes
  [f]
  (let [thread (Thread/currentThread)
        start (thread-allocated-bytes thread)]
    (f)
    (- (thread-allocated-bytes thread) start)))

(comment
  
  (let [t (Thread.  (fn []
                      (let [v (vec (range 1e8))])
                      (Thread/sleep 1000) (println "DONE.")))]
    (println "Thread allocated - before start:" (thread-allocated-bytes t))
    (.start t)
    (println "Thread allocated - after start:" (thread-allocated-bytes t))

    (Thread/sleep 100)
    (println "Thread allocated - after 100 ms:" (thread-allocated-bytes t))

    (when (< 1e6 (thread-allocated-bytes t))
      (println "Stopping the thread...")
      ;; note that `(.interrupt t)` isn't enough here - the thread is busy allocating and cannot be interrupted
      (.stop t)
      (println "Thread stopped")
      (println "Thread allocated - after stop:" (thread-allocated-bytes t)))

    (Thread/sleep 200)
    (println "Thread allocated - after 300 ms:" (thread-allocated-bytes t))

    (Thread/sleep 800)
    (println "Thread allocated - after 1100 ms:" (thread-allocated-bytes t)))
  ,)
;; prints:
;; Thread allocated - before start: -1
;; Thread allocated - after start: 14064
;; Thread allocated - after 100 ms: 125412240
;; Stopping the thread...
;; Thread stopped
;; Thread allocated - after stop: -1
;; Thread allocated - after 300 ms: -1
;

Alex Miller (Clojure team)20:02:09

Java security is built into the jvm and can do a lot of things. there is clojail as well, which puts some additional clojure restrictions over that

borkdude20:02:46

I have used clojail to run snippets from clojuredocs to verify specs in speculative, using this updated fork: https://github.com/borkdude/clojail (been a while)

ghadi20:02:24

can also sandbox at an outer layer using something like Firecracker, the VM technology that AWS uses for Lambdas

ghadi20:02:39

depending on the use-case

borkdude20:02:03

you can also run a dedicated JVM for the sandboxes and set -Xmx maybe?

ghadi20:02:38

...what is the use-case?

ghadi20:02:30

supporting user-submitted scripts in a multitenant runtime (a la Jenkins extensions)? implementing something like an interactive tutorial site (a la http://repl.it or 4clojure)? ?

roklenarcic20:02:26

basically in a mini-game I’d like to have players submit their own routines in clojure which would operate in a very closed world

roklenarcic20:02:52

unfortunately it’s very easy to nuke the server by saying (last (range 10000000000))

borkdude20:02:42

@roklenarcic this is something I have thought about for a while in sci and I haven't been able to come up with a satisfactory solution - it should be solved at a deeper level, like you say, e.g. use a timeout with Thread#stop or even an lower level

jumar20:02:31

The best approach is usually run those as standalone processes. For Clojure this might mean a non-trivial overhead of course, especially in terms of memory usage and also the execution time (maybe GraalVM could help with this) it depends on how much resources do you want to give them and the latency you can tolerate. Here's a related question: https://stackoverflow.com/questions/6599985/is-there-a-way-to-put-maximum-memory-usage-limit-on-each-thread-in-java#:~:text=To%20place%20a%20limit%20on,memory%20each%20thread%20is%20using.

borkdude20:02:15

If you go that road, maybe setting it up as a CGI-ish solution works. I bet they have solved similar problems there.

dpsutton20:02:30

just randomly, would it be possible to use clojure's asm to create classes that defeat what clojail is attempting to do?

borkdude20:02:09

@dpsutton I guess you can try that out in 4clojure

andy.fingerhut21:02:14

It would be surprising if clojail could not be compromised in some way. I don’t think it was submitted to any kind of sustained attack effort by experts before

devn01:03:41

well, aside from a lot of us in IRC trying to break it all the time for years, I agree 😄

borkdude21:02:12

This is why I like offering such solutions in browsers. If people mess up, it's their problem ;)

sova-soars-the-sora23:02:15

If I were to make a Peer-to-peer application in Clojure, is the state-of-the-art the java interop with pastry?

lukasz23:02:11

jGroups perhaps? http://www.jgroups.org - was pretty pleasant to use via the interop

🎉 3
🐝 3
sova-soars-the-sora00:02:36

Thanks, I'm checking it out now. It's fun trying to wrap one's mind around P2P 😄