Fork me on GitHub
#clojure
<
2021-09-23
>
Nikhil Warke05:09:18

I found this https://fly.io/blog/how-we-got-to-liveview/ about liveview in phoenix. I was wondering if we have a framework at similar level of abstraction in Clojure. Clojurescript as ecosystem (browser/nodejs) seems like an option, with lower level built-ins (3rd party like Re-agent/Re-frame for higher level). LiveView seems to very useful to get an idea off the ground with less resources.

tatut05:09:32

the idea in that is you can develop using hiccup components (like you would in reagent) and use sources on the server

noisesmith15:09:47

> to get an idea off the ground with less resources that's resources as in developer time, right?

Nikhil Warke10:09:05

@U051SS2EU Yes, developer/creator's time

hiredman05:09:34

Depending on what exactly you are tracking with "presence", I don't think a full crdt is required. I wrote(wrote and rewrote, many times of the course like a month) the presence system we use at work. What we had is pretty crdt like, accept we don't need to track causal history because information for a given connection/session/whatever is only ever updated by the server where that connection lives

hiredman05:09:50

We use http://netty-socket.io as a transport (which does long polling or websockets, and will try and upgrade from long polling to ws). Strictly as a transport, we don't use any of the room/distribution features of http://socket.io. And I kind of hate both http://netty-socket.io (java server for speaking http://socket.io) and http://socket.io (the protocol, technically a ridiculous stack of protocols)

hiredman05:09:02

I have at various times started writing a js library + clojure server that communicate via a custom protocol that does polling and/or websockets and a pure clojure implementation of the http://socket.io protocol

hiredman05:09:06

Still makes me smile to have figured out this bug, and to have been able to reflectively patch it https://github.com/mrniko/netty-socketio/issues/754

hiredman06:09:25

Something to keep in mind is websockets are inherently more fragile than polling, they are more easily disrupted by weird middle boxes, flakey mobile internet, roving between networks, etc

hiredman06:09:13

One of my problems with the http://socket.io protocol is the "upgrade" from polling to websockets is one way, if the upgrade succeeds, but then the websockets goes away for some reason, it doesnt try to keep the "connection" going by reverting to polling

Nikhil Warke06:09:36

@hiredman Will check out http://netty-socket.io. But I referring to the availability of such reliable tools baked into one framework. Think Python/Clojure built-in libraries, but at a higher abstraction (diff patching, presence etc). Using these common tools, you then build interactive apps/websites

oxalorg (Mitesh)11:09:38

Looking into it! Should be back up soon :)

🙏 2
👍 2
oxalorg (Mitesh)11:09:16

It’s back online! If you can’t see it yet, add any query param to break the cache 😊

3
baibhavbista11:09:54

Great!!! Thanks 😄

nikolavojicic14:09:35

What are required steps to make a function discoverable by clojure.repl/source? I have it saved in a file and loaded in REPL. Function has :file metadata. But (.getResourceAsStream (clojure.lang.RT/baseLoader) (:file (meta (resolve my-fn)))), which is used by clojure.repl/source-fn, returns nil. I'm using Leiningen.

nikolavojicic14:09:21

Btw source works for clojure core and libs and for dependencies.

nikolavojicic14:09:34

When I do lein classpath I see src directory there.

borkdude14:09:09

what's the name of the file?

borkdude14:09:22

perhaps there is something weird with underscores and hyphens or so

borkdude14:09:49

there's also one issue with source where it can't deal with auto-resolved keywords

p-himik14:09:24

I think the file has to be on the classpath - it can't be a random file that you load in your REPL. But if you're loading a file from src that's on classpath, then it's something else.

nikolavojicic14:09:59

Hmm now it's working, not sure what was wrong. Thanks anyway!

dpsutton14:09:59

If you evaluate forms through the repl they won’t have the required metadata. How can a form sent through the repl have a line number

nikolavojicic14:09:50

I wrote function in a file and loaded that file in the repl

dpsutton14:09:06

How did you load it?

nikolavojicic14:09:21

Sorry, C-c C-k is for file

nikolavojicic14:09:27

from Emacs Cider

nikolavojicic14:09:15

I guess it calls clojure.core/load-file function, not sure

borkdude14:09:24

this is probably nREPL

FiVo14:09:37

I am having trouble finding the bottlenecks in a highly concurrent application. I have tried https://visualvm.github.io/ and it's nice for monitoring how busy certain threads are but for example the profiler just dies on me when trying to profile cpu usage. I currently have always around 20% cpu usage but would like to push that towards the 100%. How do people usually go about this these days?

Ben Sless15:09:08

you can try JFR

Alex Miller (Clojure team)15:09:58

do you have reason to believe your application can be compute-bound? if not, you are unlikely to get high cpu usage

FiVo15:09:42

no not necessarily, could also be io or network bandwidth (although the later is very unlikely as I only use a fraction of what is available)

Alex Miller (Clojure team)15:09:11

if so, you should be looking for concurrency bottlenecks - are those threads contending for locks/queues/access to resources/etc

FiVo15:09:01

mainly using clojures stm primitives for things that are shared

Alex Miller (Clojure team)15:09:45

YourKit has some pretty interesting locking monitors that can help

Alex Miller (Clojure team)15:09:19

but really the most important thing is having a clear mental model of how threads are being managed and how they communicate

FiVo15:09:42

no not necessarily, could also be io or network bandwidth (although the later is very unlikely as I only use a fraction of what is available)

noisesmith15:09:39

you could try collecting metrics on retries or blockage of agents

noisesmith15:09:25

also, the amount of time from an incoming input to a calculated result is likely more interesting than percentage of CPU used - most apps I've worked on spend nearly all their time waiting for work to do

kwladyka17:09:23

I don’t know if it is good recommendation, but I always find the best solution to fix such kind of issue is to experiment with the code, add more logs, make code simpler, a little refactoring etc. I am afraid there is no gold solution. PS It can take long days to figure this out or even more depends on your use case.

Jade19:09:14

Hi! I'm trying to understand extension of protocol with metadata. So metadata are associated with values. I don't see any direct link in the doc between the protocol and the value with metadata. Do I understand it correctly it means in order to decide if a value "extends" a protocol that in addition Clojure is now looking if all the fully qualified protocol function have a definition in the metadata? What happens if only some of them are defined?

Jade19:09:30

And sorry I'm still learning 😅

hiredman19:09:50

Protocols can be partially satisified

1
p-himik19:09:09

And I think even if you fully satisfy a protocol via metadata, satisfies? will still return false.

Jade19:09:11

I get that part, but what is the behavior then in the case of let's say a multimethod using the protocol but a value only implementing some of it via metadata ?

Jade19:09:44

What are the results of satisfies? Or extends? And such when only partially satisfied?

Jade19:09:22

I guess I should just give it a try. It was just unclear to me when I was reading the doc

p-himik19:09:43

satisfies? will return false. And you can't use extends? because it requires a type - metadata extension works only on values.

Jade19:09:08

That helps

seancorfield19:09:11

It's a relatively new addition to the language so the core docs are pretty minimal and I don't think there's much out there in terms of community tutorials/blog posts. FWIW, we use metadata for protocols quite a lot at work (can't share that, unfortunately) but I can point you at next.jdbc.connection which uses metadata to partially satisfy Component's Lifecycle protocol: https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/connection.clj#L319-L328

Jade19:09:42

Thank you Sean!

Jade19:09:27

And sorry if the question was dumb

seancorfield19:09:02

Perfectly reasonable Q given how recent that feature is.

Jakub Holý (HolyJak)20:09:32

Hello! How to translate Character.UnicodeBlock.of(xxx) to Clojure? I thought (Character$UnicodeBlock/of xxx) but that does not work... 🙏

1
p-himik20:09:45

(import java.lang.Character$UnicodeBlock)
=> java.lang.Character$UnicodeBlock
(Character$UnicodeBlock/of 32)
=> #object[java.lang.Character$UnicodeBlock 0x797cc02e "BASIC_LATIN"
What do you see instead?

Jakub Holý (HolyJak)20:09:53

ah, I have not realized I need the import, thanks!

👍 1
seancorfield20:09:26

Yeah, while the java.lang top-level classes are auto-imported, none of their nested classes are.

seancorfield20:09:05

Ah, apparently Thread$State and Thread$UncaughtExceptionHandler are auto-imported!

p-himik20:09:06

Huh, cool. How did you check? I just found them in RT.java so at least now I know where to look. :)

seancorfield20:09:17

I datafy'd a namespace in Portal which showed me all the :imports 🙂

seancorfield21:09:37

You can do it in a bare REPL:

user=> (require '[clojure.datafy :as d])
nil
user=> (-> 'user (find-ns) (d/datafy) :imports (keys) (->> (filter #(re-find #"\$" (name %))))) 
(Thread$State Thread$UncaughtExceptionHandler)
user=> 

❤️ 1
p-himik21:09:08

Oh, nice, thanks!

vncz21:09:23

Folks! I could probably use some help about how to push data into a Kafka Topic. I am downloading some JSON garbage from the internet and, once processed, produce a Kafka value. Since the JSON payload is an array (and with a lot of elements) I thought of leveraging the JSON Streaming capabilities of cheshire and then put a transformation pipeline (through a transducer xf ) My doubts is about the f that reduces the elements together; because my ultimate aim is to push to Kafka without loading the entire dataset in memory, I came up with something like this (see code below) I am essentially creating a step function that returns the producer at the first step, and in the successive ones pushes the value and returns the producer again. While I think theoretically this achieves my aim (avoid loading the entire dataset in memory) I am not that confident if this is the right approach. Did anybody found themselves in the same situation?

vncz21:09:47

Bashfully asking for advices on this again, maybe somebody has some insights? 🙂

seancorfield21:09:39

Did you try the #apache-kafka channel? (I've no idea how active it is)

vncz21:09:30

I can try to post the question there, but this is more a transducer question

hiredman21:09:00

that is not all that different from how core.async channels work

hiredman21:09:54

they have a reducing function that is something like (fn [channel item] (add channel item) channel) and then transducers are applied to that

vncz21:09:37

They who?

vncz21:09:05

Assuming you’re talking about the core-async, then it kinda looks similar to what I am doing in my code, isn’t it?

vncz14:09:35

@hiredman I took another look and it seems like this is the way to go. The only other change I did was to use completing instead of defining my own function override. Thank you!