Fork me on GitHub
#clojure
<
2022-08-12
>
escherize05:08:12

I want to go from a symbol that maps to a function var, to the form of the source code, but clojure.repl/source says that it cannot find the source. I can look it up from the metadata on the var from the symbol. But is that a good idea? Why doesn’t clojure.repl/source do that?

escherize05:08:42

(defn f [x] (if x true false))

(with-out-str (repl/source f))
;; => "Source not found\n"

(defn rresolve [s]
  (try (resolve s)
       (catch Exception e (try (requiring-resolve s)
                               (catch Exception e nil)))))

(let [{:keys [file line]} (meta (rresolve 'cycloj.core/f))]
  (edn/read-string (str/join "\n" (drop (dec line) (str/split-lines (slurp file))))))
;; => (defn f [x] (if x true false))

vemv12:08:35

If you re-define a defn in the repl, there's nothing to look up, there's no (useful) file/line/column so code (re)loading has to be file-based in order for repl/source to work. There are many ways to do this, require is one (assuming the ns you're requiring wasn't defined in the repl) Your editor should have a means of code loading as well. And tools.namespace refresh also is file-based

dpsutton13:08:17

CIDER uses load file for some actions so check and see what your shortcuts do in your workflow. Load file (I think) is almost like evaling forms in the repl so no line information

pinkfrog05:08:12

I wonder when using agents, do you guys consider back pressure? What if the queue is keep compiling up? Or just from a practical stance, normally the traffic won’t surge so the issue is just ignored?

Alex Miller (Clojure team)06:08:15

I think most people just don't use agents

Alex Miller (Clojure team)06:08:35

But certainly core.async has more tools here

pinkfrog06:08:28

I guess one might provide its own executor for some monitoring and resource control.

pinkfrog06:08:16

@U064X3EF3 async is a more full-fledged way. But I wonder the purpose of agent nowadays.

didibus07:08:49

Backpressure handling is the main advantage of concurrent sequential processes over alternative concurency abstractions such as Agent or Actors

didibus07:08:43

Because there's a two way communication, publisher waits for a consumer to have consumed before producing the next one (or up to some buffer)

didibus07:08:55

You can also generally swap the executor with set-agent-send-off-executor

didibus07:08:52

I think it's just harder to find use-cases for Agents. If you just need to run a handful of background jobs, future does the trick. If you need to manage callback hell, core.async does the trick (or something like Promesa) Agents live in this middle of the road between the two. Though I think Agents could shine for keeping track of state in say a kind of multi-user scenario. Say you run a tic-tac-toe game server. Each game board can be an Agent. That would be a good use of them I feel. You could have client spectator, and clients that are playing, the playing clients would make a move, causing a send to the agent, other clients would either hot-poll or with add-watch when the agent changes they can be notified and their board updated to reflect the new state.

didibus07:08:49

I would think similarly for a chat or any kind of live service. Like for a chat, keep the chat in an Agent, multiple clients posts messages, send to the agent, other clients have a watch, get notified of new messages, the agents guarantees their order, all clients see the same order of messages.

didibus07:08:05

You can even have a client post a message, then have them await, which will wait for everything up to their message to be committed to the agent, and then you can return.

pinkfrog07:08:56

Define a state with atom, and using future to do some stuff and finally swap the result. Agent sounds like duplicated.

pinkfrog08:08:35

> Because there’s a two way communication, publisher waits for a consumer to have consumed before producing the next one (or up to some buffer) Interested to know. Could you give a more concrete example?

didibus08:08:17

Agent will order all changes. Each future run concurrently with one another. So which future will be first to change the atom? Agents will not run concurrently, all changes to the same agent is done one after the other.

didibus08:08:12

For core.async, think of it like a handshake. When you >!!into a channel, you block until some other thread <!!from it, and voce versa, if you <!!from a channel, you block until another thread >!!. I think of it like a cook and a waiter. The cook makes a dish and needs to hand it off to the waiter, but there's no table, so the cook has to hold onto it waiting for the waiter to come grab it from them. If waiter arrives to grab it first, they'll have to wait there waiting for the cook to finish preparing the plate and hand it over. If you want, you can add a buffer in-between, so maybe you buy a small table, now the cook can place the plate on the table and start making another plate right away, and the waiter can grab it from the table, but again, if the table is full and the cook made another plate they'll have to stand there with the plate in their hands and wait for the waiter to grab something from the table. Or if the waiter comes and the table is empty, they'll also have to wait.

didibus08:08:13

Same thing if using >! and <! with a Go process.

pinkfrog08:08:46

> So which future will be first to change the atom? Atom will guarantee the atomicity, everything can be put in the f of swap!. Yet, ordering is another concern. Speaking of this, why shall f of a swap! to be side-effect free. Or to say, why does Clojure make concurrent access on an atom during swap! ? Since its the same resource, so one is doomed to fail and retry.

pinkfrog08:08:45

> Same thing if using >! and <! with a Go process. I see, you were talking about rate limiting with blocking.

didibus09:08:49

Well, it's not exactly a rate limit, but kinda. It's more that the different process wait on each other to continue.

didibus09:08:14

So you can do things like alternate them and other stuff like that.

didibus09:08:24

Well, with Agents you can also do side-effects, since they are guaranteed to happen one after the other. With future and atom it has to be side-effect free logic only. And ya, you could batch the changes by putting them all in the single f of swap!, but that works only ahead of time. If you do it from requests coming in concurrently, you can all batch them, how many will come in? How long to wait before you think you have a batch? Atom handles all that, and then let's you wait on everything till now to be processed.

didibus09:08:41

The reason swap! on atom has to be side effect free is actually more that it has to be idempotent. swap! does optimistic locking, if it fails, it'll retry against the new state of the atom. So the f of swap! could be called more than once. If it did side effects, it would therefore possibly do the side effects multiple times.

didibus09:08:52

If your side-effect is idempotent or you don't care if it happens multiple time, you can do it.

didibus09:08:20

If it didn't do that, it would need to either do what Agent does, and then it would just be an Agent 😝, or it would need to lock, and then it's just a lock, which you also have in Clojure. So you have all options: • Pessimistic locking: locking • Optimistic locking: Atom • Lock-free async sequential update: Agent

thanks3 1
didibus09:08:48

The locking is a bit less convenient to use, but generally lock-free has better throughput, like I'm reality chances that two things swap at the exact same time are low, so lock-free is.ofgen better. Only if you know you'll have very high contention then locking might be faster, or possibly Agent.

thanks3 1
pinkfrog09:08:45

Yup.I missed it, Optimistic locking: Atom. Optimistic for performance optimization due to low contention rate.

Joshua Suskalo14:08:22

As someone who has actively used agents a number of times for a number of projects, if you need backpressure, use a channel and a go loop.

Alex Miller (Clojure team)14:08:55

agents are probably best used for simulation use cases (true original "oo" type use cases where agents communicate via messages), but I've only worked on maybe one system where that was actually a good fit

Alex Miller (Clojure team)14:08:15

but the secret superpower of agents is that the STM system is aware of them and you can use agents to safely do side-effecting work from a ref transaction (where agent calls are held until the transaction completes), and I have leveraged that several times

Joshua Suskalo14:08:17

The place where I found it most useful was in representing rate limits on API endpoints when the actual # of endpoints is unknown (in this case for Discord).

Joshua Suskalo14:08:43

I didn't know about the STM thing though, that seems quite useful.

Alex Miller (Clojure team)14:08:29

you don't even need the agent to have state - you can just use it as a way to run a function on transaction completion (although watchers can do that too I guess)

didibus14:08:01

I feel like there might be a secret throve of things Agents could be used for. At least conceptually, they seem so close to Actors if you use them with a multi-method. Once Loom arrives, they'll be running of a lightweight executor of virtual threads as well, and I'm really curious at that point if you couldn't design an entire OTP-like system around them.

steffan09:08:21

Where should I add type hints for return values when using defprotocol? Is it (defprotocol p (^MyValue foo [this])) or (defprotocol p (foo ^MyValue [this])) ?

Athan10:08:25

Measuring Transactions Throughput with Datomic in-memory database and Datalevin storage engine Hi, I thought you might find this interesting. First https://clojurians.slack.com/archives/C03RZMDSH/p1660170818216469?thread_ts=1660170818216469&amp;cid=C03RZMDSH below to see the experiment and the results. Before you dive deeper into my analysis, it would be good to know that the main reason I have been attracted to Clojure is that I discovered Datomic Datalog query language. That brought me decades back to the university era. 🙂 It also seems that many other Clojurians started learning the language because of Datalog. But there are many variants of Datalog and it has not been standardized. Datomic is leading the race here but in my opinion there is a serious limitation regarding to the speed of asserting new facts. 1. It's important to explain quickly why I think there is such bad performance with Datomic in-memory client and I expect analogous results using https://github.com/tonsky/datascript. It's because items in data structures are immutable objects. To maximize the speed of writing in memory one has to use different memory management and data structures similar to those used in pyarrow, numpy. So, how about building an in-memory Datalog query engine on top of pyarrow ? 2. It was relatively easy to deploy, configure and test transaction throughput of a key-value storage engine (LMDB) of https://github.com/juji-io/datalevin. I would expect Datomic transactor on AWS DynamoDB Local or https://docs.xtdb.com/storage/rocksdb/ to have similar performance.

;; 5.2 sec for 23 cols x 10000 rows
;; 3.2MB books.csv file
;; Elapsed time: 5.2 secs
;; datoms inserted: 229,956
;; transactions throughput: 229,956/5.2 datoms/sec = 44222 datoms/sec
I have met similar problems in the past when I was testing the writing performance of redis KV storage engine. All these KV engines (redis, dynamodb, lmdb) are very good on point queries but they perform really bad when you want to write (import) a big volume of data. You may argue that writing performance is not critical for a transactional (OLTP) DBMS but it becomes super important when you want to import your data from another system/project, or you want to integrate a big volume of data from other sources, or you want to do analytics without adding another storage engine. In fact what we are discussing here is the price you pay for having a flexible universal data model based on EAV/RDF triplets. Which is a similar case when you try to construct a relational, tuple based data model on top of a KV storage engine or object like memory structure (Python/Clojure). The physical layout must be appropriate for such data model and the best candidate I found from my personal research and experiments is to use a columnar layout. Why not adding Datalog query engine on top of a columnar database engine based on LSM tree data structures, such as Clickhouse and Singlestore(MemSQL) ?

pinkfrog11:08:26

Why prefix function names with - ? Not only promesa, I’ve also seen it in some metosin projects. Is it for denoting something is used internally, or avoiding name clashing with core functions?

ghadi11:08:44

it's a convention that usually means "internal", no touchy

thanks3 2
🙃 1
💯 1
ikitommi13:08:49

We have used this convention at Metosin: • defn kikka - public/user api • defn -kikka - extension api • defn- -kikka - private api

👍 1
Drew Verlee14:08:45

I'm curious, when people want private functions, why not use a closure?

(letfn [(private [x] x)]
  (defn public [x] (private x))

(public 1)

cursork14:08:06

Because you can't test it on the repl.

Drew Verlee14:08:26

I think you can test it, probably a couple ways. Here is one:

(letfn [(private [x] x)]
  (private 1)
  #_(defn public [x] (private x)))
;; => 1
I think i'm just not sure what's meant by private if the intention is the function be only usable in a certain context, then scoping it like this does that slightly more thoroughly then a defn- as this private function is private to only that public function, as opposed to the whole ns. I think the bigger downside of this approach is that it's nice to visually scan a ns at the top level and expect to see all the defns there. You can get this information back (cider-browse-ns) for example, but it requires more effort. It's not even clear you would want to test a private function, as your worried about the contract of the public function not the private one. But practically speaking, this probably happens all the time. it would be interesting to think about having the tests be in scope to.
(letfn [(private [x] x)]
  (assert (= 1 (private 1)) ;; ideally would get compiled away. 
  (defn public [x] (private x))
I don't think it matters that much either way, i'm comfortable reading both. I think it's usually more important to provide a narrative around how the functions are meant to be used, rather then trying to restrict how they shouldn't be.

Joshua Suskalo14:08:25

You might not want to test a private function from a test ns, but it's perfectly reasonable while you are writing it to test it from a repl, where making it a private defn makes a lot of sense. I myself do not often write comment blocks or other changes into the file to test things, I prefer to call my functions to test them from the actual repl buffer.

Joshua Suskalo14:08:41

That is to say, you may not wish to have any automated tests for private functions, but you still need to see if it works when you write it.

Yehonathan Sharvit09:08:23

What do you mean by extension api @U055NJ5CC ?

ikitommi09:08:22

better naming split might be basic / advanced. for basic use cases, the public api is all you need. if you are extending (the library) or doing something non-trivial, use the advanced api. Example:

;; accepts anything that can be coerced into schema
(m/walk schema walker)

;; a protocol method (no coercion, needed if you create your own schema-types + better perf, not needed if you "just use the lib")
(m/-walk schema walker [] nil)

Yehonathan Sharvit10:08:54

Why does -main start with a dash then?

Joshua Suskalo16:08:25

-main starts with a dash because the :gen-class clause in your main ns defaults to looking for vars prefixed with - to make public methods on the class.

erwinrooijakkers14:08:33

So a side effect of using defn- -kikka for private vars is that they are compiled to public methods on the class if you add the gen-class attribute? Or can’t they be private?

grzm15:08:54

How do people manage agent jars (as opposed to library dependencies)? I’m thinking of things like https://github.com/open-telemetry/opentelemetry-java-instrumentation . I’d rather not have the jar as some blob in our git repository. Is there a maven-like story here? I’m really ignorant of this type of thing, so if you’re thinking “oh, you just blah blah blah and Bob’s your mother’s brother”, that’s probably exactly the answer I’m looking for 🙂

dpsutton15:08:28

a little script to download it and an alias that adds the jvm opts

/tmp on ☁️  metabase-query
❯ cp ~/Downloads/opentelemetry-javaagent.jar .

/tmp via ☕ v17.30 on ☁️  metabase-query
❯ clj -Sdeps '{:aliases {:telemetry {:jvm-opts ["-javaagent:opentelemetry-javaagent.jar"]}}}' -A:telemetry
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2022-08-12 10:50:09:441 -0500] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.16.0
Clojure 1.11.1
user=>

dpsutton15:08:42

i’d have a gitignored directory and put it in there

dpsutton15:08:35

You don’t need the jar in your repo. Just a way to get it. And this definitely feels dev time. So just a way to run it for those interested right?

grzm15:08:11

Yeah, that’s definitely the kind of experience I’m looking for. Other considerations (and this is me thinking out loud and not necessarily Clojure-specific): • I’d like to pin the agent version, so I think I’ll want to stash the jar someplace like S3—maybe we can rely on github releases in this case? and fetch it from there. • We’ll need it at build-time as well when we package it up for deployment (similar type of fetch would work there, too)

grzm16:08:08

I’m right in thinking that this isn’t something that maven really is meant to deal with, correct? People don’t generally end up with agent jars in ~/.m2 ?

Alex Miller (Clojure team)16:08:49

I don't know of any Maven support specifically for agents

dpsutton16:08:12

You can pin with the release version: . They serve releases from github. Just have your download script grab whatever version you want instead of “latest”

dpsutton16:08:37

it would actually be way more difficult to not pin

grzm16:08:11

Agreed re: grabbing specific releases. I’m getting increasingly paranoid in my dottage and want all of my stuff in places I store it if it makes sense 🙂 Internet outages, github being down, all that jazz 🙂

dpsutton16:08:16

ah vs . Just use links from their releases

dpsutton16:08:54

Seems to me you are talking random jars in s3 vs just grabbing from their official release pages?

dpsutton16:08:02

not sure which sounds more paranoid fighting

dpsutton16:08:14

and both are susceptible to internet being down and other jazz

dpsutton16:08:42

and if you download it to a gitignored folder it’s essentially there until you download it again right?

grzm16:08:56

Agreed. Though I’ve had less issues with S3 than github. For the record, I’m likely not going to pull it into someplace else.

grzm16:08:25

The gitignored thing locally, yes. Our build environments are more ephemeral.

dpsutton16:08:01

does your build succeed if github is down?

grzm16:08:25

I’ve dragged us off-topic, though. The basic agent thing has been answered.

dpsutton16:08:29

oh nice. well maybe s3 is your choice then 🙂

grzm16:08:07

Thanks, again as always, @U11BV7MTK 🙂 Remind me to spot you a beverage of your choosing next time we see each other.

👍 1
dpsutton16:08:01

unfortunately can’t be strange loop this year (new stickers look so good too. really want that shirt 😞 ) but at conj next spring most likely

grzm16:08:23

I’ll be there with parens on!

grzm16:08:20

(there are stickers?)

grzm16:08:03

I know a guy. I might be able to get you some.

dpsutton16:08:25

haha. i’m gonna place an order. they are selling shirts for people who can’t attend (or want more i guess)

seancorfield16:08:17

I'll share what we do at work:

(ns ws.newrelic.update-libs
  "A process to update the New Relic agent library in our server installation."
  (:require [ :as io]
            [clojure.tools.deps.alpha :as t]))

(set! *warn-on-reflection* true)

(def ^:private newrelic-agent 'com.newrelic.agent.java/newrelic-agent)
;; keep in sync with clojure/components/newrelic/deps.edn:
(def ^:private agent-version  "7.9.0")

(def ^:private ws-root "/var/www/worldsingles")

(defn- get-agent-jar-file
  "Returns a File pointing to the New Relic Java Agent JAR."
  ^java.io.File
  []
  (-> (t/find-edn-maps)
      :root-edn
      (assoc :deps {newrelic-agent {:mvn/version agent-version}})
      (t/calc-basis) ; download and tell us where it is
      ;; (doto tap>) ; how I debugged the basis
      (get-in [:libs newrelic-agent :paths])
      (first)
      (io/file)))

(comment
  (get-agent-jar-file)
  )

(defn run
  [_]
  (let [jar (get-agent-jar-file)]
    (println "Copying" (.getName jar) "...")
    (io/copy jar (io/file (str ws-root "/build/lib/newrelic-agent.jar")))))
Our deployment process runs clojure -Sdeps '{:paths [\"build/src\"]}' -X:deps ws.newrelic.update-libs/run as part of setup. Then we have the agent JAR in a known place and our app startup scripts contain the -javaagent option pointing to that location. We have a number of small scripts in build/src related to deployment tasks.

grzm18:08:15

Oh, awesome. Thanks, @U04V70XH6!

Ted Ciafardini16:08:19

clojure string question…

; Trying to turn a string like addr:  
(def addr "123 Main St, Sippie Town, MD 21043")
;into => "123 Main St\, Sippie Town\, MD 21043"

Tried the following - not quite there...

  (string/replace addr #"," "\\,")
    ;=> "123 Main St, Sippie Town, MD 21043"

  (string/replace addr #"," "\\\\,")
    ;=> "123 Main St\\, Sippie Town\\, MD 21043"
Is there a way to get just one backslash before the commas?

dpsutton16:08:51

I find the simplest way to understand this is that image string literals are a little language. You start the literal with a " and then all things inside of the matching " which closes this little language mean “themselves”. But how could you represent a literal " inside of the string. It means end the string so cannot show up in it. We need a special character that instructs this little language that “this is special”. that character is the slash: \. So slashes talk to the language interpreter and are not characters in the language. But now we have one more problem: what if we want to include a slash in the language and we don’t want to “talk to the language interpreter”. Well it’s just \\ We use the special character to talk to the language and then tell it “i want a slash”. So those two slashes mean “include a single slash” in the string

dpsutton16:08:58

(count "\\a") -> 2

Ted Ciafardini16:08:39

interesting, okay. thanks

dpsutton16:08:32

and of course the string is exactly what you think it is:

user=> (println "\\a")
\a
nil
user=> 
but when representing the string literal, we have to use the escape character for the slash

🙏 1
telekid16:08:22

I feel like I remember seeing a Clojure library for running regexes on data (rather than strings) ages ago. I know that I could use spec for this, but curious if anyone happens to know the name of the library that I may be remembering?

telekid16:08:14

I think it was https://github.com/cgrand/seqexp actually. Thank you though!

Derek16:08:59

oops — totally misread you

hiredman16:08:44

spec (comes with clojure) can do this to some degree as well

Joshua Suskalo16:08:06

spec is great for this, just wish it had lookahead ops

respatialized18:08:51

1. https://github.com/metosin/malli also has very good support for sequence expressions and high performance 2. FYI #find-my-lib is a good channel for these questions

teodorlu17:08:50

Hi 🙂 I'm hoping someone can help me understand a piece of behavior that's really surprising. I have two small Clojure files, user.clj and script.clj :

(ns user)
(ns script)

(defn run [{}]
  (print :hello))
When I clj -X script/run I see :hello on the terminal -- all good. But! I want to see metadata when working in the REPL. So I change user.clj to this:
(ns user)

(set! *print-meta* false)
Now, clj -X script/run crashes:
clj -X script/run
Exception in thread "main" Syntax error macroexpanding at (user.clj:3:1).
	at clojure.lang.Compiler.load(Compiler.java:7665)
	at clojure.lang.RT.loadResourceScript(RT.java:381)
	at clojure.lang.RT.loadResourceScript(RT.java:368)
	at clojure.lang.RT.maybeLoadResourceScript(RT.java:364)
	at clojure.lang.RT.doInit(RT.java:486)
	at clojure.lang.RT.init(RT.java:467)
	at clojure.main.main(main.java:38)
Caused by: java.lang.IllegalStateException: Can't change/establish root binding of: *print-meta* with set
If I keep user.clj empty, and instead "opt-in to metadata" in script.clj like this:
(ns script)

(set! *print-meta* false)

(defn run [{}]
  (print :hello))
I no longer see the error. ----- Are there special behavior / rules for what's allowed in user.clj? Why?

teodorlu17:08:33

And where can I read more to understand this?

hiredman17:08:49

it is what it says on the tin

hiredman17:08:15

user.clj isn't loaded inside a binding of *print-meta*

teodorlu17:08:04

But everything else is?

hiredman17:08:15

it depends, but yes, unless you are calling load directly outside of a repl it is likely every place you are loading code sets up a surrounding binding for a number of vars like *print-meta*