Fork me on GitHub
#clojure
<
2020-04-29
>
mll01:04:24

Hi! I have a question. Let us say that i have a macro that replaces all occurrences of 'transact' to '\', sth like this:

(defmacro transmuting-macro [& body]
  (let [transmuted-body (clojure.walk/postwalk 
                         #(if (and (list? %) (= 'transact (first %))) 
                            (-> % rest (conj `/)) %) body)]
    transmuted-body)) 
Now i can do this:
(defn generate-error []
  (transmuting-macro (let [x 1
               b (transact 1 3)]
           (transact 3 0))))
But exception's stacktrace cannot track what is inside, it stops at the macro invocation (line 2) and then immediately jumps to divide without providing any meaningful line number (in this case line 4). If i put just body instead of transmuted-body in macro definition's last line the stack is preserved. So, I reckon, there is some hidden info somewhere that allows the compiler to introspect some macros but not others. I have tried to look at meta as well as special &form and &env variables but to no avail. Since my transmutation doesnt change anything about line numbers, just copying this information would suffice. Does anyone know how it works?

seancorfield02:04:13

My first thought is that (-> % rest (conj ...) ...) is losing any metadata that was on the original form

seancorfield02:04:15

@mll Did you try something like

(with-meta (-> % rest (conj `/)) (meta %))
?

seancorfield02:04:30

(and you may need to capture the metadata from (first %) in that expression and add it to the / symbol)

mll02:04:55

the problem is that meta of the body is nil...

mll02:04:45

I thought about the same but cannot find the meta. Where is it?

seancorfield02:04:08

It may be that postwalk itself doesn't preserve metadata?

mll02:04:56

but i have looked at the original body

seancorfield02:04:16

Hmm, I'll have to go digging. I ran into similar issues when working on expectations.clojure.test. I seem to recall needing to read metadata from &form (which is where the source form for the macro originates) but I can't remember how I propagated the data.

seancorfield02:04:27

Hmm, no, looks like I sidestepped all of that in the end. Sorry @mll I have no suggestions right now.

mll02:04:04

I did some digging and it seems there is a certain pattern to metas inside the form but not outside of it

mll02:04:11

not form

seancorfield02:04:52

(with-meta transmuted-body (meta &form)) would be my best guess at this point @mll for the body of your let instead of just transmuted-body

dominicm03:04:55

You're right that postwalk doesn't preserve meta

Alex Miller (Clojure team)03:04:05

there is a bug filed for this

Alex Miller (Clojure team)03:04:25

or at least I thought there was, not seeing it

Alex Miller (Clojure team)04:04:26

seems like this came up recently, but I don't know where then, don't see it in ask or jira

Alex Miller (Clojure team)04:04:47

actually I am seeing it preserved in postwalk?

dominicm04:04:37

https://github.com/ztellman/potemkin/blob/master/src/potemkin/walk.clj I've had to use this a couple times to fix problems. Maybe it was when I was using map & similar on the sequences though :).

dominicm04:04:05

@alexmiller there we are, lists & seqs lose their meta:

user=> (clojure.walk/postwalk identity (with-meta (list 1 2 (with-meta (list 3 4) {:bar true})) {:foo true}))
(1 2 (3 4))
user=> (clojure.walk/postwalk identity ^:foo (list 1 2 ^:bar [ 3 4]))
(1 2 ^{:bar true} [3 4])

Alex Miller (Clojure team)04:04:23

well, afaik there is no ticket about this

dominicm04:04:26

I can log one in the am, it's bitten me a couple of times.

dominicm04:04:40

I thought I found a ticket, but hazy memory.

mll06:04:35

Big thanks for spelling it out. I will use the potemkin for now then.

kwladyka10:04:15

(s/def ::generate-labels (s/keys :req-un [::label-template
                                          ::data]
                                 :opt-un [::api-version]))
is it possible to force (clojure.spec.gen.alpha/generate (s/gen ::generate-labels)) to generate data with all :opt and :opt-un? Also nested deeper. Just I would like to be able to genere full example of data.

kwladyka10:04:50

It could be very helpful as a doc then

shooit11:04:30

@kwladyka I don’t believe so, but you could write a custom generator with s/with-gen

kwladyka12:04:49

this is not what I really want to 😉

kwladyka12:04:27

even then I will have to write this for all main map specs

kwladyka12:04:33

this will be nice feature

Alex Miller (Clojure team)13:04:33

you could write an alternate spec where the optional things are required, then use its generator as the generator here

kwladyka13:04:50

thanks for the suggestion. Probably the best is to write macro which will create this spec using real one.

Spaceman17:04:36

How to unbind a macro in the repl?

Spaceman17:04:39

I required the cljs deftest and tried unmapping it like so: (ns-unmap 'cljs.test 'deftest)

Spaceman17:04:58

But when I try to require the clj deftest (require '[clojure.test :refer [deftest]])

Spaceman17:04:04

deftest already refers to: #'cljs.test/deftest in namespace: user

dpsutton17:04:55

isn't the problem that you have a user/deftest and you need to unmap that?

dpsutton17:04:07

cljs.test/deftest is a fine mapping and you probably don't want to unmap that one

Spaceman17:04:07

Okay so how do I require the clj deftest in user ns given that the cljs deftest is already defined in that namespace?

dpsutton17:04:58

you have a user/deftest and you need to remove that right?

Spaceman17:04:28

got it thanks

noisesmith17:04:59

you can use (ns-unmap 'some.ns 'sym) or (def sym nil) from inside that ns

noisesmith17:04:12

ns-unmap actually removes the var entirely, def just happens to remove all aspects of the var that are usually a problem (eg test or macro metadata)

abdullahibra19:04:53

Hello everyone,

abdullahibra19:04:12

I have general question, about consuming messages from Kafka topic, is it better to collect all messages into async channel and handle it by pure methods which take channel and process it and return new output channel and so on or what is the best approach to model that ?

noisesmith19:04:28

the gotcha of lifting a kafka consumption into eg. core.async is that you either immediately confirm you received the message (effectively lying) or risk that you might get booted from kafka because you took too long to consume the async message and confirm it (leading to error prone and complex recovery code)

noisesmith19:04:29

the main question is what you want to do with those messages - accumulate a running state? treat them as requests for tasks to perform?

noisesmith19:04:47

process some new data and put that back onto a kafka topic?

abdullahibra12:04:07

@noisesmith will fetch the messages and do some processing on it and then save into db

noisesmith12:04:03

it's not safe to do db calls inside a go loop or a kafka stream - in the go loop case you risk starving the go threadpool, in the kafka stream you risk being kicked out of the cluster

abdullahibra12:04:33

so what do you think good approach to model that?

noisesmith12:04:13

inside a go loop you can safely do longer calls with (<! (async/thread ...)) , thread returns a channel you can read and will offer a value when it completes

noisesmith12:04:23

regarding kafka, you could confirm the consumption of the message immediately, then dispatch to a future or send a message with put! and handle it in the consumer (probably via thread...)

noisesmith12:04:57

but you've already told kafka you're done with the message, so if you need fault tolerance (eg. picking up where you left off if there's a crash or the task can't complete etc.) you need something separate, as far as kafka is concerned your consumer group is no longer interested in that message

noisesmith12:04:44

(that's the gotcha I ran into when I tried to combine kafka with core.async to make a microservice / task management infrastructure)

lilactown19:04:41

can anyone ELI5 how I should use :volatile-mutable?

hiredman19:04:58

Volatile mutable is useful internally (which is why it makes fields private) when defining complicated deftypes

lilactown19:04:05

I want to build a custom ref type (similar to atom). my main use case is CLJS, but I think it would be cute to support CLJ as well

hiredman19:04:16

You might use volatile mutable for that, but it will depend on the specifics. You should be very careful implementing mutable stuff shared between cljs and clj, you can get away with a lot on js that is very unsafe with multiple threads

noisesmith19:04:32

@lilactown so something like volatile?

lilactown19:04:08

I'm at the level right now of "googling what the java volatile keyword does again" because I really don't live too much in the JVM runtime

noisesmith19:04:33

@lilactown yeah, to add to what @hiredman is saying, I see a continuum from single process js, to multi process jvm, to multi server distsys, where every concurrency semantic gets harder to implement, and more gotchas come into play

lilactown19:04:45

but ostensibly I want the same thread safety guarantees as a clojure atom

noisesmith19:04:59

and why are you not using atom?

hiredman19:04:02

It is unlikely that you custom ref type will be safe to use based just on volatile(but possible), so you may was well just assume you need a lock or an atomic ref

lilactown19:04:35

I want some special behavior (reactivity) on deref/swap/etc. I suppose I could have my custom type contain an atom and pass some of the methods through

phronmophobic19:04:12

fwiw, compare-and-set! is available if you want to change the semantics slightly

phronmophobic19:04:47

it allows you have to slightly lower level access to the CAS mechanism

hiredman19:04:56

The way to get what an atom has is using an atom

lilactown19:04:00

or I could pass in a java.util.concurrent.atomic.AtomicReference in the constructor and copy the general pattern in clojure.lang.Atom

noisesmith19:04:25

@lilactown reactivity other than what add-watch or set-validator gives you?

noisesmith19:04:18

I have a hard time imagining reactive semantics that aren't covered by those, I'd be fascinated to hear what you mean

lilactown19:04:55

it's mainly API/syntax. my goal: play with building a reagent ratom-like data structure, outside of the reagent view lib, and cljc

lilactown19:04:25

so e.g.:

(def foo (r/atom 0))

(def bar (r/reaction (* @foo 2))

(add-watch bar :log (fn [_ _ _ v] (prn :bar v))

(swap! foo inc)

;; out:
;;  :bar 4

phronmophobic19:04:56

you may be interested in javelin/hoplon https://github.com/hoplon/javelin

phronmophobic19:04:31

there is also a version for the jvm in progress

lilactown19:04:55

yeah I want to write my own 😄

👍 4
phronmophobic19:04:58

if nothing else, the folks in #hoplon are doing some interesting experimentation and are pretty responsive if you wanted to discuss ideas in a similar space

lilactown19:04:23

that helps! thank you

lilactown19:04:57

I guess the AtomicReference class is where I should start

hiredman19:04:14

Atomic reference isn't going to be safe for that

hiredman19:04:43

Because you need to change the value and do your reaction stuff atomically

lilactown19:04:58

is it the and do your reaction stuff part that will be difficult to do atomically using just an AtomicReference?

hiredman20:04:08

I mean, I guess you could build a spin lock using atomic reference, but at that point you can just use a Lock

hiredman20:04:36

Atomic ref only lets you compare and set a reference atomically

lilactown20:04:53

I guess I’ll have to think about if what I really want is an atom, or something more akin to a fancy agent

ghadi20:04:46

Atom uses an AtomicReference internally

hiredman20:04:30

I just noticed atomic reference seems to have grown a Java version of swap!

ghadi20:04:33

start at ^:volatile-mutable, end up at a UI framework 🙂

😅 4
😄 4
ghadi20:04:53

@hiredman yeah, there are a few helpers in there.

ghadi20:04:23

I reimplemented Atom with VarHandles instead of the AtomicReference

ghadi20:04:31

AtomicReference now uses VarHandles anyhow

hiredman20:04:08

It might also be interesting to look at a rules engine like clara

lilactown20:04:04

when I think high level what I actually want in CLJS, I don’t know if I want to guarantee that changes are synchronously invoked

lilactown20:04:53

it could be useful for React apps to have external state that participates using the React scheduler

phronmophobic20:04:53

do you have an example of how that would be used?

lilactown20:04:36

some API like:

(def state (r/agent 0))

(defn my-component []
  (let [[current send] (r/use-agent state)]
    [:button {:on-click #(send inc)} "+"]))
updates to state would not be immediately available, so logging the value of current or even @state inside the handler would have the old state

lilactown20:04:06

the update inc would be schedule to be run at some future time and would trigger a re-render in the component

phronmophobic20:04:13

re-frame sort of does this

lilactown20:04:22

it does, but it is very opinionated 🙂

phronmophobic20:04:14

although, I think this is the wrong direction. imho my-component should just be a normal pure function

phronmophobic20:04:05

I know hooks and subscriptions are the popular mechanism these days, but I don’t see why my-component doesn’t just take num as a parameter

lilactown20:04:33

you have to bring in the external state to the component tree at some point

lilactown20:04:20

I’m not interested in bike shedding about where and when to do that, it’s a simple example

4
lilactown20:04:22

which AFAICT is basically how an agent works but you can’t block on deref in CLJS

plins20:04:31

hey lets say I have a namespaced keyword :dojo.domain.error/not-found I want to convert it to a string but (str ns-k) -> ":dojo.domain.error/not-found" and (name ns-k) -> "not-found" I could drop the first char of the string of course but I wonder if there a better way of doing this

phronmophobic20:04:19

(str (namespace ns-k) "/" (name ns-k))

ghadi20:04:40

(str (symbol :your/kw))

phronmophobic20:04:42

does that work? I get an error:

> (symbol :foo/bar)
ClassCastException class clojure.lang.Keyword cannot be cast to class java.lang.String

ghadi20:04:52

Clojure 1.10.2-alpha1
user=> (symbol :asdf/we)
asdf/we
user=> (str (symbol :asdf/we))
"asdf/we"

plins20:04:31

worked like a charm thanks @U050ECB92

phronmophobic20:04:20

ok, I’m still on clojure 1.9. guess I need to upgrade

ghadi20:04:16

I think 1.10 has those goods

hiredman20:04:35

Rules engines often have the idea of quiescence, meaning you introduce new information, rules fire for a while asserting and retracting, until a stable state is reached

hiredman20:04:52

which is kind of like a transaction (introduce novelty -> react to novelty in private -> expose new start of the world atomically) which might lead you to wanting something like more like refs then atoms

lilactown20:04:59

yeah I like that. I think that transactionality of the reactive graph would be a great guarantee to provide, to avoid getting into broken/inconsistent states

lilactown20:04:21

I guess that kind of makes me want to reify the dependency graph

lilactown20:04:00

I need some way of computing an entire transaction without actually mutating anything

lilactown20:04:25

until the state settles

lilactown20:04:50

the funny thing is already started building a reactive lib that reifies the graph in a map and left it unfinished (it’s slow / has a lot of work to be done). then today I was like, “I don’t need all that, I just want ratoms without reagent” and here I am again

lilactown20:04:28

I think I’ve tried to scratch this itch and gotten to the same place like 3 or 4 times now

hiredman20:04:49

I am very skeptical of a lot of this dataflow ui stuff (but I also don't write a lot of uis), because it seems to end up building a system of continuous change which seems harder to understand than a discrete one

lilactown20:04:30

I sort of agree

lilactown20:04:39

I think one of the benefits of reifying edges in the graph is the ability to observe changes to the whole system much more easily

jjttjj20:04:09

I'm trying to create a web front end for a clj repl process. I will send code strings from the cljs client to the backend. I believe I have a lot to learn and feel kinda lost. I think I can start a clojure.main/repl process on the server. Is it just a matter of wrapping that clojure.main/repl call a binding that binds *in* and *out* to things that can be "sent" input and "receive" output? Any pointers for how I can start to approach this?

phronmophobic20:04:40

I would 100% recommend starting an nrepl server and then connecting to the nrepl server rather than working with raw *in* and *out*

hiredman20:04:12

it has been a while since I was tinkering with it, but I think the main thing missing is the public/private key authentication bits

hiredman20:04:51

wrap-repl is ring middleware, and http-repl is a function you call to connect your repl to a remote wrap-repl

jjttjj20:04:53

that looks great thanks, I think half the hard part for me is figuring out the java classes to use for the in/out stuff

noisesmith20:04:52

don't forget *err*

noisesmith20:04:08

you could use inputstream / outputstream pairs to interact with the repl, yeah

noisesmith20:04:36

to talk to the repl, you have an outputstream that pipes to an inputstream that feeds *in*

noisesmith20:04:49

and the reverse for *out* and *err*

noisesmith20:04:25

also, a repl (read, eval, print, loop) is something you can just construct, it's not magic

noisesmith20:04:01

read the data from client, call eval on that, print (prn) the result to the client

noisesmith20:04:48

(though there is bookkeeping for things like current ns dynamic vars etc...)

jjttjj20:04:09

Yeah that was my first approach but I was having issues evaluating ns forms so I figured I'd jump into the main repl stuff

Chris Lester20:04:58

Has anyone seen an example of a macro that merges the (cond-> behavior with (-> such that it threads the value from each expression into the next conditional?

Chris Lester20:04:27

Would like to reduce nesting so looking at that and potentially things like a maybe monad.

Chris Lester20:04:45

(and these are external service calls dependent on prior values)

jjttjj20:04:49

but I agree with @noisesmith and usually just jam together as->and cond->

noisesmith21:04:15

I misunderstood the question and accidentally said something relevant

😂 4
Chris Lester21:04:17

Thx. Yes, I could use those but given these are expensive I want short circuit vs (let ... (if () (let ... (if ... ()))). I kind of want a (cond-as->

noisesmith21:04:42

oh, then you want some-> in there

Chris Lester21:04:10

they are external service calls, don't want to make them ahead of time. I also want control over the return value .. so (some-> would work with some munging (to allow a predicate to determine instead of nil)

noisesmith21:04:48

I think what you want isn't a syntax rewrite, and all these threading operators can do is syntax

noisesmith21:04:53

none of them have real semantics

noisesmith21:04:35

you can use run! or reduce to call each function in a series, and then use an exception or reduced to force short circuiting

noisesmith21:04:49

at least that has meaning at runtime beyond moving parens around

Chris Lester21:04:07

True, will try that route first. Hadn't thought of running them as a sequence in reduce.

noisesmith21:04:40

(reduce (fn [state f] (f state)) init [f1 f2 ...])

noisesmith21:04:01

if any of the fs return (reduced v) that will prevent further functions from running, and will be the return value of reduce

noisesmith21:04:26

if you find it ugly, you could make a macro that takes each form in a body and puts it in a two-arg function

Chris Lester21:04:01

We'll see, I've done this for other things just not sequences of server calls. I can hand it a sequence of partially applied fns and avoid the macro as they all return the same shape. Thx!

noisesmith21:04:34

also (defn rapply [state f] (f state)) might be nice to in that sort of idiom

noisesmith21:04:08

a silly little dsl

(cmd)user=> (defn rapply [state f] (f state))
#'user/rapply
(ins)user=> (def pt (partial apply partial))
#'user/pt
(ins)user=> (defn states [init & fs] (reduce rapply init (map pt fs)))
#'user/states
(ins)user=> (states 0 [+ 1] [* 3] [/ 2.0] [reduced] [#(throw (ex-info {:oops %}))]])
0.6666666666666666
(ins)user=> 

noisesmith21:04:11

I think I've been reading too much scheme code

dpsutton21:04:39

you still hacking on / reading the scheme shell code?

noisesmith22:04:05

I've been messing with guile lately, for a weekend project that sneaks into my real time

g21:04:50

is there a way to cleanly exit a swap operation if an exception is thrown during it?

g21:04:35

namely,

> (def a (atom 1))
> (swap! a (fn [a] (throw Exception)))
> @a
1

g21:04:20

i’m trying to see if it’s possible to do multiple swaps within a dosync (which is probably an abuse of dosync, if possible)

noisesmith21:04:35

what's unclean about that?

noisesmith21:04:40

swap! and dosync are unrelated

noisesmith21:04:24

@gtzogana to be clear, what you have above already happens (except the exception goes all the way up, isn't caught by swap! itself)

g21:04:56

ugh, i was making a dumb syntax error. my bad

g21:04:12

is there any notion of transactional operation on more than one atom? or am i just using the wrong primitive

noisesmith21:04:22

that's what refs are

jjttjj21:04:22

I think you want ref

g21:04:35

yeah, figured. ok, thanks!

awb9921:04:47

Is it possible configure the clojure reader, so it would not ignore comment only lines?

awb9921:04:58

(+ 6 6 ; add numbers )

awb9921:04:19

; add numbers

awb9921:04:48

the comment in the first version is available in the read meta-data

awb9921:04:00

but in the second version the comment is completely gone.

awb9921:04:21

I am trying to get the second version to give me back comments only as well.

andy.fingerhut22:04:14

@hoertlehner I was not aware that the Clojure reader would give back comments, ever, in any form, metadata or otherwise. How did you get it to do so for the first string?

awb9922:04:55

If you could lead me to some kind of direction, that would be great!

awb9922:04:19

I think it still should be possible to somehow extend the reader.

awb9922:04:35

give the reader a different handler that parses comments...

andy.fingerhut23:04:45

I have not actually used a 3rd party Clojure library to get the contents of comments from Clojure code or data, but you could see whether https://github.com/cgrand/sjacket does what you want, with the warning that it might not.

andy.fingerhut23:04:23

It is possible to extend the built-in reader, if you are willing to copy it, modify it yourself for that purpose, and if you ever distribute it, do so under the EPL license that Clojure code is released under.

andy.fingerhut23:04:11

Depending on whether you prefer to write Java or Clojure code (the Clojure reader is implemented Java), the tools.reader library does most the same thing, but implemented in Clojure. It also does not do anything to preserve comments for you..

Michael J Dorian23:04:02

There are also other clojure parsers, I'm using https://github.com/xsc/rewrite-clj and I've heard good things about https://github.com/carocad/parcera

andy.fingerhut22:04:04

And depending on what your goal is, there are 3rd party libraries that might already do what you want.

Alex Miller (Clojure team)22:04:51

The reader’s job is to turn strings into data. Retaining comments or other white space is outside its main job. As Andy said, there are parsers that retain this context that might be a better match for your needs.

awb9903:04:01

Thanks Alex. Could you let me know which libraries might work for this ?

bfabry22:04:16

just my humble opinion but I think you'd be better off extracting to a function and letting the function name be your comment or putting docs on the function. unless this is a you-dont-control-the-src situation