Fork me on GitHub
#clojure
<
2020-04-08
>
pinkfrog03:04:16

one thing I found inconvenient is the short variable name s

pinkfrog03:04:49

s could be a parameter for string, or for the clojure.string namespace.

pinkfrog03:04:09

so often times have to think about is it really this s or that s .

potetm03:04:37

s -> param for a string, s/SOMETHING -> reference to a namespace

potetm03:04:11

really kind of impossible to mix them up, IMO

seancorfield03:04:18

It's considered "good style" to generally use the last segment of a namespace as its alias, as long as that would be unique in your file and consistent across namespaces -- with some short aliases being "common practice". I've not seen s as an alias for clojure.string -- but I have seen str as alias for it very commonly. Where I've seen s as a common alias is for clojure.spec.alpha -- it's all over the http://clojure.org docs.

seancorfield03:04:54

But, yeah, you can't confuse foo for a symbol with foo/ for a namespace qualifier really...

seancorfield03:04:08

The REPL guide on http://clojure.org uses str for clojure.string.

Alex Miller (Clojure team)03:04:00

s as an alias can only be used in the qualifier position of a keyword or symbol - there is no place in the language where it can be ambiguous with an unqualified local symbol or var name

Alex Miller (Clojure team)03:04:20

these are independent naming domains

seancorfield04:04:23

I can see how beginners might find (str "Hello, " "World!") and (str/join ["Hello, " "World!"]) confusing on first glance -- but the context is fundamental and just one of those things you get used to.

pinkfrog05:04:59

is using the function str and the string namespace as str a common practice that is not frowned upon?

seancorfield05:04:57

It is very common and definitely not frowned on. I'm on my phone right now but I'll check my clojure books tomorrow and see how various books alias clojure.string

Alex Miller (Clojure team)05:04:42

str, s, and string are all common aliases for clojure.string

Alex Miller (Clojure team)05:04:10

I nearly always use str

didibus07:04:49

Yup, I mostly use str, though sometimes I go with string. It is fine to have an alias "clash" with a function, like others have said, they don't actually shadow each other, the context of each is pretty clear str/wtv or str. But s for me right now is dedicated to Spec, and I do get confused if I see a s/... that isn't for Spec.

👍 4
seancorfield16:04:11

Books: Clojure in Action str, Joy of Clojure str and string, Clojure Cookbook str, Clojure Programming str, Clojure Applied (no alias), Functional Programming Patterns in Scala and Clojure str, Getting Clojure (no mentions of clojure.string at all), Programming Clojure str, Web Development with Clojure string. Several of these also use the full namespace with no alias. I don't have Living Clojure. I did not search any of my Packt Clojure books (I have several). So str seems to be the most popular with authors 🙂 @i

pinkfrog01:04:18

@U04V70XH6 thanks for that. really great efforts to compile such a list.

seancorfield01:04:07

Hey! Yeah, I can search the Packt books too if you want -- but I mostly don't think they're very good books in general so I don't pay much mind to them.

ryan echternacht04:04:00

@i As someone who started spamming clojure about about a month ago, I agree that clojure has some different conventions. And then one day it clicks and you’re good to go

Jin14:04:13

hello All I have several projects on my firebase console, but now all gone.... projects are all gone! firebase is downed?

Braden Shepherdson14:04:00

Firebase, and Google Cloud more broadly, are having a major outage; ~none of the management APIs or consoles are working.

Braden Shepherdson14:04:02

and it seems to be recovered now, more or less.

Jin14:04:24

Yes disaster

Spaceman14:04:19

I have the following docker file: and I'm trying to deploy on heroku. But I get the error that clojure.main not found.

Spaceman14:04:59

Even when running locally using: docker build -t test . and docker run -ti test I get the same error

Spaceman14:04:17

What am I doing wrong and how do I fix this

pinkfrog14:04:25

how do you cope with multiple if guards?

pinkfrog14:04:35

I feel some-> and cond still not elegant

the2bears15:04:09

I find cond quite elegant. As an alternative you could try writing a macro that meets your aesthetic requirements.

nick15:04:11

@i I really enjoyed reading a blog post on similar issue a few days ago https://adambard.com/blog/five-mistakes-clojure-newbies-make/ "attempt-all" - a monadic error-checking construct

didibus16:04:12

What do you mean multiple if guards?

pinkfrog00:04:04

@U0K064KQV saw your post: https://clojureverse.org/t/favorite-macros/2716/2 . very enlightening. multiple if guards means several error checks so the function early returns.

didibus00:04:18

Oh thanks 🙂

didibus00:04:32

I see what you mean now. That's an interesting ask, I know I've seen others before ask for things in that vein. I think though, because of Clojure trying to not fight Java too much, and embrace it, is why you don't have more of those.

didibus00:04:24

For example, you have some-> and some->> to guard against nil. You can see those a bit like the safe navigating operator some languages have, sometimes being it is .? https://en.wikipedia.org/wiki/Safe_navigation_operator

didibus00:04:50

That one is really useful when working with Java objects actually, but it can be used in Clojure too, if you'd have a function that doesn't take nil and another that can return it.

didibus00:04:15

So I'm guessing here though, you're asking where are the similar "safe navigators" for errors?

didibus00:04:20

I think the answer here is that by default, Clojure models errors using Java's exceptions, and those are always short-circuiting, no need for any special operator for that.

didibus00:04:50

But, I know some people like to model their error as values, maybe you return :error, or something else that you cooked up which is meant as an error. The thing is, since those type of values as error are all custom, you can't really build a generic safe navigator for them

pinkfrog00:04:49

ah.. you speak out a point that clojure throws exception instead of returns error values.

pinkfrog00:04:07

“Clojure models errors using Java’s exceptions” a common practice ?

didibus00:04:16

That's where you could build your own though. And a lot of these "value error" libraries do provide such macro already. I think failjure being the most popular one: https://github.com/adambard/failjure

pinkfrog00:04:20

for example, in validating the user input

didibus00:04:55

Yes, I'd say it is more common to model errors by throwing exceptions

didibus00:04:06

Normally you'd throw an ex-info

pinkfrog00:04:29

thanks. then i’d go with the exception routine.

didibus00:04:34

Personally, I think its the smoothest route. The problem is, whenever you don't go with them, you hit a point where they still leak out into your code, because underneath there is Java or JS, and both throw exceptions

didibus00:04:53

If you're going to go the other way, I do recommend failjure, it does provide nice utilities to deal with the leaking exceptions from Java and JS, and other nice ones for working with value errors. But still, I'd say best to just accept exceptions and work with them

didibus00:04:01

Hum... though you gave me an idea 😛

didibus01:04:21

(defmacro unless->
  [pred & [expr & forms]]
  (let [g (gensym)
        steps (map (fn [step] `(if (~pred ~g) nil (-> ~g ~step)))
                   forms)]
    `(let [~g ~expr
           [email protected](interleave (repeat g) (butlast steps))]
       ~(if (empty? steps)
          g
          (last steps)))))

didibus01:04:46

(defn might-error [e]
  (if (rand-nth [true false])
    e
    :error))

(unless-> #{:error} 10 inc might-error inc)
nil

didibus01:04:49

So now you can give a predicate, and between each invocation, it will check the result against it, and if it is true, it will short-circuit and return nil.

didibus01:04:27

Kind of a more generic version of some->:

(defn returns-nil [_] nil)
(some-> 10 inc returns-nil inc)
;; => nil
(unless-> nil? 10 inc returns-nil inc)
;; => nil

didibus01:04:55

This one might be more useful:

(defmacro until->
  [pred & [expr & forms]]
  (let [g (gensym)
        steps (map (fn [step] `(if (~pred ~g) ~g (-> ~g ~step)))
                   forms)]
    `(let [~g ~expr
           [email protected](interleave (repeat g) (butlast steps))]
       ~(if (empty? steps)
          g
          (last steps)))))

didibus01:04:30

It returns the last value instead of nil, so with the above you get:

(defn might-error [e]
  (if (rand-nth [true false])
    e
    :error))

(until-> #{:error} 10 inc might-error inc)
;; => :error

Jakub Holý (HolyJak)15:04:46

Hello! I get the feared StackOverflowError with a stack trace that only contains some high-level app code and "Caused by" has just a useless repetition of

...
               [clojure.lang.LazySeq sval "LazySeq.java" 42]
               [clojure.lang.LazySeq seq "LazySeq.java" 51]
               [clojure.lang.RT seq "RT.java" 535]
               [clojure.core$seq__5402 invokeStatic "core.clj" 137]
               [clojure.core$concat$fn__5493 invoke "core.clj" 725]]
Can I do something somewhere to not get these clojure.* frames and get rather my app frames that called core? Now I have no idea where in the app this happened...

Alex Miller (Clojure team)15:04:47

do you really have no frames that are outside core? sometimes you might need to actually look at the exception stack directly vs using pst or the data representation (.printStackTrace *e)

Alex Miller (Clojure team)15:04:28

the big clue above though is "concat" - that probably narrows it down quite a bit

Jakub Holý (HolyJak)15:04:24

Thank you for the tip, I will try. I use concat at few places so I could wrap them with try-catch and retry...

Alex Miller (Clojure team)15:04:49

the most common way concat gets into something like this is if you're doing some kind of recursive concat - that creates a stack of suspended "firsts" which will blow up on first realization

Alex Miller (Clojure team)15:04:19

so look at whether you're doing concats in a loop/recur or something equivalent (and stop doing that :)

Jakub Holý (HolyJak)15:04:57

There is no obvious place where I would be doing that so I must search more... I have looked at the stack trace but there only seems to be core:

(first (seq (.getStackTrace *e)))
=> [clojure.lang.LazySeq seq "LazySeq.java" 51]
(last (seq (.getStackTrace *e)))
=> [clojure.core$concat$fn__5493 invoke "core.clj" 725]

andy.fingerhut18:04:18

Also note that poring over stack traces is a good reason to avoid using the source file name core.clj for your own code.

Jakub Holý (HolyJak)15:04:23

Hm, I have replaced two concat s with into and now it works. Thank you!

Alex Miller (Clojure team)15:04:19

well it would certainly be good to understand why that helped

p-himik17:04:31

Can someone help me understand this docstring at https://aleph.io/codox/aleph/aleph.flow.html#var-utilization-executor ?

utilization-executor

(utilization-executor utilization)
(utilization-executor utilization max-threads)
(utilization-executor utilization max-threads options)

Returns an executor which sizes the thread pool according to target utilization, within
`[0,1]`, up to `max-threads`.  The `queue-length` for this executor is always `0`, and by
default has an unbounded number of threads.
It sounds like it's just like a fixed-thread-executor where N = max-threads * utilization. But I really doubt that because then there would be no reason for a different implementation.

p-himik17:04:57

Ah, I think I understand. It allows max-threads but it will try to keep the pool size just so that the utilization is at the preferred level.

noisesmith17:04:18

it's probably easier to just understand the source - after chasing down var imports and thin abstractions you get this https://github.com/ztellman/manifold/blob/master/src/manifold/executor.clj#L83

p-himik17:04:03

Yeah, I got to that part. I just blanked on the concept of utilization in this context.

Johnny Hauser18:04:27

I am greatly confused by the completion aspect of transducers. It makes sense when transducing synchronous collections, but not collections which produce their member values over time. You put such a value into the transduce function and immediately get the result back, just like synchronous collections, but the input collection doesn't complete until later (if ever). That makes the semantics described by Rich impossible for such a collection, right? Because the completion function would have to be called immediately, and the return value of it should be the return value of the call to transduce, but that is before all the step functions would have been called, since those are happening later. On the other hand, calling the completion function when the input process actually completes seems favorable for those types, but that breaks compatibility among different collections. Am I missing something?

phronmophobic18:04:39

> You put such a value into the transduce function and immediately get the result back you don’t always get a result back immediately. if it can’t produce a value, the transducer will hold state until the next value

💯 4
noisesmith18:04:23

exactly, see for example partition-all

Johnny Hauser18:04:37

Hmm. Let me try to digest this for a few... It doesn't seem like we're talking about the same things.

phronmophobic18:04:56

(defn partition-all
  "Returns a lazy sequence of lists like partition, but may include
  partitions with fewer than n items at the end.  Returns a stateful
  transducer when no collection is provided."
  {:added "1.2"
   :static true}
  ([^long n]
   (fn [rf]
     (let [a (java.util.ArrayList. n)]
       (fn
         ([] (rf))
         ([result]
            (let [result (if (.isEmpty a)
                           result
                           (let [v (vec (.toArray a))]
                             ;;clear first!
                             (.clear a)
                             (unreduced (rf result v))))]
              (rf result)))
         ([result input]
            (.add a input)
            (if (= n (.size a))
              (let [v (vec (.toArray a))]
                (.clear a)
                (rf result v))
              result))))))
  ... other arities)

Johnny Hauser18:04:16

Yeah, I'm pretty lost. To try to say it again, in case it helps, I put in a collection and immediately get back a transformed collection, but neither input nor output collection have any values yet, because they come to exist later on. It's just that the transformed collection that was returned will have transformed values at those times, in contrast to the input collection.

Johnny Hauser18:04:50

Which means that I was able to produce a transduced collection immediately, but the iteration and so step functions happen after that

phronmophobic18:04:28

now, I’m not sure I follow. do you have a small code example?

Johnny Hauser18:04:45

Unfortunately, I'm from JavaScript, so I couldn't offer any clojure 😕

Johnny Hauser18:04:28

RxClojure is probably a good point of reference

Johnny Hauser18:04:23

If you were to transduce an Rx Observable, you would put an observable in and get an observable back immediately.

phronmophobic18:04:43

one difference between transducers and similar ideas elsewhere (like in observable) is that the transducer (or step function) is just the step function and isn’t coupled to an observable, collection, or anything else

Johnny Hauser18:04:52

Rx for JS used to have a transduce function, which would take an observable and a transducer/pipeline and return an observable that, on each value from the input observable, would run it through the step function and if it reaches the end, it goes into the resulting observable. Then when the input observable completes, it calls the transducer completion function.

phronmophobic18:04:07

in Rx, the map they provide is coupled to an observable

Johnny Hauser18:04:25

Yes... fwiw, I have written a whole transducer lib. I'm not that much a noob, I hope 🙂

Johnny Hauser18:04:50

I particularly don't want operators specific to my frp lib - which is why I wrote the transducer lib.

Johnny Hauser18:04:37

A lot of stuff works.

phronmophobic18:04:22

yea, hopefully I’m not just telling you a bunch of stuff you already know 😬. I’m just trying to understand the crux of the question.

Johnny Hauser18:04:37

I am just confused in that the protocol seems to believe that the return value and the completion of the input are one and the same.

phronmophobic18:04:14

in the clj transducer protocol? or the Rx protocol?

Johnny Hauser18:04:21

transducer protocol

Johnny Hauser18:04:01

it builds up the result, then passes that result into the transducer result fn, and returns the value from it

Johnny Hauser18:04:08

exactly as Rich said in the talk about transducers

Johnny Hauser18:04:24

His slide said the transduction has to return the result from the completion function

Johnny Hauser18:04:46

But that means that the return value from the completion function and the return value from the transduce call have to be the same, and that's impossible for observables.

Johnny Hauser18:04:11

You get the resulting observable immediately/synchronously, and the values/steps/iteration come later

Johnny Hauser18:04:15

and so completion is later

phronmophobic18:04:32

per the docs: > The inner function is defined with 3 arities used for different purposes: > Init (arity 0) - should call the init arity on the nested transform rf, which will eventually call out to the transducing process. > Step (arity 2) - this is a standard reduction function but it is expected to call the rf step arity 0 or more times as appropriate in the transducer. For example, filter will choose (based on the predicate) whether to call rf or not. map will always call it exactly once. cat may call it many times depending on the inputs. > Completion (arity 1) - some processes will not end, but for those that do (like transduce), the completion arity is used to produce a final value and/or flush state. This arity must call the rf completion arity exactly once. the completion is separate from the step. there is a transduce function that will do a bunch of this stuff for you by calling the step function for each value of the supplied collection and then calling the completion function

phronmophobic18:04:50

you don’t have to use transduce to use your transducer, if that makes sense

ghadi18:04:24

those arities in the docs refer to a reduction function, also known as a stepped process

Johnny Hauser18:04:50

That doesn't seem to mention this thing I'm referring to.

ghadi18:04:02

transducers are about altering or augmenting a process

ghadi18:04:18

transducers do not see values from the collection

ghadi18:04:25

transducers are called with processes

ghadi18:04:33

and return processes

ghadi18:04:39

it is the process that sees the input / collection

phronmophobic18:04:58

my point is that you can use a transducer without ever calling its completion function if a completion function doesn’t make sense for your use case

ghadi18:04:20

my point is being precise about terminology

ghadi18:04:28

transducers don't have completing functions

ghadi18:04:51

it's the processes that transducers return that have the completing processes

Johnny Hauser18:04:15

weelll, it's a losing battle, because that slide is directly from Rich and muddies it all up

ghadi18:04:29

that slide is a bit muddled, IMHO

Johnny Hauser18:04:38

It's the "must return what it returns" thing that I'm referring to

Johnny Hauser18:04:55

He also specifically mentioned that transducers may want to modify the final accumulated value

ghadi18:04:59

so that last slide should read:

ghadi19:04:05

last bullet, rather:

ghadi19:04:04

the completion operation of the updated process returned by a transducer...

ghadi19:04:44

...must call the original process's completion op

ghadi19:04:11

transducers are process -> process

Johnny Hauser19:04:15

So we agree that returning the result from the completion function as the result of the transduction is not necessary?

ghadi19:04:45

give an example?

phronmophobic19:04:12

it depends on the transducer, but I don’t think the completion function ever has to be called if your input never completes

Johnny Hauser19:04:45

I had linked to the js transducer lib above, where it passed the accumulated value into the completion function and returns that

ghadi19:04:08

completion must be called because processes can abort early

Johnny Hauser19:04:11

return xf["@@transducer/result"](acc);

// vs.

xf["@@transducer/result"](acc)
return acc

Johnny Hauser19:04:27

I don't mean whether it's called - I mean whether it's returned.

ghadi19:04:35

consider the take transducer

phronmophobic19:04:55

that’s for arrayReduce. presumably, its a finite array and calling the completion function does make sense

Johnny Hauser19:04:11

I have the take transducer and it works fine for my arrays and observables

ghadi19:04:15

even if the input collection is infinite, the a reducing function transformed by the take transduder will abort

Johnny Hauser19:04:26

heh, I still feel like my point is missed

ghadi19:04:32

and when it aborts, must call the completion op

ghadi19:04:40

to flush any nested processes

Johnny Hauser19:04:02

Of course we should call the completion function, I'm asking whether the return value of the completion function is supposed to be the return value of the transduction

Johnny Hauser19:04:07

Rich said it should be

ghadi19:04:43

yes, that is correct

Johnny Hauser19:04:58

But that's the paradox

ghadi19:04:01

some reduction contexts do not reveal the completed value though

ghadi19:04:04

like channels

Johnny Hauser19:04:30

You have an input observable, you get back an immediate observable, but there hasn't even been any step functions yet, and those will happen later and the completion even later. You can't have both.

phronmophobic19:04:31

the completion function doesn’t return the full result, it emits any accumulated values that might need to be flushed

phronmophobic19:04:20

so values* can have been emitted long before a completion function is called or even if the completion function is never called

ghadi19:04:31

do you mind restating the question here?

Johnny Hauser19:04:06

const transduce = transducer => etc => input => {
  const result = reduceSomehow(transducer.step, input, etc)
  return transducer.complete(result) // this!
}

Johnny Hauser19:04:17

for the sake of demonstration - obviously not real and coherent code

Johnny Hauser19:04:54

You get the result immediately, and return it immediately, but Rich says you're supposed to pass it through the completion function and return that

Johnny Hauser19:04:05

but all the code there is synchronous

Johnny Hauser19:04:31

inside that reduction is where the setup is done which will hook up future values to the transformation pipeline

ghadi19:04:32

that code above is not a good representation of how transduce works

ghadi19:04:37

transducers don't have steps

ghadi19:04:41

processes have steps

Johnny Hauser19:04:30

I was trying to get to the meat of it without dealing with that stuff

ghadi19:04:57

(defn transduce
  [xf f init coll]
  (let [tricked-out-process (xf f)]
    (-> (reduce tricked-out-process init coll)
      (tricked-out-process)))) <- completion step

Johnny Hauser19:04:04

yes, you need a builder, and pass it into the transducer to get back, what I have usually heard referred to as the transformer, which has the step, result, init properties

ghadi19:04:06

transducer is xf

ghadi19:04:12

process is f

phronmophobic19:04:24

here’s an example that is maybe more concrete that we can fix:

(def my-transducer (map (fn [x] (inc x))))

(def my-atom (atom []))
(defn add-to-atom! [atm val]
  (swap! atm conj val))

(def process (my-transducer add-to-atom!))

(process my-atom 1)
@my-atom ;; [2]

(process my-atom 2)
@my-atom ;; [2 3]

phronmophobic19:04:53

so you can see that every time we call process, the steps are emitting value

ghadi19:04:31

....keep going

Johnny Hauser19:04:53

You know what I just noticed. Rich said a process may want to do a final transformation of the value being built up

ghadi19:04:56

keep in mind add-to-atom! is missing the completion arity

phronmophobic19:04:19

here’s another example:

(def my-transducer (comp (map (fn [x] (inc x)))
                         (partition-all 2)))

(def my-atom (atom []))
(defn add-to-atom! [atm val]
  (swap! atm conj val))

(def process (my-transducer add-to-atom!))

(process my-atom 1)
(process my-atom 2)
(process my-atom 1)
(process my-atom 2)
(process my-atom 1)

;; notice the there is a 1 missing
@my-atom ;;[[2 3] [2 3]]

Johnny Hauser19:04:39

What does he mean by a process?

Johnny Hauser19:04:12

I understand your examples, btw

ghadi19:04:18

process is something with: a beginning (arity-0), step (arity 2) called many times, completion (arity 1)

Johnny Hauser19:04:35

ah yeah, so back in the same boat

ghadi19:04:38

sometimes loose usage of process refers to the step (arity 2)

Johnny Hauser19:04:51

In other words, it's the thing that takes the next thing and returns a thing

ghadi19:04:14

in your example add-to-atom! needs to return the atom for the next step

ghadi19:04:31

you're manually calling it with the same atom every time

ghadi19:04:43

but you couldn't feed that function into transduce/reduce and have the intended behavior

Johnny Hauser19:04:47

Then, in your example, the issue is that any process that wants to modify the final accumulated value can't do it

ghadi19:04:18

the process's arity-1 is exactly when it can complete the accumulated value

ghadi19:04:26

look at into:

phronmophobic19:04:30

like this?

(def my-transducer (comp (map (fn [x] (inc x)))
                         (partition-all 2)))

(def my-atom (atom []))
(defn add-to-atom! [atm val]
  (swap! atm conj val)
  atm)

(def process (my-transducer add-to-atom!))

(-> my-atom
    (process 1)
    (process 2)
    (process 1)
    (process 2)
    (process 1))


@my-atom ;;[[2 3] [2 3]]

Johnny Hauser19:04:38

Is this example not making sense?

Johnny Hauser19:04:42

return xf["@@transducer/result"](acc);
// vs.
xf["@@transducer/result"](acc)
return acc

ghadi19:04:48

yes @U7RJTCH6J that is more correct

👍 4
phronmophobic19:04:40

I can’t connect how the js example from arrayReduce applies to your question

ghadi19:04:07

the "into" process does: make a mutable collection (arity-0) add to the collection (the step) make the mutable collection persistent (completion / arity1)

Johnny Hauser19:04:35

In the first case, the accumulated value is passed to the completion fn and the result from the completion fn is returned, and in the second case, the completion fn is called, and it receives the accumulated value, but it's return value goes out into the void. Nothing cares. The accumulated value is just returned, independent of the completion fn.

ghadi19:04:39

(transduce
  identity ;;  <-- don't modify the process!
  (fn 
    ([] (transient []))
    ([tc] (persistent! tc))
    ([tc v] (conj! tc v)))
  coll)

Johnny Hauser19:04:19

Rich said the first one, which that array reduce does, it what is supposed to be done

Johnny Hauser19:04:22

You know, I've got an idea of a solution

ghadi19:04:27

what is the first case vs. second case?

Johnny Hauser19:04:42

heh, fun times

phronmophobic19:04:28

here’s the example above using the completion function:

(def my-transducer (comp (map (fn [x] (inc x)))
                         (partition-all 2)))

(def my-atom (atom []))
(defn add-to-atom!
  ([atm]
   atm)
  ([atm val]
   (swap! atm conj val)
   atm))

(def process (my-transducer add-to-atom!))

(-> my-atom
    (process 1)
    (process 2)
    (process 1)
    (process 2)
    (process 1)
    (process))


@my-atom ;; [[2 3] [2 3] [2]]

Johnny Hauser19:04:10

perhaps when he said "a process may want to do a final transformation of the value being built up", there is the rule that it is a mutation of that value, and not a replacement of it. If so, then my concern is irrelevant.

Johnny Hauser19:04:24

Imagine a completion fn that, no matter what, just returns the number 7. You could put any array through that process, and you're just going to get back a 7 with that transducer lib I linked, because it returns the result of passing the accumulated value into the transformation fn.

Johnny Hauser19:04:42

And so it should be true that no matter what you're operating on, you should just get back a 7.

Johnny Hauser19:04:22

But that would only be true if you called the completion fn immediately/synchronously, which is only relevant to some collections. Which means something is whacked.

Johnny Hauser19:04:11

Given a collection that is an observable, you would get back an observable, but given a collection that is an array, you would get back a 7? Bananas!

ghadi19:04:42

there are different types of transducing contexts, and not all of them even expose the completed value

ghadi19:04:35

core.async channels vs reduce/transduce -- totally different

ghadi19:04:57

the general idea of transducers being "process transformation" is the key

ghadi19:04:15

in a channel the "process/step" is adding to the channel

Johnny Hauser19:04:21

Well, maybe I'll settle this in my mind from another angle. I would like to generalize some common operations, for example, last

ghadi19:04:43

in reduce/transduce, the process is the reducing function that the user passed

Johnny Hauser19:04:49

I am surely coming across a lot dumber than I hopefully am

phronmophobic19:04:09

last doesn’t make sense for an infinite collection. right?

Johnny Hauser19:04:32

It would be dumb to do it haha

Johnny Hauser19:04:45

An unresolved promise would be a comparable idea

ghadi19:04:47

again transducers and processes have nothing to do with the input

phronmophobic19:04:05

I guess the last value of an infinite number of 1s might make sense, but I’m not sure that’s what you’re talking about

Johnny Hauser19:04:06

I really, really do understand transducers

Johnny Hauser19:04:38

It'd be amazing if I wrote a transducer lib and am transducing all kinds of things and don't actually understand transducers lol

ghadi19:04:09

you could have a reducing process that threw away all prev values except the immediate one

Johnny Hauser19:04:45

and you could return a promise of the one that it is at the completion of the input collection

phronmophobic19:04:38

i hope we’re not coming off as patronizing. I’m really just trying to help and understand the question. I feel kinda dumb for not understanding the question exactly

phronmophobic19:04:07

as @U050ECB92 said, I think most-recent-value makes more sense than last for an infinite collectino, but maybe that’s just a naming thing

Johnny Hauser19:04:37

What you need for totally collection generic last is just that collections provide a reduce fn (so you can step it), a thing to keep holding the latest value of the step, and access to the value in a way that accords with the semantics of the collection, which would just be synchronous or async access to it, meaning, you could get it down to either a value now or a promise of a value.

phronmophobic19:04:54

you can do that

Johnny Hauser19:04:45

Yeah, I have done it, but some confusing stuff with the transducer protocol is preventing me from really moving forward with that stuff.

Johnny Hauser19:04:21

I really suspect the rule I'm looking for would mean that that array reduce I had linked could have (and should have) passed the accumulated value into the completion function and returned it after that, rather than returning the result of the completion function.

Johnny Hauser19:04:15

because that completion function should just be mutating that value, if anything

ghadi19:04:33

imho transducers.ITransformer in the transducers-js library is a really bad name

Johnny Hauser19:04:42

never, for example, saying "well, I've decided the result of this operation is going to be a 7, instead of the array I was given"

ghadi19:04:02

IProcess would have been better

phronmophobic19:04:31

I think those semantics make sense for arrayReduce, but another consumer of the transducer can use the transducer in a different context with different rules

Johnny Hauser19:04:39

Another way of asking my original question would be whether the return value of transduce has to be the same thing that is (thisResultRightHere, value) => in the step/reducer.

Johnny Hauser19:04:56

Do you have an example of a different context?

Johnny Hauser19:04:08

I'd really like that not to be the case... seems like a bad idea.

ghadi19:04:20

the return value of transduce has to be the result of calling complete on the accumulator

Johnny Hauser19:04:46

sigh... but that's once again, impossible for some collections

phronmophobic19:04:58

the two main contexts for transducers are: 1. concrete collections 2. core.async channels

ghadi19:04:12

right, arrayReduce is a particular transducing context

ghadi19:04:55

in Clojure, arrays know how to be reduced, and transduce is built on top of that

phronmophobic19:04:07

but they’re really flexible, so you can use the same transducers (map, filter, partition-all) in other contexts (eg. observables)

Johnny Hauser19:04:33

by 'context', do you mean having a different transduce fn altogether?

Johnny Hauser19:04:53

That seems to be the common thing to do, but so far, I don't agree with it.

ghadi19:04:24

yeah, things like Iterators have a "stepping process" that can be transformed

ghadi19:04:49

same with channels

ghadi19:04:03

I suspect observables too, but haven't considered them

Johnny Hauser19:04:05

In either case, do you not put one in and immediately get one back?

ghadi19:04:20

put what in?

Johnny Hauser19:04:31

input a channel, output a channel

Johnny Hauser19:04:47

no return value?

ghadi19:04:06

channels have values flowing through them

ghadi19:04:20

a transducer on a channel changes the flow

ghadi19:04:32

like (partition-all 500) would batch things

Johnny Hauser19:04:43

Ofc, just like with observables

ghadi19:04:49

you put in a value ,but no one sees it on the other side until 500 values are put in

ghadi20:04:02

channels do not expose the completion value

Johnny Hauser20:04:18

Can you show me like a two liner code sample for this, because that doesn't many any sense.... that you don't get a return value

ghadi20:04:20

which is very different than the collection transducing context

phronmophobic20:04:45

(def my-transducer (comp (map (fn [x] (inc x)))
                         (partition-all 2)))
(def my-atom (atom []))
(defn add-to-atom! [atm val]
  (swap! atm conj val))
(def process (my-transducer add-to-atom!))
(process my-atom 1)
(process my-atom 2)
(process my-atom 1)
(process my-atom 2)
(process my-atom 1)
;; notice the there is a 1 missing
@my-atom ;;[[2 3] [2 3]]

phronmophobic20:04:54

notice the missing 1

Johnny Hauser20:04:15

channelA = makeAChannel()
channelB = transform(ChannelA) // no??

ghadi20:04:30

(sequence (halt-when even? (fn [ret i] {:ret ret :i i})) [1 3 5 2])

ghadi20:04:37

not channels, but a one-liner ^

Johnny Hauser20:04:43

dudes, I really have no problem understanding abstract special collection semantics 🙂

ghadi20:04:20

(you keep bringing it up, but no one is making any judgements on your capacity)

ghadi20:04:42

(sequence xf coll) <-- returns a lazy seq subject to a transducer

Johnny Hauser20:04:55

We just keep getting back to examples talking about how the transduction is independent of the collection semantics, but sort of naturally affects them in cool ways. I get that.

Johnny Hauser20:04:02

I thought we were almost to a good thought, but we got sidetracked somehow

ghadi20:04:03

notice halt-when is a transducer that allows you to barge in a different completed value

ghadi20:04:26

but the sequence transducing context never shows the completed value

ghadi20:04:36

I believe you are conflating source collections with transducible contexts

Johnny Hauser20:04:56

strong possibility that is true

Johnny Hauser20:04:15

What's the difference?

ghadi20:04:23

-`transduce` (the function in clojure, is a context) - async channels (another context) - sequenceanother context

phronmophobic20:04:34

> input a channel, output a channel within the context of core async channels, the channel is still returned, but it’s not a new or different channel (it’s the exact same channel). the important part is the side effect of stepping with the channel, which may have had 0 or more values put onto it

Johnny Hauser20:04:39

That seems like equivocation

ghadi20:04:14

transducible contexts just means a place where a process is exposed to a user's transformation

ghadi20:04:34

channel's process is the "put this thing in the channel" operation

ghadi20:04:07

sequence's process is the act of pulling from a source

ghadi20:04:20

transduce's process is directly supplied by the user

phronmophobic20:04:27

not every step with a channel will put a value on the channel though (eg. if your transducer is filtering as part of its step).

Johnny Hauser20:04:40

Let's say you have a channel, you can transduce it to put a bunch of stuff into it, and then transduce it again afterward and put more stuff in?

👍 4
ghadi20:04:51

sequence & transduce contexts allow a user to supply the source collection

ghadi20:04:00

channel contexts do not allow the user to supply the source

phronmophobic20:04:33

yes, you can use a transducer with a channel, put some stuff on it, then wait, then put more stuff on it later. however, you won’t use the transduce function to do this

ghadi20:04:35

your sentence doesn't parse to me

ghadi20:04:10

someone produces values into a channel, and a consumer sees values subject to transformation

phronmophobic20:04:21

so I wouldn’t say you were “transducing” the channel, but hopefully the idea makes sense

Johnny Hauser20:04:29

Yeah, that's like an observable, except mutable 😕

ghadi20:04:57

I mean, channels are synchronization tools

Johnny Hauser20:04:16

It's an observer/emitter kind of thing, right?

ghadi20:04:18

"mutable" doesn't really capture it -- certainly some of the transformations are stateful

ghadi20:04:39

takeor partition being the classic stateful transducers

ghadi20:04:28

(channels are like channels in golang)

ghadi20:04:56

anyways... I gotta run but let me recap:

Johnny Hauser20:04:26

What I mean is, this is how I would do that sort of thing:

const a = Event.create()
// this is the impure/nasty thing -
// putting values 'in', but this is the only nasty thing
a.occur(1)
a.occur(2)
const b = T.transduce(T.map (etc), T.filter (etc)) (a) // ah, delicious purity

Johnny Hauser20:04:07

occur being like pushing/mutating an array, sure, but that's only at those source events. b is exactly what it says it is and only what it says it is.

Johnny Hauser20:04:27

It would be horror nightmare code if you could do occur on b, in that example

ghadi20:04:49

there exist different transducible contexts, that can act slightly differently, they all have a "step" processes are things with a init,step,complete transducers: process -> process Not all transducible contexts expose the value returned from the 0/1 arities

ghadi20:04:05

all of them will call the arity-1

ghadi20:04:24

but might not reveal that return value (sequence & channels do not)

Johnny Hauser20:04:12

Thanks for chatting! @U050ECB92

ghadi20:04:25

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/TransformerIterator.java ^ a transducible context for Iteration (not directly exposed in Clojure)

noisesmith18:04:24

fundamentally, how would partition-all (for example) do anything that partition doesn't if there's no "completion" (that is, if the source never ends)

noisesmith18:04:47

if a source of data can stop, then a completion is relevant, if it can't, a completion can't be relevant

Ben Grabow20:04:15

I'm not sure this is accurate. Consider Rich's (take-while not-ticking?) example from his talk. Even if you have an infinite stream of bags coming to the baggage handler, the handler can still stop the loading process if it encounters a ticking bag.

Ben Grabow20:04:14

Completion is relevant if either the source of data is finite, or if the transducing process decides the process should terminate early.

noisesmith20:04:26

thanks - that's true, apologies for overgeneralizing

noisesmith18:04:00

also, pedantically, transducers are not tied to collections - that's a big part of the point

OrdoFlammae18:04:24

They aren't? I don't understand. I thought that transducers had to operate over collections. Am I missing something?

noisesmith18:04:43

transducers operate on a data context, see for example core.async

noisesmith18:04:22

thanks to transducers, we were able to deprecate async/map, async/filter etc. which were otherwise re-implementing the basic collection operations, now we can reuse the transduce version of those on channels

noisesmith18:04:58

a channel can take a transducer, if provided the transducer manages the data on that channel, no need for a collection conversion in between

jumar18:04:58

We have an error reporting mechanism which can be used from multiple threads and which writes error details in JSON format to a file. We got some bug reports from customers suggesting that the JSON data stored in the error file are corrupted. My colleague analyzed the issue and come up with this fix to prevent multiple threads overwriting their data:

(ns concurrency
  (:require [ :as io]
            [cheshire.core :as json]
            [taoensso.timbre :as log]))

(let [lk (Object.)]
  (defn- known-errors-from 
    "Read known errors from file - call only with lock taken!"
    [destination]
    (if (.exists (io/as-file destination))
      (with-open [r (io/reader destination)]
        (doall (json/parse-stream r keyword)))
      []))
  
  (defn- write-errors
    "Write errors to file - call only with lock taken"
    [errors destination]
    (with-open [w (io/writer destination)]
      (json/generate-stream errors w)))
  
  (defn- with-lock-persist-to-file
    "Persisting errors to file must be thread safe.
     Use a local lock and make sure everything is:
     - realized before taking the lock,
     - persisted when releasing the lock"
    [path-fn log-entry]
    (let [destination (path-fn "error.json")]
      (locking lk
        (let [known-errors (known-errors-from destination)
              updated-errors (conj known-errors log-entry)]
          (write-errors updated-errors destination)
          updated-errors)))))

(defn persist-to-file
  [{:keys [path-fn] :as _context}
   category
   {:keys [description] :as details}]
  (try
    (with-lock-persist-to-file path-fn {:category category :error (:error details)})
    (catch Throwable t
      (log/error t "Failed to persist the reported error: " description))))
I instinctively didn't like the (let [lk (Object.)] part; the lock is then used via locking ; I didn't find a better way to do it though; my first thought was to at least lock something more meaningful like the destination file but for that to work strings representing the same destination path would have to be the same object (which would require "interning" I think). Notice it's not possible to simply append to the error file (I think) because the format is JSON so you basically need to "insert" new element at proper position (although I believe in this case it's really just inserting it before the final ] marking the end of JSON array) So my question is: Does the solution look reasonable or do you know a better way to do it?

Alex Miller (Clojure team)19:04:41

Some people use an agent to write logs (as agents already force single queue)

💡 4
Alex Miller (Clojure team)19:04:56

The locking / Object. thing is common in Java, no inherent problem with that (I usually put it in a private defonce though rather than use the big let)

jumar19:04:20

Great, thanks! Yeah, I'd define it like ordinary def I think. Agent-based solution looks interesting - I also thought about somehow using a queue for that but it felt a bit too complicated at a first sight.

Alex Miller (Clojure team)19:04:19

Yeah, agents have that all built in

👍 4
lukasz19:04:37

If you're using something like logback, you can define remote agents for ingestion (fluentd etc) or use logback's own configuration to write to a file, setup logrotate etc)

jumar19:04:44

Another minor thing - why the time macro casts to double?

(prn (str "Elapsed time: " (/ (double (- (. System (nanoTime)) start#)) 1000000.0) " msecs"))
It looks to me as if it would be perfectly fine without it (the same Double as a result of division by 1000000.0) :
(prn (str "Elapsed time: " (/ (- (. System (nanoTime)) start#) 1000000.0) " msecs"))

noisesmith20:04:13

I think the precision of double is greater than the precision of OS timing reports, so there's nothing to lose in the cast ?

jumar03:04:51

I mean, both these expressions yields Double so I'm wondering why would one want to cast it at all...

(let [start (System/nanoTime)] 
  (type (/ (double (- (System/nanoTime) start)) 
           1000000.0)))
;;=> java.lang.Double

(let [start (System/nanoTime)]
  (type (/ (- (System/nanoTime) start)
           1000000.0)))
;;=> java.lang.Double

leonoel07:04:09

there's actually a very subtle bug in this code, and it's not related to double casting

jumar14:04:43

@U053XQP4S By "this" you mean the first version (`clojure.core/time`) or the second version? And do you have an idea why the cast is there?

leonoel16:04:59

they both have the same problem

leonoel16:04:33

read the doc for System/nanoTime and -

leonoel16:04:44

I have no idea why the cast, it looks redundant to me as well

jumar04:04:06

I’m not sure what you meant by that. The docs for - looks pretty harmless to me apart from “Does not auto-promote longs, will throw on overflow” which given the range for longs should never happen. And System.nanoTime docs suggest the same approach for computing duration. So the only difference I can see is that clojure’s - throws an overflow error, while Java’s - will silently overflow The fact that System.nanoTime can return negative values (if the origin is in the future) is surprising but it should still work:

(- -10 -20)
;;=> 10
(And I doubt it’s actually ever implemented like that but of course I can never know in what kind of weird environment a java program might run)

leonoel06:04:17

The problematic situation is when the period start is positive and the period end is negative, then - will throw. System/nanoTime explicitly allows that and relies on arithmetic overflow to stay consistent in that case. I've no idea how problematic it is in the wild but I would be pretty worried to have that kind of code in production.

jumar07:04:11

Ah, right - I interpreted javadoc as either "positive" (the common case) or "negative", but never both. Thanks for clarification!

potetm19:04:16

Yeah, that was my instinct as well. There’s been a lot of work around coordinating writes to a file on the JVM. I would start there 😉

jumar19:04:40

But it's not an ordinary log file; it's a custom JSON file which serves the purpose of presenting some errors to end users. But maybe logback offers something in that space I'm not familiar with...

lukasz19:04:58

It helps with coordinating writes to a file - what's in that file is a different matter, as you can configure appenders for different namespaces, with different formatters and output formats. Admittedly - I'm not a logback expert, but I've seen some impressive configs in the wild

potetm19:04:18

I mean, you basically set the pattern to a no-op

potetm19:04:42

and get all the benefits of a logging framework (e.g. optimized writes, possible buffering, etc)

jumar19:04:22

Interesting, I haven't thought about that - I'll have a look at it later to see what can be done 🙂

potetm19:04:55

looks like

%message
would work

potetm19:04:37

so you serialize to JSON before you write to the log

jumar19:04:34

In this particular case at least part of the problem is that you need to read the whole file before you can append the next element

potetm19:04:27

Yeah, other people get around that by parsing line-by-line (not assuming proper JSON)

potetm19:04:46

hang on, searching

👍 4
potetm19:04:31

Yeah, I dunno. It’s kind of an interesting question. Streaming to a file is kind of antithetical to JSON.

potetm19:04:44

e.g. what happens if the proc just dies before it prints the final ]?

potetm19:04:45

At any rate, feel free to ignore. You happened to hit one of my soapbox issues: Just Use Logback! 😄

potetm19:04:07

It sounds like you have interesting constraints here.

potetm19:04:19

(e.g. logback)

kwladyka20:04:50

(println "!!!!"
           (total-sell-rate room-rate)
           (/ 9626 100)
           (class (money/minor-of (total-sell-rate room-rate)))
           (type (money/minor-of (total-sell-rate room-rate))))
print
!!!! #USD 96.26 4813/50 java.lang.Long java.lang.Long
but
(println (/ (money/minor-of (total-sell-rate room-rate)) 100))
CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: [email protected], compiling money is clojurewerkz.money.amounts form clojure library in github
(defn ^long minor-of
  "Returns the amount in minor units as a long"
  [^Money money]
  (.getAmountMinorLong money))
What I miss here? Why I see this exception?

Alex Miller (Clojure team)20:04:25

^long is being resolved to the long function in this location

Alex Miller (Clojure team)20:04:36

(defn minor-of
  "Returns the amount in minor units as a long"
  ^long [^Money money]
  (.getAmountMinorLong money))

kwladyka20:04:05

Not my function, this is library function

kwladyka20:04:15

what exactly happening here? I don’t get it.

Alex Miller (Clojure team)20:04:10

var metadata is evaluated

Alex Miller (Clojure team)20:04:27

here, the hint is evaluating not to a long type hint, but to the long function object

Alex Miller (Clojure team)20:04:47

because the hint is in the wrong place

kwladyka20:04:56

hmm and it works for class type and is able to print this. This is the part which confuse me. Why it works at all.

Alex Miller (Clojure team)20:04:14

hints are hints, and typically ignored if not right

Alex Miller (Clojure team)20:04:50

depending how you're calling it, you may be running into a situation where you're causing it to matter (primitives invoke a different execution path)

Alex Miller (Clojure team)20:04:25

you haven't shared enough context for me to know what you're doing to actually get the error, so I'm just guessing

Alex Miller (Clojure team)20:04:33

but the root cause is that the code is bad

kwladyka20:04:39

ok, I understand the general idea. Not exactly why the issue exactly with this call ,but I understand the point. Thank you.

otwieracz21:04:01

I need to create private API for my full-stack app (clj+cljs). I'd rather avoid creating full-blown REST APIs as it makes no sense here. Ideally I'd like to have something taking advantage of cljc, so somehow backend and frontend uses the same code in many places... I'd like to avoid having "server" and "client" in one project. Do you have any suggestions what to use?

noisesmith21:04:30

what's the advantage of having client and server in separate projects if they can only be used together?

noisesmith21:04:00

of course you could have a client lib (that the server uses to get the js payload to send to the frontend)

noisesmith21:04:19

and bundle the cljc either in the client lib, or in a third lib that both use

noisesmith21:04:31

I don't understand the benefit offered by this split though

otwieracz21:04:45

Oh, you misunderstood me!

otwieracz21:04:01

I am exactly trying to avoid such split within one project.

otwieracz21:04:11

I am looking for something like in Fulcro.

noisesmith21:04:40

clj and cljs will both use cljc files if present on classpath when loading, there's no special setup needed

otwieracz21:04:41

But it's not very useful in my case as I have to rely heavily on react components.

otwieracz21:04:35

Yes, but my problem is: if I have backend and ui in one project, then in this single project I will have to create "explicit" API server and API client.

otwieracz21:04:17

Imagine creating full-blown REST API with documentation, swagger, etc in project only to be used by itself.

noisesmith21:04:14

so you're looking for something that abstracts the communication between the two, and also the definition of the two processes (browser and server)?

otwieracz21:04:29

Something like that.

otwieracz21:04:12

It feels stupid to build REST with all its HTTP sheningans only to talk between clojure and clojurescript.

noisesmith21:04:14

I haven't done cljs work in years, but back then sente worked decently for sharing clojure native data between client and server with an event sourcing communication model

👍 4
noisesmith21:04:57

then each side effectively runs a reduce across the events offered by the other

otwieracz21:04:09

Thanks, I will have a look.

seancorfield22:04:10

sente could be good for simple transfer of data between the UI and backend. You could also just use Ring + Compojure for a very simple HTTP server with a few routes (and pass JSON).

jsa-aerial22:04:22

@slawek098 You should have a look at https://github.com/jsa-aerial/hanasu. It's much simpler than sente and based on what you describe sounds like it may be exactly the sort of thing you want.