Using bidi and yada you can get this, I suspect bidi with Ring would get you most of the way there too. If I understand the question correctly

G’day all. Is the closure.algo.generic really the only standard-ish library fmap? That’s astounding! Is there a transducer for it? I like the look of @gerritjvv but might be a bit much stuff for all I need (at the moment)... mind you I’ll probably use ap and lift soon enough.

@delitescere by fmap you mean monads? Then yes, monad usage is quite rare in Clojure except when no other approach will work (for example in test.check).

In general Clojure tends to drive abstractions with data, and data interpreters/compilers instead of directly with functional composition.

hey all. I’ve got a function that provides me with a list of all the possible combinations of values from two lists:

(defn all-combinations
  "Return all possible combinations of values from as and bs collections."
  [as bs] (apply concat (for [a as] (for [b bs] [a b]))))

I’d like to do this to a list of lists rather than just a pair of lists.

Has anyone got any hints for doing so neatly?

@triss Perhaps https://github.com/clojure/math.combinatorics

Thanks @curlyfry cartesian-product looks just the job

Plz help cteate and use data base. Telegram Bot. Morse dep

🦌

🧐

😳

@djabbat Are you the beginning of the singularity? Morse dep, bro!

Hello, wanted to hear your opinions on honeysql and yesql. I want to build a small web app on top of a RDBMS and I wonder how should I handle the queries. Any thoughts or past experience?

I prefer #hugsql

you can also ask in #sql

i have a long list of let bindings where a conditional could branch an error at any time options: 1. I have many-nested (let [...] (cond ... (let [...] (cond ... 2. would it be idiomatic to build the let "context" as a record getting passed down a (cond->? 3. something else?

actually, step 2 doesn't seem to work like I want

basically, I'd just like a long list of let bindings, that depend on things before them, but if I have a conditional branch during that process, I can "short circuit" the whole thing

Is there a way for a deftype to extend Atom, or do I need to have an atom in its parameters and implement the lang.IAtom interface?

I assume I should do that instead of manually using mutable fields? (using set!?)

@ghopper, there are some cases where manually setting mutable fields might be more straightforward (eg. java interop)

why won’t using a regular atom work for your usecase?

@josh.freckleton let-later from useful works for that kind of thing https://github.com/amalloy/useful/blob/develop/src/flatland/useful/utils.clj#L224

@smith.adriane I'm using deftype, because there are a number of other protocols to be implemented and modifications to the normal atom functions. I didn't know if there was a way for it to extend an atom, since its base data is just an atom.

the typical approach is to extend all the interfaces / protocols atom implements, instead of concrete inheritance from atom itself

I usually implement IDeref when I want a thing that acts like an atom

=> (supers (class (atom nil)))
#{clojure.lang.IRef java.lang.Object clojure.lang.IMeta clojure.lang.AReference clojure.lang.ARef clojure.lang.IDeref clojure.lang.IReference clojure.lang.IAtom}

it depends what you need - IDeref doesn’t help with eg. swap! or add-watch

isn’t swap! specifically for atoms and not any other reference type?

it depends what someone means by “acting like an atom” here - it’s fully in our power to make something that is a drop in replacement for an atom

right. I ask mostly because most of the time when I’ve seen someone try to implement IAtom, they don’t plan to also implement all of the same semantics. if you’re not implementing all of the same semantics, I think it probably makes more sense to just use another function to manipulate your reference type just like the other clojure reference types each have their own functions, (eg. vswap!, send, ref-set, etc)

Thanks, guys, this helps.

@ghopper going back to your initial question, proxy can extend an atom, or you can implement all the right protocols / interfaces to replicate an atom (maybe even using a closed over atom) with deftype, reify, defrecord, etc.

but it really depends on what you are trying to do, which behaviors you need, what you exactly you wanted from IAtom

because extending a clojure built in tends to be a bit tedious because of how the interfaces / protocols are factored

Closed over meaning the deftype includes an atom as a parameter?

as a parameter, or just creating a reify inside a let block that defines the atom - there’s a few ways to “capture” an atom for internal usage

:thumbsup:

@noisesmith When implementing IAtom's swap functions, is there a way to reduce duplication of swapping code? Do I need the 4 different arrity versions?

since you can’t control the caller, and the interface supports all those arities, I think the only safe thing is to define them all (even if they all end up calling the same variadic function)

clojure will let you skip arities or even methods, but it will lead to a runtime error if they get called

Is there a performance benefit of not colling the other arities of the type (duplicating the function). (e.g. should I call deref on the this argument, or does it make sense to reimplement the logic of the deref inline?)

Oh, so an external variable arity function could be called by all of them. That's probably the cleanest.

that’s a more general software design question isn’t it? sometimes it’s worth it to avoid the abstraction for performance / resource reasons, sometimes it makes sense to abstract and avoid the repetition

right, yeah, that’s probably cleanest

Yeah, I suppose it is. Thanks!

@noisesmith You mind giving me your opinion on this?:

(defn- swap-g-counter [this f & args]
  (let [oldv @this
        newv (apply f oldv args)]
    (assert (number? newv) "Swap! G-Counter function must return a Number")
    (swap! (.p this) update (.n this)  + (- newv oldv))
    @this))

(defn- compare-and-set-g-counter [this oldv newv]
  (let [value @this]
    (if (not= oldv value)
      false
      (do
        (swap! this + (- newv value))
        true))))

(deftype G-Counter [p n]
  clojure.lang.IDeref
  (deref [this]
    (reduce + (vals @p)))

  clojure.lang.IAtom
  (swap [this f]
    (swap-g-counter this f))
  (swap [this f a]
    (swap-g-counter this f a))
  (swap [this f a b]
    (swap-g-counter this f b))
  (swap [this f a b args]
    (apply swap-g-counter this f b args))
  (compareAndSet [this oldv newv]
    (compare-and-set-g-counter this oldv newv))
  (reset [this newv]
    (swap! this + (- newv @this))
    newv))

(defn gcounter []
  (new G-Counter (atom {:id 0}) :id))

using @ and swap! in the same function is usually a race condition

On the reset function? So the @this should be bound before swap!?

you probably want compare-and-set! which allows more complex logic

@ghopper at a quick skim, every single function that uses @ and swap! in the same body is a likely race condition

I mean, you even have compare-and-set in the name, so implementing via compare-and-set! is much more likely to be correct

I'm not following how it would be a race condition.

and yes, looking closer, those are definite race conditions in both of the first two functions

Right, I didn't think about how the normal compare-and-set! would use my custom reset logic. I'll switch to that.

@ghopper (+ newv oldv) - this is a race condition, oldv came from a deref, the swap! itself will retry, but your function won’t so it will use stale data in a race

(not= oldv value) - the truth value of this can change before the swap! runs

@noisesmith Oh! Thanks for clerifying that. I'm following now.

I'd completely neglector the retrying functionality of atoms.

@noisesmith Oh, I can't use the normal compare-and-set!, because I'm not wanting the atom reset to the newval; only a subset of the atom can ever be set.

compare-and-set! uses a java interop .compareAndSet

you can’t use it to implement your full functionality, but to preserve expected atom semantics it needs to be your building block

It is? compareAndSet

Between swap and reset

you can’t just mix @ and swap! of the same atom and expect correctness of any sort

but you can get it by using compare-and-set, with your own retry condition, etc.

Oh, I think I'm following. I'll try implementing that.

the idea is to use compare-and-set to wrap both the read and the modification, and then recur as your retry if the operation did not succeed

is there a reason you want your reference type to use the atom functions instead of creating new functions to interact with your reference type?

Mostly as a proof of concept / out of curiosity at the moment.

it seems like instead of putting counter related logic inside of swap! and deref, it might make more sense to put that logic in another function that takes an atom as an argument

Well my thought was to abstract it, so users can completely ignore the implementation details of the crdt. 🤷 I think I'll do it both ways and see how it feels.

I would actually probably have the functions work on immutable data

and then let the consumers of the counter decide if they want to use an atom or other reference type

Yeah, that's likely how I'd do it.

How can I get two swap!s to run at the same time to test retrying?

Oh, it was just too fast... A Thread/sleep did the trick.

yeah - adding sleeps (maybe randomized) is a quick way to get race conditions to happen

Randomized is a good idea

combining randomized timeouts / sleeps with looped calls is a great way to stress test things that might misbehave

(in terms of race conditions that is)

@noisesmith So I'm still not quite seeing the purpose of compare-and-set!. Is there a reason I shouldn't just implement reset! inside it with the new value if the old value is the expected value?

the point of compare-and-set! is that atoms don’t lock, and you need to be able to retry if the value changes before your update completes

Which reset! would do, right?

no

how would it even know?

Oh, unless you have some sort of abstract logic based on the value?

I think I'm starting to understand.

reset! never retries, and the condition you would need to detect in order to retry is outside the scope of the reset! call

which is why you have compare-and-set!

@noisesmith Mind giving this another look?

(defn- project-g-counter [p]
  (reduce + (vals p)))

(defn- swap-g-counter [p n f & args]
  (let [oldv (project-g-counter p)
        newv (apply f oldv args)]
    (assert (number? newv) "Swap! G-Counter function must return a Number")
    (assert (>= newv oldv) "G-Counter is grow only")
    (update p n + (- newv oldv))))

(deftype G-Counter [p n]
  clojure.lang.IDeref
  (deref [this]
    (project-g-counter @p))

  clojure.lang.IAtom
  (swap [this f]
    (swap! p swap-g-counter n f)
    @this)
  (swap [this f a]
    (swap! p swap-g-counter n f a)
    @this)
  (swap [this f a b]
    (swap! p swap-g-counter n f a b)
    @this)
  (swap [this f a b args]
    (apply swap! p swap-g-counter n f a b args)
    @this)
  (compareAndSet [this oldv newv]
    (loop []
      (let [pval @p]
        (if (not= oldv (project-g-counter pval))
          false
          (or (compare-and-set! p pval (swap-g-counter pval n + (- newv oldv)))
              (recur))))))
  (reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    (swap! this (partial + (- newv @this)))
    newv))

(defn gcounter []
  (new G-Counter (atom {:id 0}) :id))

that is all race conditions

@ghopper I can see you are trying but @hiredman is right - all the code that uses deref and swap! in the same block of code is setting up a race of some sort

swap returns the value swapped in, derefing after the swap could return anything

what I meant about compare-and-set! is that to make this code correct the whole thing needs to be inside a compare-and-set! loop which can either let the caller know if it failed or retry if there was any modification outside

Oh, right, I need to use project-g-counter on the return value of swap! instead. Is there something somewhere else?

the various places you call swap! then return @this for example

I realize that problem now:

clojure.lang.IAtom
  (swap [this f]
    (project-g-counter (swap! p swap-g-counter n f)))
  (swap [this f a]
    (project-g-counter (swap! p swap-g-counter n f a)))
  (swap [this f a b]
    (project-g-counter (swap! p swap-g-counter n f a b)))
  (swap [this f a b args]
    (project-g-counter (apply swap! p swap-g-counter n f a b args)))

someone posted and example of using compare-and-set! recently, https://github.com/plumatic/plumbing/blob/master/src/plumbing/core.cljx#L360

Is there something wrong with the implementation of compare-and-set! above?

you don’t seem to understand why I was mentioning compare-and-set! - in order to check an arbitrary condition and retry if the value is modified and also return an arbitrary value other than the thing you just modified, compare-and-set! is the ideal tool

it lets you keep the logic you wanted without having all the race conditions (as long as you use it properly)

Oh, you're saying compare-and-set! should be used not only for the implementation of compareAndSet, but also for the other implementations too?

right - it lets you coordinate in a more flexible way than swap!

I apologize for not reading that more carefully. Makes sense. 🙂

I didn't even know compare-and-set! was a thing until I went to implement IAtom.

@noisesmith Like this?

(defn- project-g-counter [p]
  (reduce + (vals p)))

;; (defn- swap-g-counter [p n f & args]
;;   (let [oldv (project-g-counter p)
;;         newv (apply f oldv args)]
;;     (assert (number? newv) "Swap! G-Counter function must return a Number")
;;     (assert (>= newv oldv) "G-Counter is grow only")
;;     (update p n + (- newv oldv))))

(defn- swap-g-counter [this f & args]
  (let [p (.p this)
        n (.n this)]
    (loop []
      (let [oldval @this
            newval (apply f oldval args)
            pval @p
            nval (update pval n + (- newval oldval))]
        (if (compare-and-set! p pval nval)
          newval
          (recur))))))

(deftype G-Counter [p n]
  clojure.lang.IDeref
  (deref [this]
    (reduce + (vals p))
    ;; (project-g-counter @p)
    )

  clojure.lang.IAtom
  (swap [this f]
    (swap-g-counter this f)
    ;; (project-g-counter (swap! p swap-g-counter n f))
    )
  (swap [this f a]
    (swap-g-counter this f a)
    ;; (project-g-counter (swap! p swap-g-counter n f a))
    )
  (swap [this f a b]
    (swap-g-counter this f a b)
    ;; (project-g-counter (swap! p swap-g-counter n f a b))
    )
  (swap [this f a b args]
    (apply swap-g-counter this f a b args)
    ;; (project-g-counter (apply swap! p swap-g-counter n f a b args))
    )
  (compareAndSet [this oldv newv]
    (loop []
      (let [pval @p]
        (if (not= oldv (project-g-counter pval))
          false
          (or (compare-and-set! p pval (swap-g-counter pval n + (- newv oldv)))
              (recur))))))
  (reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    (swap! this (partial + (- newv @this)))
    newv))

(defn g-counter []
  (new G-Counter (atom {:id 0}) :id))

I don’t get what’s happening in reset but the rest looks better at a first glance

Rreset needs to take the value the user wants the counter to be and find the difference between the that and the current value of the counter, then it adds that to the value in the map the user is allowed to modify.

(reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    (swap! this #(+ % (- newv @this)))
    newv)

Can a swap function deref the atom it's swapping?

It's weird, because derefing doesn't return the same value as the swap function accepts.

I don’t know if that’s correct yet, but fyi you can always replace (swap! a #(f % b)) with (swap! a f b)

Oh yeah, good point. :thumbsup:

Much cleaner

Actually, swap does accept the current counter value. There's no need for the @this

a deref of the same atom inside the function arg to swap! is safe because if the atom changes you will retry the swap!

but yes, that too

That's what I thought, which is why I hadn't thought about it.

(reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    ;; (swap! this + (- newv @this))
    (swap! this #(+ % (- newv %)))
    newv)

... I'm not sure why I'm doing this. It just needs to return the newv, and the swap! will handle the difference stuff.

(reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    (swap! this (constantly newv))
    newv)

won’t the swap! call already return newv?

Uh, yeah, it will.

Well, that shrunk a lot...

It feels a little weird to use swap for reset, though I suppose that allows me to keep the difference logic in the same place.

I'm pretty happy with that. For anyone interested:

(defn- swap-g-counter [this f & args]
  (let [p (.p this)
        n (.n this)]
    (loop []
      (let [oldval @this
            newval (apply f oldval args)
            pval @p
            nval (update pval n + (- newval oldval))]
        (assert (number? newval) "Swap! G-Counter function must return a Number")
        (assert (>= newval oldval) "G-Counter is grow only")
        (if (compare-and-set! p pval nval)
          newval
          (recur))))))

(deftype G-Counter [p n]
  clojure.lang.IDeref
  (deref [this]
    (reduce + (vals p)))

  clojure.lang.IAtom
  (swap [this f]
    (swap-g-counter this f))
  (swap [this f a]
    (swap-g-counter this f a))
  (swap [this f a b]
    (swap-g-counter this f a b))
  (swap [this f a b args]
    (apply swap-g-counter this f a b args))
  (compareAndSet [this oldv newv]
    (loop []
      (let [pval @p]
        (if (not= oldv (project-g-counter pval))
          false
          (or (compare-and-set! p pval (swap-g-counter pval n + (- newv oldv)))
              (recur))))))
  (reset [this newv]
    (assert (number? newv) "Reset! G-Counter value must be a Number")
    (swap! this (constantly newv))))

(defn g-counter []
  (new G-Counter (atom {:id 0}) :id))

Thanks, @noisesmith, @smith.adriane, @hiredman! I appreciate the help.

how does the deref method work? isn’t p an atom?

seems like calling vals on p in the deref method would throw an exception

Yeah, that's a typo. I've fixed it:

clojure.lang.IDeref
  (deref [this]
    (reduce + (vals @p)))

I'm not sure when that got changed.

i might be reading it wrong, but seems like there’s still a race condition in your swap-g-counter function

the compare-and-set! will fail if p no longer olds pval

that’s the point of using that function

p is dereferenced twice

inside a loop that retries if the compare-and-set! fails

and the compare-and-set! will fail if p changes

so if it changes between the 1st and 2nd dereference, but not between the 2nd dereference and compare-and-set!

it would be inconsistent

since the new value is derived from both the 1st and 2nd dereference

and the first and second dereference might have different values

no- if it changes between binding oldval and compare-and-set! the compare-and-set will fail

oldval isn’t passed into compare-and-set!, pval is

oh, tricky

yeah, this is weird

Yeah, I need to get a better naming convention.

i think the code would be simplified a lot by having functions that work on immutable counter data

I think you need nested compare-and-set! calls to ensure consistency on both dereferences?

and using a plain ol’ atom to hold the data

if you really wanted it to work with this interface, you could have pval use the data from the oldval dereference

Yeah, I think I'm going to need to switch to immutable functions anyways for my merging logic. At the moment I'd need to have a function which takes one of these and a p (from somewhere else) and merge them. I'd also need another function to get the p out to send elsewhere.

the nice thing about atoms is that they just encapsulate identity. I don’t think you want to mix the identity and state pieces into one thing

pval can't use oldval, because oldval is just an integer.

Yeah, I think you're right.

right, I guess you would have to have oldval derive from pval

Good call, that's something that it could do.

Though I'd need to bring back project-g-counter for the reduction function.

yea

or as long as you dereference pval before you do the dereference for oldval, it would work

actually, nvmd

i take that back

I think you have to do it with a single dereference

other wise it could change in between dereferences and then change back

(defn- swap-g-counter [this f & args]
  (let [p (.p this)
        n (.n this)]
    (loop []
      (let [pval @p
            oldval (reduce + (vals pval))
            newval (apply f oldval args)
            nval (update pval n + (- newval oldval))]
        (assert (number? newval) "Swap! G-Counter function must return a Number")
        (assert (>= newval oldval) "G-Counter is grow only")
        (if (compare-and-set! p pval nval)
          newval
          (recur))))))

Yeah, I'm pretty sure it's needed.

:thumbsup:

I'm using wrap-json-response, but I'd like to opt out and return Content-Type: text/html sometimes, is there an easy way out?

(oh, I'm working within Compojure-API)

so you have one endpoint, that would sometimes return json and sometimes return html?

@noisesmith correct, based on their request header of Content-Type, so they can have json or html

I think I need to just pull out this route from the main app, and add it back in separate without the offending middleware

is there a less awkward solution tho, perhaps?

there’s no rule saying you can’t generate json inside your handler (or have a conditional in your handler that either dispatches to an html function or a json function)

(the ring function content-type was a no go)

those functions are just modifying data in a hash-map, if using them is at all awkward just skip them

oh no, I mean, I'd like wrap-json-X most of the time, but on one route, reset it. I think what I'm setting is getting over-written though by the middleware

@josh.freckleton I don’t know if that’s wrong, but when I need to return a different content-type I just change manually and return something like this: {:status 200 :headers {“content-type” “text/html”} :body {:message (:message (any-function))}}

yeah, if you want to conditionally make json you don’t want a middleware that always returns json

if you do that, the content-type is changed?

that middleware isn’t doing as much as you think it is, just skip it