hey anyone here has any experience with plumatic/schema?

@igor.larcs Some, a long time ago. Back when it was Prismatic/Schema I think.

was there some safety issues in reading strings from external source into keywords in Clojure? e.g. keywordize json keys.

I believe it used to be that you could fill up memory by pumping unique JSON keys into code that keywordized them all, but I also believe that got fixed at some point -- changing keywords so they get GC when nothing refers to them?

is it a legitmate/idiomatic thing to do to create stateful transducers by wrapping existing transducers with the state, ie

(let [vold (volatile! nil)]
                    (map (fn [x]
                           (let [old @vold]
                             (vreset! vold x)
                             ;;trigger if last value was below but current
                             ;;value is above
                             (if (and old
                                      (< (get old :ema) (get old :sma))
                                      (> (get x :ema) (get x :sma)))
                               (assoc x :trigger true)
                               x)))))

@jjttjj Well, the answer to that is… complicated.

Take, for example, the map-indexed transducer:

([f]
   (fn [rf]
     (let [i (volatile! -1)]
       (fn
         ([] (rf))
         ([result] (rf result))
         ([result input]
          (rf result (f (vswap! i inc) input)))))))

You see, there is some state that it tracks. And, if you know about transducers, there is a specific place to initialize that state.

Now, your example is a little different than that.

You’re wanting to look at a stream of values (e.g. from a channel or a seq), look at two values at a time: previous and current

It just so happens that’s a concept clojure covers: partitioning

Unfortunately, clojure doesn’t ship with a partition+step transducer

Luckily, cgrand has given us: https://github.com/cgrand/xforms

IMO xforms pretty much a necessity when doing significant work w/ transducers.

@potetm Yes, I believe clojure core partition transducers support a "step", i've used partition from https://github.com/cgrand/xforms which does work, as in:

(defn map-cross-over [k1 k2 f]
  (comp (x/partition 2 1 (comp
                          (x/into [])
                          (map (fn [[old new]]
                                 (if (and (< (get old k1) (get old k2))
                                          (> (get new k1) (get new k2)))
                                   (f new)
                                   new)))))))

I guess I'm mainly trying to figure out if there's any reason i'm missing to have the state fully outside of the transduer, rather than between the (fn [rf] ... and the other 3 arity transducing function?

I’m having trouble imagining a scenario where either partitioning or reduction doesn’t cover it.

At any rate: the whole concept of keeping state is antithetical to a mapping operation

ok the first step is setting the triggers with the mapping above, the second step i need to start at each trigger, mark that map as an "entry" (by associng something), count six more X's from that and mark an exit, ignoring any triggers between the entry and exit. I think i definitely need some state in here that's not easily expressed by a more general function?

but i might be missing a something obvious

complete example:

(into []  (comp
           (map-indexed #(assoc %2 :i %1))
           
           (let [vold (volatile! nil)]
             (map (fn [x]
                    (let [old @vold]
                      (vreset! vold x)
                      ;;trigger if last value was below but current
                      ;;value is above
                      (if (and old
                               (< (get old :ema) (get old :sma))
                               (> (get x :ema) (get x :sma)))
                        (assoc x :trigger true)
                        x)))))
           (let [n  5
                 vi (volatile! nil)]
             (map (fn [x]
                    (let [i @vi
                          i (when i (vswap! vi inc))]
                      (cond
                        (and (not i) (:trigger x))
                        (assoc x :enter true :held (vreset! vi 0))

                        (and i (= i n))
                        (do (vreset! vi nil)
                            (assoc x :held i :exit true))

                        (and i (< i n)) (assoc x :held i)
                        :else           x))))))

      [{:ema 67.15, :sma 67.15}
       {:ema 67.14619047619048, :sma 67.13}
       {:ema 67.15321995464853, :sma 67.16}
       {:ema 67.15767519706296, :sma 67.17}
       {:ema 67.13884898781886, :sma 67.128}
       {:ema 67.1180062270742, :sma 67.09333333333333}
       {:ema 67.09057706259095, :sma 67.05571428571429}
       {:ema 67.06766496139181, :sma 67.03}
       {:ema 67.03455401268783, :sma 66.99555555555555}
       {:ema 66.9969774400509, :sma 66.96000000000001}])

Having thought about it for all of 5 seconds: I would start with transduce (or maybe even regular reduce) and shove a lot of that logic into the reducing fn.

In general, reduce is the go-to for tracking state across items.

yeah originally i was using a reducing function, with a sort of context map to build up "state". but now i'm trying to make things work asynchronously and considering just putting everything in transducers so i can just stick it in a core.async chan

so the con is that i have to use mutable state but the pro is i can use the same functionality for old data and live streaming data

Not sure how the operations you just described could be easily parallelized.

but what about if the new data doesn't exist yet, ie i have a chan just waiting for data i'd like to apply this to

Then I would look at a specialized stateful transducer. (Not composing from existing fns like map.)

But remember: stateful transducers in clojure.core cannot be parallelized.

volatile gives consistent reads, but no write guarantees. So you’ll lose information for sure.

gotcha, but is there a reason why say my way with the let wrappers containing state around (map ...) is worse than kind of just copying the source for the transducer version of map and sticking my state inside that? Is it just not idiomatic?

Two reasons: The semantics of maps don’t allow for state. The semantics of transducers are that they’re arbitrarily composable.

So take this example:

(into []
      (comp (map #(do (println %)
                      (inc %)))
            (filter odd?))
      (range 100))

In this case, you get a println for every item, even though the output is only half the items.

Imagine now that, instead of println, you kept some state.

Well it turns out you were tracking state about items that were never emitted by the transducer chain.

And that map operation has no way of knowing what another operation will do.

It’s kinda like the question, “Why can’t I keep state in a mapping operation on a sequence?” Well, if you happen to force the whole sequence, that might just work. But the semantics of lazy evaluation doesn’t guarantee they’ll get run at all. So you’re kinda setting yourself up for confusing bugs.

Gotta run for now but thanks a lot for this input!

Has anyone worked with Onyx? I have some questions.

There's an #onyx channel here @jclavijo -- not sure how active it it. I think there was a pretty active Onyx channel on Gitter, at least a while back.

https://gitter.im/onyx-platform/onyx is one of them. There's a learn-onyx channel on Gitter too. looks like that channel hasn't been active for a few years