OK, so we've just found something really weird with accumulators: retract-fn is never used, though documented in acc/accum.

Because retract-fn does not appear in the Accumulator record.

I'm going to guess this is not how it's supposed to work?

@eraserhd odd that it isn’t on the record definition, not sure I know why there, but it is called when provided (well expected to be)

do you have a custom accumulator you are trying to work with?

or did you try to add this to the existing acc/accum

We are getting ready to build an accumulator with accum, and I was looking for how to extract those functions again so we can test drive it, and I found that retract-fn gets lost.

uhhh, whoa. I had no idea that you could add arbitrary fields to a defrecord with map->X.

you are using clara.rules.accumulators/accum to build an accumulator?

No, doing nothing but research at this point.

yeah, defrecord in clj is often considered more of a perf optimization for a common group of fields you access, they work as regular maps in many cases, and having any number of keys is one of them. the keys not defined on the record are just accessed slower on the impl side

and you can’t reflect for them etc

clara.rules.accumulators/accum supports passing the :retract-fn in

have to be a bit careful on how you do retraction though if I recall correct

if you don’t provide one, Clara will default to rebuilding things minus the facts retracted

:retract-fn is meant for when you know how to more efficiently remove a retracted fact from your reduced structure

If you had an example situation where you think :retract-fn wasn’t used though, that’d be easier to look at and walk through

ohhhh, there may be cases where it isn’t used at all

checking, I’m forgetting things

We have the ability to do more efficient retraction and combining, so we're writing a custom accumulator to get around the thrashing from yesterday. It also allows us to get rid of some intermediate facts that are frequently retracted.

when you use the accumulator

what sort of form does it have

[?acc <- custom-acc :from [<this part>]]

Are you using some joining conditions there like [X (test-something ?y z)], or just [X], or equality based join, like [X (= ?y z)]?

The point is there are 2 variants of accumulate nodes in the rules engine, one for the simpler join cases using = or no joining at all

that one is optimized for those cases because they can be done better

the other variant handles the rest, which are arbitrary other joining criteria

At this point, if I recall all the impl details correctly, providing your own :retract-fn is only a potential perf optimization. If you use the optimized, easier variant of an accumulator, it will be used, otherwise, it is ignored because it can’t be used from within the more generalized case

or really, it can’t be used to help in the more general case, it would only be slower

Does the accumulator maintain the reduced value before being passed to the :convert-return-fn for the purposes of handing it to the :retract-fn?

(assuming both are defined)

To rephrase, should the :retract-fn expect the output of the :reduce-fn or the :convert-return-fn?

@jody > To rephrase, should the :retract-fn expect the output of the :reduce-fn or the :convert-return-fn? It should expect the output of :reduce-fn

it’s part of the “reducing phase”, :convert-return-fn is only done after everything else

Under the current implementation (would have to double check), I don’t think working memory for the accumulate node ever maintains the :convert-return-fn value actually, just the :reduce-fn/`:retract-fn` returned value state

the reason is that the :convert-return-fn value is not considered useful to use for any subsequent changes

the reducing state is

Is it possible for an accumulator to get a net negative, e.g. more retractions than insertions. I'm wondering because maybe this is implied by the "combine" function?

I believe the combine function is just a performance optimizer for the reduce phase. I’d expect a net-negative to represent a bug in truth maintenance. I might be mis-understanding the question though.

I mean, obviously, if you combine all batches, you can't have more retractions than insertions. But depending on how you are batching, you could have one batch with, say three insertions and another batch with two retractions. That would mean that two retractions were applied to the initial value in that case.

Alternately, you could say that Clara enforces that retractions occur in the same batch as insertions. I'm not sure how it could, though I don't understand how this is being called.

I guess one possibility is that Clara only uses this for joins where each reduced value is a selection of attributes, such that insertions and retractions are always sorted into the same batch. That actually makes the most sense.

@eraserhd when retracting facts, with respect to an accumulator, the accumulator is only re-ran on facts that were retracted and were found to have previous matches

so the facts retracted and processed by an accumulator, should only be facts that were processed by the same accumulator before, when they were inserted

if you batched up retracts of facts that were not inserted yet, the accumulator wouldn’t need to retract anything

and it wouldn’t try

facts are considered the “same” via =

but respecting cardinality of them

ie insert 2 = objects, retract 1, you still have 1 left

if the retracted is identical? to 1 of = instances, that’d like be the one removed - but I think that’s more due to perf optimizations than anything

@mikerod OK, that makes sense. Thanks!