Fork me on GitHub
#clara
<
2018-03-01
>
wdullaer08:03:46

So I fired up visualVM

wdullaer08:03:16

and it seems to be spending all the time in clara.rules.engine$retract_accumulated.invoke()

wdullaer08:03:04

which lines up with the theory that implicit retractions are causing this

wdullaer08:03:02

I can share the snapshot if that would be interesting for people

wdullaer08:03:20

I'll spend some time trimming the example down to a bare bones app I can put in a gist

wdullaer12:03:59

when making this I noticed that the 2 million case is not exactly the same as 2x 1 million, because of how the attributes are randomly generated

wdullaer12:03:17

2x 1 million will generate much more consent for a given personId

wdullaer12:03:27

(giving the accumulator more work)

wdullaer12:03:45

2x 1 million goes fast enough if I make sure they generate distinct sets

wdullaer12:03:18

however 1x 2 million tweaked to have the same overlap in consent as 2x 1 million still goes an order of magnitude faster

wdullaer12:03:13

I know clara doesn't deduplicate facts, but I think in this case that would prevent me from running into this scenario

mikerod14:03:08

@wdullaer thanks for the details

mikerod14:03:58

I will take a look. It may be about an hour or so though. Doing a few other things at the moment.

wdullaer14:03:09

no worries, take all the time you need

wdullaer14:03:21

I can supply the vm snapshot as well (just need to find a quick place to upload it)

mikerod14:03:46

Yeah, if you have one you can share, that’d be good to take a glance at.

mikerod14:03:50

How to share hmm

wdullaer14:03:06

I'll put it into dropbox, just a sec

mikerod16:03:28

@wdullaer I’ve only got to look at this briefly so far today

mikerod16:03:29

It seems like these collections created via acc/distinct would be fairly large. Also, there are many of them after the first 1mil facts. There’d be a big acc/distinct collection associated with each personId + purpose grouped Consent facts. When the next 1mil facts come in, I believe they will contribute to many (or perhaps even all) of those same big collections.

mikerod16:03:14

The work from the first 1 mil facts is stored in working memory. Clara builds the new collections on the 2nd wave of inserts. It then must remove the old accumulated work it did from working memory - in order to replace it with the updates.

mikerod16:03:51

I think the fact that there are many of these large collections in working memory is making it get really expensive to remove them all from working memory. Clara tries to be efficient in finding and removing things in memory. I think huge collections may be a pitfall.

mikerod16:03:19

I’d like to look a little deeper at it, but haven’t been able to yet. I’m not sure what a reasonable workaround may be, other than inserting everything in one batch (as you’ve found).

mikerod16:03:37

It’s an interesting case to explore some more for sure.

wdullaer17:03:42

thanks for the input

wdullaer17:03:56

I can see how this is a worst case scenario

wdullaer17:03:03

we probably won’t see this amount of massive updates in a real life scenario, and if it does (say a big data migration), we can always consider starting from scratch

wdullaer17:03:49

it was a bit surprising that updating these structures is a lot more expensive than creating them

wdullaer17:03:19

if I can help in any way here, just let me know

wdullaer17:03:34

I have a few other benchmarks I’m going to run over the coming days as well

mikerod17:03:27

I need to look deeper at what is happening to have a stronger sense of what could be done

wparker19:03:39

I’ve logged https://github.com/cerner/clara-rules/issues/385 for this @mikerod @wdullaer , I’d suggest that we post findings there when we have them

wparker19:03:10

From the snapshot it looks like the memory is the bottleneck, which isn’t too surprising. From my first glance over it I suspect there are some performance optimizations we can make for cases like this, but it may be a bit before I have time to write my thoughts down in a sufficiently articulate form. The benchmark and snapshot are really helpful and make diagnosing things like this much easier - thanks for that. It is useful to have benchmarks of realistic rule patterns that stress-test Clara.