Fork me on GitHub
#clara
<
2017-04-21
>
devn19:04:55

@wparker we built something hackish to build graphviz, and for the entire session the graph was way too big to be useful

devn19:04:20

then there was an attempt at using cytoscape.js to make it traversable

devn19:04:26

so you could see a few nodes away from YOU ARE HERE

devn19:04:38

what i found myself wanting was a way to define views over the graph, like, i may not care about all the edges and nodes between Node A and Node B, so I'd like to define a view that shows me Node A -> Node B without all of the intermediate steps

devn20:04:13

I imagine having something like: (def views {:everything [FactA FactB ...], :simple [FactA FactC FactF]})

devn20:04:14

:simple automatically collapses nodes and edges along the path of A->C->F

devn20:04:38

@wparker another thing that I think would be nice would be to generate a static graph

devn20:04:33

of all the facts in the project, regardless of whether they are ever used

devn20:04:14

but maybe your PR achieves this?

devn20:04:11

what we built relied on the data returned from inspect, which IIRC required us to fire-rules

devn20:04:07

im interested in building the graph of all the relationships, not just the relationships that survive after running the rules

wparker20:04:53

The graph would include all logical insertions regardless of whether they were used for anything, but unconditional/external insertions wouldn’t be if they weren’t used for anything

wparker20:04:15

Since Clara allows them to be garbage-collected if they aren’t used

wparker20:04:00

My general line of thinking here was to build a graph, and then

wparker20:04:43

provide functions to retrieve particular subsets of the graph, potentially modify it, etc.

wparker20:04:43

and then look at how to display those views. Basically have a data-oriented approach to the problem that different views could layer on top of

zylox20:04:49

you can always reduce the data in a graph, thats the wonder of them

wparker20:04:29

In particular, in our case we’d probably want to combine fact graphs with some other indicators that our specific to our internal frameworks

wparker20:04:50

so I’m looking for composability

wparker20:04:55

I’m not clear on what :simple would be - do you mean that if you had A -> B -> C you’d have a direct edge of A -> C?

wparker20:04:56

Also, @devn I’d be curious about why cytoscape.js versus any other graph visualization options you tried or considered - the visualization layer isn’t something I know much about yet

devn21:04:05

@wparker I did not directly participate in the decision to use cytoscape, but as I recall it was chosen partly for performance, and partly because it made the traversal of subgraphs relatively easy.

devn21:04:32

@wparker RE: direct edge on A->C, yes, that was what I was thinking

devn21:04:27

or perhaps you have A->[summary collection of intermediate nodes]->C

devn21:04:54

basically anything to make it easy to skip the display of 382910 nodes and edges that don't actually tell you much that you care to know about getting from A to C

devn21:04:37

i think it's often the case that the fact that you got to C suggests everything you'd need to understand about B

devn21:04:34

for instance, there is a big fan out and fan in between A and C, like A->[B1, B2, B3, B4, B5, BN]->C If C is a specific fact like: MoreThanThreeBs, then who really cares what those Bs were? all I really care about is that by virtue of A, I have the possibility of many Bs, and all I care to know is if I get to state C from A

devn21:04:55

We have a lot of scenarios where we need to look at hundreds of thousands of facts on our way to a state, but in the end all we really care about is that given a Lab, we determine whether it is normal or abnormal according to a massive set of facts, and then produce an AbnormalLabResult or NormalLabResult. Labl->AbnormalLab/NormalLab is how I'd prefer to view it in a UI rather than render the 500k+ nodes and corresponding edges in between

devn21:04:39

being able to express something like: "Provided #{A}, render direct edges to #{C, D} through nodes of type #{B}" would be nice

devn21:04:43

Stop me if what I'm saying doesn't make any sense

devn21:04:20

I suppose I imagine that for any collection of rules and facts, there are layers to the view. For instance, I don't care about seeing the source of a rule by default, that's another layer. Similarly, I don't care which specific constraints in that rule were met, unless I do.

wparker21:04:12

I can see where that would be useful for some rules patterns, yes. I’m thinking that it probably wouldn’t be too hard to transform the graph down to that level though?

devn21:04:29

@wparker no, sure, that's doable

devn21:04:51

I'm just listing stuff that I don't think would be totally out of bounds for clara.tools.*

wparker21:04:54

By the way, I think having transformation functions in the fact-graph namespace could be useful, I just haven’t gotten to that point yet

devn21:04:15

I totally understand for instance, that if you have the raw data, you can do whatever you please, but it would be really nice to not leave that as an exercise for the reader, and instead have a couple of basic built-in tools for specifying what you want to look at

wparker21:04:24

or other namespace as appropriate as you say “clara.tools.*”

devn21:04:31

Like, a function which dumps graphviz, even if there is no dependency within clara to actually render it, seems like it might be useful

devn21:04:42

and then adding on to that: Dumps graphviz where you specify the starting node (the type of fact), and how many hops should be rendered

devn21:04:22

or: Dumps graphviz, and lets you set the color for nodes based on their fact type

wparker21:04:36

For that, I’m thinking you’d do something like (-> original-graph remove-intermediate-steps render-to-graphviz)

devn21:04:12

sure, yeah

devn21:04:08

It would be interesting to generate some anonymized graphs for all of our projects just to get a sense for what transformation functions would be generally useful

devn21:04:08

I know there are Reasons™ to not include the transformation kitchen sink or unnecessary deps, but I feel like a simple toolkit for rules engine graph generation would be super helpful to have, maybe even as part of core

devn21:04:00

as opposed to an external lib

wparker21:04:46

Yeah, there is often a tradeoff of not tying things together too much versus not providing enough functionality especially if you get into needing other libraries as dependencies. Adding generic data transforms doesn’t seem like it adds too much burden though since it is very easily ignorable if you don’t need them

wparker21:04:40

I’m guessing the types of transforms needed will depend heavily on the domain. For example, in our case, I think a lot of it would revolve around taking a given fact and showing everything either upstream or downstream from that fact, and potentially “backtracking” a bit from some joins on that path, but not the rest of the rules session

wparker21:04:01

If you have a fact A that flows through rules B->C->D->E->F and you don’t get the expected F it would be nice to be able to easily see where it “stopped”

wparker21:04:42

or in the case of accumulation how it interacted with other facts in that chain

wparker22:04:00

The first step is to make the full data set available for manipulation though

wparker22:04:25

You could get most of it from inspect already but not when you go through accumulators

wparker22:04:37

which we use a lot of