Fork me on GitHub
#off-topic
<
2020-01-22
>
mloughlin09:01:40

Do clojurians prefer wrapping interop calls in clojure functions and giving them their own namespace, or calling interop methods in-situ?

p-himik09:01:10

If it's something one-time, in-situ. If it's something that's used in multiple places, a separate function.

lispyclouds09:01:41

also in cases like the java method takes varags and passing it via clojure is a bit unclean. https://stackoverflow.com/a/11702298

mloughlin09:01:44

I've run into the varargs/array stuff working with the Java nio lib. It sticks out like a sore thumb from the surrounding Clojure (into-array ALL/THOSE_ENUMS)

dharrigan10:01:32

I have interop with Elasticsearch HighLevelRestClient - all the functions live in their own namespace (ns foo.bar.elasticsearch)

dharrigan10:01:12

Not that I choose to isolate away the interop, it just felt natural to put all that stuff in its own namespace

vemv11:01:13

> Do clojurians prefer wrapping interop calls in clojure functions and giving them their own namespace, or calling interop methods in-situ? The peril is when the wrapper loses nuances, introduces opinionated defaults, decreases performance etc. If those are addressed a wrapper isn't bad

mokr18:01:09

Hi! I’m toying with an idea and could really need some pointers in the right direction as my googling hasn’t provided anything useful. The general idea is to simulate how one or more nodes failing in a network (think servers, VMs, services, ..) affects other nodes and services. My assumption is that such a network can be described as a directed acyclic graph with multiple edges in and out, but I’m no graph theory expert. All I find is typically related to finding friends of friends or the quickest way to the airport. To be useful it would need to have some way to capture redundancy mechanisms as well. This must have been solved countless times, and it might be way too complicated to achieve for my real use case, but I would at least enjoy fiddling with a simplified version. Any ideas regarding where to look and what to look for?

hiredman18:01:35

most approaches to that kind of thing aren't network centric like that, you don't describe a graph of nodes, you describe nodes and the messages they exchange and the network graph is implicit in the exchange of messages

hiredman18:01:09

jepsen is a great tool for testing properties of real systems, for more formal stuff there is tla+, and a new tool might be something like alloy

mokr18:01:17

@deleted-user Thanks, I’ll need to investigate that. After a quick look it wasn’t immediately obvious what that solves/how it works.

andy.fingerhut18:01:54

@mokr Are you interested in answering questions like "how many network links need to break before certain pairs of hosts can no longer communicate"?

andy.fingerhut18:01:30

Or it sounds more like "which software processes stop operating correctly when other software processes they depend upon quit responding?"

mokr18:01:59

@hiredman Thanks, I’ll have a look. Describing the messages is not an option in this case as this is a “huge” pre-existing network, not something that is feasible to document message by message.

mokr18:01:11

@andy.fingerhut I work in Telecom, so this is ultimately about seeing how alarms on a system might affect end user services.

hiredman18:01:25

what is the difference in volume between saying there is a link between A and B and saying A can send messages to B?

hiredman18:01:07

in a formal framework you'll like say something like for all messages M the structure of the message is Y and if the sender is A the receiver can be B sort of things

mokr18:01:31

If I don’t have to detail what is sent, then I guess there is not much of a difference.

andy.fingerhut19:01:06

Are you hoping to be able to answer questions like "exactly how does service A misbehave when services B and D fail?"

andy.fingerhut19:01:25

Or are you only looking to model coarser relationships like "for service A to be fully operational, service B and D must be fully operational. Therefore, tell me the the sets of nodes that, if they die, then service A is no longer fully operational?"

mokr19:01:07

The latter.

andy.fingerhut19:01:20

The first question requires knowing details about how service A, B, and D are implemented, or at least their specified behavior. The second is much less detailed.

mokr19:01:20

I know I need to start out very, very simple.

hiredman19:01:43

I'll sometimes just model systems as core.async processes(basically just write a smaller simpler version of whatever) communicating over channels, which isn't formal, and can't use logic to deduce properties, but it can be useful to feel things out

mokr19:01:32

@hiredman Hmm, I haven’t thought of using core.async. That might actually be a good start. Thanks.

andy.fingerhut19:01:12

And you want to explicitly model some kinds of redundancy like "For A to be fully operational, at least one of B1, B2, or B3 must be fully operational, AND at least one of D1, D2, D3, or D4 must be fully operational" ?

mokr19:01:37

I think my start will be quite binary. If there is an active alarm for a node, I say it is down. Any nodes depending on it will be down as well.

mokr19:01:18

Later, I would like to take redundancy into account. Node C is down if both node A and B is down.

mokr19:01:30

If one of them is up, then C is up as well.

andy.fingerhut19:01:56

Do you need a combination of AND and OR in the statements that determine if a node is up?

hiredman19:01:21

(sounds like maybe a sat solver)

mokr19:01:11

@andy.fingerhut Maybe both. Depending on how I end up modelling the graph. The problem is that a Telco network has a lot of complexity. Way more than I can solve, but if I can find some simplification that at least can indicate what might be affected based on an alarm, then I have something.

mokr19:01:06

What I’m describing is starting to sound like something similar to “game of life”, doesn’t it?

andy.fingerhut19:01:13

If you have a bunch of statements like this one: "For A to be fully operational, at least one of B1, B2, or B3 must be fully operational (i.e. not giving any alarms), AND at least one of D1, D2, D3, or D4 must be fully operational (i.e. not giving any alarms)" and then you have a collection of statements like "B2 is giving alarms". and "D4 is giving alarms", you can use boolean variables like B1=true to mean B1 is not giving alarms, and B2=false to mean B2 is giving alarms.

andy.fingerhut19:01:13

Then if you evaluate the formula "A=(B1 or B2 or B3) and (D1 or D2 or D3 or D4)", you get true, meaning A is fully operational (because of the redundancy in this case -- assuming it is correct that B1 being in a not-giving-alarms state is good enough for A, even though B2 is giving alarms)

mokr19:01:07

How can something like that be generated for 10k/100k nodes? I aware that I keep introducing constraints here, but I tried to keep the intro short. Sorry.

hiredman19:01:48

https://en.m.wikipedia.org/wiki/Boolean_satisfiability_problem "Nevertheless, as of 2007, heuristic SAT-algorithms are able to solve problem instances involving tens of thousands of variables and formulas consisting of millions of symbols"

andy.fingerhut19:01:02

Are you asking "How would one determine the boolean formulas for 10k/100k nodes?" I think the answer is "knowing something about the services involved, and which ones depend upon the others"

andy.fingerhut19:01:40

If that isn't documented anywhere, then you have a big fact-finding mission ahead of you, or you need to guess based upon information you do have.

hiredman19:01:03

people do things like taking package dumps and generating network diagrams from that

mokr19:01:15

I need to read up on SAT solvers.

andy.fingerhut19:01:43

I have used one called Z3 and its Python bindings, and it is definitely capable of solving some pretty impressive things.

andy.fingerhut19:01:33

In this particular case, if you had the formulas, and the state of which nodes were giving alarms, you would not need a SAT solver. You would just need to plug in the true/false values for variables and evaluate the formulas.

andy.fingerhut19:01:19

but a SAT solver might be useful for answering other kinds of questions

mokr19:01:19

I know I’m probably in way over my head, but I still would like to have a go at it. I plan to dump data from a few different sources (pretty much two CMDBs), try do normalise them so they can both contribute to the full dependency graph. After creating the model, I take the alarms from the network and use them to mark nodes in the model as affected (down). Having this I plan to use CLJS + Re-frame + D3js to visualise the state of the network and services. Especially en user services like SMS, calls, roaming, …

mokr19:01:50

@hiredman That UML generation is very much in the same spirit as what I’m trying to do. In the sense that you use available data to generate instead of creating by hand.

mokr19:01:42

Thanks for the input, all of you! This has been very inspiring and given me some actual ideas on where to start.