Fork me on GitHub
#clara
<
2019-02-19
>
eraserhd22:02:08

Over the next week or so, we're going to be working on a library to translate pull expressions into Clara rules productions. Any thoughts on that now could shape how this works.

dominicm22:02:06

Why do you need this? I'm missing something I think

eraserhd23:02:39

The primary system I work on models tasks and data dependencies. Inputs and output for a task are described as pull expressions, and this allows us to compute task dependencies. We couldn't do this with raw Clara queries, since they model general computation so halting problem yada yada.

eraserhd23:02:14

Pull expressions are a really nice, natural way to describe the shape of data wanted, IMHO.

eraserhd23:02:05

We are moving to Clara handling more of the computation, because its truth maintenance property solves a lot of problem for us, and one of the new things we want to know is when a tasks inputs have changed, in order to retrigger our task.

eraserhd23:02:25

So we're building this library to have Clara compute the inputs for a task in real time and update them as new data arrives.

adam62203:02:17

@ sounds like an interesting project. Do the task inputs and outputs change frequently, preventing the use of a simple dependency graph? Can you say more about the business domain? If you haven’t seen this, you may find it useful for working with pull expressions: https://github.com/edn-query-language/eql

dominicm06:02:49

This sounds quite similar to something I've been working on in my spare time!

eraserhd13:02:19

@ the problem with a hand-coded dependency graph is that the tasks are sort of "locally owned" by different teams. This project coordinates tasks across the org.

eraserhd13:02:19

eql looks really neat, definitely something to investigate

dominicm14:02:05

@ are you familiar with terraform at all? would you say your tool is similar?

eraserhd14:02:48

I am familiar, we use it. I have big rants on terraform. Our thing is much higher level.

eraserhd14:02:05

Our thing's tasks are like, "we need a marketing site". That integration might use terraform.

eraserhd14:02:40

There's some similarities, except that we really lean on forward-chaining rules. We need to have things built as early as possible, and we incrementally discover the information we need to do things.

eraserhd14:02:30

A terraform-like thing using Clara would be nice!

dominicm14:02:52

I want to write a terraform like thing in clojure. So I may look closer at what you're saying here.

adam62214:02:16

@ Pulumi uses Terraform under the hood (and therefore takes advantage of the existing Terraform ecosystem), perhaps adapting Pulumi to Clojure would be a reasonable route to take https://github.com/pulumi/pulumi, https://github.com/pulumi/pulumi-terraform

dominicm14:02:30

Interesting, will take a look

dominicm14:02:51

@ we probably share complaints about terraform, and also the level at which you are describing things sounds similar to a project I tried to tackle things.

eraserhd14:02:10

My main complaint is that it is trying to be declarative. I think that's a mistake in that domain. We frequently need one object per thing in some database, for example, and that leads to terrible duplication and weird text ETL procedures.

eraserhd14:02:00

Those ETL procedures defeat the idea of what terraform is for. So they've added things like loops and getting external data, but it is kind of terrible and they don't want you to use it.

dominicm14:02:03

Oh I know. And the modules don't even help! Count doesn't even work for the "loop construct" in a number of cases, so you have to take explicit counts for everything

eraserhd14:02:08

In reality, it should be pure functional.

dominicm14:02:56

The terraform language sucks, generally. I'd love it if they had a minimal extensible core.

dominicm14:02:16

I quite like the idea of figuring out some primitives for building your own tf :)

eraserhd14:02:03

In Clojure, terraform really just becomes a data structure and a plan/apply procedure. The data structure can be generated by a function.

eraserhd14:02:32

I mean, Clara is probably good here :D But not necessary.

eraserhd14:02:02

The data structure needs some kind of ?var to bind results of some operations to inputs of others. Unlike the go people, we know about proper unification :D

dominicm14:02:41

So, I'm looking at clara to do the dependency graph of work semi-automatically. For example, when do you go fetch the list of GitHub users? Well, if something wants to query them (and then you cache it). If you've diffed the list of users, and something inserts a new user, then you need to change your actions based on that. This is a pretty half-baked idea though 🙂

eraserhd14:02:42

So, are you using it ahead-of-time to build a plan? Or are you using it during execution?

dominicm15:02:10

thinking about trade-offs, I decided that a plan is possible to do as an auxiliary based on a secondary set of rules which inform you of the "optimistic write output" of a task.

eraserhd22:02:55

We're thinking about depending on clara_eav, since this will need a triplet type, and I'm considering the EAV record in clara_eav the "canonical triplet type" for Clara.

eraserhd22:02:16

It'll likely use nothing else from it, though.