Fork me on GitHub
#off-topic
<
2017-04-18
>
qqq03:04:51

@roberto @noisesmith : erlang vs clojure: fight! šŸ™‚

donaldball03:04:07

I like functions and values as substrate, objects for stateful components, and queues between systems.

noisesmith04:04:14

@didibus

Clojure uses the software transactional memory concurrency model. That is, each access to data is made inside a transaction, and must either go through fully, or not at all.
- data writes to refs, yes. But we tend to read a lot more than write, and reads can be uncoordinated, and furthermore almost nobody actually uses refs.

didibus05:04:04

That's true, but the only part that's novel is STM. I could have mentioned CSP, but I feel that would have gotten too deep in details of actors vs csp

didibus05:04:07

Maybe I could reword this a bit though. It could sound like that's the only way to access data.

tbaldridge05:04:44

Actors are async OOP, why take something as painful as mutable objects then make them async?

mpenet05:04:34

There's nothing oop about actors, unless you consider anything that deals with state oop

mpenet05:04:06

OOP's definition can be quite fuzzy

qqq06:04:16

@tbaldridge : I feel like in Clojure, if you have a bunch of different agents/atoms/refs, you need to coordinate their communication. They don't talk to each other until you tell them to.

qqq06:04:33

In an "erlang/actor" model, I get the impression that up front, you tell them who to talk to, ... and then they just talk to each other.

qqq06:04:53

This mess of 'routing messages/updates' between parts -- if manually handled in Clojure and seems defined once and behind the scenes in actor models.

tbaldridge12:04:26

@qqq that's one of the reasons why it's highly recommended to have one atom that holds all the state of your application. I don't understand how the answer to hard engineering problems needs to be "more mutability! More non-deterministic code!"

qqq13:04:01

@tbaldridge : Most complex systems I see are individual components acting on their own. "One centralized atoms" feels alot like a mayor of a city dictating the action of every citizen via Sim 3 or something.

tbaldridge14:04:15

But that's the way "real life" works šŸ™‚

tbaldridge14:04:43

To quote "Are we there yet", you don't send a message to the quarterback of the super bowl to ask him what the state of the football is.

tbaldridge14:04:02

The "real world" can be viewed as a succession of "universe states", with each old state existing forever as an immutable fact, and future states being produced by (f state) => new-state.

qqq14:04:51

@tbaldridge: in https://www.youtube.com/watch?v=096pIlA3GDo&amp;feature=youtu.be&amp;t=30m19s what does "userspace" mean ? (in the context of actors, dataflow, core.async)

tbaldridge14:04:04

And to be fair, that's exactly how most games are programmed. They may use mutable datastructures, but they are synchronous, it's very hard to enforce game rules on an async concurrent system.

tbaldridge14:04:32

@qqq yeah I didn't define it in the talk, I mean "outside of library or framework development".

tbaldridge14:04:46

Or the code in which you spend 80% of your time and comprises your business logic.

qqq14:04:04

@tbaldridge : aren't you glad that the internet is an actor model and not a single atom šŸ™‚

tbaldridge14:04:23

But it's not the actor model

qqq14:04:25

I feel like at certain levels of compelxity, one can't micro manage everty part, and just code components and let them interact

tbaldridge14:04:43

many machines have multiple IP addresses (mailboxes) and a server can't magically spawn new servers.

tbaldridge14:04:09

the internet is more closely related to CSP than actors.

qqq14:04:17

@tbaldridge : I must be misunderstanding "actor model" then; what do you call something like the internet with lots of connected machines, each with their own state, and can only talk via message passing

tbaldridge14:04:34

Not to mention that actors often assume infinate mailbox sizes and ignore backpressure.

qqq14:04:56

I believe in finite mailboxes.

qqq14:04:23

If you were to simulate the internet in clojure, what would you use to represent each indivudla machine?

qqq14:04:33

And how would you simulate the 'routing layer' ?

leonoel14:04:09

@tbaldridge in your talk you mention immutable / purely functional actors, do you mean stateful processes are a bad thing ?

tbaldridge14:04:45

So to start with, I'd recommend everyone watch this informal talk given by the guy who helped invent actors: https://channel9.msdn.com/Shows/Going+Deep/Hewitt-Meijer-and-Szyperski-The-Actor-Model-everything-you-wanted-to-know-but-were-afraid-to-ask

tbaldridge14:04:06

So we probably have to start with the problem that what is described in that talk ^^ is rather different from Akka, Erlang, etc.

tbaldridge14:04:00

@leonoel but yes, one of the biggest problems I have with actors is that they (like CSP go loops) are very hard to debug because there's no good way to get at their internal state.

tbaldridge14:04:38

So you can improve the model given by Erlang/Akka by defining actors as a application of a function over old state to produce a new state, function and messages to be sent

tbaldridge14:04:10

(f state msg) => [next-f next-state [[dest msg] [dest2 msg2] ...]]

tbaldridge14:04:06

And then the system that runs the actors could allow you to deref @state just like any other clojure ref

leonoel14:04:15

well, given that actors (like go blocks, or agents) process one message at a time, they're free of concurrency issues, so they're not that hard to get right

leonoel14:04:32

what you describe looks like agents, btw

tbaldridge14:04:04

As that talk mentions ^^ "conceptually one message at a time", the actor system is free to optimize and run many messages through an actor at once if it knows that the actor will only ever read from the state

qqq14:04:52

@tbaldridge: that video is tedious

tbaldridge14:04:18

well if you want to understand hard concepts sometimes you got to do some research šŸ˜›

qqq14:04:45

yeah, but I think there's probably 2 pages of dense math equations somewhere instead of these three guys chatting

tbaldridge14:04:56

Perhaps, but I'd rather sit and talk to "these three guys" talk anyday. Remember, one guy helped invent actors, the other helped write parts of Haskell, Linq, and C#.

tbaldridge14:04:21

But I think these very abstract "chats" are a good way of removing conceived notions of what actors are.

tbaldridge14:04:09

But at the core actors are: 1) a process 2) that can send messages to other actors 3) based on a message it receives

qqq15:04:34

@tbaldridge : yeah, this video is not for me; but what I did get is that csp (communicating sequential processes) != actor; and, as you mentioned erlang != actor, go != actor ... right?

tbaldridge15:04:00

go lang is csp, that's for sure. Erlang bends a lot of actor rules, and adds things. For example Erlang allows for mailbox searching, and IO inside of an actor's message processing loop.

tbaldridge15:04:35

Mailbox searching can result in degrading performance over time. And IO inside an actor has the same problems of any IO inside a function, it's impure.

qqq15:04:05

where does core.async go loops fall in this? I feel it's a "strict subset" of go, in the sense that <! and >! can only happen in the "top level of the go loop" and not "nested in functions" since go is some magical macro

tbaldridge15:04:21

Both go and core.async are implementations of CSP.

tbaldridge15:04:07

The problem isn't nested functions, but that both CSP and actors often recommend a pattern of programming that is impure: namely IO inside functions that perform decision logic. This is why transducers are fantastic, we can test transducers in isolation, in a deterministic manner, then we can use them in a impure context like a channel. This allows us to test our impure code in one place, our transform logic in another.

tbaldridge15:04:11

So for example (chan 1 (map f)). How do we test this? We don't. We know chan is tested by core.async, and map is tested by clojure.core. So all we need to do is test f. Once we compose this channel into a larger system we can also use integration tests to make sure it all works correctly.

qqq15:04:36

Is the crux of your argument: 1. pure functions = easier to test than impure functions 2. <! and >! perform IO and are thus impure 3. any function with <! or >! nested inside of them are impure and thus hard to test 4. thus, we'd like <! and >! to be top level, so the other functions involved are pure (if so, I buy the argument)

tbaldridge15:04:02

yep, and that's about 50% of my talk at Clojure/West šŸ™‚

tbaldridge15:04:25

Don't get me wrong, I love the idea of actors, and continue to play with them from time-to-time. However I have yet to run into a situation where actors made a problem clearer than it would have been using immutable data and "normal" clojure code.

leonoel15:04:27

@tbaldridge a channel with a transducer is not unlike an actor

tbaldridge15:04:53

They are nothing alike

leonoel15:04:27

imo it's better, as you said features are decoupled so more easily testable

leonoel15:04:44

transducers are composable, actors are not

leonoel15:04:53

channels are generic, actors are not

tbaldridge15:04:06

Actors have unbounded message boxes, unknown outputs, can "become" new actors, have internal state.

tbaldridge15:04:20

Most of those are completely different from channels + transducers

leonoel15:04:43

transducers can be stateful

leonoel15:04:43

you can view the channel output as the new messages to send

qqq15:04:57

@tbaldridge : IIRC (can't find reference), Alan Kay once drew a reference between OO and biological cells. Cells (1) have their own state (2) do their own thing (3) can't view/poke the state of other cells and (4) communicate only via message passing. This also describes individual machines on the internet. Using core.async, how would one model cells/individual-machines via go blocks ?

tbaldridge15:04:25

You haven't watched my talk from Clojure/West have you?

qqq15:04:37

I have not.

qqq15:04:45

Which talk are you referring to? You have many talks.

tbaldridge15:04:09

It's a little over 35min of talking about different models that handle different tasks better than raw core.async or even actors.

qqq15:04:29

Where is the talk? I'll put it on background while I code.

tbaldridge15:04:27

So in my case, what I prefer is the Dataflow model. Nodes with explicit inputs/outputs connected via a configuration graph. True it's not as clean in a distributed model. But even stuff like Onyx can be viewed as a dataflow graph.

qqq15:04:08

I'll watch through the video and get back to arguing with you after wards. šŸ™‚

jjmojojjmojo15:04:52

"using emacs leaves me with no closure" (facepalm)

leonoel16:04:21

@tbaldridge the model you describe as "dataflow and FRP" has working clojure implementations ?

leonoel16:04:00

or is this just hammock time ?

tbaldridge16:04:43

Although I have a non-OSS implementation that is 100% immutable.

leonoel16:04:44

ok, so I guess e.g reagent or Rx fall in this category as well ?

tbaldridge16:04:58

Yes they can. Rx is often built by attaching objections into chains, so yes, it looks somewhat like javelin.

tbaldridge16:04:46

In most situations I prefer to build up a dataflow graph as data, but that's an implementation detail.

tbaldridge16:04:05

If you store the entire state of the dataflow graph in immutable data you can do step-by-step debugging, time travel, etc.

leonoel16:04:37

well, what annoys me with frameworks like Rx is that you handle "live" objects, and the graph itself is a mutable thing

tbaldridge16:04:10

Which interestingly enough is also a problem with actor systems.

leonoel16:04:31

anyway, if you release your implementation as OSS, I will have a serious look at it

tbaldridge16:04:18

It's currently part of a rules engine, I should just pull it out as a separate library. Maybe I'll look into doing that this Friday.

baptiste-from-paris16:04:45

hello guys, I am looking to extract some PDF document in Clojure. I did some proof of concept with Apache Tika and PDFBox. For both solution I can easily extract text but I have no clue how to get back titles from the original PDF document

baptiste-from-paris16:04:11

any ideas on how to achieve this ?

Ed20:04:52

Hi ... is anyone using gradle and emacs/cider together? ... could do with help jacking-in to an existing project šŸ˜‰

mobileink20:04:40

what does your gradle thingie do now?

mobileink20:04:39

also: there's a #cider channel as well.

Ed20:04:22

there's a long build.gradle script that i'm not really sure i want to dig into too far yet ... so I downloaded clojuresque.jar v1.7.0 and put it in the gradle home dir (/usr/share/gradle/lib/plugins)

Drew Verlee20:04:31

@tbaldridge you said... > So in my case, what I prefer is the Dataflow model. Nodes with explicit inputs/outputs connected via a configuration graph. True itā€™s not as clean in a distributed model. But even stuff like Onyx can be viewed as a dataflow graph. prefer as opposed to what? you linked your talk to core.async, but iā€™m having a hard time comparing core.async to onyx. [edit] I mean, couldnā€™t someone build a dataflow model using core.aysnc?

Ed20:04:45

@mobileink fair point ... i'll ask there too if I get no joy here ... thanks šŸ˜‰

mobileink20:04:38

so you're trying to add some clojure to a java project that uses gradle, or what?

tbaldridge20:04:47

@drewverlee the gist of the talk is that core.async can be used to create higher level constructs like dataflow, pedestal interceptors, etc. But just like bare actors don't give you a whole lot, the same is true with core.async.

tbaldridge20:04:30

So I'd prefer to find a higher-level model that allows me to not have to think about concurrency or parallelism on a daily basis. I should be able to write pure functions, and then something else can take care of those details for me.

Ed20:04:39

@mobileink nope ... i'm joining a clojure project that uses gradle as a build tool

Ed20:04:45

i've never used gradle

mobileink20:04:20

@l0st3d no! using clojuresque already?

tbaldridge20:04:40

@drewverlee Onyx is the same way, you define nodes that do things: read from files, sum results, etc. And then wire them together with a config. What Onyx does with that config is unrelated to the work performed by the nodes themselves.

mobileink20:04:51

you poor slob! šŸ˜‚

Ed20:04:59

not as far as i can tell... everyone seems to use cursive ... i'm the only emacs user on the team

Ed20:04:05

šŸ˜‰

Ed20:04:29

(that i've met so far)

mobileink20:04:43

i don't suppose you can show the gradle file?

Ed20:04:21

probably not ... its 425 lines long and probably contains a bunch of things that shouldn't be shared in a public forum

mobileink20:04:08

using gradle for clojure, omg, i spent a bunch of time learning gradlecin order to use clojuresque and ended up running screaming for the exits.

Ed20:04:14

the last place i worked, there were a bunch of java devs moving to it from maven ... the stuff i worked on was all lein šŸ˜‰

mobileink20:04:26

do your wholevteam a big favor, and get a big raise, by rewriting it in boot. or leinengen.

Ed20:04:48

yeah ... maybe ... but not on day 2 šŸ˜œ

mobileink21:04:17

yeah, be strategic šŸ˜‰

Ed21:04:24

there's a bunch of things that are much more important than rewriting the build system šŸ˜œ

Ed21:04:19

i'm sure i can extract the classpath from gradle, then add nrepl & cider middleware and just run java -cp ....

Ed21:04:38

it won't be too hard šŸ˜œ

mobileink21:04:40

but if you've got gradle stuff that alreafy works, can't you just run it and cider-connect?

Ed21:04:18

it won't add the cider middleware into the deps

Ed21:04:26

and i'm not sure there's a repl task

mobileink21:04:50

i'd look at writing a boot wrapper around the gradle stuff.

mobileink21:04:19

but it's been a long time since i felt the gradle pain.

Ed21:04:20

maybe ... i just need something that injects the correct version of nrepl, the cider middleware and maybe the clj-refactor stuff ... emacs knows what those versions are, and i think i can work out how to get the cp out of gradle ... but it's 10pm here, and i should get some sleep ... got an early flight this morning, and could do with the sleep ... thanks for your sympathy @mobileink šŸ˜‰

mobileink21:04:16

let us know what you come up with.

Drew Verlee21:04:28

@tbaldridge Ok yea, that makes sense. Iā€™m not sure Onyx supports a multithreaded local version. I mean, they run their tests using core.async, so its possible to simulate the model locally, but i donā€™t know what it would be like to use it in production. There was a talk at clojure remote about a higher level abstraction on top of core.async. https://www.youtube.com/watch?v=-SMHkPAEGIk I just confirmed that isnā€™t opensource though šŸ™‚

cfleming21:04:23

@l0st3d @mobileink Gradleā€™s actually not that bad, I use it but with a plugin I wrote: https://github.com/cursive-ide/gradle-clojure

cfleming21:04:38

Itā€™s very flexible.

mobileink21:04:36

de gustibus non disputandem. me, i 'd rather slug drano than so much as look at groovy again. simple_smile

mobileink21:04:05

did not know about gradle-clojure. did that come out after clojuresque? looks good, the more the merrier.

mobileink21:04:55

thing is, once you've seen boot you never want to go back.

mobileink21:04:35

if you want more capabilities you have to write more gradle plugins?

mobileink21:04:02

my foggy recollection is that one reason i came to hate gradle was because it was almost impossoble to figure out how to write plugins.

mobileink21:04:43

and even if you figgered it out you have to use groovy.

sveri21:04:45

Hm, gradle plugins can be plain groovy / java code. So technically they have the same power as boot.

mobileink21:04:17

sure, so does assembler. it's about expressivity.

mobileink21:04:16

sorry, i concieved a great hatred for groovy while working with gradle.

sveri21:04:38

I agree that clojure is more expressive than java / groovy. But you can do the same stuff with gradle that you do with boot, thats all I said.

mobileink21:04:39

my personal thing ymmv.

sveri21:04:39

Hehe, one year ago we switched from several 1000 lines of ant to gradle and it was mostly straight forward, compared to the ant stuff šŸ™‚

mobileink21:04:44

i don't doubt that, best wishes to amybody who wants to. but there is a pain factor.

mobileink21:04:45

fwiw i found gradle/groovy almost totally impenetrable. which is not what you want in your build tools.

sveri21:04:47

As with every framework, as long as you stick with the suggested / proven paths its fine. Everything beyond gets messy, complex and hard to maintain.

mobileink21:04:24

that's exactly the problem. frameworks are great, until they're not. i gradually discovered that gradle did not in fact solve the problem, it just moved it, begat a different kind if complication. boot is a whole nother thing, it actually solves the problem.

mobileink21:04:54

put it this way: show me anybody who has mastered both boot and gradle (or any other fancy bld system) and i will bet my next paycheck that said person will pick boot.

mobileink21:04:23

the whole idea that we need specialized "build systems" like gradle or whatever is wrong from the git-go, imho. it's all just peogramming.

roberto21:04:32

I havenā€™t mastered boot, but when I tried playing with it I didnā€™t like it and went back to lein for clojure and gradle for java/kotlin projects. šŸ™‚

mobileink21:04:55

@roberto: how much of "didn't like it" is attributable to sucky docs? curious, cause i think that's a big problem.

mobileink21:04:37

i admit the learning curve is too steep. i think that can be fixed.

roberto21:04:08

that could well be the biggest turn off for me. It has so much hype but I found it much harder to even grasp the mental model and do anything with it.

roberto21:04:53

It is difficult to justify investing time on it these days with so many things coming our way: build systems, deployment pipelines (kubernetes, docker, vault, etc), front end frameworks, other programming languages

mobileink21:04:00

i sympathize. i have to admit, i spent a lot of time and effort getting to good.

mobileink21:04:04

hah! the Firehose! hell i can't even keep up with clojure! (transducers? huh?)

roberto21:04:09

learning clojure and emacs already required some effort, and to pile learning boot on that was just too much for me.

roberto21:04:23

oh yeah, I can barely grasp transducers, and now spec ā€¦.

roberto21:04:47

I realize I canā€™t fight all the battles

mobileink21:04:52

heh, just now singeing my paws on spec.

roberto21:04:47

plus, we are a consultancy, so sometimes we have to learn new languages rather fast. Had to learn Kotlin for a project while on the job. So time is very limited.

mobileink21:04:17

ouch, but also: cool!

roberto21:04:24

I decided to not learn any new js framework, no new testing framework (no matter which language, if using Kotlin/java stick to JUnit, if using clojure sticking to clojure-test)

roberto21:04:39

and the tools with the most mindshare.

mobileink21:04:46

love minimalism

roberto21:04:45

I do tend to go off the deep end learning new concepts, right now Iā€™m fascinated by type systems, especially the concept of expressing a program as a state machine and focusing on making impossible states ā€¦. impossible ā€¦. (I think that is what the creator of OCaml calls it).

roberto21:04:33

I think Dave Thomas gave a talk recently at GOTO expressing the same concepts, but I think his lack of familiarity with more advanced type systems hurt his presentation

roberto21:04:08

he was using Elixir to show this new ā€œenlightmentā€ of his.

mobileink21:04:11

fascinating thing: you might think that as time goes on we would settle on the best stuff and be done with it. instead we see the opposite: massive proliferation of new languages and tools. it's about 3 fulltime jobs just to keep an eye on things, let alone master anything.

mobileink21:04:00

you wanna blow your mind start in on coinduction and codata. šŸ˜‰

roberto21:04:32

I am actually enjoying this new age of innovation in programming languages. It is the proliferation of different tools that do the same thing that Iā€™m tired of. Especially in the JS space and the ā€œdevopsā€ space.

roberto21:04:05

There are all these tools in the devops space that I can hardly understand what they do

roberto22:04:09

last year we were using Mesos, this year kubernetes seems to be the thingā€¦..

mobileink22:04:27

heh. fortunately i can ignore the js churn. devops? i hear its cool, that's about as far as i go. i've heard of this thing they call "docker", does that count?

roberto22:04:00

hehehe, yeah, I actually like docker. It is the orchestration tools that I find intimidating.

mobileink22:04:42

you could probably make a good living just knowing what those things mean and bsing clients. šŸ˜‰

roberto22:04:56

yeah, that is what we do šŸ˜›

mobileink22:04:10

you're hired!

mobileink22:04:37

it's all about buzzword compliance.

mobileink22:04:02

hey, there's a website to be made of that.

roberto22:04:25

We started using Kafka before it became hype, and it was a headache. I remember having to read the source code if we wanted to do anything with it.

roberto22:04:33

that is a good idea

mobileink22:04:49

wish i had the time!

roberto22:04:05

it is exhausting after a while. Even when you are paid to do it.

mobileink22:04:11

submit your homepage, we'll vet it for buzzword compliance - and alert you to the latest! That'll be $20K, please.

roberto22:04:56

hahaha, yeah, it is like that ad in Silicon Valley making fun of all the startups

roberto22:04:31

E.g. of an actual job description that reminds of the buzzword compliance idea:

We are a multi-award winning information and advisory company that helps leading global companies to improve their strategic decision-making and operational implementation through the best use of data and analytical techniques.


 our open plan offices enable our people to collaborate with industry leaders and incredible minds

mobileink23:04:31

roberto: "industry leaders, and also people with incredible minds"

mobileink22:04:36

lovely. but they left out "world class". izzat outdated now?

mobileink22:04:23

i like "multi-award winning". what is a multi-award?

noisesmith22:04:41

@mobileink clearly itā€™s an award with arbitrary first class dispatch using either the implicit global hierarchy of award types or optionally a local first class hierarchy argument.

mobileink22:04:14

noisesmith:

mobileink22:04:16

noisesmith:

mobileink22:04:15

sorry, fat fingered my blasted phone

cfleming23:04:39

@mobileink Actually gradle-clojure is written in Kotlin, and you can now use Kotlin for build files too.

mobileink01:04:31

Great Caesarā€™s Ghost! Thatā€™s truly horrifying.

cfleming02:04:42

Why is that?

roberto13:04:07

I think moving to Kotlin is a good thing.

cfleming19:04:50

Absolutely, no doubt. If only for the better IDE support.

mobileink19:04:41

depends on your needs, i guess. if you're developing clojure code, why would a kotlin build tool be a good thing when you already have boot and leinengen?

mobileink19:04:50

@roberto you mean moving from groovy to kotlin? anything is better than groovy.

cfleming03:04:29

@mobileink Because I need access to lots of other JVM stuff I donā€™t want to have to reimplement myself (like compiling Kotlin, running profilers etc). Basically, my project is a JVM one, not pure Clojure.

mobileink03:04:13

that makes sense

cfleming23:04:54

And yes, itā€™s pretty recent - late last year.