Fork me on GitHub
#off-topic
<
2019-04-13
>
john04:04:32

Ola Bini got arrested in Ecuador

chokheli08:04:38

https://www.imf.org/en/News/Articles/2019/03/11/ecuador-pr1972-imf-executive-board-approves-eff-for-ecuador March 11, 2019 - IMF Executive Board Approves US$4.2 Billion Extended Fund Facility for Ecuador

chokheli08:04:38

https://www.imf.org/en/News/Articles/2019/03/11/ecuador-pr1972-imf-executive-board-approves-eff-for-ecuador March 11, 2019 - IMF Executive Board Approves US$4.2 Billion Extended Fund Facility for Ecuador

nathantech200517:04:18

Think that’s linked to Assange?

chokheli21:04:34

Who can prove it’s not?

val_waeselynck08:04:56

In Lisbon for a few days, hotel's a bit gloomy. Anyone know a nice place with wi-fi to work?

trevor13:04:14

The Block is quite a nice place to work.. http://theblock.cafe/

val_waeselynck14:04:44

Unfortunately Bitcoin's a dealbreaker for me 😞

trevor17:04:10

How so? They accept euros

paulocuneo18:04:09

Working with multiple micro-services, many teams, multiple contexts for the same business entity, and multiple sources for the same entity. It seems like the tower of babel story, we've build something so big that every teams starts to speak its own language. I see that this independence brings success, but it also seems I'm writing structure transformations all the time. I've also seen this happen at my previous job, and also many coworkers have seen this before at different scales. ¿Have you deal with these "tower of babel integration" problem? ¿Did you enforce a common API across all teams, or did you solve it by automating the transformations, or making a catalog? ¿Or do you just code the transformations?

gklijs19:04:45

How I think it would work best is all using Kafka for messages, with the same 'type' of massages. For example strings as keys, and values avro encoded with schema's. And on each topic there is only one producer. Of all JVM you could use specific records if you want. If you also makes sure the schema's are always backwards compatible I think most things are tackled. Depends a bit on what the product is, but towards the front-end I would expose multiple topics with GraphQL.

richiardiandrea20:04:20

This is a big topic for me also, another question I had was: do the microservices need to be written in the same language? I would say of course not but many Enterprise companies are worried that it would be a mess recruiting people...dunno

matt.wistrand21:04:20

What do you mean by “multiple sources for the same entity”? Different teams are duplicating work so there are multiple models for the exact same data?

matt.wistrand21:04:57

@ I think it’s fine for microservices to use different languages, but for a number of reasons I advocate that companies limit “approved” languages to two or three with very few exceptions.

nathantech200521:04:00

Could you please describe a specific example of a structure transformation? I think using Message Brokers like RabbitMQ and ZeroMQ can really help reduce duplicate code. And you could even use foreign language interfaces… e.g. how Clojure can call other languages inside of it. I am researching Vert.x because I could write it in Clojure, Python/Jython or even Jruby/Ruby. You can compare and contrast that to Elixir Umbrella Projects that try to combine monolithic with micro services.

paulocuneo22:04:53

Kafka: I guess a big enterprise bus is one way to solve it, or at least for a subsystem. Polyglot: multiple languages is not an issue.(Just use one lang for service) I guess the issue I tried to mention is being a mediator between multiple "bounded-contexts", that's not exactly model duplication, but thats the part where we start to speak different tongues

nathantech200522:04:15

You might try Vert.x because it builds a message bus into the tool. They built it for providing micro services (reactive programming) and polyglots. It supports most of the JVM languages. But I need to do a proof of concept… to make sure it’s not another callback hell async problem. It has a promises/futures library to fix that, I believe.

paulocuneo23:04:44

you really love Vert.x 😁

nathantech200523:04:31

I love the idea of it. But I can’t say yet until I put it in production and load test it.

nathantech200523:04:57

It’s also not the easiest to setup. But sometimes the easier stuff to setup is a headache in the long term.

nathantech200523:04:33

The only disadvantage for me so far with Vert.x is that it doesn’t auto scale as easily as Elixir. But Vert.x has so many more advantages. JVM, libraries, machine learning, foreign language interfaces, actor model + reactive model + blocking thread worker pool… and more

nathantech200523:04:55

Elixir gives you rapid dev with a monolith and umbrella style micro services. But it’s not a general purpose tool for machine learning, cpu number crunching. And Vue JS can give me the rapid monolith advantage.

gklijs05:04:22

From my experience using Kafka is a lot easier then VertX. The Kafka ecosystem is huge making it easy to connect basically whatever you want. If you have a legacy service you still need, you can create events from the changes to the the database. Also you have access to almost all languages for clients thanks to librkafka. But maybe most importantly Kafka allows parallel async programming as a whole, with execution of 'simple' synchronous code.

nathantech200521:04:19

Can anyone explain to me… if a computer has 8 cores, it can have 16 threads (if hyper threading, intel). But why do I see code setting up 100's of threads? Are the threads virtualized and pooled? And there is a difference between actual real world parallel threads versus pooled threads?

lennart.buit21:04:24

Yes they are, its the task of your operating system to schedule threads and processes on the CPU

lennart.buit21:04:46

but the story is not often as simple

nathantech200521:04:13

Ahh ok. So it’s a virtualized thread count. They CPU interleaves them (schedules them) on the actual hardware threads?

lennart.buit21:04:42

Well, what your operating system is concerned, a octa-core with HT is actually just 16 cores

lennart.buit21:04:03

it doesn’t “know” that half of them are not physical cores

nathantech200521:04:31

ahh, the hyper threading makes it look like 2 cores from the operating system’s perspective?

nathantech200521:04:11

I am researching Clojure threading primitives in comparison to just the actor model. e.g. clojure agents

paulocuneo22:04:38

from what I remember, JVM will map each thread to an O.S. managed thread.

paulocuneo22:04:23

threads are "abstraction" for "concurrency", but we you go down the hardware there is a limit to what is actually done in "parallel"

k.i.o07:04:51

JVM will introduce (cooperative) fibers\\continuations in near future, which is a lightweight replacement for (pre-emptive) threads, it will allow JVM to manage async tasks in more efficient way, mapping hundreds of thousands fibers to a thread pool matched to real cores in size.

hi89513:04:17

@ so something similar to what Go or Erlang does? That sounds amazing! What I was able to find was "Project Loom". Very curious when this will be available in a stable JVM release 🙂

k.i.o14:04:08

@ correct, in a year or two, also probably tail call recursion, since they changing how stack frames being handled

hi89514:04:48

Can't wait! Also curious how this will be integrated into Clojure since Clojure still needs to support older JVM versions..

k.i.o14:04:35

¯\(ツ)

nathantech200518:04:18

Java Fibers will be similar to Actors in Erlang where no locking is needed? I am not sure how Golang fits under the Actor model since it’s really CSP (Communicating Sequential Process) and actually relies heavily on locking/mutexes?

nathantech200518:04:43

Reading this… it seems Java Fibers are the equivalent of actors with async ability?

k.i.o03:04:11

@ java fibers is green threads scheduled by JVM with ability to pause and continue by saving fiber stack on heap

nathantech200503:04:20

ahhh ok. there’s that “green threads” word again. They are virtual threads

k.i.o03:04:05

@ pretty much, regular Thread will become lower level of abstraction rather for continuous calculations

nathantech200503:04:34

I am researching if Vert.x can take advantage of fibers

nathantech200503:04:33

wow. Vert Sync allows for async non blocking calls but with a syncronous style, no callback hell. I am reading dzone website.

nathantech200503:04:06

It says Vert Sync uses Quasar for Fibers

nathantech200504:04:17

it says the Fiber is blocked… but the underlieing thread is not blocked. So you get the best of both worlds. And it doesn’t turn your code into spaghetti callback hell

lennart.buit21:04:44

yeah, I was just about to bring up Green threads

jaihindhreddy21:04:13

And don't forget, there's also threads (Intel Hyperthreading) 😂

lennart.buit21:04:16

but generally with green threads, its your app, or host platform (f.e. the JVM) that does the scheduling

lady3janepl21:04:16

there should be a guide for beginners… “Learning the ropes of threading”

lennart.buit21:04:25

instead of the operating system’s scheduler

jaihindhreddy21:04:11

^^ I'm gonna steal that if you don't mind.

lennart.buit21:04:46

stealing is not stealing if you ask 😛

lady3janepl21:04:14

let’s not discriminate against British/Canadian thieves!

lady3janepl21:04:48

they can’t help being polite…

lady3janepl21:04:33

@jaihindhreddy ping me a link if you end up writing a threads overview 🙂

nathantech200522:04:31

what are green threads? virtual threads?

lennart.buit22:04:23

threads that don’t map directly to threads in your operating system

nathantech200522:04:46

hmm. like software level threads?

nathantech200522:04:43

I am curious because this weekend is my concurrency/threading study time. I need to write a deadlock on Clojure to learn how to simulate it.

lennart.buit22:04:46

(lol, google is showing me an image of a green ball of yarn when looking for “green threads”)

lady3janepl22:04:41

you’ve been yarn bombed

lennart.buit22:04:07

(I think I fell out of my filter bubble)

lennart.buit22:04:17

(please send help)

lady3janepl22:04:45

as I understand it, it’s your programming environment pretending threads exist

lady3janepl22:04:45

as I understand it, it’s your programming environment pretending threads exist

lennart.buit22:04:13

yeah this exactly

nathantech200522:04:00

thanks for threading this

lennart.buit22:04:21

yeah, gotta keep the concurrency going

nathantech200522:04:21

we are in a green thread now

lennart.buit22:04:45

its not proper multithreading unless people are majorly confused 😉

lady3janepl22:04:57

is green thread basically a more ecological way of threading then?

lennart.buit22:04:50

a deadlock occurs when two threads are waiting on eachother

lennart.buit22:04:27

huh where did the question go

nathantech200522:04:30

Clojure can still deadlock? It just happens less often in Clojure?

nathantech200522:04:42

Sorry, I deleted it. I didn’t want to derail Jane

lennart.buit22:04:53

well, everything can deadlock, its just that Clojure has a mitigation strategy when sharing memory between threads

nathantech200522:04:02

I am asking because I am trying to figure out if I need lower level threading like Clojure provides. Lower level than actor model.

lennart.buit22:04:22

(its actually funny that you deleted the question, you kinda illustrated the problem that threaded execution can have, if you are not careful the “world” can change under your feet)

lady3janepl22:04:35

also idk what you’re using to learn Clojure, but http://shop.oreilly.com/product/0636920013754.do has a really (insert punhusky here) thorough chapter on concurrency

nathantech200522:04:42

My conclusions, faulty as they can be, is that Clojure rules the realm of data crunching because of these lower level concurrency patterns?

nathantech200522:04:06

I was reading that book. Clojure, Brave and True.

nathantech200522:04:42

I installed Spacemacs because I knew VIM. And I really don’t want to go back to Jetbrains since that.

nathantech200522:04:10

I will try to write that dinner problem in Clojure… Right now I am comparing Elixir to Vert.x

lady3janepl22:04:13

I think I will bow out; I’m not a specialist on concurrency / threaded programming enough to compare languages and say which one is better 🙂

nathantech200522:04:29

I am either going to put Clojure on a message bus… or use Clojure inside Vert.x

lennart.buit22:04:53

yeah, same. I have a theoretical basis because of my CS degree, but I wouldn’t know the ins and outs of the concurrency model used in various languages

nathantech200522:04:15

Elixir sticks to the Actor Model, message passing

nathantech200522:04:25

no shared memory, no data crunching

nathantech200522:04:05

but Clojure can give you lower level concurrency, shared memory and also use actor model

lady3janepl22:04:20

You’ve got 3 different models in Clojure, plus anything you can steal from Java, plus lazy evaluation which is kinda generators painted over, and generators are kinda coroutines painted over if you squint 😉

nathantech200522:04:13

yea, seems like Clojure has the most robust concurrency solutions

lady3janepl22:04:04

Anyway, I’m off to sleep b/c it’s near midnight here. Thanks for bringing this up, it reminded me I need to explore this area in Clojure more 🙂

nathantech200522:04:04

the only reason I’d use Elixir is because it can do micro services (umbrella) without the headaches

nathantech200522:04:28

yes, thank you @ I learn a lot here

lady3janepl22:04:02

(if you’re wondering about what language to choose for a project, my advice would be: don’t look at the language, look at the libraries/ecosystem.)

nathantech200522:04:21

Clojure/JVM rules in my opinion for that

lennart.buit22:04:41

the JVM rocks!

nathantech200522:04:49

My new stack probably will be Vue JS, Vert.x and Clojure

nathantech200522:04:18

ya, Internet of Things needs the JVM too

lady3janepl22:04:29

oh yeah, also this is an interesting library in development: https://github.com/reborg/parallel

nathantech200522:04:32

Too difficult to write native code for so many different chips

lady3janepl22:04:45

and with that I’m truly off 😄 o/

nathantech200522:04:45

Ahh, reading it now. The first thought is… actor model alleviates the pausing garbage collection problem because the entire actor’s memory is freed up.

seancorfield22:04:27

Pausing is much less of a problem with the G1 collector than some of the previous collectors -- and GC is always always improving in the JVM.

nathantech200522:04:13

Wow, I did’t know that. I haven’t looked at Java in 5 years. This is super good.

nathantech200522:04:42

I think 1 use case for no pausing GC is high frequency trading algorithms on the NYC exchanges

seancorfield22:04:03

Yeah, there are some people who think that HFT and Oracle's JVM are not compatible. The makers of the Azul JVM claim that theirs is suitable for HFT.

seancorfield22:04:45

(I was just reading a Computer World UK article about that from six years ago)

nathantech200522:04:53

I am thinking of how an email client will thread a bunch of sent emails when I hear green threads?

nathantech200522:04:37

green threads really sound like concurrent operations?

lady3janepl22:04:02

see also: fibres, coroutines

hobosarefriends22:04:13

oh, there’s a lot of these.

nathantech200522:04:50

1 more major question. Has anyone tweaked the JVM for Jython?

paulocuneo22:04:29

whatever you do leave some ram for off-heap memory. JVM store "the objects" in "heap", but it also needs off-heap memory

nathantech200522:04:17

off heap memory is for other operating system work?

paulocuneo22:04:01

JVM store the "class-metadata" off-heap, also needs off-heap for the GC operation

nathantech200522:04:30

so the GC can do its analysis

nathantech200522:04:44

e.g. walking down tree structures

nathantech200522:04:10

I need to learn when the heap is used versus the stack

paulocuneo22:04:13

also JVM does JIT, so needs off-heap memory for code cache

nathantech200522:04:53

want to know a very interesting comparison of Clojure versus Elixir data type for lists?

paulocuneo22:04:30

sure, tell me

nathantech200522:04:03

Clojure lists use tries, but Elixir lists use linked lists

nathantech200522:04:25

tries are fast with reads, linked lists are fast with writes

paulocuneo22:04:56

linked list dont do so well with cache-line

nathantech200522:04:11

what is cache line?

paulocuneo22:04:34

the procesors core has a cache memory

nathantech200522:04:48

oh, yes, that cache. gotcha

nathantech200522:04:05

ok, so then Clojure would use l1, l2 better than Elixir

paulocuneo22:04:07

array usually fits in cache-block

nathantech200522:04:29

hmm, I wonder if tries (trees) work well with cpu cache

paulocuneo22:04:31

will only know by measuring

nathantech200522:04:53

it seems to me, Elixir is more geared towards write fast, read slow

nathantech200522:04:06

and Clojure is geared towards read fast, share memory

paulocuneo22:04:21

mmm I think clojure is geared towards ease

nathantech200522:04:33

I am leaning towards putting Clojure on top of Vert.x if my proof of concept shows me I can scale it across machines

nathantech200522:04:02

I want to find out if I can run Jython and Clojure side by side. And not have to use Cython. I want to utilize JVM for my entire stack.

lennart.buit22:04:07

you may be interested in Graal and its python implementation

lennart.buit22:04:23

but Graal is pretty … experimental

nathantech200522:04:28

well, this would be for machine learning… not user facing

lennart.buit22:04:07

yes, but Graal is an experimental polyglot VM that allows you to run jvm languages and python intertwined. That said, it is limited in some senses. For one, it disallows some forms of reflection

lennart.buit22:04:11

so, you may end up having some difficulties because of that

lennart.buit22:04:42

and thats why I was warning, its early — but really cool — technology

nathantech200523:04:30

Ahh ok. Java reflections. Is Java Generics based on reflections?

lennart.buit23:04:16

no, generics are really only a concept in the Java compiler. As soon as your program is compiled, they are gone

nathantech200523:04:24

I should try a performance test with Graal versus Cython

lennart.buit23:04:25

That would be interesting indeed

nathantech200523:04:36

To give you an idea…

nathantech200523:04:20

Cython tree recursion was as fast as Clojure lazy eval. But I don’t know if Cython did some under the hood compile optimization.

nathantech200523:04:46

Both Cython and Clojure took 1 second to find the 40th nth term in fibonacci.

nathantech200523:04:02

Python took 1 minute 30 seconds

seancorfield22:04:57

You can certainly run multiple languages side by side on the JVM. Sometimes the interop between them is awkward tho', depending on how they compile their constructs for the JVM.

seancorfield22:04:01

For example, some aspects of Scala can be difficult to call from other languages.

nathantech200522:04:19

would you advise against calling languages inside other languages? And just use a message bus instead?

seancorfield22:04:00

It depends on the languages involved and the JVM-level APIs they expose in terms of data structures etc.

nathantech200522:04:27

I did have experience with Jruby/Ruby, foreign language interfaces…

nathantech200522:04:57

Writing the code was the easy part. But supporting Jruby on the server was a headache

nathantech200522:04:32

It doesn’t work with monoliths because every single library needs to be thread safe in Jruby… I am pretty sure

nathantech200523:04:33

Ok, thank you everyone, for your help @seancorfield @paulocuneo @lennart.buit I will draw an architecture diagram this week and share it.