Fork me on GitHub
#off-topic
<
2019-04-13
>
john04:04:32

Ola Bini got arrested in Ecuador

agigao08:04:38

https://www.imf.org/en/News/Articles/2019/03/11/ecuador-pr1972-imf-executive-board-approves-eff-for-ecuador March 11, 2019 - IMF Executive Board Approves US$4.2 Billion Extended Fund Facility for Ecuador

🙁 4
hipster coder17:04:18

Think that’s linked to Assange?

agigao21:04:34

Who can prove it’s not?

val_waeselynck08:04:56

In Lisbon for a few days, hotel's a bit gloomy. Anyone know a nice place with wi-fi to work?

trevor13:04:14

The Block is quite a nice place to work.. http://theblock.cafe/

👍 4
val_waeselynck14:04:44

Unfortunately Bitcoin's a dealbreaker for me 😞

trevor17:04:10

How so? They accept euros

paulocuneo18:04:09

Working with multiple micro-services, many teams, multiple contexts for the same business entity, and multiple sources for the same entity. It seems like the tower of babel story, we've build something so big that every teams starts to speak its own language. I see that this independence brings success, but it also seems I'm writing structure transformations all the time. I've also seen this happen at my previous job, and also many coworkers have seen this before at different scales. ¿Have you deal with these "tower of babel integration" problem? ¿Did you enforce a common API across all teams, or did you solve it by automating the transformations, or making a catalog? ¿Or do you just code the transformations?

gklijs19:04:45

How I think it would work best is all using Kafka for messages, with the same 'type' of massages. For example strings as keys, and values avro encoded with schema's. And on each topic there is only one producer. Of all JVM you could use specific records if you want. If you also makes sure the schema's are always backwards compatible I think most things are tackled. Depends a bit on what the product is, but towards the front-end I would expose multiple topics with GraphQL.

richiardiandrea20:04:20

This is a big topic for me also, another question I had was: do the microservices need to be written in the same language? I would say of course not but many Enterprise companies are worried that it would be a mess recruiting people...dunno

mjw21:04:20

What do you mean by “multiple sources for the same entity”? Different teams are duplicating work so there are multiple models for the exact same data?

mjw21:04:57

@U0C8489U6 I think it’s fine for microservices to use different languages, but for a number of reasons I advocate that companies limit “approved” languages to two or three with very few exceptions.

hipster coder21:04:00

Could you please describe a specific example of a structure transformation? I think using Message Brokers like RabbitMQ and ZeroMQ can really help reduce duplicate code. And you could even use foreign language interfaces… e.g. how Clojure can call other languages inside of it. I am researching Vert.x because I could write it in Clojure, Python/Jython or even Jruby/Ruby. You can compare and contrast that to Elixir Umbrella Projects that try to combine monolithic with micro services.

paulocuneo22:04:53

Kafka: I guess a big enterprise bus is one way to solve it, or at least for a subsystem. Polyglot: multiple languages is not an issue.(Just use one lang for service) I guess the issue I tried to mention is being a mediator between multiple "bounded-contexts", that's not exactly model duplication, but thats the part where we start to speak different tongues

hipster coder22:04:15

You might try Vert.x because it builds a message bus into the tool. They built it for providing micro services (reactive programming) and polyglots. It supports most of the JVM languages. But I need to do a proof of concept… to make sure it’s not another callback hell async problem. It has a promises/futures library to fix that, I believe.

paulocuneo23:04:44

you really love Vert.x 😁

hipster coder23:04:31

I love the idea of it. But I can’t say yet until I put it in production and load test it.

hipster coder23:04:57

It’s also not the easiest to setup. But sometimes the easier stuff to setup is a headache in the long term.

hipster coder23:04:33

The only disadvantage for me so far with Vert.x is that it doesn’t auto scale as easily as Elixir. But Vert.x has so many more advantages. JVM, libraries, machine learning, foreign language interfaces, actor model + reactive model + blocking thread worker pool… and more

hipster coder23:04:55

Elixir gives you rapid dev with a monolith and umbrella style micro services. But it’s not a general purpose tool for machine learning, cpu number crunching. And Vue JS can give me the rapid monolith advantage.

gklijs05:04:22

From my experience using Kafka is a lot easier then VertX. The Kafka ecosystem is huge making it easy to connect basically whatever you want. If you have a legacy service you still need, you can create events from the changes to the the database. Also you have access to almost all languages for clients thanks to librkafka. But maybe most importantly Kafka allows parallel async programming as a whole, with execution of 'simple' synchronous code.

hipster coder21:04:19

Can anyone explain to me… if a computer has 8 cores, it can have 16 threads (if hyper threading, intel). But why do I see code setting up 100's of threads? Are the threads virtualized and pooled? And there is a difference between actual real world parallel threads versus pooled threads?

Lennart Buit21:04:24

Yes they are, its the task of your operating system to schedule threads and processes on the CPU

Lennart Buit21:04:46

but the story is not often as simple

hipster coder21:04:13

Ahh ok. So it’s a virtualized thread count. They CPU interleaves them (schedules them) on the actual hardware threads?

Lennart Buit21:04:42

Well, what your operating system is concerned, a octa-core with HT is actually just 16 cores

Lennart Buit21:04:03

it doesn’t “know” that half of them are not physical cores

hipster coder21:04:31

ahh, the hyper threading makes it look like 2 cores from the operating system’s perspective?

hipster coder21:04:11

I am researching Clojure threading primitives in comparison to just the actor model. e.g. clojure agents

paulocuneo22:04:38

from what I remember, JVM will map each thread to an O.S. managed thread.

paulocuneo22:04:23

threads are "abstraction" for "concurrency", but we you go down the hardware there is a limit to what is actually done in "parallel"

Ivan Koz07:04:51

JVM will introduce (cooperative) fibers\\continuations in near future, which is a lightweight replacement for (pre-emptive) threads, it will allow JVM to manage async tasks in more efficient way, mapping hundreds of thousands fibers to a thread pool matched to real cores in size.

Jorin13:04:17

@UH16CGZC2 so something similar to what Go or Erlang does? That sounds amazing! What I was able to find was "Project Loom". Very curious when this will be available in a stable JVM release 🙂

Ivan Koz14:04:08

@U8ZN5EHGU correct, in a year or two, also probably tail call recursion, since they changing how stack frames being handled

Jorin14:04:48

Can't wait! Also curious how this will be integrated into Clojure since Clojure still needs to support older JVM versions..

Ivan Koz14:04:35

¯\(ツ)

hipster coder18:04:18

Java Fibers will be similar to Actors in Erlang where no locking is needed? I am not sure how Golang fits under the Actor model since it’s really CSP (Communicating Sequential Process) and actually relies heavily on locking/mutexes?

hipster coder18:04:43

Reading this… it seems Java Fibers are the equivalent of actors with async ability?

Ivan Koz03:04:11

@UGQJZ4QNR java fibers is green threads scheduled by JVM with ability to pause and continue by saving fiber stack on heap

hipster coder03:04:20

ahhh ok. there’s that “green threads” word again. They are virtual threads

Ivan Koz03:04:05

@UGQJZ4QNR pretty much, regular Thread will become lower level of abstraction rather for continuous calculations

hipster coder03:04:34

I am researching if Vert.x can take advantage of fibers

hipster coder03:04:33

wow. Vert Sync allows for async non blocking calls but with a syncronous style, no callback hell. I am reading dzone website.

hipster coder03:04:06

It says Vert Sync uses Quasar for Fibers

hipster coder04:04:17

it says the Fiber is blocked… but the underlieing thread is not blocked. So you get the best of both worlds. And it doesn’t turn your code into spaghetti callback hell

3Jane21:04:15

there’s threads and there’s threads. https://en.wikipedia.org/wiki/Green_threads

💯 4
Lennart Buit21:04:44

yeah, I was just about to bring up Green threads

jaihindhreddy21:04:13

And don't forget, there's also threads (Intel Hyperthreading) 😂

Lennart Buit21:04:16

but generally with green threads, its your app, or host platform (f.e. the JVM) that does the scheduling

3Jane21:04:16

there should be a guide for beginners… “Learning the ropes of threading”

8
Lennart Buit21:04:25

instead of the operating system’s scheduler

jaihindhreddy21:04:11

^^ I'm gonna steal that if you don't mind.

3Jane21:04:29

y/w

👏 4
Lennart Buit21:04:46

stealing is not stealing if you ask 😛

3Jane21:04:14

let’s not discriminate against British/Canadian thieves!

3Jane21:04:48

they can’t help being polite…

3Jane21:04:33

@jaihindhreddy ping me a link if you end up writing a threads overview 🙂

hipster coder22:04:31

what are green threads? virtual threads?

Lennart Buit22:04:23

threads that don’t map directly to threads in your operating system

hipster coder22:04:46

hmm. like software level threads?

hipster coder22:04:43

I am curious because this weekend is my concurrency/threading study time. I need to write a deadlock on Clojure to learn how to simulate it.

Lennart Buit22:04:46

(lol, google is showing me an image of a green ball of yarn when looking for “green threads”)

3Jane22:04:41

you’ve been yarn bombed

Lennart Buit22:04:07

(I think I fell out of my filter bubble)

Lennart Buit22:04:17

(please send help)

3Jane22:04:45

as I understand it, it’s your programming environment pretending threads exist

Lennart Buit22:04:13

yeah this exactly

hipster coder22:04:00

thanks for threading this

👌 4
Lennart Buit22:04:21

yeah, gotta keep the concurrency going

hipster coder22:04:21

we are in a green thread now

Lennart Buit22:04:45

its not proper multithreading unless people are majorly confused 😉

3Jane22:04:57

is green thread basically a more ecological way of threading then?

😂 8
Lennart Buit22:04:50

a deadlock occurs when two threads are waiting on eachother

Lennart Buit22:04:27

huh where did the question go

hipster coder22:04:30

Clojure can still deadlock? It just happens less often in Clojure?

hipster coder22:04:42

Sorry, I deleted it. I didn’t want to derail Jane

Lennart Buit22:04:53

well, everything can deadlock, its just that Clojure has a mitigation strategy when sharing memory between threads

hipster coder22:04:02

I am asking because I am trying to figure out if I need lower level threading like Clojure provides. Lower level than actor model.

Lennart Buit22:04:22

(its actually funny that you deleted the question, you kinda illustrated the problem that threaded execution can have, if you are not careful the “world” can change under your feet)

😆 4
3Jane22:04:35

also idk what you’re using to learn Clojure, but http://shop.oreilly.com/product/0636920013754.do has a really (insert punhusky here) thorough chapter on concurrency

hipster coder22:04:42

My conclusions, faulty as they can be, is that Clojure rules the realm of data crunching because of these lower level concurrency patterns?

hipster coder22:04:06

I was reading that book. Clojure, Brave and True.

hipster coder22:04:42

I installed Spacemacs because I knew VIM. And I really don’t want to go back to Jetbrains since that.

hipster coder22:04:10

I will try to write that dinner problem in Clojure… Right now I am comparing Elixir to Vert.x

3Jane22:04:13

I think I will bow out; I’m not a specialist on concurrency / threaded programming enough to compare languages and say which one is better 🙂

hipster coder22:04:29

I am either going to put Clojure on a message bus… or use Clojure inside Vert.x

Lennart Buit22:04:53

yeah, same. I have a theoretical basis because of my CS degree, but I wouldn’t know the ins and outs of the concurrency model used in various languages

hipster coder22:04:15

Elixir sticks to the Actor Model, message passing

hipster coder22:04:25

no shared memory, no data crunching

hipster coder22:04:05

but Clojure can give you lower level concurrency, shared memory and also use actor model

3Jane22:04:20

You’ve got 3 different models in Clojure, plus anything you can steal from Java, plus lazy evaluation which is kinda generators painted over, and generators are kinda coroutines painted over if you squint 😉

hipster coder22:04:13

yea, seems like Clojure has the most robust concurrency solutions

3Jane22:04:04

Anyway, I’m off to sleep b/c it’s near midnight here. Thanks for bringing this up, it reminded me I need to explore this area in Clojure more 🙂

😴 4
hipster coder22:04:04

the only reason I’d use Elixir is because it can do micro services (umbrella) without the headaches

hipster coder22:04:28

yes, thank you @U82DUDVMH I learn a lot here

3Jane22:04:02

(if you’re wondering about what language to choose for a project, my advice would be: don’t look at the language, look at the libraries/ecosystem.)

hipster coder22:04:21

Clojure/JVM rules in my opinion for that

Lennart Buit22:04:41

the JVM rocks!

hipster coder22:04:49

My new stack probably will be Vue JS, Vert.x and Clojure

3Jane22:04:51

seriously

hipster coder22:04:18

ya, Internet of Things needs the JVM too

3Jane22:04:29

oh yeah, also this is an interesting library in development: https://github.com/reborg/parallel

hipster coder22:04:32

Too difficult to write native code for so many different chips

3Jane22:04:45

and with that I’m truly off 😄 o/

hipster coder22:04:45

Ahh, reading it now. The first thought is… actor model alleviates the pausing garbage collection problem because the entire actor’s memory is freed up.

seancorfield22:04:27

Pausing is much less of a problem with the G1 collector than some of the previous collectors -- and GC is always always improving in the JVM.

👍 4
hipster coder22:04:13

Wow, I did’t know that. I haven’t looked at Java in 5 years. This is super good.

hipster coder22:04:42

I think 1 use case for no pausing GC is high frequency trading algorithms on the NYC exchanges

seancorfield22:04:03

Yeah, there are some people who think that HFT and Oracle's JVM are not compatible. The makers of the Azul JVM claim that theirs is suitable for HFT.

seancorfield22:04:45

(I was just reading a Computer World UK article about that from six years ago)

hipster coder22:04:53

I am thinking of how an email client will thread a bunch of sent emails when I hear green threads?

hipster coder22:04:37

green threads really sound like concurrent operations?

3Jane22:04:02

see also: fibres, coroutines

Mno22:04:13

oh, there’s a lot of these.

hipster coder22:04:50

1 more major question. Has anyone tweaked the JVM for Jython?

paulocuneo22:04:29

whatever you do leave some ram for off-heap memory. JVM store "the objects" in "heap", but it also needs off-heap memory

hipster coder22:04:17

off heap memory is for other operating system work?

paulocuneo22:04:01

JVM store the "class-metadata" off-heap, also needs off-heap for the GC operation

hipster coder22:04:30

so the GC can do its analysis

hipster coder22:04:44

e.g. walking down tree structures

hipster coder22:04:10

I need to learn when the heap is used versus the stack

paulocuneo22:04:13

also JVM does JIT, so needs off-heap memory for code cache

hipster coder22:04:53

want to know a very interesting comparison of Clojure versus Elixir data type for lists?

paulocuneo22:04:30

sure, tell me

hipster coder22:04:03

Clojure lists use tries, but Elixir lists use linked lists

hipster coder22:04:25

tries are fast with reads, linked lists are fast with writes

paulocuneo22:04:56

linked list dont do so well with cache-line

hipster coder22:04:11

what is cache line?

paulocuneo22:04:34

the procesors core has a cache memory

hipster coder22:04:48

oh, yes, that cache. gotcha

hipster coder22:04:05

ok, so then Clojure would use l1, l2 better than Elixir

paulocuneo22:04:07

array usually fits in cache-block

hipster coder22:04:29

hmm, I wonder if tries (trees) work well with cpu cache

paulocuneo22:04:31

will only know by measuring

hipster coder22:04:53

it seems to me, Elixir is more geared towards write fast, read slow

hipster coder22:04:06

and Clojure is geared towards read fast, share memory

paulocuneo22:04:21

mmm I think clojure is geared towards ease

hipster coder22:04:33

I am leaning towards putting Clojure on top of Vert.x if my proof of concept shows me I can scale it across machines

hipster coder22:04:02

I want to find out if I can run Jython and Clojure side by side. And not have to use Cython. I want to utilize JVM for my entire stack.

Lennart Buit22:04:07

you may be interested in Graal and its python implementation

Lennart Buit22:04:23

but Graal is pretty … experimental

hipster coder22:04:28

well, this would be for machine learning… not user facing

Lennart Buit22:04:07

yes, but Graal is an experimental polyglot VM that allows you to run jvm languages and python intertwined. That said, it is limited in some senses. For one, it disallows some forms of reflection

Lennart Buit22:04:11

so, you may end up having some difficulties because of that

Lennart Buit22:04:42

and thats why I was warning, its early — but really cool — technology

hipster coder23:04:30

Ahh ok. Java reflections. Is Java Generics based on reflections?

Lennart Buit23:04:16

no, generics are really only a concept in the Java compiler. As soon as your program is compiled, they are gone

hipster coder23:04:24

I should try a performance test with Graal versus Cython

Lennart Buit23:04:25

That would be interesting indeed

hipster coder23:04:36

To give you an idea…

hipster coder23:04:20

Cython tree recursion was as fast as Clojure lazy eval. But I don’t know if Cython did some under the hood compile optimization.

hipster coder23:04:46

Both Cython and Clojure took 1 second to find the 40th nth term in fibonacci.

hipster coder23:04:02

Python took 1 minute 30 seconds

seancorfield22:04:57

You can certainly run multiple languages side by side on the JVM. Sometimes the interop between them is awkward tho', depending on how they compile their constructs for the JVM.

seancorfield22:04:01

For example, some aspects of Scala can be difficult to call from other languages.

hipster coder22:04:19

would you advise against calling languages inside other languages? And just use a message bus instead?

seancorfield22:04:00

It depends on the languages involved and the JVM-level APIs they expose in terms of data structures etc.

hipster coder22:04:27

I did have experience with Jruby/Ruby, foreign language interfaces…

hipster coder22:04:57

Writing the code was the easy part. But supporting Jruby on the server was a headache

hipster coder22:04:32

It doesn’t work with monoliths because every single library needs to be thread safe in Jruby… I am pretty sure

hipster coder23:04:33

Ok, thank you everyone, for your help @seancorfield @paulocuneo @lennart.buit I will draw an architecture diagram this week and share it.

👍 12