Fork me on GitHub
#onyx
<
2016-05-31
>
michaeldrogalis02:05:04

Patch for reducing number of ZooKeeper connections to 1 per machine is passing the full test suite. Moving into code review and Jepsen testing this week, hopefully releasing early next week. I'll make a blog post to talk away why this is a big step forward for scalability, since the patch also contains more detailed changes about how we handle local replica manipulation.

jeroenvandijk07:05:06

Does it apply for all zookeeper connections or just the ones that are required for Onyx internals. I.e. we have 25 kafka partitions requiring 1 virtual peer per partition, and thus 1 zookeeper connection per partition. Can this connection also be shared? It will probably be a different Zookeeper cluster than the Onyx Zookeeper cluster

lucasbradstreet07:05:15

I think you will end up with one ZK connection per kafka peer, however I think that generally the Kafka peers don’t need to keep the connection open (this may still be a problem at startup however)

jeroenvandijk09:05:02

@lucasbradstreet: Ok cool, will give it a try when it’s ready 🙂

lucasbradstreet09:05:59

We'll see how long it takes to pass jepsen. It's already found one issue :)

bcambel11:05:54

is the first task should always be a onyx/type :input ? Is there a way to run a :function to start the job ?

lucasbradstreet11:05:16

You can put a function task on an output node, but not an input node

lucasbradstreet11:05:01

Unfortunately it depends on a bunch of protocol functions being defined on the task. If you put a function on the input node you’ll end up with it trying to read segments from the messenger

lucasbradstreet11:05:05

onyx-seq is the go to here

lucasbradstreet11:05:35

That thing is a swiss army knife

bcambel11:05:39

well the reason was, I was going to download a s3 file to local and then pass it to the onyx-seq

lucasbradstreet11:05:01

Can’t you just do that in the lifecycle function?

bcambel11:05:21

hmm that's a possibility, so before the task start, run this

drewverlee13:05:03

@lucasbradstreet or @michaeldrogalis. There is a podcast called “Software Engineering daily” hosted by jeff meyerson, which I have really enjoyed the last 4 months. There seems to be a heavy focus on distributed systems. On a whim, I asked him if he would he thought a show on Onyx would work and he seemed excited about the idea. So obviously no pressure, but if the team wants to investigate that option he seems open to it.

lucasbradstreet13:05:03

I like and listen to it. He really manages to pump them out (I guess that’s the daily part)!

aspra15:05:19

I have a question about exception handling

aspra15:05:22

When we want to kill the onyx job in case of an exception, should the handle exception fn return false or :kill?

lucasbradstreet15:05:52

It should be kill

lucasbradstreet15:05:59

What do you mean by different versions?

aspra15:05:30

The second link says " This function must return true or false"

aspra15:05:53

so I got a bit confused 🙂

lucasbradstreet15:05:05

Oh, sorry, I didn’t see the second link. The second doc is out of date

lucasbradstreet15:05:28

cheat sheet is almost always right. We really need to get the docs generated from the information model / cheat sheet info

lucasbradstreet15:05:36

I’ll fix that up. Thanks!

aspra15:05:50

ah right, ok thanks for clarifying!

lucasbradstreet15:05:56

(almost always more correct than the main docs I should say :P)

michaeldrogalis15:05:59

@drewverlee: Thanks for the heads up!

aaelony18:05:13

Hi - fyi, I'm going through the README.md steps generated by lein new onyx-app my-app-name -- +docker +metrics. The ./script/build step worked fine and the docker-compose up command ran for a while before complaining

ERROR: Service 'kafkacat' failed to build: The command '/bin/sh -c apt-get update -y &&     apt-get install $BUILD_PACKAGES -y &&     git clone  &&     cd kafkacat &&     ./bootstrap.sh &&     make install &&     cd .. && rm -rf kafkacat &&     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*' returned a non-zero code: 100 
But re-running docker-compose up seems to have corrected things and gone past this error.

aaelony18:05:37

Now, I'm at the

mysql -h $(echo $DOCKER_HOST|cut -d ':' -f 2|sed "s/\/\///g") -P3306 -uroot
phase

aaelony18:05:53

which gives

Can't connect to MySQL server on

aaelony18:05:19

... is there a way to unblock the connection? (and perhaps add this to the excellent docs?)

gardnervickers18:05:19

Are you on Linux or OS X?

aaelony18:05:30

el capitan

gardnervickers18:05:32

And you can see mysql running when you docker ps?

aaelony18:05:54

mysql is not running in docker ps

gardnervickers18:05:17

when you look at the output of docker-compose up, do you see mysql starting up?

aaelony18:05:08

yeah, the mysql apparently doesn't start up correctly or exits for some reason

gardnervickers18:05:45

Could you publish the log lines where that’s happening?

aaelony18:05:27

wait ... now it's working??

aaelony18:05:31

what's the difference between docker-compose exec db mysql -uroot and mysql -h $(echo $DOCKER_HOST|cut -d ':' -f 2|sed "s/\/\///g") -P3306 -uroot

aaelony18:05:55

pretty cool indeed 🙂

gardnervickers18:05:26

One’s on your OS, one’s run inside the container I would think?

gardnervickers18:05:56

docker-compose exec runs inside the mysql container

aaelony18:05:59

yeah, I think it suddenly started working... unclear why, but nice!

gardnervickers18:05:12

It probably just needed to bootstrap

aaelony18:05:27

perhaps I was impatient

gardnervickers18:05:44

If you don’t see it explicitly exiting or throwing errors then it’s usually just still working

aaelony18:05:08

👍 really great demo app

aaelony18:05:32

sorry, one last question... I can start things up...

16-May-31 11:52:13 Avram-Aelonys-MacBook-Pro.local INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-May-31 11:52:13 Avram-Aelonys-MacBook-Pro.local INFO [onyx.log.zookeeper] - Stopping ZooKeeper client connection
Submitted job:  #uuid "5a2c8f5f-220e-4883-8b77-6bf006021200" 
but checking the mysql table I don't see the data captured to recentMeetups What am I doing wrong?

aaelony22:05:21

works now.. need to understand better why it fails at times then works at other times

gardnervickers22:05:51

Did you restart docker machine?

aaelony23:05:57

at this point yes, and shut it down.. I'm new to docker so still have yet to try all the combinations

gardnervickers23:05:42

There’s a problem with docker-machine i’ve run into where the DNS gets messed up and you need to restart the docker-machine

gardnervickers23:05:02

happens especially when I sleep my laptop and change networks, docker-machine wont change the DNS server