Links to papers and images on a changelog to explain a new implementation. You all are the best! 👏

@andrewhr Thanks! Much of the main docs still need to be revamped before the release, so that README will have to do for now

Let us know if there’s anything confusing in it

Morning guys.

We're testing onyx a bit at the workplace and we're running into the persistent bookkeeper cookie in zk issue. We are using docker containers to wrap our onyx workers and are a bit unsure what would constitute a "clean shutdown".

As our expectations was that stopping the container should lead to cookie removal, if the delete-server-data? was set to true.

Has anybody run into this recently?

I assume you're running ZK outside your containers @boyanb?

The delete-server-data option sort of assumes you're running in-memory zookeeper, where ZK is created anew for each run. BookKeeper stores data in ZK with an TTL. When BK is killed and started in quick succession, it see's the old data in ZK and assumes there is already a BK bookie running for that "identity".

It's part of how BK maintains consistency in the face of node failure. Unfortunately that makes it difficult to provide a generic solution for the various development/testing configurations folks use.

If you are using something like docker-compose for this, I suggest just deleting and remaking Zookeeper every time you delete and remake your Onyx cluster. Another option is to run BookKeeper in its own docker container, so it won't rapidly restart with your Onyx peers.

Has anyone tried consuming gzip files as messages through the Kafka-8 plugin? I’m deserializing with what I’d expect to work

(-> bytes
ByteArrayInputStream.
GZIPInputStream.

line-seq
vec)

Thanks gardner, indeed, we are running zk "semi-persistent". It's provisioned outside of the onyx containers.

We are using ansible, and yes, the solution we came up with is removing the container at each provisioning.

but we wanted to make sure that there isn't a way to shut down in a graceful way, that would clean up the zk state.

Ahh ok. Onyx namespaces clusters in ZooKeeper by the :onyx/tenancy-id. You could re-deploy your Onyx peers with a different :onyx/tenancy-id to achieve the same effect as clearing the ZK state.

The zk instance is serving only the onyx workers in this case, so dropping and recreating is not an issue.

Fantastic, that’s the best option IMO.

Thanks again.

@boyanb 0.10 drops BookKeeper and RocksDB for what it’s worth, so whatever can get you around the problem for the short term is a good solution.

Really excited to start trying out .10

I am making dashboard new UI

@mariusz_jachimowicz Nice 🙂