This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (7)
- # boot (39)
- # braid-chat (3)
- # braveandtrue (1)
- # cider (27)
- # cljsjs (15)
- # cljsrn (6)
- # clojars (18)
- # clojure (307)
- # clojure-art (1)
- # clojure-brasil (1)
- # clojure-italy (2)
- # clojure-poland (3)
- # clojure-russia (61)
- # clojure-sdn (2)
- # clojure-taiwan (4)
- # clojure-uk (5)
- # clojurebridge (7)
- # clojurescript (19)
- # core-async (1)
- # core-matrix (1)
- # cursive (35)
- # datomic (3)
- # emacs (51)
- # euroclojure (3)
- # hoplon (20)
- # jobs (1)
- # keechma (1)
- # mount (3)
- # off-topic (2)
- # om (177)
- # onyx (96)
- # parinfer (4)
- # pedestal (4)
- # re-frame (19)
- # reagent (7)
- # untangled (5)
@michaeldrogalis @gardnervickers: I'd like to embed the onyx version number into our jars somehow. Doing it properly may be a little tricky. The goal is two fold. Firstly we should write the version when peers join, like we do with the job scheduler. This would mean that the dashboard can play assert that the correct dashboard version is used. Secondly, I want to link to the correct version of our docs for help with some errors.
The dashboard playback will be a tricky one btw. At some point I think we're going to need to be able to dynamically play back against the correct version of our peer code. Otherwise it's going to be annoying when you roll out against a new version and lose the ability to load a tenancy id in the dashboard.
That said, I'd rather throw an error than potentially show an incorrect replica.
Maybe we should try to use boot pods with the dashboard https://github.com/boot-clj/boot/wiki/Pods
For the frontend stuff boot seems to make more sense. I’ll have to read up on pods though
As for versioning aspect, yup, seems like there’s probably away. I just don’t know what it is!
Then it’s like
(.. x .getClass .getPackage .getVersion) or something like that
I think we’ll need to ensure that the minor version matches too, because the peer has to play back exactly, meaning it would be too easy to make breaking changes fixing bugs.
Really we should be checking whether we’re running on the same clojure version too
@gardnervickers @michaeldrogalis: new PR up https://github.com/onyx-platform/onyx/pull/555. This one adds the event map, and trigger events to the information model, and auto generates schemas for them from the information model. The latter part is a bit of a work in progress but seems to work pretty well so far.
One thing I think the cheat sheet needs now is to link from different parts, like when we describe function arguments
This is better than what I was doing with generating the information_model from the schema
Yeah, I hope I didn’t step on your toes. I just had to work on the event-map and state-event information models and thought I might just give the schema generation a try once they were there
It’ll be nice for the schemas and the information models to always be in sync
Alright cool I’ll give this a shot with the plugins later tonight. Working on the template stuff right now
If anyone has any suggestions of interesting stuff to do with a twitter stream besides wordcount i’m all ears
windowing/triggers is probably the best thing. If I think of any flow conditions ideas I’ll let you know
Flow conditions to sort country code and aggregation to figure out what country is using the most emoji's
aggregation kinda directs the country code stuff already. I was thinking maybe you only want the US and maybe another country, then group-by-key to group on something else?
@lucasbradstreet: perhaps this is useful in version listing if you haven't considered it already... https://github.com/pallet/lein-sha-version/blob/develop/README.md
I think that it doesn’t quite do what we want it to do, which is we still want regular version numbers, but we want for them to be set somewhere in the code where it can be looked up
Looks like it’s just read from the properties file, which is probably set by maven https://github.com/clojure/clojure/blob/master/src/resources/clojure/version.properties
I think we need to either be able to get at the pom.xml somehow (either through the jar or otherwise), or just set it separately in a def that we substitute via our release process
at least useful for ideas.. maybe the git sha is a lookup for the release version (or something)
Agreed. I think this might be a better fit https://github.com/trptcolin/versioneer
(defn get-version [dep] (let [path (str "META-INF/maven/" (or (namespace dep) (name dep)) "/" (name dep) "/pom.properties") props (io/resource path)] (when props (with-open [stream (io/input-stream props)] (let [props (doto (Properties.) (.load stream))] (.getProperty props "version")))))) (get-version 'org.onyxplatform/onyx)
Choosing the EPL for our license is kinda cargo culting license choices, but it really is nice to be able to use EPL code directly
Seems to work from within the dashboard (version/get-version "org.onyxplatform" "onyx")
pods will be pretty key if we want to use a different version for each tenancy we load
We could probably build that part outselves, but boot does nice things like manage pools of pods for reuse
Lucas, it is most expected plugin for me! I'm an bigdata architect in a bank and software stack of our division based on Hadoop technologies. So if HDFS plugin will be available, I would start test Onyx in our lab.
If I can help with HDFS plugin I can spent every holidays to work with Onyx team to help in implementing this plugin.
I assume you’re interested in batch jobs? Do you want to operate over a single file or multiple files per job?
I can definitely see why someone would want to be able to read HDFS data in Onyx, I’m just trying to figure out a little more about how it should be organised
https://github.com/apache/storm/tree/master/external/storm-hdfs#hdfs-spout this documents how Storm’s hdfs spout works. Maybe you could give me your thoughts
I don’t think we necessarily need to follow what they do, but it might be useful to hear your thoughts
Looking at the README, I don’t really like how they move files around to signal whether the file is “done"
Well, my interest goes beyond... I'm a full stack Clojure developer, but unfortunately my current team is a Scala guys. Scala is a mian stream but I like Clojure. My goal is to create Clojure team in our bank, but there are not so much Big data tools. I believe that Onyx can help me solve Bigdata tasks (that Spark does). I've seen Clojure tools like Sparkling, Pigpen... but some internal (intuitive) thoughts tell me that Onyx will be a real competitor of Spark. So I'm expecting HDFS plugin...
TypicalHDFS task for our team: batch job for processing tsv/csv or json files from different bank sources. We must solve various analytical tasks about our clients based on internal and external data sources. We have streaming data like click stream from site of our bank, and we have offline sources ( db replica's). I've tried to use Spark from Clojure but unfortunately all power of Spark is available from java or Scala.
@lucasbradstreet: @gardnervickers Re: embedded version in the jar. I believe the standard way to do this is to create a text file in the resources directory. In
project.clj, you do
(defproject blah ~(read-string (slurp "version.txt")))
@michaeldrogalis: that's fine too, though the above library works fine by reading the maven data we need. I'm mostly trying to avoid modifying our release plugin but either way it's fine
We'll chat about it at stand up tomorrow, I'm indifferent as to how this gets done.
Textfile in resources + modifying the release script is probably the way to go then, but we'll talk about it tomorrow
Cool, I think cheat sheet just needs to incorporate the docs once that's in. That plus updating the triggers doc will mean we're sufficiently doc'd for 0.9.0
Which issue are you referring to? Jepsen was testing fine after modifications. Running the new job-complete trigger signal overnight
Ah k. I think it was mostly just getting it into shape with 0.9.0 and making sure we pass before release