This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-06-08
Channels
- # aleph (52)
- # beginners (74)
- # boot (8)
- # cider (4)
- # clara (3)
- # cljs-dev (1)
- # cljsjs (2)
- # cljsrn (1)
- # clojars (2)
- # clojure (300)
- # clojure-argentina (1)
- # clojure-dev (9)
- # clojure-italy (10)
- # clojure-nl (1)
- # clojure-russia (77)
- # clojure-sg (9)
- # clojure-spec (38)
- # clojure-uk (70)
- # clojurescript (108)
- # core-async (12)
- # cursive (9)
- # data-science (4)
- # datascript (7)
- # datomic (37)
- # defnpodcast (4)
- # emacs (11)
- # graphql (6)
- # jobs (3)
- # jobs-discuss (1)
- # juxt (3)
- # keechma (1)
- # klipse (4)
- # lein-figwheel (1)
- # lumo (1)
- # off-topic (3)
- # om (5)
- # onyx (10)
- # parinfer (3)
- # pedestal (1)
- # perun (1)
- # protorepl (3)
- # re-frame (35)
- # reagent (19)
- # spacemacs (4)
- # specter (2)
- # uncomplicate (279)
- # unrepl (32)
@whilo I'd need to see at least a hello-world-ish code of those ideas to be able to form an opinion.
@blueberry http://www.robots.ox.ac.uk/~fwood/anglican/examples/viewer/?worksheet=gaussian-posteriors
I can see that the example takes 20.000 samples of one-dimensional distribution. That's a fairly trivial problem. See Doing Bayesian Data Analysis book - it shows many practical hierarchical models and many examples go into 50+ dimensions. All the examples compute in a fraction of second with bayadera (the same examples run for minutes with Stan, which does all fancy variational + hamiltonian mcmc etc. in C++ called from R). Anyway, for vision problems, which is perception, I do not see how anything can beat deep neural nets...
But you are right, that bayadera is much more focused on performance. For vision this might be very helpful.
I just want to point out that the work around anglican is fairly broad and innovative in regard to language embedding and composability.
I can totally imagine having a highly optimized bayadera model being part of it. As far as I understand the models are composable at the boundaries in general.
the problem is that those computations are so demanding that performance is THE metric. It doesn't matter if anglican can create more complex models if those models run in days. BTW, can you please do a quick microbenchmark and tell me how long does it take anglican to take those 20.000 samples (do not forget to unroll the sequence with doall)
another thing I do not like with anglican's approach is that it does not support regular clojure functions, but does some sort of model compilation that require the model code to be somehow special (i.e. not clojure)
when I talked about hello world I ment the demonstration of those fancy ideas (variational inference, NN, etc.) and the comparison with some baseline wrt. some metrics
I'm not too much into probabilistic programming, I am more interested in probabilistic data analysis, and probabilistic machine learning. these things are based on the same theory, but are not the same.
The problem is not so much whether to do it in anglican or in bayadera, but whether to build on Clojure at all.
I agree about the programming aspect, although I think it is possible to have fast probabilistic machine learning in such a language and optimize inference with it through "compilation".
From that direction anglican might be much more approachable to embed in some small problems, than going full machine learning.
probabilistic techniques might be interesting next layer, that could do some reasoning on the output of the vision layer
btw. next on my reading list: https://arxiv.org/pdf/1703.04977.pdf
MC methods are inefficient in general and not respected much in machine learing (at least from my environment)
This is their newest take: https://arxiv.org/abs/1705.10306
And, that is the Gaussian distribution, which is the easiest distribution to sample from (after the Uniform)
I know. But I don't think bayadera couldn't be described by the anglican syntax and framework. I think the big success of NNs besides initial breakthroughs is mostly due to the fact that modern toolkits allow easy composition and modelling.
Now, Bayadera can take much, much, much, more (I forgot how many) samples from Gaussian in a few dozens milliseconds, and most of this time is the communication with the GPU.
It was the GMM on iris, it was not the worksheet i have sent you. But it is still sample.
Bayadera gives you ordinary clojure functions. Why wouldn't you be able to compose them?
I see. The problem is that I can barely convince anybody to use Clojure. For Bayadera to be attractive it would help if it would be part of a bigger community. Clojure in machine learning is still a very hard sell. I can probably use something on my own, but it will be difficult to attract colleagues. Anglican is not much better in that regard, but the few really nice projects that are out there feel very isolated and fragmented. My colleague would like to go with edward, I guess, since it is sponsored by OpenAI and built on top of tensorflow (although he doesn't like tensorflow in particular).
What I like about Anglican is that I can see people using it for small data problems and inference. If this is possible with Bayadera as well, I am totally fine with it.
With people I mean everyday Clojure developers without a GPU and a background in data science.
That's why I don't like to bump people to use my (or other) tools. I'm OK with competition using inferior tools 🙂 OTOH, the best way to convince people to use some thechnology is to build useful stuff with it. When they see the result of your work, they'll ask you to tell them what you did (provided that you did something awesome, of course).
Now, it is difficult to convince people to use Clojure for ML, when there are lots of pies-in-the-sky talk, but Clojure tools like Incanter are a joke.
Hmm, yes. You are right about proving with results and the GPU. Anglican can leverage the GPU by embedding LSTMs or VAEs for its proposal distribution, just to point out.
Whoever hopes that they will be able to do ML with the level of knowledge required for Web apps and no maths, will spent years circling around punching other people's tutorials
BTW Bayadera IS useful for small data problems, and I doubt it is useful for big data problems. That goes to Bayesian methods in general.
https://arxiv.org/pdf/1705.10306.pdf Section 5.4 has a 4096 dimensional sampling space.
that's the problem with most papers. they count number of steps, without regard how much one step costs wrt computation, memory access, and parallelization
I agree that you need maths. But doing maths before seeing what you can do with machine learning can turn many people off. Esp. when you do probabilistic modelling, you need to do a lot of math, much more than for NNs.
I don't think so - NNs also require maths to understand what you do, it's only that there are popular frameworks with ready made recipes, that work well for handful of problems (mostly vision and nlp). however, what if you do not work with vision and nlp?
Well, I already have (for more than a year) large speedup over Stan, but I am not even eager to publish that. I plan to work on some commercial applications, so I do not even want to show off the technology itself, but the product.
And you are right, that is also my problem. As long as the toolboxes for black-box bayesian inference are complicated, they will never be as popular.
The biggest "doh" is that bayesian stuff isn't even that complicated, especially when you use the tools that someone other made.
Yes, that is why I like the blackbox approaches where you can specify a generative process and then have reasonable inference per default.
I think machine learning should stop being a niche thing for highly skilled specialists.
The problem with programmers is that sometimes they... well, almost like physicists. But, I do not even care. I open-sourced the technology because at least some people can "get it" (and some really do get it and use it) and I might get valuable feedback and even some contributions. However, I do not want to beg anyone to use it and why would I?
I have been motivated by LDA 4 years ago, it was a very intuitive generative model and easy to understand even without heavy math knowledge back then.
You can get the topics generating a corpus of texts and the distribution of topics per document in an unsupervised fashion from it.
Did you have any application/business ideas around that or more like research curiosity?
I am trying to be set. It is frustrating. I would like to at least work probablistically and not just throw NNs at some thing.
I don't want to just do some topic they throw at me. I like to be motivated by myself, but this is not working well this time.
Did you see the (old-ish) book by Bishop called Pattern recognition and machine learning?
but the book is probabilistically-oriented, and he discusses the probabilistic perspective of NNs
but the problem is that DL is so successfull with vision and perception in general that it is a slim chance that you'll get something with bayesian methods
Well, the paper I have first posted is from a guy (Yarin Gal), who showed that droput regularization turns the neural net into a bayesian neural net (or a GP).
maybe bayesian techniques could be useful when you have extremely scarce visual information
So you can use the uncertainty on the output by running the inference several times in a meaningful way.
An example from Anglican was to have generative model for captchas and theirs was state of the art, cracking all of them.
Your prior is a GP prior. That corresponds to a normal distribution over the weight matrices.
wait, wait. I meant, you, as a human, set some prior that describe your current (pre-data) knowledge. How do you decide on that?
But generally, let's say I am trying to estimate when people call the call center, or something like that
from that you generate a function that has some prior set on the uncertainty of the measurement.
Note that the model is pretty sure about the parameters, and they are not very probabilistic 🙂
Does it have to be a part of a narrow EU/DE/industry-backed project or you are more free in choosing the area as long as it is vision?
Solving a good vision problem, something practical. Not just doing maths or playing around.
It would be good if we can show that bayesian modelling can work for vision problems, atm. weakly supervised problems are interesting.
Do you have some problem where data is really, really poor and DL does not work well?
DL (and most of ML) fields seems to me as many soothsayers throwing bones around and reading star constellations
Isn't it more appropriate for that level to take something that has been researched well and make a good implementation?
Good question. Maybe I have stayed too long in the math department and have complexes now.
Well, I mostly see paper's and thesis from state of the art people in the field and they are really good.
but it is better to do some cool exploratory implementation as an introduction to phd
although they are all working programmers, since in Serbia there is little opportunity to fund students to work on research so they are all working full-time in industry cranking code, and do their studies part-time
I have worked part-time with Clojure over the last half year, but I was lucky to do so with friends.
but we could do datomic with fulltext-search and google places integration in the backend and two single page apps and an android and ios app.
atm it requires AMD GPU that supports OpenCL 2.0, but I'm also probably giving it CUDA backend soon-ish
although my card, R9 290X, which is quite a beast, probably costs something like 100 EUR in Germany now (second-hand)
I see. There are tons of pretrained models out there though, which could be helpful for industry applications.
Well, this should be mostly plumbing, nothing critical. pretrained models can be applied to data separately anyway.