Does anyone know of a simple example of doing multiclass text classification using TensorFlow in Clojure?

@mac as far as I know, there is no TensorFlow binding in Clojure.

http://gigasquidsoftware.com/blog/2016/12/27/deep-learning-in-clojure-with-cortex/ has example deep-learning code in clojure, but not in tensorflow

So say you were trying to do a project in clojure/clojurescript, and as you were thinking about the design, you realized that it segmented very cleanly into five or so microservices. If the project ever needs to scale, some of the services will run just fine on a a small cloud server, while at least one in particular will probably need two or three bigger ones. How do you organize the microservices within a project so that you can share code but do separate deployments?

What I'm thinking right now is to have one repository that contains several lein projects, where each project is either a microservice or a shared library

But that may be a total n00b misunderstanding of the ecosystem 🙂

I'm looking for a setup where I could either run all of the services on one JVM on one instance, or split the services across several instances if it takes off and I need to handle a lot of traffic

One thing you could do is use a single project, but have multiple "main" functions/namespaces that you can call, depending on which service you want to start.

java -jar uber.jar app.microservice.GreatThing

Something like that.

Alternatively, you have one Main and pass in params for the combos of services you want to start up.

Were I in your shoes, I’d probably do a single process to start with, but design it with isolated components that strictly communicate (after startup, at least) via queues

If you ever got to the big traffic stage, then you might look to replace those transient jvm queues with persistent message queues

Interesting. I like the idea of an entry point that takes params to indicate what to run. Lots of flexibility there.

Starting with a single process would be the intelligent way to go if development time were a concern

You could have separate lein folders inside the main repo, but I think you could also have lein use profiles for different combinations, if you want compile just what's needed for a given service over an omnibus service.

But this is a side project so it's more about learning :)

@jrheard Yes saw that one, but there appears to be a lot of complexity derived from having to deal with images. I am looking for an example of just text which I think will be a simpler place to start.

Yeah I could use profiles for that, if I even decided I care. Is there a memory usage penalty for having code loaded for services that are never started in a given instance?

I don't think so.

I think the JVM loads classes on demand.

I saw a project somewhere that did the "fat jar" thing, the masquerade technique?, but it might have been Scala.

Hm. I'm going to have to experiment

Thanks!

@zentrope: what is this 'fat jar' technique? I'm really not happy with :aot all generating hundreds of tiny class files

@qqq We were just talking about keeping a bunch of potentially separate apps in a single project, with a single uberjar, then just using paramters or main-class files to invoke the different modes/services.

@zentrope: oh, so (1) this reduces # of jar's down to 1, but (2) it does not do anything about decreasing the number of classes (expect in the case where the multiple jars share classes) ?

Nah, there's no way to reduce the number of classes. Putting them all in a jar makes it easier to use, though.

@zentrope: got it; thanks for clarifying

(com.rpl.specter/setval [:a com.rpl.specter/ALL nil?] com.rpl.specter/NONE {:a [1 2 nil 3 nil]})

doc output is {:a [1 2 3]}; my output: the nils replaced literally with ::NONE

(com.rpl.specter/setval [com.rpl.specter/ALL] com.rpl.specter/NONE [1 2])

@qqq aru you on 0.13.3-SNAPSHOT?

I'm on 0.13.2

is this a known bug?

I think the NONE feature has been just added

upgrading to 13.3-snapshot

@richiardiandrea : it works now, thanks! (would have never guessed this was the issue)

Is it possible to require namespaces deterministically? I often import the wrong lib during development and then I get namespace collisions, at which point the side-effecting defs have ruined my REPL and I need to start over to return to a point of sanity with symbols. E.g. IllegalStateException … “already refers to ’some-broken-symbol’”. My only recourse is to restart the REPL. Maybe I’m doing something wrong.

@petrus: does remove-ns work?

Doesn’t seem to (pardon the noise). Am I using it right?

(use '[launchpad.core] :reload)
clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: launchpad-connected-receivers already refers to: #'theronica.midi/launchpad-connected-receivers in namespace: theronica.core, compiling:(…<some-path>/src/theronica/core.clj:27:3)
        java.lang.IllegalStateException: launchpad-connected-receivers already refers to: #'theronica.midi/launchpad-connected-receivers in namespace: theronica.core
(remove-ns 'theronica.midi)
=> #object[clojure.lang.Namespace 0x5986fd43 "theronica.midi"]
(use '[launchpad.core] :reload)
clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: launchpad-connected-receivers already refers to: #'theronica.midi/launchpad-connected-receivers in namespace: theronica.core, compiling:(…<some-path>/src/theronica/core.clj:27:3)
        java.lang.IllegalStateException: launchpad-connected-receivers already refers to: #'theronica.midi/launchpad-connected-receivers in namespace: theronica.core

@petrus: sorry, I misunderstood, I thought you were requiring it as a qualified name, not using it (i.e. importing the names into current namespace) -- I don't know how to solve this problem

Although I personally find it annoying when people suggest I change my habits; but you might be better off using (require [... :as .. ]); I've found this actually drasticalyl reduce most of the collission problems I've had.

I want to use a transducer purely for side-effects, can I just use (constantly nil) as the reducing function?

@luxbock it will work, have you considered if your code can be refactored into (run! side-effecting-fn (eduction xform coll))?

@cgrand yeah that would probably work, I remembered that run! exists but forgot I could use it with eduction

Does anybody know how to specify custom SSL protocol and keys for clj-http?

any nice solution to handle parallelism with promise-chan based operations (seems like pipeline-* has quite different semantics, even tho it's usable for this in theory)?

like a fn that'd take input/output chans, a fn (returning a promise-chan), and a max parallelism arg. And would run inputs against the function in // , up to the limit specified

so far I have a poor mans pipeline-async like function for this, but maybe there's a built'in

Has any work on session types(http://simonjf.com/2016/05/28/session-type-implementations.html) been done in Clojure?

Hi all, is there any syntax sugar to make destructuring easier in 1.9 for ns qualified keys?

(let [{:keys [customer/name customer/company]} {:customer/name 1 :customer/second-name 2 :customer/company 3}]
  [name company])

@artemyarulin

=> (let [#:customer{:keys [name company]} {:customer/name 1 :customer/second-name 2 :customer/company 3}]
     [name company])
[1 3]

and if you have an alias you can do #::alias{:keys …}

oh, it works both ways, superb, thank you @cgrand

@artemyarulin you may prefer to use {:customer/keys [name company]} as it allows to mix several nses and still use :or and :as

cool, thx

http://clojurians.net/ this is dead

what do you guys do for testing code with database connections or api calls? You know, the classic testing impediments. I've heard some people talk about stuart sierra's component library. Is that useful for this?

@tjtolton I know of a few popular approaches 1) Protocols. You mock out the main parts of the API or Database access via protocols 2) You pass the db into each function that needs it, and you create a test database (maybe in-memory) during tests, and pass those into that function

You probably want a combination of both really though 🙂

Almost always used with-redefs

I've used with-redefs for this too - usually writing macros for test convenience, like (with-database (facts...)) etc..

Midje provides some help there too

im trying to come up with a solution that uses clojure-spec to generate a set of first line of defense generative testing

but the problem is

because we're not using datomic, I can't generate a db value

ill check out with-redefs. passing the db in would imply refactoring the entire codebase. which I would like to do, but probably not feasable.

if you are willing to use generated values instead of actually mocking up a db, the spec test instrument function has the ability to :stub @tjtolton

tjtolton this is worth a read, it uses F# but the partial application approach is definitely viable in Clojure as well: https://fsharpforfunandprofit.com/posts/dependency-injection-1/

@tjtolton just convince your boss to migrate to datomic 🙂

Not sure where to post this, but http://clojurians.net is broken. How do we invite new people to this Slack?

@tjtolton Here is an example of a function which gets data from a db. id is ignored here. (select-id 5) will return some data. Once you turn on stubbing, it will no longer be called, but instead, return values will be generated from the :ret spec. Pretty handy IMO.

(defn select-id [id]
  ; real code would connect to db, just simulating here
  {:col1 (rand-int 101)
   :col2 (rand-nth ["Joe" "Bob" "Bill" "Sally"])})

(s/def ::col1 nat-int?)
(s/def ::col2
       (s/with-gen string?
                   #(gen/such-that (complement empty?) (gen/string-alphanumeric))))
(s/def ::db-row (s/keys :req-un [::col1 ::col2]))

(s/fdef select-id
        :args (s/cat :id pos-int?)
        :ret ::db-row)

(select-id 5)
=> {:col1 37, :col2 "Bill"}

; now turn on instrumenting -- function not called, only gen'd
(stest/instrument `select-id {:stub #{`select-id}})
(select-id 5)
=> {:col1 347, :col2 "1rfT3Pw"}

@joshjones pretty neat 🙂

@solussd I just tried http://clojurians.net, and it looks like it’s working now. Maybe somebody noticed and gave it a kick?

@tjtolton with-redefs works, but you can use mount to swap states during testing / REPLing: https://www.dotkam.com/2016/01/17/swapping-alternate-implementations-with-mount/

@maxfurman lol. Very funny.

working. on. it.

Is anybody using Kafka successfully with Clojure? Or is there an alternative anybody here has used that can act as a persistent (as in durable, stored on disk, not Clojure persistent) messaging system?

@adamkowalski this should get you started https://www.youtube.com/results?search_query=clojure+kafka

I’ve looked through a lot of those videos, and while they give good high level overviews, I feel like there is some material lacking as to how to actually make them work together

@adamkowalski we use kafka quite a lot with clojure. although iirc only the consumption api's

Like I tried writing a absolute basic consumer, and I was having some issues with the polling function

@bfaber can you tell me how that experience was?

did you use a clojure library for it like clj-kafka or did you just go with the java API

I've not looked at clj-kafka so I can't comment on it specifically but I am very wary of wrapper libs. we use the kafka java sdk directly. it's like 30 lines of interop maybe, adding a dependency to avoid that is imo crazy

Yeah thats what I was planning on doing too, it seems like none of the libraries for clojure even work with the latest version either

I'll send you the code for a simpler kafka consumer component. it's using auto offset committing. once you have to manually commit offset it becomes a bit too coupled with business logic for me to feel comfortable sending

they are all 1 or 2 minor versions behind

That would be awesome, but do you mind if I pm you so we don’t spam this channel haha

ya

@adamkowalski I’m using the gregor library with good results so far https://github.com/weftio/gregor

@adamkowalski I am also using gregor, in terms of kafka in general, it's still pretty raw IMO

@adamkowalski plus one for gregor, simple and works great. I have several projects ported to it

Interesting, I will have to check it out!

What do you think it gives you over the raw java api?

i prefer to deal with maps and seqs vs objects, it's a pretty thin wrapper though

@adamkowalski besides data structures Kafka has quite confusing APIs.. 0.8 vs. 0.9, simple vs. non simple consumer, offset management, etc.. gregor does not try to solve all of it, but it makes it minimally simple

does clojure take a stand on "english vs logic", i.e. "children vs childs" "foxes vs foxs" for naming plurals

[for naming variables]

Do you have an example of existing code that does this? @qqq

@joshjones: I was writing a simple tree, and was deciding if the keys should be :node + :childs or :node + :children

child’s is the possessive of child, and children is the plural of child … so children seems to be the word you are intending

I'm used to seeing normal English plurals (`children`, foxes) in Clojure code.

Pretty much every code base I’ve worked on / with in 30+ years has tried to use English for names (and therefore English plurals etc), no matter where the code originated… with a couple of notable exceptions: a French system that was entirely written in French (apart from the programming language keywords — although I can’t remember what programming language it was in) and a German-authored system that almost entirely used English, except for -en suffices for plurals in a lot of places.