what webserver do clj projects use in proudction?

http-kit? jetty? tomcat?

my servers use aleph or http-kit, planning to move all of them to aleph

I've used http-kit for a few years, still works reasonably but with the lack of current maintenance, I wouldn't recommend starting something new on it now. Also planning to move to aleph or jetty9, haven't evaluated enough to say which yet.

I use http-kit because of the recommendation of Luminus documentation about Websockets

and it's the simplest solution

http-kit's performance is good enough, and easy to use

gae used jetty; I guess I'll stick with jetty then

@noisesmith you're moving everything to aleph?

yeah - it's better maintained than http-kit, and performs better

Thanks for the info. Hadn't heard how that stack was regarded for production

fwiw I also put it behind nginx

I don't trust the java web servers for security like I do nginx

and handles "back pressure" eagle screams

haha - I learned a lot about backpressure from ztellman yeah

a problem I haven't had the privilege of facing lol

would be a nice problem to have I suppose

more of an issue with asynchronous systems I guess, right?

I went from a server that could only handle 6 concurrent requests to being limited only by the throughput limits of external resources (db, apis) by fixing backpressure issues

@john well web servers are async if you accept two clients at once

true

the general pattern behind most of our backpressure problems was some loop that spawned a thread for each input

because someone thought "hey, concurrency will make this faster, so let's use threads" without measuring performance

wouldn't lots of internal queueing, often associated with "async servers" also pose pressure problems?

so we have an algorithm that is effectively processing a graph based on client input, you don't have to go very far in a graph to blow up, if you are starting a thread for each input

the queueing is the solution to those problems

simply starting threads with no queue logic is the cause

because the OS will happily let you start up threads until nothing works any more (think fork bomb) if you try it

right

also, this was exacerbated because someone thought "oh, things slowed down, we better find the slowest things and put them in more threads"

this happened recursively throughout the app until sane voices were heard

interesting

clojure makes it too easy to use parallelism in our case - so parallelism was used by people who don't really understand concurrency or resource issues

eventually I learned... haha

right

"let's put a pmap in there"

well no, it got more subtle "pmap limits the number of threads it will create based on your CPU count and that's not enough, let's wrap a map in a doall and start a thread for every item in the collection"

pmap would have been much better :P

we still do a lot parallelism - we need to for the throughput we need - but we now have backpressure, and configurable limits on how much work it will try to do at once

nice

another good example was taking 1000 queries, and sending them all at once to a single server (we own it, so we know it's a single server)

a high percentage of our results were error messages that snuck through insufficient validation

we fixed it by simply sending 1 request at a time

and that was faster too, because we weren't making the poor machine thrash (it was also a 1 cpu machine btw)

cargo cult parallelism

aye

my work on web workers bared that out too. Using the lib for simple scenarios would be stupid.

the coordination overhead

when we dug down, and found the slow things that people were trying to fix with parallelism, it was actually fixing improperly configured database tables, or bad queries to said db, that made the biggest difference

and I won't mention which non-leadership member of the team kept saying "stop optimizing without measuring, it's making things worse" but his avitar on slack shows him wearing a bright blue shirt

haha

lol

"you can't improve what you don't measure"

a) "we need backpressure" b) "but we can't do that optimization until it works" a) "it doesn't work because it's falling over" b) "right" a) "it would stop falling over if we had proper backpressure" b) "we need to fix it before we do optimizations like that"

lol

sorry - that was a pretty intense time for me so it gets me ranting

what kind of graph problem?

social data analysis based on apis describing social relationships and interactions

it's a fun problem that definitely isn't fully solved

nice. did you use a clojure solution for the graph?

it's clojure end to end

except one java class for dijkstra's algorithm because it's just so much faster in java with v8 lambdas

oh wow. did you use loom?

we've looked at it, but no; early team lead was a mathematician in disguise as a programmer, so he lacked the self control to use a pre-packaged solution to a hard algorithmic problem

(lol, that's a bit glib)

but in all seriousness, loom doesn't have the specific algorithms we use, and he decided it was easier to just read the papers and implement from scratch rather than see how loom's optimized apis would let us do the same thing

(but we would have had to stitch the parts together of course)

that represents a lot of lost dev time, but it's a sunk cost and that part of the code is solid, so we haven't had to go revisit it

Yeah, I read that loom was extensible in the core.matrix kind of way. But hey, if he can do it.

but eventually if we get super fast access to our apis and a much faster db, loom would be worth looking at for sure

it's just a question of when it would actually be the bottleneck, and for a while at least that's going to be the data source, not the number crunching

@markbastian The 7th icon there in the toolbar will print the stacktrace of the last error.

@noisesmith ubergraph's implementation of shortest-paths is implemented in Java; the graph data structure is Clojure/immutable.

Is there an equivalent to the (now deprecated) 'clojure.contrib.repl-utils/source?

is it different from clojure.repl/source ?

Probably not, I just found the other version first. 🙂 Thanks!

most repls should start with clojure.repl in scope, see also clojure.repl/doc and clojure.repl/apropos and clojure.repl/find-doc

Is find-doc working now? I don't know if it's the function I'm thinking of, but when I last worked in Clojure a few months ago one of the doc getters required making HTTP requests to a site that seemed not to be maintained

find-doc does a substring match or regex match on doc strings

Ah, that actually sounds quite nice

it doesn't use http, javadoc does use http to open the official java docs, but I think it can be configured if you have the html java docs local

I see. Would any external documentation for project dependencies have to be loaded manually then?

http://clojuredocs.org is better .

clojuredocs doesn't have my libraries, clojure.repl/doc does

I could swear that there was a function that was able to, at least in theory, query some generic oracle for missing documentation... which would be great if possible as a fallback

what exactly did this oracle provide?

Trying to find a namespace for it. One of the several unofficial clojure (maybe clojurescript?) sites offered documentation for a variety of libs, and the function I think was just a dumb http request in its schema format

I use the Clojure script to generate API document text files of project dependencies.

@kzeidler oh, right, arrdem's http://conj.io had an api

has?

That's the one!

it's only for a limited set of libs though

Convenient Notepad++ Query

my documentation steps are apropos if I forget the name, or find-doc if apropos didn't work; doc if I forget how to use it, or source if that's not enough; last resort is to just read the source itself, fast to look up automatically in my editor

yes the source function works well, I can see... much easier here than in Python or JS. 👍

(I guess just Python actually, JS is pretty code<->data isomorphic)

what, .toString?

that, or simply the function name resolves to the literal function definition I think

(uninvoked)

at least it does in console mode

aye

.toString on invoked JS closures and you've lost some info there

though invoked in clojure/script might be hard to get back as well

learning JS and that bit me recently

ah. the (function () { return self.invoking = true; })() kind you mean?

eh, function () { var whatever ... ; return function () ... }

can't get at whatever

aha. yeah, that is true, the inner function is lazy

after you've invoked and returned the closed over fn

JS ain't that bad though.

I've done some google apps customization with it for some clients with apps script

pretty fun

I actually got a self hosted CLJS instance running on Google Apps back end lol

it's basically node

it gets a bad rap, but it inherits a lot from scheme

probably why it gets a bad rap 🙂

lol

I think as it's matured its syntax has gotten a little more sugary, but it's the one language that makes me glad I learned how to make class-like "klazzes" in SICP once upon a time

It's just hard to optimize when the programmer is always right

of course I meant concatenation by +

What about concatenation by +? (It's super weird? Yes. Oh man.)

It's just hard for JS to catch up performance wise, when the programmer can do no wrong

When I taught JS the last time I tried introducing the + polymorphism as a gameshow in the style of numberwang

Yeah, I can see the arguments for it as a first language. #1 being reach

did you just rick roll me?

that video is hilarious

haha no

oh haha yes, numberwang I find a valuable teaching aid for explaining why, idk, the first element of [-1,0,-2, 1, 2].sort() is.... -1

and the last is... 2

THAT'S NUMBERWANG! 🎉

lol

Is there an easy way to lookup the current version of some library as you're declaring the dependency in project.clj?

I usually end up fumbling around clojars, and thinking I must be doing this wrong

I always check Clojars or GitHub. When you want to update the dependency, though, there's lein-ancient https://github.com/xsc/lein-ancient

What is the largest known clojure project by lines of code count ?

some ~20k loc projects here...

w/ 150+ projects no single project had the chance to get too large though

40+k lines pure clojure persional project, homepage: https://github.com/linpengcheng/fa

c# -> Clojure-clr -> clojure(script)

Use the R language with Clojure DSL (like hiccup, HoneySQL).

Next version add expert system core (using core.logic)

Follow up on statistics-based AI

I've only written this project in my spare time.

Is it the largest known clojure project?😀

What's the built-in method for casting a boolean to int?

@kzeidler what outcome would you expect of such a function ?

@lenoel: (let [index (toInt has-child?)] ... ), e.g.

I can use an if, I know, but I would suspect there might be a built in way to go between those two

there is no obvious coercion from bool to int, so the function does not exist looking at your example I'm not even sure what are valid values for index writing your own conversion is a one-liner, use an if or a map literal

Interesting. Huh, this is kind of bringing back some memories of Java's type system

clojure follows strong typing

Boolean in Java is... somehow related to the string type, right? not Integer?

you can't coerce a boolean in integer in java either

Gotcha. Thanks for clarifying

On a more subjective note... I'm a little rusty on Clojure conventions, but one of the things I've noticed is parameter names tend to be pretty terse

xs... coll... s...

(Boolean/compare false false) => 0

is this true even of ad hoc functions, or is that just a convention of more abstract HOF?

(Boolean/compare true false) => 1

@hcarvalhoaves: excellent. I like this much better than (if (pred?) 0 1) for some reason

yeah... since it's static it might be a good candidate for inlining

(definline b->int [^Boolean b] `(Boolean/compare ~b false))

hcarvalhoaves: Interesting. A def primitive I haven't encountered yet. :thinking_face:

seems equivalent to (defn ... ^{:inline (fn [] )} ...) without having to define the normal body

Come to think of it, what does 'static' mean in a Clojure context exactly?

This is a different use of the word 'static' than I'm familiar with, since it does indeed return a variable

it returns a symbol. what I meant is that a static method call like that (`Boolean/compare`) seems a good candidate to expand inline

I have a data structure that's sort of like a chart, [{:from 0 :to 10 :value 2} {:from 10 :to 20 :value 5} ...]. There can be no overlap between items. Does this structure have a name in computer science?

sounds like a histogram

@pupeno try using clojure frequencies

(frequencies [[0 10] [0 10] [10 20] [10 20] [10 20] [10 20] [10 20]])

It's sort of like a histogram, but it has variable size buckets.

And I'm trying to figure out a good way to go from a bunch of ranges to this sort of variable-sized bucket histogram.

data doesn't vary in clojure unless you are talking about state

in which case you will need to use one of clojure's stateful reference types

@bcbradley I'm not talking about state, I'm talking about the buckets. They are not the same size.

[{:from 0 :to 10 :value 2} {:from 10 :to 15 :value 5} {:from 15 :to 34.12 :value 500} ...]

I often hear it called a "discretized state space"

but that's for when you convert a continuous space into buckets of discrete spaces

why does a histogram need to be on same sized buckets?

frequencies will work regardless of what its counting

I'm struggling with a bit of Java interop - using akka-wamp to connect to an web socket

The Java equivalent code is fairly succinct: https://github.com/angiolep/akka-wamp/blob/master/examples/poloniex/src/main/java/PoloniexJavaClient.java

I've instantiated ActorSystem and Client, calling .connect on the Client.

From the .thenAccept on line 4, I'm not familiar with the Java syntax.

Can anyone enlighten me please? 🙂

@danp I have no idea but is line 8 working for you? \n in Java actually gets translated to a linefeed instead of newline

you might sub it with n% if it's causing you any problems

I can't actually get that far. I've got slightly further now though - I deref'ed the future that's returned from the connect to get a connection. Trying to .open on the connection but getting dependency issues now

that syntax is effectively (fn [conn] ... (fn [session] .... (fn [event] ...)))

but it's instances of Function

https://docs.oracle.com/javase/8/docs/api/java/util/function/Function.html

and it's a classic callback-road-to-perdition (not quite callback hell yet...)

anyway, it should be hard to reify Function with an actual clojure fn

err, I guess you don't need the fn part, just a reify should do it

and like anything else using generics, you just pretend generics don't exist

someone with fresher knowledge of Java syntax could you confirm or deny my suspicion that variable interpolation in Java strings is not defined for Arrays? (re: line 8)

it will just call toString

I got curious and tried looking up the API, but the API for System.out isn't quite as thorough it seems

so you get the str of two collections in a string

i've currently got

(def client (Client/create actor-system))
(-> client 
    (.connect "" "json") 
    (deref))

wait, why deref?

I would think you should have a call to thenAccept

@noisesmith: interesting. That's pretty cool

and provide an instance of Function to that

because it was returning a future

and that's about as far as my understanding of futures goes!

(i.e. you can deref them)

it's not just a future, if it has a thenAccept method...

I tried using .thenAccept, but then the syntax confusion threw me off it

(-> client (.connect ...) (.thenAccept (reify Function ....))

But from what you just said, I want to pass in a function as a param to .thenAccept - is that right?

not just a function, a java.util.function.Function but yes

it's an object with an apply method, pretty easy to make one

ok, with you so far - let's give it a go

@noisesmith I've found this: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html

which has thenAccept method

aha, future chaining

And I think I've just found something useful here: https://angiolep.github.io/projects/akka-wamp/client/futures.html#open

so yeah, thenAccept takes a Consumer, a Function is a kind of Consumer (I think they just differ by arg count)

Following on from above, I've got the following code now:

(-> client 
    (.connect "" "json") 
    (.thenAccept 
      (reify java.util.function.Consumer 
        (accept [this conn] 
          (.open conn "realm1"))))
    (.thenAccept 
      (reify java.util.function.Consumer 
        (accept [this session] 
          (.subscribe session "BTC_XMR"
                              (reify java.util.function.Consumer 
                                (accept [this event]
                                  (println (.kwargs event) (.args event)))))))))

It's not working. It returns the future, then nothing further happens in the REPL

I'm guessing I'm doing interop incorrectly for the Consumer callbacks, but not 100% how I'd go about debugging this.

Does anyone have any ideas?

Is there an alternative to Component that’s a little more straightforward? I have a Pedestal webserver I want to refresh in the REPL instead of needing to restart from scratch every time I make a change

I'm trying to create a lazy sequence of lazy sequences.

Is that even possible?

(defn lazy-gen
  []
  (letfn [(step [n]
            (println n)
            (lazy-seq (cons n (step (inc n)))))]
    (step 1)))

(defn lazy-test
  []
  (doseq [group (partition 4 (take 12 (lazy-gen)))]
    (println "group")
    (doseq [line group]
      (println ">" line))))

No ">" lines are printed out until the entire group is generated.

Is it possible to derive hash map values from adjacent values of the object?

i.e. (def obj {:a 1 :b (+ 1 (get :a obj))})

@curlyfry ya, integrant looks pretty rad, thx

you can do it with a let

How so? I've been getting around it okay using lets to precompute, and later dropping them in to the corresponding fields but it seems a little indirect

Is that the strategy you're talking about?

the var exists when the body runs, but doesn't have anything useful in it yet

yes, I'm talking about using a let block to build up the value in steps, it's the only way to do it

sure, yeah "obj" wouldn't be a valid selector in that context. Most languages expose a 'this' or 'self'...

there's not such thing in clojure

obj points to the var container itself, which holds a special undefined value until your def form returns

it's useful for self calls, that's it

(def obj (let [o {:a 1}] (assoc o :b (inc (:a o)))))

hey guys, I'm using alia to connect to cassandra, does any one encountered this expection:

java.lang.NoSuchFieldException: builder
clojure.lang.Compiler$CompilerException: java.lang.NoSuchFieldException: builder, compiling:(qbits/alia/policy/load_balancing.clj:74:16)

@noisesmith huh. How clever. I can work with this in lieu of 'this.

Thanks!

'net.cgrand.enlive-html in the stub below is the namespace, right? (And a symbol)

(ns musil.utils
  (:require [net.cgrand.enlive-html :as html])

(defproject guestbook "0.1.0-SNAPSHOT"

  :description "FIXME: write description"
  :url ""

  :dependencies [[enlive "1.1.6"]... ])

yeah, [enlive "1.1.6"] gets auto-expanded to [enlive/enlive "1.1.6"] which stands for organization-id "enlive" project-id "enlive" version string "1.1.6"

it's represented like this in maven:

<dependency>
      <groupId>group-a</groupId>
      <artifactId>artifact-a</artifactId>
      <version>1.0</version>
    </dependency>

I noticed that yeah the tokens without slashes just echoed themselves in the url schema

yeah, it's a shorthand that lein supports

so the thing that "enlive" is, would it be more appropriate to call that the groupId or the artifactId in the absence of a pair?

it's a project where the group id and the artifact id are the same

otherwise it would be group/project "version"

or, strictly speaking, artifact not project

got it. I have a function that takes those string parameters and returns the latest version but I was at a loss what to call it

hello everybody, can someone recomend me an http client that can be consumed both by clj and cl-js?

is there a wa to tell rikng.adapter.jetty/run-jetty to serve a static/resources directory?

or do I have to do this at the ring layer?

Here's a function that works fine, but has a tiny flaw:

(defn parse-int [s]
  (Integer. (re-find #"\d+" s)))
(parse-int "2.01")
=> 2
(parse-int "2.9999")
=> 2

How would I modify this to round to the nearest whole number?

[\d\.]+ ?

It's an interesting flaw because the information appears to be lost in the conversion to int, so I'm skeptical it can be fixed with any Int/Double methods

for starters, you have to grab the part of the string that includes and comes after the .

@noisesmith: d'oh!

Thank you. Thank you. I'm not sure I would've caught that right away

or maybe #"\d*\.?\d+" is closer to correct

regex is hard