Fork me on GitHub
#clojure
<
2017-05-20
>
qqq00:05:36

what webserver do clj projects use in proudction?

lincpa00:05:43

qqq: I use http-kit.

qqq00:05:49

http-kit? jetty? tomcat?

noisesmith00:05:50

my servers use aleph or http-kit, planning to move all of them to aleph

rwilson00:05:29

I've used http-kit for a few years, still works reasonably but with the lack of current maintenance, I wouldn't recommend starting something new on it now. Also planning to move to aleph or jetty9, haven't evaluated enough to say which yet.

lincpa00:05:13

I use http-kit because of the recommendation of Luminus documentation about Websockets

lincpa00:05:05

and it's the simplest solution

lincpa00:05:05

http-kit's performance is good enough, and easy to use

qqq02:05:26

gae used jetty; I guess I'll stick with jetty then

dpsutton02:05:29

@noisesmith you're moving everything to aleph?

noisesmith02:05:17

yeah - it's better maintained than http-kit, and performs better

dpsutton02:05:30

Thanks for the info. Hadn't heard how that stack was regarded for production

noisesmith02:05:39

fwiw I also put it behind nginx

noisesmith02:05:59

I don't trust the java web servers for security like I do nginx

john02:05:58

and handles "back pressure" eagle screams

noisesmith02:05:03

haha - I learned a lot about backpressure from ztellman yeah

john02:05:28

a problem I haven't had the privilege of facing lol

john02:05:35

would be a nice problem to have I suppose

john02:05:33

more of an issue with asynchronous systems I guess, right?

noisesmith02:05:12

I went from a server that could only handle 6 concurrent requests to being limited only by the throughput limits of external resources (db, apis) by fixing backpressure issues

noisesmith02:05:26

@john well web servers are async if you accept two clients at once

noisesmith02:05:00

the general pattern behind most of our backpressure problems was some loop that spawned a thread for each input

noisesmith02:05:23

because someone thought "hey, concurrency will make this faster, so let's use threads" without measuring performance

john02:05:06

wouldn't lots of internal queueing, often associated with "async servers" also pose pressure problems?

noisesmith02:05:07

so we have an algorithm that is effectively processing a graph based on client input, you don't have to go very far in a graph to blow up, if you are starting a thread for each input

noisesmith02:05:20

the queueing is the solution to those problems

noisesmith02:05:30

simply starting threads with no queue logic is the cause

noisesmith02:05:51

because the OS will happily let you start up threads until nothing works any more (think fork bomb) if you try it

noisesmith02:05:36

also, this was exacerbated because someone thought "oh, things slowed down, we better find the slowest things and put them in more threads"

noisesmith02:05:52

this happened recursively throughout the app until sane voices were heard

john02:05:15

interesting

noisesmith02:05:55

clojure makes it too easy to use parallelism in our case - so parallelism was used by people who don't really understand concurrency or resource issues

noisesmith02:05:03

eventually I learned... haha

john02:05:22

"let's put a pmap in there"

noisesmith02:05:01

well no, it got more subtle "pmap limits the number of threads it will create based on your CPU count and that's not enough, let's wrap a map in a doall and start a thread for every item in the collection"

noisesmith02:05:07

pmap would have been much better :P

noisesmith02:05:19

we still do a lot parallelism - we need to for the throughput we need - but we now have backpressure, and configurable limits on how much work it will try to do at once

noisesmith02:05:39

another good example was taking 1000 queries, and sending them all at once to a single server (we own it, so we know it's a single server)

noisesmith02:05:57

a high percentage of our results were error messages that snuck through insufficient validation

noisesmith02:05:06

we fixed it by simply sending 1 request at a time

noisesmith02:05:31

and that was faster too, because we weren't making the poor machine thrash (it was also a 1 cpu machine btw)

noisesmith02:05:47

cargo cult parallelism

john02:05:42

my work on web workers bared that out too. Using the lib for simple scenarios would be stupid.

john02:05:54

the coordination overhead

noisesmith02:05:32

when we dug down, and found the slow things that people were trying to fix with parallelism, it was actually fixing improperly configured database tables, or bad queries to said db, that made the biggest difference

noisesmith02:05:31

and I won't mention which non-leadership member of the team kept saying "stop optimizing without measuring, it's making things worse" but his avitar on slack shows him wearing a bright blue shirt

john02:05:20

"you can't improve what you don't measure"

noisesmith02:05:52

a) "we need backpressure" b) "but we can't do that optimization until it works" a) "it doesn't work because it's falling over" b) "right" a) "it would stop falling over if we had proper backpressure" b) "we need to fix it before we do optimizations like that"

noisesmith02:05:43

sorry - that was a pretty intense time for me so it gets me ranting

john02:05:04

what kind of graph problem?

noisesmith02:05:28

social data analysis based on apis describing social relationships and interactions

noisesmith02:05:05

it's a fun problem that definitely isn't fully solved

john02:05:26

nice. did you use a clojure solution for the graph?

noisesmith02:05:34

it's clojure end to end

noisesmith02:05:06

except one java class for dijkstra's algorithm because it's just so much faster in java with v8 lambdas

john02:05:58

oh wow. did you use loom?

noisesmith02:05:08

we've looked at it, but no; early team lead was a mathematician in disguise as a programmer, so he lacked the self control to use a pre-packaged solution to a hard algorithmic problem

noisesmith02:05:15

(lol, that's a bit glib)

noisesmith02:05:25

but in all seriousness, loom doesn't have the specific algorithms we use, and he decided it was easier to just read the papers and implement from scratch rather than see how loom's optimized apis would let us do the same thing

noisesmith02:05:02

(but we would have had to stitch the parts together of course)

noisesmith02:05:51

that represents a lot of lost dev time, but it's a sunk cost and that part of the code is solid, so we haven't had to go revisit it

john02:05:13

Yeah, I read that loom was extensible in the core.matrix kind of way. But hey, if he can do it.

noisesmith02:05:18

but eventually if we get super fast access to our apis and a much faster db, loom would be worth looking at for sure

noisesmith03:05:00

it's just a question of when it would actually be the bottleneck, and for a while at least that's going to be the data source, not the number crunching

cfleming04:05:28

@markbastian The 7th icon there in the toolbar will print the stacktrace of the last error.

puzzler04:05:08

@noisesmith ubergraph's implementation of shortest-paths is implemented in Java; the graph data structure is Clojure/immutable.

kzeidler05:05:35

Is there an equivalent to the (now deprecated) 'clojure.contrib.repl-utils/source?

noisesmith05:05:01

is it different from clojure.repl/source ?

kzeidler05:05:37

Probably not, I just found the other version first. 🙂 Thanks!

noisesmith05:05:26

most repls should start with clojure.repl in scope, see also clojure.repl/doc and clojure.repl/apropos and clojure.repl/find-doc

kzeidler05:05:38

Is find-doc working now? I don't know if it's the function I'm thinking of, but when I last worked in Clojure a few months ago one of the doc getters required making HTTP requests to a site that seemed not to be maintained

noisesmith05:05:08

find-doc does a substring match or regex match on doc strings

kzeidler05:05:35

Ah, that actually sounds quite nice

noisesmith05:05:41

it doesn't use http, javadoc does use http to open the official java docs, but I think it can be configured if you have the html java docs local

kzeidler05:05:16

I see. Would any external documentation for project dependencies have to be loaded manually then?

noisesmith05:05:35

clojuredocs doesn't have my libraries, clojure.repl/doc does

kzeidler05:05:55

I could swear that there was a function that was able to, at least in theory, query some generic oracle for missing documentation... which would be great if possible as a fallback

noisesmith05:05:23

what exactly did this oracle provide?

kzeidler05:05:47

Trying to find a namespace for it. One of the several unofficial clojure (maybe clojurescript?) sites offered documentation for a variety of libs, and the function I think was just a dumb http request in its schema format

lincpa05:05:54

I use the Clojure script to generate API document text files of project dependencies.

noisesmith05:05:13

@kzeidler oh, right, arrdem's http://conj.io had an api

kzeidler05:05:23

That's the one!

noisesmith05:05:24

it's only for a limited set of libs though

lincpa05:05:37

Convenient Notepad++ Query

noisesmith05:05:44

my documentation steps are apropos if I forget the name, or find-doc if apropos didn't work; doc if I forget how to use it, or source if that's not enough; last resort is to just read the source itself, fast to look up automatically in my editor

kzeidler06:05:05

yes the source function works well, I can see... much easier here than in Python or JS. 👍

kzeidler06:05:34

(I guess just Python actually, JS is pretty code<->data isomorphic)

john06:05:43

what, .toString?

kzeidler06:05:45

that, or simply the function name resolves to the literal function definition I think

kzeidler06:05:58

(uninvoked)

kzeidler06:05:03

at least it does in console mode

john06:05:51

.toString on invoked JS closures and you've lost some info there

john06:05:17

though invoked in clojure/script might be hard to get back as well

john06:05:47

learning JS and that bit me recently

kzeidler06:05:09

ah. the (function () { return self.invoking = true; })() kind you mean?

john06:05:54

eh, function () { var whatever ... ; return function () ... }

john06:05:02

can't get at whatever

kzeidler06:05:34

aha. yeah, that is true, the inner function is lazy

john06:05:41

after you've invoked and returned the closed over fn

john06:05:53

JS ain't that bad though.

john06:05:37

I've done some google apps customization with it for some clients with apps script

john06:05:48

pretty fun

john06:05:06

I actually got a self hosted CLJS instance running on Google Apps back end lol

john06:05:20

it's basically node

kzeidler06:05:56

it gets a bad rap, but it inherits a lot from scheme

kzeidler06:05:05

probably why it gets a bad rap 🙂

kzeidler06:05:41

I think as it's matured its syntax has gotten a little more sugary, but it's the one language that makes me glad I learned how to make class-like "klazzes" in SICP once upon a time

john06:05:37

It's just hard to optimize when the programmer is always right

john06:05:35

of course I meant concatenation by +

kzeidler06:05:05

What about concatenation by +? (It's super weird? Yes. Oh man.)

john06:05:00

It's just hard for JS to catch up performance wise, when the programmer can do no wrong

kzeidler06:05:13

When I taught JS the last time I tried introducing the + polymorphism as a gameshow in the style of numberwang

john06:05:09

Yeah, I can see the arguments for it as a first language. #1 being reach

john06:05:47

did you just rick roll me?

john06:05:23

that video is hilarious

kzeidler06:05:36

oh haha yes, numberwang I find a valuable teaching aid for explaining why, idk, the first element of [-1,0,-2, 1, 2].sort() is.... -1

kzeidler06:05:48

and the last is... 2

kzeidler06:05:22

THAT'S NUMBERWANG! 🎉

kzeidler06:05:23

Is there an easy way to lookup the current version of some library as you're declaring the dependency in project.clj?

kzeidler06:05:06

I usually end up fumbling around clojars, and thinking I must be doing this wrong

miikka07:05:41

I always check Clojars or GitHub. When you want to update the dependency, though, there's lein-ancient https://github.com/xsc/lein-ancient

qqq08:05:35

What is the largest known clojure project by lines of code count ?

hcarvalhoaves08:05:51

some ~20k loc projects here...

hcarvalhoaves08:05:51

w/ 150+ projects no single project had the chance to get too large though

lincpa08:05:19

40+k lines pure clojure persional project, homepage: https://github.com/linpengcheng/fa

lincpa08:05:06

c# -> Clojure-clr -> clojure(script)

lincpa08:05:06

Use the R language with Clojure DSL (like hiccup, HoneySQL).

lincpa08:05:42

Next version add expert system core (using core.logic)

lincpa08:05:13

Follow up on statistics-based AI

lincpa09:05:20

I've only written this project in my spare time.

lincpa09:05:58

Is it the largest known clojure project?😀

kzeidler09:05:43

What's the built-in method for casting a boolean to int?

leonoel09:05:03

@kzeidler what outcome would you expect of such a function ?

kzeidler09:05:42

@lenoel: (let [index (toInt has-child?)] ... ), e.g.

kzeidler09:05:38

I can use an if, I know, but I would suspect there might be a built in way to go between those two

leonoel09:05:18

there is no obvious coercion from bool to int, so the function does not exist looking at your example I'm not even sure what are valid values for index writing your own conversion is a one-liner, use an if or a map literal

kzeidler09:05:23

Interesting. Huh, this is kind of bringing back some memories of Java's type system

leonoel09:05:30

clojure follows strong typing

kzeidler09:05:33

Boolean in Java is... somehow related to the string type, right? not Integer?

leonoel09:05:03

you can't coerce a boolean in integer in java either

kzeidler09:05:06

Gotcha. Thanks for clarifying

kzeidler09:05:32

On a more subjective note... I'm a little rusty on Clojure conventions, but one of the things I've noticed is parameter names tend to be pretty terse

kzeidler09:05:45

xs... coll... s...

hcarvalhoaves09:05:22

(Boolean/compare false false) => 0

kzeidler09:05:32

is this true even of ad hoc functions, or is that just a convention of more abstract HOF?

hcarvalhoaves09:05:33

(Boolean/compare true false) => 1

kzeidler09:05:03

@hcarvalhoaves: excellent. I like this much better than (if (pred?) 0 1) for some reason

hcarvalhoaves09:05:36

yeah... since it's static it might be a good candidate for inlining

hcarvalhoaves09:05:39

(definline b->int [^Boolean b] `(Boolean/compare ~b false))

kzeidler09:05:02

hcarvalhoaves: Interesting. A def primitive I haven't encountered yet. :thinking_face:

hcarvalhoaves09:05:13

seems equivalent to (defn ... ^{:inline (fn [] )} ...) without having to define the normal body

kzeidler09:05:26

Come to think of it, what does 'static' mean in a Clojure context exactly?

kzeidler09:05:42

This is a different use of the word 'static' than I'm familiar with, since it does indeed return a variable

hcarvalhoaves10:05:55

it returns a symbol. what I meant is that a static method call like that (`Boolean/compare`) seems a good candidate to expand inline

Pablo Fernandez12:05:16

I have a data structure that's sort of like a chart, [{:from 0 :to 10 :value 2} {:from 10 :to 20 :value 5} ...]. There can be no overlap between items. Does this structure have a name in computer science?

bcbradley12:05:29

sounds like a histogram

bcbradley12:05:38

@pupeno try using clojure frequencies

(frequencies [[0 10] [0 10] [10 20] [10 20] [10 20] [10 20] [10 20]])

Pablo Fernandez12:05:00

It's sort of like a histogram, but it has variable size buckets.

Pablo Fernandez12:05:31

And I'm trying to figure out a good way to go from a bunch of ranges to this sort of variable-sized bucket histogram.

bcbradley12:05:34

data doesn't vary in clojure unless you are talking about state

bcbradley12:05:47

in which case you will need to use one of clojure's stateful reference types

Pablo Fernandez12:05:10

@bcbradley I'm not talking about state, I'm talking about the buckets. They are not the same size.

Pablo Fernandez12:05:49

[{:from 0 :to 10 :value 2} {:from 10 :to 15 :value 5} {:from 15 :to 34.12 :value 500} ...]

Lone Ranger13:05:53

I often hear it called a "discretized state space"

Lone Ranger13:05:17

but that's for when you convert a continuous space into buckets of discrete spaces

bcbradley13:05:41

why does a histogram need to be on same sized buckets?

bcbradley13:05:48

frequencies will work regardless of what its counting

danp14:05:11

I'm struggling with a bit of Java interop - using akka-wamp to connect to an web socket

danp14:05:34

I've instantiated ActorSystem and Client, calling .connect on the Client.

danp14:05:22

From the .thenAccept on line 4, I'm not familiar with the Java syntax.

danp14:05:27

Can anyone enlighten me please? 🙂

kzeidler14:05:08

@danp I have no idea but is line 8 working for you? \n in Java actually gets translated to a linefeed instead of newline

kzeidler14:05:38

you might sub it with n% if it's causing you any problems

danp14:05:26

I can't actually get that far. I've got slightly further now though - I deref'ed the future that's returned from the connect to get a connection. Trying to .open on the connection but getting dependency issues now

noisesmith14:05:46

that syntax is effectively (fn [conn] ... (fn [session] .... (fn [event] ...)))

noisesmith14:05:55

but it's instances of Function

noisesmith14:05:42

and it's a classic callback-road-to-perdition (not quite callback hell yet...)

noisesmith14:05:30

anyway, it should be hard to reify Function with an actual clojure fn

noisesmith14:05:42

err, I guess you don't need the fn part, just a reify should do it

noisesmith14:05:14

and like anything else using generics, you just pretend generics don't exist

kzeidler14:05:57

someone with fresher knowledge of Java syntax could you confirm or deny my suspicion that variable interpolation in Java strings is not defined for Arrays? (re: line 8)

noisesmith14:05:29

it will just call toString

kzeidler14:05:49

I got curious and tried looking up the API, but the API for System.out isn't quite as thorough it seems

noisesmith14:05:51

so you get the str of two collections in a string

danp14:05:51

i've currently got

(def client (Client/create actor-system))
(-> client 
    (.connect "" "json") 
    (deref))

noisesmith14:05:17

wait, why deref?

noisesmith14:05:30

I would think you should have a call to thenAccept

kzeidler14:05:36

@noisesmith: interesting. That's pretty cool

noisesmith14:05:37

and provide an instance of Function to that

danp14:05:38

because it was returning a future

danp14:05:58

and that's about as far as my understanding of futures goes!

danp14:05:10

(i.e. you can deref them)

noisesmith14:05:17

it's not just a future, if it has a thenAccept method...

danp14:05:13

I tried using .thenAccept, but then the syntax confusion threw me off it

noisesmith14:05:40

(-> client (.connect ...) (.thenAccept (reify Function ....))

danp14:05:41

But from what you just said, I want to pass in a function as a param to .thenAccept - is that right?

noisesmith14:05:57

not just a function, a java.util.function.Function but yes

noisesmith14:05:18

it's an object with an apply method, pretty easy to make one

danp14:05:59

ok, with you so far - let's give it a go

danp15:05:11

which has thenAccept method

noisesmith15:05:00

aha, future chaining

noisesmith15:05:26

so yeah, thenAccept takes a Consumer, a Function is a kind of Consumer (I think they just differ by arg count)

danp18:05:10

Following on from above, I've got the following code now:

(-> client 
    (.connect "" "json") 
    (.thenAccept 
      (reify java.util.function.Consumer 
        (accept [this conn] 
          (.open conn "realm1"))))
    (.thenAccept 
      (reify java.util.function.Consumer 
        (accept [this session] 
          (.subscribe session "BTC_XMR"
                              (reify java.util.function.Consumer 
                                (accept [this event]
                                  (println (.kwargs event) (.args event)))))))))

danp18:05:30

It's not working. It returns the future, then nothing further happens in the REPL

danp19:05:25

I'm guessing I'm doing interop incorrectly for the Consumer callbacks, but not 100% how I'd go about debugging this.

danp19:05:31

Does anyone have any ideas?

bradford19:05:10

Is there an alternative to Component that’s a little more straightforward? I have a Pedestal webserver I want to refresh in the REPL instead of needing to restart from scratch every time I make a change

curlyfry20:05:24

bradford: Check out integrant or Mount! My personal preference is integrant.

neumann19:05:44

I'm trying to create a lazy sequence of lazy sequences.

neumann19:05:01

Is that even possible?

neumann19:05:11

(defn lazy-gen
  []
  (letfn [(step [n]
            (println n)
            (lazy-seq (cons n (step (inc n)))))]
    (step 1)))

(defn lazy-test
  []
  (doseq [group (partition 4 (take 12 (lazy-gen)))]
    (println "group")
    (doseq [line group]
      (println ">" line))))

neumann19:05:53

No ">" lines are printed out until the entire group is generated.

kzeidler20:05:41

Is it possible to derive hash map values from adjacent values of the object?

kzeidler20:05:33

i.e. (def obj {:a 1 :b (+ 1 (get :a obj))})

bradford20:05:44

@curlyfry ya, integrant looks pretty rad, thx

noisesmith20:05:46

you can do it with a let

kzeidler20:05:05

How so? I've been getting around it okay using lets to precompute, and later dropping them in to the corresponding fields but it seems a little indirect

kzeidler20:05:42

Is that the strategy you're talking about?

noisesmith20:05:54

the var exists when the body runs, but doesn't have anything useful in it yet

noisesmith20:05:10

yes, I'm talking about using a let block to build up the value in steps, it's the only way to do it

kzeidler20:05:44

sure, yeah "obj" wouldn't be a valid selector in that context. Most languages expose a 'this' or 'self'...

noisesmith20:05:53

there's not such thing in clojure

noisesmith20:05:23

obj points to the var container itself, which holds a special undefined value until your def form returns

noisesmith20:05:30

it's useful for self calls, that's it

noisesmith20:05:27

(def obj (let [o {:a 1}] (assoc o :b (inc (:a o)))))

lxsameer20:05:51

hey guys, I'm using alia to connect to cassandra, does any one encountered this expection:

java.lang.NoSuchFieldException: builder
clojure.lang.Compiler$CompilerException: java.lang.NoSuchFieldException: builder, compiling:(qbits/alia/policy/load_balancing.clj:74:16)

kzeidler20:05:49

@noisesmith huh. How clever. I can work with this in lieu of 'this.

kzeidler20:05:48

'net.cgrand.enlive-html in the stub below is the namespace, right? (And a symbol)

(ns musil.utils
  (:require [net.cgrand.enlive-html :as html])
What's the correct way to refer to the keyword passed in via project.clj? Maven ID or something?
(defproject guestbook "0.1.0-SNAPSHOT"

  :description "FIXME: write description"
  :url ""

  :dependencies [[enlive "1.1.6"]... ])

noisesmith20:05:26

yeah, [enlive "1.1.6"] gets auto-expanded to [enlive/enlive "1.1.6"] which stands for organization-id "enlive" project-id "enlive" version string "1.1.6"

noisesmith20:05:08

it's represented like this in maven:

<dependency>
      <groupId>group-a</groupId>
      <artifactId>artifact-a</artifactId>
      <version>1.0</version>
    </dependency>

kzeidler20:05:43

I noticed that yeah the tokens without slashes just echoed themselves in the url schema

noisesmith20:05:59

yeah, it's a shorthand that lein supports

kzeidler20:05:18

so the thing that "enlive" is, would it be more appropriate to call that the groupId or the artifactId in the absence of a pair?

noisesmith20:05:59

it's a project where the group id and the artifact id are the same

noisesmith20:05:18

otherwise it would be group/project "version"

noisesmith20:05:50

or, strictly speaking, artifact not project

kzeidler20:05:38

got it. I have a function that takes those string parameters and returns the latest version but I was at a loss what to call it

plins21:05:04

hello everybody, can someone recomend me an http client that can be consumed both by clj and cl-js?

qqq22:05:46

is there a wa to tell rikng.adapter.jetty/run-jetty to serve a static/resources directory?

qqq22:05:52

or do I have to do this at the ring layer?

kzeidler23:05:29

Here's a function that works fine, but has a tiny flaw:

(defn parse-int [s]
  (Integer. (re-find #"\d+" s)))
(parse-int "2.01")
=> 2
(parse-int "2.9999")
=> 2
The Integer constructor ignores the mantissa. Casting to Double still ignores the mantissa, but returns 2.0 for those examples

kzeidler23:05:13

How would I modify this to round to the nearest whole number?

kzeidler23:05:27

It's an interesting flaw because the information appears to be lost in the conversion to int, so I'm skeptical it can be fixed with any Int/Double methods

noisesmith23:05:50

for starters, you have to grab the part of the string that includes and comes after the .

kzeidler23:05:23

Thank you. Thank you. I'm not sure I would've caught that right away

noisesmith23:05:09

or maybe #"\d*\.?\d+" is closer to correct

noisesmith23:05:16

regex is hard