@gmercer The ideal is if start and stop are idempotent so, yes, nil them out when you stop a system. And start can check if they’re not nil and only start things that are not nil.

Oh, I see what you’re suggesting now… in the context of the code that @fbielejec posted…

No, the handler component doesn’t need to assoc :database database — that’s done by Component itself.

And the resources in the database would be released by called stop (which happens recursively — which you can see by adding println calls in).

so can you explain the failing test

So in a real example, the database connection would be released when stop is called on Database.

The test seems incorrect to me.

Or certainly does not indicate what the OP thinks it does.

(fwiw, the cloned repo throws an exception for me, as written)

incorrect in which way - @noisesmith pointed out that you need a reference to the started system to correctly call stop - and this aligns with the examples

@seancorfield mkdir test/app && mv test/system_tests.clj test/app/

If you add println calls (in a do around each assoc call), you see

lein test app.system-tests
start the database
start the handlers
stop the handlers
stop the database

I can see the stop gets called on Database - just wondering if it the component maker's responsibility to null out references to dependencies

Also, if you add an atom in the database file so you can verify the connection is actually correct inside handler, you see that Component recreates the dependencies.

If you started and stopped a component and then held onto it “forever”, you’d keep a reference to a stale database in the handler (even tho’ its actual connection resource had already been cleaned up). But that’s not how things work in the real world.

Normally, when you stop a Component, you either throw away the whole thing or you call start which recreates the dependencies.

If you instrument the Database start and stop functions fully, you’ll see

lein test app.system-tests
start the database with connection#1
start the handlers, database is: connection#1
stop the handlers
stop the database, releasing connection#1
start the database with connection#2
start the handlers, database is: connection#2
stop the handlers
stop the database, releasing connection#2

Here you see that Component has started the database and bound it into the handler by the time it calls start on the handler — by that point, the old database dependency is no longer referenced: no leak even of the memory.

^ does that help @gmercer and @fbielejec ?

I understand that it is cleaned up - I guess @fbielejec was asking why it doesn't look like it was cleaned up

https://cb.codes/a-tutorial-of-stuart-sierras-component-for-clojure/ the defrecord Worker definitely clears out references to dependencies if @fbielejec was to follow this pattern the confusion would subside, yes?

actually that example seems to not correctly deal with the stop-chan 😢

No lisp here https://github.com/tensorflow but haskell and rust 😞

Just to make sure if I use r/map on a vector it will run in paralell? Which structures default to non-concurrency evaluation?

Same for pmap

@hlolli as long as we have java 😄 https://github.com/kieranbrowne/clojure-tensorflow

@liminal18 only r/fold works in parallel on maps and vectors. pmap will do chunked parallel execution over sequential data.

@alexmiller ok thanks so what struts does r/map do parallel execute on?

r/map doesn't do anything by itself. You need to use it with r/fold to get parallel execution.

You should read https://clojure.org/reference/reducers if you haven't yet

And I have longer explanation in Clojure Applied

Is there a Clojure library for producing plain text report - for example Household Pet Inventory, Page 1 ID NAME DIET OUNCES ------------------------------------------------- 12 Buster Tuna 1.00 1 Mimi Pounces 0.30 37 Ginger Fancy Feast 0.75 19 Ellie White Castle 9.99 The columns are aligned, decimal points may be aligned etc

@saicheong The function that does just what you need is right there in your clojure installation. cl-format

@saicheong there's also https://clojuredocs.org/clojure.pprint/print-table

@blueberry could you explain how you can use cl-format to build a table?

For the one in the example, I'd map the input sequence of data with a cl formatter function, which can finely format and align the columns.

and the OP did not even request a table just a plain text report that has nice alignment.

Out of curiosity, is there any way to execute one-off tasks via nREPL? E.g. to migrate the db after deploying the webapp:

lein repl :connect  "(app.db/migrate)"

you can host nrepl in your app, but it's easier to use clojure's built in socket server

that's just an arg to your java command

-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"

avoid using lein to run an app in production - it's a build tool

Right. But assuming the server is already running in webapp (at port 7888), is there then a way to execute a Clojure form via CLI once?

wouldn't this work? echo "(app.db/migrate)" | lein repl :connect

@metametadata yes, if it starts a repl host, and I'm recommending using a raw socket server (built into clojure) instead of using lein for the client and server

but pesterhazy's answer works if you are determined to use lein in production

I think @metametadata wants to work with an existing server, not adapt a solution where you have control of the server

well, I'm ambivalent about the specific client, it can be lein repl, boot or something else 🙂

so yeah, let's say I just want to migrate the db after deploying the app

piping actually works!

but it doesn't seem to return the error exit code if there was an exception

@metametadata don't use lein on production, but if you do at least address the points here, it's a lot simpler to use an uberjar and there are far fewer gotchas https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#server-side-projects

agree with noisesmeith that the socket server is a much better solution - you can just use netcat then

my the theoretical plan is to deploy an uberjar, and then use nREPL to perform one-off tasks, like migrations

I don't think it counts as using lein on production

OK - just making sure

in that case the socket server, built into clojure, is a lot easier to do than running nrepl inside your own app and using lein to connect to it

I used to start an nrepl server, before the socket repl option existed

Probably yes, I can use the command-line option to start the server in uberjar instead of hardcoding it in application. But there's still a question on how to easily execute tasks via CLI without interactively running an REPL session 🙂 @pesterhazy provided one of the solutions.

netcat does this with the socket repl

echo "(db/migrate)" | nc localhost 5555

ah, got it, I'm going to try this

I missed that there's a difference between socket and nREPL protocols

yeah, socket repl is just bytes in bytes out

$ echo "(println \"hello\")" | nc localhost 5555
user=> hello
nil

I bet it would even show you a stack trace if you had an error...

$ echo "(/ 1 0)" | nc localhost 5555
user=> ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
user=>$

Yup, this works, thank you! Though it also doesn't return the exit code:

ᐅ echo "(/ 1 0)" | nc localhost 5555
user=> ArithmeticException Divide by zero  clojure.lang.Numbers.divide (Numbers.java:158)
user=> %
ᐅ echo $?                           
0

And if I do want to sometimes control the server via interactive REPL then I suppose nREPL is more user-friendly? E.g. I can see that it supports autocompletions when used via lein repl :connect.

yeah - but clojure doesn't have exit codes, but it does print return values

@metametadata only if you load the right nrepl middleware to support it, but yes

I find rlwrap plus a socket repl suffices, and it reduces the complexity of my code and the size of my jar to leave nrepl out

Got it! Yes, I added (clojure.tools.nrepl.server/start-server :port 7888 :bind "localhost") into the sample project here.

yeah, but that won't give you completion - you need middlware set up for that

hmm, I doubt I added the middleware manually

if it's not in the start-server can it be the feature of lein then?

lein is setting up your nrepl middleware

I mean - maybe you are getting completion out of the box? it wasn't so simple when I last checked...

I dunno, I think I will have to re-read the docs 🙂 My impression was that middleware is added on the "server-side".

it is

but your project still has to set up the middleware, unless the completion middleware is installed by default

okay, then it looks like start-server is all I added

@noisesmith after digging in sources it looks to me like autocompletion is the feature of lein repl and it's implemented in https://github.com/trptcolin/reply

right, but it's done via middleware - you can add reply to your app

that's what I was trying to say

(not knowing the precise package adding it...)

aah, okay 🙂

and in that case it would also work with say rlwrap nc

well, for interactive I'd use telnet instead of nc

but it relies on the nrepl protocol

right right

so it's not quite that simple

nrepl is more complex than you might think, it abstracts messages and replies with data structures

is there any book / tutorial on "deploying clojure on aws" ? I find all sorts of docs on clojure web dev, but none talk about deploying to aws. Then, I find all forms of docs on AWS ec2/beanstalk/rds/.... and getting stuff to play together, but they'rea ll ruby on rails / php / python ... and not clojure.

which is something things like reply rely on

@qqq simplest is to make an uberjar, ensure that java is installed, and run it

yeah that's great for webserver

you don't need any other deps or setup to run it

but then, how do you connect it to RDS ?

RDS is a database, you give it your database config - or at least that's how I do it

so I have a RDS postgresql instance; I have a elastic beanstalk server running, but it's not clear how to get the two to talk to each other

OK you really don't need elasticbeanstalk but if you wnat that you don't want an uberjar you want an uberwar

and you have to mess with the servlet configs

uberjar is easier

I have uberjar on beanstalk

all that said, you tell your clojure app the credentials to the rds db

beansatl ksupports both tomcat war and java jars

oh, it's changed since I used it then

but you're right; maybe it'll be eaiser if I don't use beanstal at all

and just use ec2 + rds

anyway, RDS is just a db, you give your app connection config like you would with any other

it'll be easier to see how stuff fits togehter without all the magic

there might be something fancier, but I haven't needed to try it

@noisesmith : that would be complecting it 🙂

What is the different between send and send-off for agents? I know theory send is for actions CPU limited and send-off for potentially block on IO. But how exactly this functions are different? I see it is about clojure.lang.Agent/pooledExecutor vs clojure.lang.Agent/soloExecutor.

@kwladyka the executors are different, the one used by send-off is an expandable pool, the other isn't

if your work is CPU bound, expanding the pool of threads to do it in slows you down

if it's IO bound, it's good to be able to expand the pool

@noisesmith that's a little bit risky, though, right? You'd want to ensure an attacker can't cause you to spawn 10 million threads.

With the auto expanding thread pool

right, it's not a good idea to let requests map 1:1 with your code period

there should be backpressure before anything touching agents

There you go using big words again 😛

So, what you're saying is, I should be managing my send-offs behind a queue either way, right?

Which I can rate limit as necessary