Fork me on GitHub
#clojure
<
2020-09-15
>
zackteo03:09:16

Hi guys, I have this project https://istd50043.github.io/project#implementation-requirements I'm trying to consider the possibility of doing this solo (so that I can do Clojure full stack) I'm thinking the "Production System" can be done with Clojurescript. "Analytics System" can be done with Clojure. But I also need to do automation scripts for AWS - not sure how possible is it to do this with babashka :o I know might be pretty intense to tackle alone. But am wondering how intense/possible is it :x (Also not sure if this is the best channel for this)

dpritchett16:09:33

For simpler shell scripts you might consider using #joker and wrapping the AWS CLI tool rather than importing the AWS SDK into your clojure of choice.

seancorfield03:09:05

@zackteo If you're not already a member, I suspect the #aws and maybe #aws-lambda channels will be helpful to you. I don't know if there are channels specific to automating ops stuff on AWS.

zackteo03:09:34

In that case, I'll move this there

seancorfield03:09:26

It's a big enough project that different parts of it will likely be best served by asking questions in several different channels but I think some of the core questions about AWS will need answering first and those are almost certainly "non-Clojure" questions. I think it's an interesting project and should be a good challenge.

zackteo03:09:10

I think it will be very good for me to attempt to tackle it myself. Especially with groups you end up missing out on learning the different parts. But the wonder is will it be too overwhelming a task that it would be better to take a step back

seancorfield03:09:13

There are some Clojure-based projects for managing AWS ops stuff (Pallet comes to mind, there's a #dda-pallet but I don't know how active it is) but I'm not sure how active any of those ops projects are. I think people went back to more mainstream tech for that sort of thing?

borkdude06:09:19

@zackteo aws is a recurring topic in #babashka - welcome to join and thinking about it

zackteo09:09:58

@seancorfield am gonna do the project in Clojure after all! :D I miraculously found 2 peers to join me to form a 3 man team :) so it isnt as ridiculously daunting anymore

6
🎉 3
Freddy13:09:13

What would be the best way to organize such a program in clojure ? I want to concurrently build a set of unique things existing across a cluster of "locations" (DBs in reality). Let's say for simplification that I have an expensive func (get-ids location) which might take a while to return. I have a list of locations, say a few thousands of them, and I want to share the load of calling get-ids for each of them, across, say, 16 workers, and finally get a unique set of values out of this. I have to limit the number of concurrent workers because in effect, each worker opens a connection.

Freddy13:09:51

that looks great! thanks

Freddy13:09:51

just out of curiosity, how would you go about organizing such a thing with the core libs, as in, for an ad-hoc solution for a one time problem?

jsyrjala13:09:57

My first attempt would be something like this

jsyrjala13:09:06

(require '[com.climate.claypoole :as cp])
(let [adresses [address1 address2 address3]]
  (cp/upmap 10 (fn [address] (connect-address address)) addresses))

Freddy13:09:08

yep ok, using claypoole then, Tbf it looks simple enough to be pulled in, thanks

Freddy13:09:19

Tried my way around with pipeline but ended up having bloated parts to rebuild the final list of results out of channels that I'm not sure they're done fetching.

benoit13:09:59

I would write a pipeline that fetches the location in a thread pool or asynchronously and plug the distinct transducer on the output channel.

Freddy13:09:42

I think I went more or less that way, wasn't sure how to ensure my output channel gave me all expected results though

Freddy13:09:21

(so I could dump my final list to a file and exit the program)

benoit13:09:12

If you close your input channel after posting your locations, the pipeline-blocking function will close your destination channel.

benoit13:09:16

So when your code detects that the destination channel closes, you can assume that you have the complete set of ids and can dump them in the file.

Freddy13:09:51

ah yes, good idea

benoit13:09:55

And if you build your input-channel with to-chan on the collection of locations. you don't even have to bother closing this input channel, it will be done automatically.

Freddy13:09:12

would you assume that the chan is closed when you're taking nils out of it ?

Freddy13:09:40

yep, feels weird but makes sense, thanks a lot

Alex Miller (Clojure team)13:09:33

nil is not an allowed value in a chan

Alex Miller (Clojure team)13:09:42

so if you take and get a nil, you know the chan is closed and drained

👍 3
nivekuil16:09:24

what's a good way to express "at least 2" elements in a sequence? I believe count will traverse an entire seq?

delaguardo16:09:56

you can use bounded-count

ivana16:09:01

bounded-count came from one recent release, if you have older clojure, you can use simply (->> x (take 2) count)

nivekuil16:09:56

oh, cool thanks :) tbh I wish there were a many? in clojure.core that did this; I find myself having to sprinkle this around in a few places for UI-related stuff since we humans like our grammatical numbers

ivana16:09:20

it seems that nobody else needs it

nivekuil16:09:46

actually, (count (range ..)) appears to be constant time in cljs, so it seems to always be ￱~￱4x faster to use count instead of bounded-count there -- not that it's meaningfully slow either way, at about 25 nanos for bounded-count according to simple-benchmark.

dpsutton16:09:15

range implements ICounted so it will be constant time

Alex Miller (Clojure team)16:09:49

only finite long ranges

dpsutton16:09:50

ICounted
  (-count [rng]
    (Math/ceil (/ (- end start) step)))

Alex Miller (Clojure team)16:09:09

infinite ranges obvs can't be ICounted

dpsutton16:09:05

its nonsensical but it works

(count (range))
1.7976931348623157e+308

dpsutton16:09:07

yes. in cljs it seems they are never infinite. ([] (range 0 (.-MAX_VALUE js/Number) 1))

dpsutton16:09:38

never delved into this before so fun to learn

Alex Miller (Clojure team)16:09:41

I mean, technically even in Clojure counting is int-bounded due to Java collection apis right now

isak16:09:12

Would tell you guys what it returns in babashka, but still waiting for the answer

borkdude17:09:16

bb is JVM-based so same as clj

3
dpsutton16:09:13

ah, the clojure version of (range) is (iterate inc 0) which obviously won't implement Counted whereas the "infinite" version of (range) in cljs does

Alex Miller (Clojure team)16:09:34

also, the non-long version of range in Clojure does not implement ICounted

Alex Miller (Clojure team)16:09:44

as there are many weird edge cases

dpsutton16:09:56

yeah i remember you saying you spent a lot of work in there

dpsutton16:09:16

thanks for that tedious work 🙂

Alex Miller (Clojure team)16:09:46

user=> (counted? (range 0.0 10.0))
false
user=> (counted? (range 0 10))
true

nivekuil16:09:25

user> (counted? '(1 2 3)) true user> (counted? `(1 2 3)) false

nivekuil16:09:30

that I really don't understand.

Alex Miller (Clojure team)16:09:38

the first is a persistent list, which is realized (and thus countable, like all persistent colls)

Alex Miller (Clojure team)16:09:54

the second is a sequence

Alex Miller (Clojure team)16:09:12

sequences are not (necessarily) realized so not necessarily countable

Alex Miller (Clojure team)16:09:58

why it's a sequence is pretty obscure and probably should be considered an implementation detail

nivekuil16:09:00

very interesting. think that's mentioned here https://clojure.org/reference/reader#syntax-quote

mauricio.szabo22:09:40

Hi, I'm trying to test a highly asynchronous test, and found out some bizarre behavior of clojure.test: sometimes, my assertions simply are not registered if they are on another thread. I was able to shring to the following code:

(t/deftest bar
  (t/testing "Why this does not work?"
    (.submit (java.util.concurrent.ForkJoinPool/commonPool)
             ^Runnable (fn []
                         (Thread/sleep 200)
                         (t/do-report {:type :fail, :expected 10, :actual 12})))
    (Thread/sleep 1000)))

mauricio.szabo22:09:06

The task submitted to the ForkJoinPool, when it calls do-report, prints the failure on my terminal (but not on nREPL / Socket REPL) but does not register on clojure.test that a failure occurs

ghadi22:09:19

fail how?

mauricio.szabo22:09:21

It doesn't register that there was an assertion, nor register the failure. So run-tests thinks that it's an empty test and returns a success.

ghadi22:09:44

that pool submission code above does not convey dynamic bindings...

ghadi22:09:10

guess: do-report uses dynamic bindings?

seancorfield22:09:48

@mauricio.szabo I think @ghadi is right -- clojure.test is full of dynamic bindings so that's likely to be your problem.

seancorfield22:09:36

*testing-vars* and *testing-contexts* are the two used by do-report that jump out at me immediately.

mauricio.szabo22:09:16

Oh well... So I'll need to bring these asserts into the main thread, right?

hiredman22:09:27

That is usually for the best, but you can also look at bound-fn

hiredman22:09:59

If you don't put the assertions on the main thread you also have to make sure your assertions on other threads have run to completion before the main thread exits, which can also be tricky

mauricio.szabo23:09:31

@hiredman yes, I was in fact working on a library to ease async tests in ClojureScript, and wanted feature-parity with Clojure. Seems that it'll not really be possible, unfortunately