any idea why advanced compilation might fail to connect to the repl even when the browser correctly shows the cljs page at localhost:9000?

clj --main cljs.main --optimizations advanced --compile cloudier.core --repl just stays forever at Waiting for browser to connect to ...

I would not be surprised if the repl stuff isn't part of advanced compliation builds

huh, yeah I guess that makes sense

Yeah I think you're probably right

I don't know for sure, just a guess

you made me realize that the quickstart guide omitted the --repl command when it added the optimizations flag

so, I think you're right

is there a clj option to just run a node script directly? like compile and run in one step? I guess I could just use && :thinking_face:

yeah, &&

@idiomancy Yeah, you can't have a REPL with :advanced ClojureScript builds. One of the main reasons is that much of the ClojureScript standard library (the functions in cljs.core) are eliminated if unused (dead code elimination, or DCE). So, for example if you were to to try to evaluate, say (simple-ident? :xyz) it is possible that simple-ident? no longer exists. A second reason is that even if the functions still theoretically "exist", they are often inlined or their names shortened via mangling to reduce code size.

If you are using cljs.main you can run CloureScript programs directly just like you can run Clojure programs directly, simply by passing the name of the source file. So, for example:

clj -m cljs.main -re node foo.cljs

Hi. I would like to use hiccup and garden to output to local files (for serving them statically later), auto-compile them after changes (so that I can watch the directory and live reload) and, if possible, avoid using leiningen or boot. Any ideas on how to do this or if a solution exists somewhere? I've tried hard to find a resource for doing just this (writing good old HTML and CSS with Clojure only in the simplest manner). All tutorials either assume you are writing a SPA and need re-frame/etc or that your server will be rendering the Clojure code.

The idea is to have front-end friends to use Clojure to write static assets and actually understand the plumbing (that's why I would like to avoid build tools), so they can slowly get the idea of it.

@anantpaatra you can use the same libraries for server side Clojure, and just write it to a file instead of serving it over HTML

Does the -> thread macro keep all intermediary collections in memory? I'm working with a huge pair of vectors of maps in a deep merge and I believe a thread macro might be the culprit behind my current memory heap exhaustion (JS)

would it be useful to work with large collections as a multiple separate operations instead of using the thread macro?

-> is just a syntactic rearrangement of code that happens during macroexpansion

The -> macro basically re-writes (-> foo bar baz boom) into (boom (baz (bar foo))), as I understand (and oversimplify) it.

That’s not over simplified at all

Just so

well, I just meant -> also handles more complex stuff too

Could it be something between these?

Could be, if your maps are big enough

is there a way to break this up while still being able to merge by groups?

should i return the grouped vector then transduce over the results with the deep merge?

not sure what - if anything - can be done about the memory exhaustion

instead of (for ...ing it

well, for returns a lazy seq, so by itself it’s probably not exhausting your memory, but depending on what you do with the result of calling group-and-merge, you could be holding on to the head, and that could cause problems.

the group-and-merge is applied as an core.async/map on two chans

what does it mean to "hold on the head"?

You’re familiar with linked lists?

a little, yes

Ok, a lazy seq is like a linked list — each element has a pointer down to the next element.

ok

The first element in the list is the head, and everything after that is the tail

following

If you are processing the list and releasing each element as you process it, the garbage collector can trim nodes off as you go, and keep memory free

that sounds good 🙂

but if you retain a reference to the first element, the garbage collector says, “Nope, still in use, can’t recover”

does for maintain the head?

so the whole list remains in memory

for by itself won’t hold onto the head, it just generates the initial list

It’s what you do with the list after you get it back from for that’s potentially a problem

so for generates a linked list/lazy seq

right

would applying the function as map be better?

No, for 2 reasons: you want a lazy list because that helps you avoid memory exhaustion, and also map also returns a lazy list 😉

shucks

There’s a mapv that returns the entire list as a vector, all at once, which means it immediately consumes all the memory needed to hold the list. Which is the opposite of what you want, but it’s nice to know about map vs mapv

Anyway, back to your problem…

hmm... but perhaps mapv would be ok, because it seems that the function being applied to the lazy list is what's exhausing memory... the list itself should only be about (max) 100mb

Hmm, mapv uses more memory, so I’d be surprised if it solved the problem

ok, so no bueno

but thank you for bringing mapv into my lexicon 🙂

Can you share a gist or something with more context? It’s hard to tell where your problem might be with just that snippet

sure, sec...

@loganpowell @manutter51 ClojureScript holds head; it doesn’t have locals clearing

ah, so there's no way around it?

Ah, that’s right, forgot that.

Two ideas. One is dirty and describe in http://blog.fikesfarm.com/posts/2016-01-15-clojurescript-head-holding.html Second is if you can be clever with transducers you might be able to pull it off. See the example in https://clojurescript.org/news/2018-03-26-release#_reducible_sequence_generators

Thank you guys! Let me read those

If you are using the transducer trick, it is based on some new sequence generators that were added in 1.10.238

A third idea (which seems less reliable, but sometimes works, depending on the JavaScript VM), is to write a low-level loop / recur, attempting to overwrite previous pointers

This is great, at least I see a glimmer of light at the end of the tunnel 🙂

It is a great opportunity to grok transducers 🙂

I am using transducers elsewhere, but I do copy a lot of older code 😄

The TL;DR is to eliminate the sequence (or at least intermediate ones)

yes, that's what I want to do

so, I would use a transducer instead of for?

I think Christophe has one...

https://github.com/cgrand/xforms

But if you don't want to take on a dep, I would just try to rewrite the logic using core transducer functionality

Dude, thank you

I will try it with xforms to see if it solves the problem and if so try it myself

I ❤️ the Clojure community

If you are processing two collections, I can't recall but I think Steve Miner came up with a cool transducer-based approach to a similar problem we were pondering on the Apropos podcast... seeing if I can dig up that gem

yes, that's exactly my bottle neck right now... merging two very long lists of maps

Right, the fundamental question might be: How do I get my two long lists into a single thing that I can transduce over, without producing yet another sequence?

yessir

Dang, I think it was this reference https://twitter.com/miner/status/975781512393252865

I can look through episode 3

It might be the case that we discussed this very problem on that episode 🤞

Otherwise, if you just need to merge two long lists of maps, can you do one, and then do the other? I don't know if you posted a precise (but smaller) example of the data transformation you need to make.

the two lists contain "partners"/compliments of each other they are only made complete by merging them together (not two of the same lists)

I can share a snippet, but it might be a PITA to understand... I'm not sure how well I even understand it 😄

should I?

Let me dig through some of the resources you've shared

Is it something like this?

(map merge [{:a 1} {:b 2}] [{:c 3} {:d 4}])

yep

Does the two lists have the same size? Do items occur in the same order?

(map merge-with...

prior to the merge they are grouped by a common key

Too bad you can’t do (into [] (map merge) coll1 coll2)

I augmented each of the maps with a composite key that allows them to be grouped

ok so it’s not (map f coll1 coll2) territory siince the list order is accidental, what matters is your grouping-key

ah

Maybe you could still loop / recur to walk both lists

so try to merge-with a single map across the second list of maps, one at a time?

instead of grouping?

Another question in my mind: Are the two long lists already in RAM, or are they generated / fetched, and is that's whats blowing out the heap?

How big are the input lists? Are they stored in memory or lazy? (same question as Mike’s)

they will each be in a chan that's merged in a core.async/map function

so, I believe they'd both be in memory separately

Another related question: Does your output list need to be completely in RAM, or can it be consumed lazily?

it's returned as single GeoJSON featurecollection

once it's merged

Your input lists may not be in RAM if, say they involve fetching elements from a backend one at a time.

the two lists are created from two separate APIs

but they are fetched as a single request each

the APIs return lists

OK, it is probably safe to assume they are both in RAM. And at that point the heap can hold them.

The question seems to boil down to: How do I produce this third collection? If it magically appears, can the two input lists and the output collection all fit in heap?

The biggest responses are @ 100mb

you got it

"wishful thinking" ™️

Hmm… maybe your input collections are lazy and you are the only one holding onto them?

If so, you could walk them trying to let go of their heads as you do so

if I can just figure that part out 😄

ClojureScript probably doesn’t help either, especially if those collections are being referenced in function parameters

as to @cgrand s point, could the group function be the culprit?

what's educe?

eduction?

i suspect a typo of reduce?

perhaps

how about fold?

Did you try to capture two input lists, put them in vars and play with them at the repl?

hmm, let me give that a try

(trying to figure out if heap is saturated beacuse of “infrastructure” code, or just the computation + data)

the REPL to the rescue

did you just update your xforms project?

I'm looking at it now 😄

First, here is a saner function to replace your deep merge:

(defn deep-merge [a b]
  (if (map? a)
    (into a (for [[k v] b] [k (deep-merge (a k) v)]))
    b))

that will do a merge-with?

seeing this code makes me feel really dumb

😄

=> (deep-merge {:a 1 :b {:x 1}} {:a 2 :b {:y 4}})
{:a 2, :b {:x 1, :y 4}}

that should do it

ok, let me use that and do some REPL

your deep-merge works like a charm

still get the heap overflow, but I love the new function

Check list 1/ you have your two lists in two vars 2/ you can call count on these vars without blowing the heap 3/ manually picking one map of each and merging it with deep-merge doesn’t blow the heap

I've shared it on the gist: https://gist.github.com/danielpcox/c70a8aa2c36766200a95#gistcomment-2711849

You can use merge-with

(defn deep-merge [a b]
  (if (map? a)
    (merge-with deep-merge a b)
    b))

shared as well

@cgrand let me give that a go

and a variation on group-and-merge

(defn group-and-merge [coll1 coll2]
  (let [coll1-by-key (into {} (for [x coll1] [(:your-key x) x]))
        coll2-by-key (into {} (for [x coll1] [(:your-key x) x]))]
    (vals (deep-merge coll1-by-key coll2-by-key))))

How can I multithread applying a function to members of a collection?

I have a sequence of files that I'd like to read through and make REST calls for.

I have everything working, just not sure what the right way to make it execute in multiple threads.

pmap, reducers/fold or async/pipeline-blocking

@scott.archer I believe the last one is the better fit for what you described

So with pmap i would just end up with a sequence of nil

That you would have to consume to drive computation.

I need to call rest services, so I think it would all just be side effects, nothing to consume unless i'm misunderstanding the use of pmap.

Pmap is lazy so if you don’t consume (E.g. dorun) it nothing will be computed, no requests sent.

I'm doing clojure and the brave. In the chapter on macros the following is written

(defmacro code-critic
  "Phrases are courtesy Hermes Conrad from Futurama"
  [{:keys [bad good]}]
  `(do ~@(map #(apply critize-code %)
             [["Great squid of Madrid, this is bad code:" bad]
              ["Sweet gorilla of Manila, this is good code:" good]])))

that function gets passed a map as its single argument

one reason is that if you just make your parameters [bad good] then you always have to pass them both. By passing a map you can assume that any parameter which doesn't correlate to a given key in the map is nil.

Also, you can use {:keys [...] :or {...}} for defaults

I would say it's useful in cases where you want params to be optional, or when there are a lot of params it is easier to change your code later if you have passed them in as a map rather than maintaining a lot of different parameters being passed around

Stuart Sierra’s advice: https://stuartsierra.com/2010/01/15/keyword-arguments-in-clojure

So when you do it that way you have to call code-critic the following way?: (code-critic #{bad (1 + 1), good (+ 1 1)})?

Since it expects a map?

#{...} is a set, {...} is a map

(code-critic {bad (1 + 1), good (+ 1 1)}) than?

Because that prints nil

(because bad and good are nil)

bad and good should be keywords

Oh yeah it works if use :bad and :good

(defn foo 
  [{:keys [bad good]}]
  [bad good])

=> (foo {:bad 1})
[1 nil]

Hello. very beginner question. Clojure begs for top-down planning and implementation. Which is ok, as I can't know all the tiny small "bottom" functions from the beginning. But if I start with top-down design and functions like "setup-game, game-turn" how can I have some testing capability against such high level functions if I don't have a meat on the bones (and even bones are more like skin)

Another question, as Clojure lets you play with maps, lists and mostly all the system data might end up in multi-level complex compound map/list, how such structures can be tested? Test data prepartion itself is huge work. The same "game-turn" function might take whole game state inside, and I still don't see some convinient method of testing such function? Does it means that as big functions are composed small functions, it follows that tests should be done the same way?

"The same "game-turn" function might take whole game state inside, and I still don't see some convinient method of testing such function?" I think in this case you would write a function that updates a certain value of the game state, then test the transformation, for example:

(defn next-turn [game-state]
  (update game-state :turn inc))

You could test this by simply asserting (= (next-turn {:turn 10}) 11), so while it takes in the entire game state when it runs you want to test each transformation on values as a pure function.

At the company I work at, we generally do thorough testing of the smaller functions, then for the more broad functions you just need to test what they are doing - rather than the output of each smaller fn.

@christian.gonzalez Ahh makes sense. If my "big function" does nothing, so I can test like "empty" nothingness, but at the moment when it starts doing even small value changes, I can also have test exactly for that small change without "looking" at the big container? Makes sense. Thanks.

@trailcapital But how you design system from the beginning? Most likely you start at pretty high level of abstraction? So how tests evlove against such design? Or there is different approach?

I am interested in the evolutio of tests against actual system, at the same time I would like to not box myself into "thinking in 5 turns ahead" situation - thats why I want to dive into Clojure in the first place.

One thing that really helped me wrap my head around how to transform, maintain, and test an application state in Clojure is the Clojurescript Re-frame library... maintaining state for a SPA. The idea of having an application state, updating, assoc-ing, dissoc-ing values based on user interaction with the application. An important concept is separating the pure functions from stateful, impure functions so that you focus on testing idempotent functions.

" maintaining state for a SPA." <- What is SPA?

single page application

It's just a suggestion, there are other ways to practice writing Clojure if you have other interests.

I still need to do simple programs, web part would be next step - when I want to provide some transport layer to the "outer world", for the beginning I would be happy to write simple text-based game, but complex enough to cover most of Clojure basics... what I want to learn first, to make my brain thing in Clojure... not to try turn Clojure into some sort of self-invented Java and write Java

@janis.pauls If you're already used to a TDD workflow, I think you can apply that "as-is" to Clojure design/development. Another tool you might find handy for dealing with "complex" state and test data is clojure.spec -- if you write a (minimal) spec for your data (and refine/add to it as you grow your program), then you can also leverage spec to generate test data for you and verify the results are as expected.

That's a great idea, I think games are well suited for functional programming.

I think one of Clojure's great strengths is that you can "grow into" the available tooling as you "grow into" the language itself -- because so many things are composable (and not inherently coupled).

I hear abuyt that clojure.spec - probably have to look at it

When I learned Haskell I was tasked to write a Sudoku solver/ui

I downloaded Uncle Bob's Martin space war game written in Clojure, but it is a bit complicated for me yet to read and get my head around it

Great way to “grow” into a functional mindset

The second task was writing a super simple compiler

my original background is Java/Android and I am sick of Android classess with zillion of member variables, all changing arbitrary, impure functions here and there... and I am learning Clojure more and more, and hope my thinking will change into making programs as Functional as possible (by functional mostly meaning - composition of functions, instead of line-by-line impeartive style)

also the game implies some graphical interface - does it mean I just take java awt and swing components and do stuff there, or is there some nice graph libraries out there already wrapped for Clojure - I mostly will need to display game board with pieces on it and text, no dynamic 60fps shooting, explosions etc.

There was a talk I watched when I first started learning Clojure, it might be of help https://www.youtube.com/watch?v=0GzzFeS5cMc

in short - want to try to implement Cave Troll board game (not AI yet, just for players to enter turns and display results)

Ahh nice video, going to watch. Thanks people... Have a good night

So, when we are talking about organisation anyway what kind of conventions do you guys have for aliasing namespaces

I started sort-of shortening them to three letter acronyms, but I am starting to get collisions and have to juggle many acronyms in my head at the same time

I also made it kinda convension to have protocol aliasses ending in p, but well I got myself in trouble when an acronym of an implementation of a protocol also ended in p

I find using an IDE or something like Cider in a text editor helps identify things that are already aliased or reserved.

For example not setting clojure.string to str since str is a function.

well if the ns is called organization.myapp.section.stuff.thing I usually do stuff.thing or thing

as much as possible, I find using the last segment of the namespace as the alias reduces confusion

gah we said the same

and using a standard set of “short” aliases for things you use a lot

s for spec, str for string, jio for http://java.io, etc

really, I think the most important thing is for everyone on a project to use the same set of aliases, whatever they are :)

> not setting clojure.string to str since str is a function. I always alias clojure.string to str. they don't conflict at all

same with clojure.set and set

It doesn't conflict, I just prefer not to....

I have run into people who had a hard time getting over the fact that a namespace can be be aliased with name of a pre-existing function without a problem, so I try to avoid it as well - for others sake

what a weird thing to think, you can never use ns aliases as a value

I really have to find a balance between not beating a dead horse with naming (e.g. (message-protocol/get-message ...)) and having a garbled mess of acronyms (e.g. (mp/get-message ...))

if mp is used a lot it's worth it, else I'd just go with message-protocol, which isn't half bad

Hmm true also, I’ll try to find that partition

naming things remains hard…

I better not tell you about cache invalidation then

I only read half of that message, the rest was parts of your previous

I find it helpful to think about naming functions in a namespace, in the context of the last segment of the namespace name itself (i.e., assume the "default" alias will be used and see how functions look when called that way).

That can also help when choosing namespace names since you focus on the last segment of the name as something all your client code will be using.

Can you give an example?

set/union

Ahh yeah, ofcourse

I’ll try some of these suggestions tomorrow, thanks!