Fork me on GitHub
#beginners
<
2018-11-14
>
idiomancy01:11:32

any idea why advanced compilation might fail to connect to the repl even when the browser correctly shows the cljs page at localhost:9000?

idiomancy01:11:04

clj --main cljs.main --optimizations advanced --compile cloudier.core --repl just stays forever at Waiting for browser to connect to ...

hiredman01:11:47

I would not be surprised if the repl stuff isn't part of advanced compliation builds

idiomancy01:11:44

huh, yeah I guess that makes sense

idiomancy01:11:52

Yeah I think you're probably right

hiredman01:11:05

I don't know for sure, just a guess

idiomancy01:11:03

you made me realize that the quickstart guide omitted the --repl command when it added the optimizations flag

idiomancy01:11:09

so, I think you're right

idiomancy01:11:50

is there a clj option to just run a node script directly? like compile and run in one step? I guess I could just use && :thinking_face:

mfikes04:11:48

@idiomancy Yeah, you can't have a REPL with :advanced ClojureScript builds. One of the main reasons is that much of the ClojureScript standard library (the functions in cljs.core) are eliminated if unused (dead code elimination, or DCE). So, for example if you were to to try to evaluate, say (simple-ident? :xyz) it is possible that simple-ident? no longer exists. A second reason is that even if the functions still theoretically "exist", they are often inlined or their names shortened via mangling to reduce code size.

mfikes04:11:57

If you are using cljs.main you can run CloureScript programs directly just like you can run Clojure programs directly, simply by passing the name of the source file. So, for example:

clj -m cljs.main -re node foo.cljs
This will compile and run foo.cljs under Node, all in one fell swoop.

felipebarros04:11:26

Hi. I would like to use hiccup and garden to output to local files (for serving them statically later), auto-compile them after changes (so that I can watch the directory and live reload) and, if possible, avoid using leiningen or boot. Any ideas on how to do this or if a solution exists somewhere? I've tried hard to find a resource for doing just this (writing good old HTML and CSS with Clojure only in the simplest manner). All tutorials either assume you are writing a SPA and need re-frame/etc or that your server will be rendering the Clojure code.

felipebarros04:11:39

The idea is to have front-end friends to use Clojure to write static assets and actually understand the plumbing (that's why I would like to avoid build tools), so they can slowly get the idea of it.

pradyumna05:11:44

not sure if i understand your requirement. i would just write a small command line program in clojure to watch a folder and generate the static files. this program can be built with lein or boot. then others would just run this uberjar and won't have to worry about any tooling setup.

lilactown06:11:25

@anantpaatra you can use the same libraries for server side Clojure, and just write it to a file instead of serving it over HTML

Logan Powell12:11:50

Does the -> thread macro keep all intermediary collections in memory? I'm working with a huge pair of vectors of maps in a deep merge and I believe a thread macro might be the culprit behind my current memory heap exhaustion (JS)

Logan Powell12:11:42

would it be useful to work with large collections as a multiple separate operations instead of using the thread macro?

Alex Miller (Clojure team)12:11:36

-> is just a syntactic rearrangement of code that happens during macroexpansion

manutter5112:11:57

The -> macro basically re-writes (-> foo bar baz boom) into (boom (baz (bar foo))), as I understand (and oversimplify) it.

Alex Miller (Clojure team)12:11:30

That’s not over simplified at all

manutter5112:11:24

well, I just meant -> also handles more complex stuff too

Logan Powell12:11:15

Could it be something between these?

manutter5113:11:39

Could be, if your maps are big enough

Logan Powell13:11:34

is there a way to break this up while still being able to merge by groups?

Logan Powell13:11:21

should i return the grouped vector then transduce over the results with the deep merge?

Logan Powell13:11:59

not sure what - if anything - can be done about the memory exhaustion

Logan Powell13:11:31

instead of (for ...ing it

manutter5113:11:48

well, for returns a lazy seq, so by itself it’s probably not exhausting your memory, but depending on what you do with the result of calling group-and-merge, you could be holding on to the head, and that could cause problems.

Logan Powell13:11:42

the group-and-merge is applied as an core.async/map on two chans

Logan Powell13:11:27

what does it mean to "hold on the head"?

manutter5113:11:00

You’re familiar with linked lists?

manutter5113:11:48

Ok, a lazy seq is like a linked list — each element has a pointer down to the next element.

manutter5113:11:02

The first element in the list is the head, and everything after that is the tail

manutter5113:11:06

If you are processing the list and releasing each element as you process it, the garbage collector can trim nodes off as you go, and keep memory free

Logan Powell13:11:17

that sounds good 🙂

manutter5113:11:31

but if you retain a reference to the first element, the garbage collector says, “Nope, still in use, can’t recover”

Logan Powell13:11:38

does for maintain the head?

manutter5113:11:43

so the whole list remains in memory

manutter5113:11:07

for by itself won’t hold onto the head, it just generates the initial list

manutter5113:11:23

It’s what you do with the list after you get it back from for that’s potentially a problem

Logan Powell13:11:23

so for generates a linked list/lazy seq

Logan Powell13:11:40

would applying the function as map be better?

manutter5113:11:31

No, for 2 reasons: you want a lazy list because that helps you avoid memory exhaustion, and also map also returns a lazy list 😉

manutter5113:11:54

There’s a mapv that returns the entire list as a vector, all at once, which means it immediately consumes all the memory needed to hold the list. Which is the opposite of what you want, but it’s nice to know about map vs mapv

manutter5113:11:03

Anyway, back to your problem…

Logan Powell13:11:47

hmm... but perhaps mapv would be ok, because it seems that the function being applied to the lazy list is what's exhausing memory... the list itself should only be about (max) 100mb

manutter5113:11:44

Hmm, mapv uses more memory, so I’d be surprised if it solved the problem

Logan Powell13:11:58

ok, so no bueno

Logan Powell13:11:35

but thank you for bringing mapv into my lexicon 🙂

manutter5113:11:43

Can you share a gist or something with more context? It’s hard to tell where your problem might be with just that snippet

mfikes13:11:39

@loganpowell @manutter51 ClojureScript holds head; it doesn’t have locals clearing

Logan Powell13:11:29

ah, so there's no way around it?

manutter5113:11:11

Ah, that’s right, forgot that.

mfikes13:11:49

Two ideas. One is dirty and describe in http://blog.fikesfarm.com/posts/2016-01-15-clojurescript-head-holding.html Second is if you can be clever with transducers you might be able to pull it off. See the example in https://clojurescript.org/news/2018-03-26-release#_reducible_sequence_generators

Logan Powell13:11:18

Thank you guys! Let me read those

mfikes13:11:14

If you are using the transducer trick, it is based on some new sequence generators that were added in 1.10.238

mfikes13:11:03

A third idea (which seems less reliable, but sometimes works, depending on the JavaScript VM), is to write a low-level loop / recur, attempting to overwrite previous pointers

Logan Powell13:11:50

This is great, at least I see a glimmer of light at the end of the tunnel 🙂

mfikes13:11:02

It is a great opportunity to grok transducers 🙂

Logan Powell13:11:23

I am using transducers elsewhere, but I do copy a lot of older code 😄

mfikes13:11:29

The TL;DR is to eliminate the sequence (or at least intermediate ones)

Logan Powell13:11:37

yes, that's what I want to do

Logan Powell13:11:59

so, I would use a transducer instead of for?

mfikes13:11:13

I think Christophe has one...

mfikes13:11:13

But if you don't want to take on a dep, I would just try to rewrite the logic using core transducer functionality

Logan Powell13:11:42

Dude, thank you

Logan Powell13:11:14

I will try it with xforms to see if it solves the problem and if so try it myself

Logan Powell13:11:45

I ❤️ the Clojure community

mfikes13:11:02

If you are processing two collections, I can't recall but I think Steve Miner came up with a cool transducer-based approach to a similar problem we were pondering on the Apropos podcast... seeing if I can dig up that gem

Logan Powell13:11:01

btw, I subscribe to Apropos! It would be really great if you guys could add the topics of your discussion in the titles?

Logan Powell13:11:55

yes, that's exactly my bottle neck right now... merging two very long lists of maps

mfikes13:11:51

Right, the fundamental question might be: How do I get my two long lists into a single thing that I can transduce over, without producing yet another sequence?

Logan Powell13:11:08

I can look through episode 3

mfikes13:11:34

It might be the case that we discussed this very problem on that episode 🤞

mfikes13:11:53

Otherwise, if you just need to merge two long lists of maps, can you do one, and then do the other? I don't know if you posted a precise (but smaller) example of the data transformation you need to make.

Logan Powell13:11:16

the two lists contain "partners"/compliments of each other they are only made complete by merging them together (not two of the same lists)

Logan Powell13:11:03

I can share a snippet, but it might be a PITA to understand... I'm not sure how well I even understand it 😄

Logan Powell13:11:34

Let me dig through some of the resources you've shared

mfikes13:11:12

Is it something like this?

(map merge [{:a 1} {:b 2}] [{:c 3} {:d 4}])

cgrand13:11:54

Does the two lists have the same size? Do items occur in the same order?

Logan Powell13:11:02

(map merge-with...

Logan Powell13:11:22

prior to the merge they are grouped by a common key

mfikes13:11:29

Too bad you can’t do (into [] (map merge) coll1 coll2)

Logan Powell13:11:07

I augmented each of the maps with a composite key that allows them to be grouped

cgrand13:11:17

ok so it’s not (map f coll1 coll2) territory siince the list order is accidental, what matters is your grouping-key

mfikes13:11:56

Maybe you could still loop / recur to walk both lists

Logan Powell13:11:30

so try to merge-with a single map across the second list of maps, one at a time?

Logan Powell13:11:31

instead of grouping?

mfikes13:11:59

Another question in my mind: Are the two long lists already in RAM, or are they generated / fetched, and is that's whats blowing out the heap?

cgrand13:11:12

How big are the input lists? Are they stored in memory or lazy? (same question as Mike’s)

Logan Powell13:11:32

they will each be in a chan that's merged in a core.async/map function

Logan Powell13:11:26

so, I believe they'd both be in memory separately

mfikes13:11:24

Another related question: Does your output list need to be completely in RAM, or can it be consumed lazily?

Logan Powell13:11:59

it's returned as single GeoJSON featurecollection

Logan Powell13:11:07

once it's merged

mfikes13:11:08

Your input lists may not be in RAM if, say they involve fetching elements from a backend one at a time.

Logan Powell13:11:30

the two lists are created from two separate APIs

Logan Powell13:11:41

but they are fetched as a single request each

Logan Powell13:11:50

the APIs return lists

mfikes13:11:51

OK, it is probably safe to assume they are both in RAM. And at that point the heap can hold them.

mfikes13:11:51

The question seems to boil down to: How do I produce this third collection? If it magically appears, can the two input lists and the output collection all fit in heap?

Logan Powell13:11:52

The biggest responses are @ 100mb

Logan Powell13:11:21

"wishful thinking" ™️

mfikes13:11:35

Hmm… maybe your input collections are lazy and you are the only one holding onto them?

mfikes13:11:59

If so, you could walk them trying to let go of their heads as you do so

Logan Powell14:11:22

if I can just figure that part out 😄

mfikes14:11:41

ClojureScript probably doesn’t help either, especially if those collections are being referenced in function parameters

Logan Powell14:11:35

as to @cgrand s point, could the group function be the culprit?

valerauko14:11:43

i suspect a typo of reduce?

Logan Powell14:11:39

how about fold?

cgrand14:11:41

Did you try to capture two input lists, put them in vars and play with them at the repl?

Logan Powell14:11:27

hmm, let me give that a try

cgrand14:11:28

(trying to figure out if heap is saturated beacuse of “infrastructure” code, or just the computation + data)

Logan Powell14:11:38

the REPL to the rescue

Logan Powell14:11:09

did you just update your xforms project?

cgrand14:11:23

I ipdated it 1 or 2h ago, adding x/time

Logan Powell14:11:15

I'm looking at it now 😄

cgrand14:11:51

First, here is a saner function to replace your deep merge:

(defn deep-merge [a b]
  (if (map? a)
    (into a (for [[k v] b] [k (deep-merge (a k) v)]))
    b))

Logan Powell14:11:17

that will do a merge-with?

Logan Powell14:11:34

seeing this code makes me feel really dumb

cgrand14:11:33

=> (deep-merge {:a 1 :b {:x 1}} {:a 2 :b {:y 4}})
{:a 2, :b {:x 1, :y 4}}

Logan Powell14:11:07

that should do it

Logan Powell14:11:00

ok, let me use that and do some REPL

Logan Powell14:11:42

your deep-merge works like a charm

Logan Powell14:11:07

still get the heap overflow, but I love the new function

cgrand14:11:44

Check list 1/ you have your two lists in two vars 2/ you can call count on these vars without blowing the heap 3/ manually picking one map of each and merging it with deep-merge doesn’t blow the heap

jaihindhreddy-duplicate14:11:04

You can use merge-with

(defn deep-merge [a b]
  (if (map? a)
    (merge-with deep-merge a b)
    b))

Logan Powell14:11:12

shared as well

Logan Powell14:11:31

@cgrand let me give that a go

cgrand14:11:03

and a variation on group-and-merge

(defn group-and-merge [coll1 coll2]
  (let [coll1-by-key (into {} (for [x coll1] [(:your-key x) x]))
        coll2-by-key (into {} (for [x coll1] [(:your-key x) x]))]
    (vals (deep-merge coll1-by-key coll2-by-key))))

ScArcher14:11:47

How can I multithread applying a function to members of a collection?

ScArcher14:11:28

I have a sequence of files that I'd like to read through and make REST calls for.

ScArcher14:11:53

I have everything working, just not sure what the right way to make it execute in multiple threads.

cgrand14:11:29

pmap, reducers/fold or async/pipeline-blocking

cgrand14:11:04

@scott.archer I believe the last one is the better fit for what you described

ScArcher14:11:07

So with pmap i would just end up with a sequence of nil

cgrand14:11:49

That you would have to consume to drive computation.

ScArcher14:11:38

I need to call rest services, so I think it would all just be side effects, nothing to consume unless i'm misunderstanding the use of pmap.

cgrand15:11:12

Pmap is lazy so if you don’t consume (E.g. dorun) it nothing will be computed, no requests sent.

RobinVdB20:11:06

I'm doing clojure and the brave. In the chapter on macros the following is written

(defmacro code-critic
  "Phrases are courtesy Hermes Conrad from Futurama"
  [{:keys [bad good]}]
  `(do ~@(map #(apply critize-code %)
             [["Great squid of Madrid, this is bad code:" bad]
              ["Sweet gorilla of Manila, this is good code:" good]])))
What is the advantage of [{:keys [bad good]}] over [bad good]

Lennart Buit20:11:56

that function gets passed a map as its single argument

enforser20:11:05

one reason is that if you just make your parameters [bad good] then you always have to pass them both. By passing a map you can assume that any parameter which doesn't correlate to a given key in the map is nil.

Lennart Buit20:11:17

Also, you can use {:keys [...] :or {...}} for defaults

enforser20:11:55

I would say it's useful in cases where you want params to be optional, or when there are a lot of params it is easier to change your code later if you have passed them in as a map rather than maintaining a lot of different parameters being passed around

RobinVdB20:11:27

So when you do it that way you have to call code-critic the following way?: (code-critic #{bad (1 + 1), good (+ 1 1)})?

RobinVdB20:11:44

Since it expects a map?

Lennart Buit20:11:50

#{...} is a set, {...} is a map

RobinVdB20:11:31

(code-critic {bad (1 + 1), good (+ 1 1)}) than?

RobinVdB20:11:56

Because that prints nil

RobinVdB20:11:04

(because bad and good are nil)

enforser20:11:25

bad and good should be keywords

❤️ 4
RobinVdB20:11:12

Oh yeah it works if use :bad and :good

enforser20:11:19

(defn foo 
  [{:keys [bad good]}]
  [bad good])

=> (foo {:bad 1})
[1 nil]

JanisOlex20:11:57

Hello. very beginner question. Clojure begs for top-down planning and implementation. Which is ok, as I can't know all the tiny small "bottom" functions from the beginning. But if I start with top-down design and functions like "setup-game, game-turn" how can I have some testing capability against such high level functions if I don't have a meat on the bones (and even bones are more like skin)

JanisOlex20:11:14

Another question, as Clojure lets you play with maps, lists and mostly all the system data might end up in multi-level complex compound map/list, how such structures can be tested? Test data prepartion itself is huge work. The same "game-turn" function might take whole game state inside, and I still don't see some convinient method of testing such function? Does it means that as big functions are composed small functions, it follows that tests should be done the same way?

chrisulloa20:11:31

"The same "game-turn" function might take whole game state inside, and I still don't see some convinient method of testing such function?" I think in this case you would write a function that updates a certain value of the game state, then test the transformation, for example:

(defn next-turn [game-state]
  (update game-state :turn inc))

chrisulloa20:11:24

You could test this by simply asserting (= (next-turn {:turn 10}) 11), so while it takes in the entire game state when it runs you want to test each transformation on values as a pure function.

enforser20:11:43

At the company I work at, we generally do thorough testing of the smaller functions, then for the more broad functions you just need to test what they are doing - rather than the output of each smaller fn.

JanisOlex20:11:17

@christian.gonzalez Ahh makes sense. If my "big function" does nothing, so I can test like "empty" nothingness, but at the moment when it starts doing even small value changes, I can also have test exactly for that small change without "looking" at the big container? Makes sense. Thanks.

JanisOlex20:11:58

@trailcapital But how you design system from the beginning? Most likely you start at pretty high level of abstraction? So how tests evlove against such design? Or there is different approach?

JanisOlex20:11:44

I am interested in the evolutio of tests against actual system, at the same time I would like to not box myself into "thinking in 5 turns ahead" situation - thats why I want to dive into Clojure in the first place.

chrisulloa20:11:00

One thing that really helped me wrap my head around how to transform, maintain, and test an application state in Clojure is the Clojurescript Re-frame library... maintaining state for a SPA. The idea of having an application state, updating, assoc-ing, dissoc-ing values based on user interaction with the application. An important concept is separating the pure functions from stateful, impure functions so that you focus on testing idempotent functions.

JanisOlex20:11:49

" maintaining state for a SPA." <- What is SPA?

chrisulloa20:11:55

single page application

chrisulloa20:11:23

It's just a suggestion, there are other ways to practice writing Clojure if you have other interests.

JanisOlex20:11:00

I still need to do simple programs, web part would be next step - when I want to provide some transport layer to the "outer world", for the beginning I would be happy to write simple text-based game, but complex enough to cover most of Clojure basics... what I want to learn first, to make my brain thing in Clojure... not to try turn Clojure into some sort of self-invented Java and write Java

seancorfield20:11:23

@janis.pauls If you're already used to a TDD workflow, I think you can apply that "as-is" to Clojure design/development. Another tool you might find handy for dealing with "complex" state and test data is clojure.spec -- if you write a (minimal) spec for your data (and refine/add to it as you grow your program), then you can also leverage spec to generate test data for you and verify the results are as expected.

chrisulloa20:11:45

That's a great idea, I think games are well suited for functional programming.

seancorfield20:11:17

I think one of Clojure's great strengths is that you can "grow into" the available tooling as you "grow into" the language itself -- because so many things are composable (and not inherently coupled).

JanisOlex20:11:17

I hear abuyt that clojure.spec - probably have to look at it

Lennart Buit20:11:41

When I learned Haskell I was tasked to write a Sudoku solver/ui

JanisOlex20:11:42

I downloaded Uncle Bob's Martin space war game written in Clojure, but it is a bit complicated for me yet to read and get my head around it

Lennart Buit20:11:54

Great way to “grow” into a functional mindset

Lennart Buit20:11:10

The second task was writing a super simple compiler

JanisOlex20:11:48

my original background is Java/Android and I am sick of Android classess with zillion of member variables, all changing arbitrary, impure functions here and there... and I am learning Clojure more and more, and hope my thinking will change into making programs as Functional as possible (by functional mostly meaning - composition of functions, instead of line-by-line impeartive style)

JanisOlex20:11:40

also the game implies some graphical interface - does it mean I just take java awt and swing components and do stuff there, or is there some nice graph libraries out there already wrapped for Clojure - I mostly will need to display game board with pieces on it and text, no dynamic 60fps shooting, explosions etc.

chrisulloa20:11:13

There was a talk I watched when I first started learning Clojure, it might be of help https://www.youtube.com/watch?v=0GzzFeS5cMc

JanisOlex20:11:30

in short - want to try to implement Cave Troll board game (not AI yet, just for players to enter turns and display results)

JanisOlex20:11:16

Ahh nice video, going to watch. Thanks people... Have a good night

Lennart Buit20:11:13

So, when we are talking about organisation anyway what kind of conventions do you guys have for aliasing namespaces

Lennart Buit20:11:23

I started sort-of shortening them to three letter acronyms, but I am starting to get collisions and have to juggle many acronyms in my head at the same time

Lennart Buit20:11:42

I also made it kinda convension to have protocol aliasses ending in p, but well I got myself in trouble when an acronym of an implementation of a protocol also ended in p

chrisulloa20:11:48

I find using an IDE or something like Cider in a text editor helps identify things that are already aliased or reserved.

chrisulloa20:11:06

For example not setting clojure.string to str since str is a function.

joelsanchez20:11:19

well if the ns is called organization.myapp.section.stuff.thing I usually do stuff.thing or thing

Alex Miller (Clojure team)20:11:20

as much as possible, I find using the last segment of the namespace as the alias reduces confusion

joelsanchez20:11:29

gah we said the same

Alex Miller (Clojure team)20:11:32

and using a standard set of “short” aliases for things you use a lot

Alex Miller (Clojure team)20:11:46

s for spec, str for string, jio for http://java.io, etc

Alex Miller (Clojure team)20:11:04

really, I think the most important thing is for everyone on a project to use the same set of aliases, whatever they are :)

joelsanchez20:11:21

> not setting clojure.string to str since str is a function. I always alias clojure.string to str. they don't conflict at all

joelsanchez20:11:08

same with clojure.set and set

chrisulloa20:11:14

It doesn't conflict, I just prefer not to....

enforser20:11:54

I have run into people who had a hard time getting over the fact that a namespace can be be aliased with name of a pre-existing function without a problem, so I try to avoid it as well - for others sake

joelsanchez20:11:42

what a weird thing to think, you can never use ns aliases as a value

Lennart Buit21:11:12

I really have to find a balance between not beating a dead horse with naming (e.g. (message-protocol/get-message ...)) and having a garbled mess of acronyms (e.g. (mp/get-message ...))

joelsanchez21:11:55

if mp is used a lot it's worth it, else I'd just go with message-protocol, which isn't half bad

Lennart Buit21:11:28

Hmm true also, I’ll try to find that partition

Lennart Buit21:11:10

naming things remains hard…

joelsanchez21:11:26

I better not tell you about cache invalidation then

Lennart Buit21:11:59

I only read half of that message, the rest was parts of your previous

😂 4
seancorfield21:11:53

I find it helpful to think about naming functions in a namespace, in the context of the last segment of the namespace name itself (i.e., assume the "default" alias will be used and see how functions look when called that way).

☝️ 4
seancorfield21:11:21

That can also help when choosing namespace names since you focus on the last segment of the name as something all your client code will be using.

Lennart Buit21:11:39

Can you give an example?

Lennart Buit21:11:07

Ahh yeah, ofcourse

Lennart Buit21:11:15

I’ll try some of these suggestions tomorrow, thanks!