Fork me on GitHub
#clojure
<
2019-08-27
>
alfredx00:08:11

@m131 I have a tiny test coverage tool that does this with namespace finding and then alter-var-root. You could have a look.

noisesmith01:08:14

@alfred.xiao cool - one small thing I notice reading - you could just use clojure.core/run! instead of your doseq-with function - it's designed for exactly that type of usage (though it takes the args in flipped order)

noisesmith01:08:45

also, since nothing can access the return value of a finally clause, you could use run! there too (instead of the mapv in evaluate-test-coverage)

👍 4
alfredx01:08:28

Thanks @noisesmith Didn’t know run! was there

noisesmith01:08:44

yeah, IMHO it's underutilized

nori01:08:19

Today I Learned to Load File before I Load Block

quadron01:08:32

is cascalog dead?

noisesmith02:08:12

36 open issues on gh and last commit was 1/29/2016 - that looks close to it

quadron02:08:38

what are people using?!

quadron02:08:53

mapreduce-wise?

noisesmith02:08:47

flambo has been updated more recently https://github.com/sorenmacbeth/flambo

noisesmith02:08:01

but honestly if it is java you can use an actively maintained lib directly

✔️ 4
noisesmith02:08:56

I don't know if onyx is still live, but that was a distributed processing system made in clojure

quadron02:08:07

hmm, onyx seems to be liveliest one. hope it does not add a ton of complexity.

quadron02:08:20

i'll probably use interop.

ahungry02:08:17

thanks for the link @alfred.xiao - looks interesting

ahungry04:08:37

https://github.com/ahungry/determinism - that's what I am thinking/working on

ahungry04:08:12

with results that are trackable (and reflective of runtime code paths) I think you could easily trace a path and see if a potential return value of some fn is incompatible with the input value of another fn to safeguard against regressions

sogaiu05:08:00

@m131 somewhat related, i have been working on something to modify / transform source code in various ways. one motivator was to produce detailed execution traces (e.g. by running code after inserting tap> or some other logging form at the beginning of function bodies), save them, and analyze them for various purposes. source modification was the approach considered because it seemed likely to work for jvm / clr clojure as well cljs.

ahungry05:08:51

Nice, that would be great to have

alfredx07:08:00

This maybe a dumb question, but keen to see how others’ view on this: From a pure functional approach perspective, should a map over a structure, like a vector or a set, return the same structure? In Clojure, obviously (map inc #{1 2}) returns a seq rather than a set.

henrik09:08:09

Well, mapv coerces the output to a vector. You could (def maps (comp set map)) if you wanted something that coerces to a set.

henrik09:08:22

Sorry, not the question, now that I re-read what you wrote.

jumpnbrownweasel14:08:08

In some languages (Haskell) a map returns the same type as the input. But in Clojure a sequence is returned, so this may be something to get used to. Sequences are lazy in Clojure, while sets, maps, vectors are not, so that may be part of the reason -- but I'm guessing.

alfredx01:08:55

That behavior (in Haskell) is more natural to me. map over a Tree you still see a Tree, map over a Graph, you still see a Graph

jumpnbrownweasel01:08:33

I agree, especially when you're surprised by it initially, but I think Clojure's approach -- sometimes using sequences and laziness, and sometimes not -- is very practical and makes sense once you get used to it.

henrik05:08:01

Like others pointed out, you would run into trouble when mapping over sets.

alfredx05:08:57

It is not necessary a trouble, for example, a math function that maps from int set to bool set, you would know you would not have a output set bigger than 2 elements.

henrik07:08:52

Sure, it depends on what you need I suppose. If you need to answer “does this set contain even numbers?“, mapping over #{1 2 3 4 5} would tell you that (though map seems like the wrong way to answer that question). If you need to answer, “which numbers are even?“, returning a set would strip that information. Of course, you could cast the set into something else before mapping over it to get around it.

alfredx07:08:25

Absolutely, there is no definite right or wrong here. The second question sounds more like a filter question rather than map question though.

henrik10:08:48

Yeah, true. Pointless unless you also need the odd numbers, and you need them in the same seq. I.e., (map #(vector % (even? %)) #{1 2 3 4 5})

henrik10:08:30

Otherwise, (group-by even? #{1 2 3 4 5}) is also fine.

henrik10:08:20

Which incidentally gives groups of vectors, since the entire thing needs to be realized anyhow. Yeah, Clojure certainly isn’t about beauty or purity as much as pragmatic compromises.

alfredx13:08:34

Yes, agree with your point regarding purity and pragmatism.

alfredx13:08:58

Perhaps that is why Haskell is less popular.

lodin19:08:55

@alfred.xiao Rich rephrased the "100 functions" quote to be about a data abstraction rather than a data structure. And in Clojure the core abstraction/interface is the seq interface. map is defined for seqable collections, and vectors, maps, lists, sets are all seqable. map immediately gets the seq view of the input before doing anything. So it effectively takes a seq and returns a seq. Most core functions are written so that you don't need to call seq manually on them, so you get the impression that you map over a set, but you actually map over a seq.

jumpnbrownweasel21:08:17

@U8J6W2SC8, that's a really clear way to look at it, thanks.

alfredx23:08:57

@U8J6W2SC8 thanks for pointing that out and it is helpful in understanding it. However, this does not necessary mean it is the best way of thinking of things though. What I am trying to say is how much abstraction we need is a balancing art and subjective. For example, in OO/Java concept, everything is a object/class, is that good?

lodin21:08:43

@alfred.xiao I'm sorry, I'm lost.

alfredx00:09:01

@U8J6W2SC8 Sorry but never mind, just wanna raise question of whether the usefulness of seq is being exaggerated

henrik06:09:46

I’ve been thinking about the design decisions in Clojure recently, and I realize I’m not experienced enough to judge what particulars of Clojure are due to how the underlying VM works, and what are there due to them being a good decision regardless of platform constraints. Maybe everything would have been lazy if it had been possible? Maybe it’s better to have one a separate collection type that encodes laziness (as it is right now)? I’m not sure.

henrik06:09:50

As for the comparison to everything being an object (in Java, or Ruby), I think that’s the kind of idealism that Clojure specifically doesn’t strive for. I.e., everything is not a lazy sequence in Clojure.

lodin10:09:39

@alfred.xiao How is it exaggerated (or by whom)?

lodin10:09:47

@U06B8J0AJ To make everything lazy (not just sequences) would be a huge change. E.g. PureScript is not lazy because it would require a runtime on top of JS.

henrik11:09:17

Yeah, definitely. It’s also hard to imagine what a how a lazy map would behave, if not returned as a sequence.

alfredx12:09:30

@U8J6W2SC8 The official Clojure guide says ‘Clojure defines many algorithms in terms of sequences (seqs).’ This implies most algos/funcs accept, return, in other words, optimized for`seq`. E.g. It is not uncommon to see real world code that map over a (hash) map then into a {} to make it a (hash) map again. Actually in this process, seq plays a key role and is pretty much unavoidable if we do map. Perhaps it is really unavoidable for cases where laziness is a must - e.g. infinite range. I like the logical sense that everything can be viewed as a sequence, but real code always need to concern data structure, hence many times a mix of seq and concrete data structure, like (hash) map, set, etc.

lodin08:09:00

@U06B8J0AJ I guess it would require (?) everything to be lazy, as in Haskell.

lodin08:09:06

@alfred.xiao If you want to avoid (into {} (map ...)) you can write (as many have) a map-vals function. Perhaps you'd want to use reduce-kv instead though, if you're using an unary function. (Works for vectors as well, although if you know it's a vector then you can use mapv.)

alfredx10:09:04

@U8J6W2SC8 Yeah, map-vals is a solution. I guess what I am really thinking is that seq is sort of invasive in a processes like this: Given a (hash)map, I need to filter, add one entry, then map the values to something else, this process should still give output as a (hash)map. By invasive I mean, the programmer has to know where seq plays a part, and when to convert back to a concrete data structure, because many funcs/algos accepts various data structures but returns seq only. It is good that they accept various data structures as input, but they also strip some key information from input.

lodin10:09:08

@alfred.xiao I think the key here is that you see it as it strips key information from the input. map is a seq function. It's not a map/vector/set function.

lodin10:09:43

The way I see it is that it calls seq for you as a convenience.

lodin10:09:06

Many times you want the return value from the function given to map to be not a map entry.

henrik11:09:40

Filtering the entries of a map is rather unusual I would say. While it would be handy to have it stitched back into a map in the end, filtering is a sequential operation when you come down to it. For more advanced data structure manipulation, https://github.com/redplanetlabs/specter might come in handy. It’ll navigate your data and do transformations while leaving the structure intact, and will often do so in a more performant way than the core library functions.

henrik11:09:18

Out of curiosity, if you have something like,

{:hello :world
 :some-things [{:a [1 2]} 
               {:b [3 4]} 
               {:c [1 2 3 4 5]}]}
In Haskell, will it update the stuff in say [:some-things 2 :c] in a lazy manner?

lodin17:09:40

@U06B8J0AJ Filtering is not super uncommon though. I've done it several times, and in PureScript I've used https://pursuit.purescript.org/packages/purescript-ordered-collections/1.6.1/docs/Data.Map#v:update which is a combo of map and filter.

lodin17:09:30

Regarding the Haskell question, I'm not sure exactly what you mean. Generally everything is lazy. For instance, if you have

lodin17:09:56

haskell
let x = 3
    y = f x
in x

lodin17:09:04

then f will never be called. So it's lazy everywhere. So if you update something deep in a structure, then yes, it would typically be lazy.

lodin17:09:04

Another example: snd (f x, g y) will not invoke f either. Because it never has to.

lodin17:09:40

Correction, I meant https://pursuit.purescript.org/packages/purescript-ordered-collections/1.6.1/docs/Data.Map#v:mapMaybe . The other one is for a specific key (that's useful too!).

andy.fingerhut07:08:44

I'm not sure what you mean by "From a pure functional approach perspective"? If you give Clojure map a pure function, then map's operation is also a pure function.

alfredx09:08:11

I was not referring to side effect in this context but more like from a math or category perspective. My thought for map was: Given a collection of things, apply a change to each thing only. e.g. Given a Tree of nodes, map over the Tree returns the same Tree structure but each node has changed somehow.

andy.fingerhut07:08:09

Are you asking whether other functional programming languages have an operation like map that returns the same collection type that it is given as input?

danielneal08:08:47

If I map over a set I’d expect to have one result per item in the set. For example, if I did (map (fn [x] (odd? x)) #{1 3 5 8}) I’m not sure I’d want the result #{true false} as opposed to (true true true false)

alfredx09:08:21

Good point on set. I was thinking a Tree kind of thing, where the mapped outcome is still a tree.

danielneal08:08:44

You can find fmap implementations that do this though

Lone Ranger10:08:48

part of the deal with map is that it is lazy, which is extremely handy. For instance, you can map over an infinite sequence (useful for when you're dealing with input from a server/user) and for deferred computation. Most of the other datatypes are eager, so they can't be used this way.

borkdude10:08:59

@alfred.xiao I find it useful to think of the seq functions to have the “type” seqable? to seqable?. Seq functions (like map) call seq on their arguments and don’t bother converting the return value back into a concrete type, since that’s a waste when you are composing lots of seq functions

borkdude10:08:57

If you want to transform concrete types you’re often better off using transducers

alfredx13:08:58

Have never used/seen transducers in projects worked/working in, any other compelling reason to use it other than the reason mentioned above?

borkdude13:08:33

performance

borkdude13:08:54

managing resources (lazyiness is often a problem when trying to control the scope of when a resource can be closed)

emccue12:08:58

Random thought

emccue12:08:29

Could spark be monkey patched to accept clojure forms on workers instead of serialized bytecode?

emccue12:08:16

That way the requirement that code be serializable wouldn't exist for clojure code

emccue12:08:51

And you could make an incredibly interactive spark session

kah0ona14:08:33

hi, if I repl into a running jar to execute a long running command. would that continue to run if my ssh connection to the box dies, and thus the repl connection dies?

kah0ona14:08:42

it probably stops right?

kah0ona14:08:09

oh it doesn’t stop

noisesmith15:08:52

what stops on EOF is the procedure consuming your input (the read / eval / print loop), anything started in that session already exists and isn't auto-destroyed when that stream closes

kah0ona14:08:13

rephrase command as in: just call a function, not something in a separate thread or go loop

rickmoynihan16:08:41

I’m looking to add server sent events to an existing application that uses ring and good old blocking I/O… However, I clearly don’t want to serve the events from the existing server with long poll requests eating a thread per connection. I only need to send a few simple events from the app, so I’d like something pretty minimal. Ideally with relatively few dependencies (to minimise conflicts), and to be relatively lightweight in terms of overall size. I’d also like to run both the existing jetty server and the SSE server in the same JVM process. Last time I did SSE in clojure was many years ago; so not sure what people would recommend these days… http-kit, aleph, pedestal?

rickmoynihan16:08:59

ahh interesting pedestal can work with a jetty backend… was wondering if I might be able to leverage the async capabilities of the current jetty server

Joe Lane16:08:32

@rickmoynihan My experience about 2 years ago with pedestal SSE was fantastic. Lots of things like replaying events to catch a client up after a dropped connection "Just Worked".

👍 4
sova-soars-the-sora16:08:03

i have a strange question, is it possible for clojure to work on really low level architectures in the future? what's the minimum amount of architectural complexity required to provide the luxury of map, filter, and reduce?

rickmoynihan16:08:29

having map, filter and reduce doesn’t mean you have clojure. The minimum required for the luxury of map, filter and reduce is essentially something like the 7 special forms, and high order functions. So you probably want to look into minimal lisp dialects that target small machines.

hiredman16:08:12

https://clojure.org/about/rationale#_languages_and_platforms is a good kind of summary of the facilities that clojure rests on top of

mrchance16:08:30

There are some very low-level "almost-clojure" implementations though: https://ferret-lang.org/

😮 8
mrchance16:08:30

And languages like carp also go in the direction of high performance, low level code from a high level lisp, didn't do too much with it to judge though

😲 8
mrchance16:08:05

It takes some inspiration from Clojures syntax

sova-soars-the-sora16:08:00

Ferret can run on 2KB of ram! what!

hiredman16:08:16

"real low level architectures" is also very hand wavy

hiredman16:08:57

the jvm was originally created for embedded devices, and a computer with gigs of ram and multiple cores fits in your pocket today

parens 4
djtango17:08:18

My dev workflow involves hosting a socket repl and then piping code to the repl via my editor using ncat. I get an exception if I run the following code twice in a row via my repl: Is this a bug/known-issue?

(ns socket-test.core-test)
  (do
   (require 'socket-test.core-test :reload)
   (run-tests))
Exception:
socket-test.core-test=> Execution error (SocketException) at java.net.SocketOutputStream/socketWrite (SocketOutputStream.java:118).
Socket closed
Minimal repro: https://gist.github.com/djtango/f5b81183d269eaf5eaa73122796cc061

djtango17:08:22

Essentially, it seems clojure.test/run-tests and/or (require 'ns :reload) don't seem to be compatible with the socket-repl

noisesmith17:08:35

that execution error means that nc is exiting while the java process is trying to write its output back right?

djtango17:08:58

yes potentially, but why does nc exit early?

hiredman17:08:19

if I recall there is some option you need to pass to nc

hiredman17:08:22

no I am thinking of something else

hiredman17:08:42

my guess is you are not stopping the jvm from exiting

hiredman17:08:35

the socket repl starts up, which if I recall spins up the future threadpool, which can stop the jvm for exiting from a minute or two, but unless you have something keeping some main threads running it will exit

hiredman17:08:51

(some non-daemon threads)

noisesmith17:08:16

ahh, so you could fix it with the @(promise) trick or something

hiredman17:08:55

the socket repl doesn't spin up the future threadpool

Alex Miller (Clojure team)17:08:12

(.join (Thread/currentThread)) ;)

djtango17:08:30

:thinking_face: Do I just prepend this to my code that I inject to nc?

hiredman17:08:04

you need to do it in the -main of your "server"

hiredman17:08:14

but something else likely is, or something else is stopping the jvm from immediately exiting

hiredman17:08:09

and the threads the socket repl creates are specifically marked as daemon threads, so they don't stop the jvm from exiting if nothing else is going on

Alex Miller (Clojure team)17:08:59

btw, there is a flag to choose whether they should be daemons or not

Alex Miller (Clojure team)17:08:26

actually two flags, :server-daemon for whether the acceptor thread is a daemon and :client-daemon for whether the socket client threads are daemons

djtango17:08:21

I'm so sorry, I lie: a vanilla lein repl with socket enabled doesn't reproduce the issue The issue seems to be an interaction with rebel-readline

noisesmith17:08:28

lein repl starts a non-daemon nrepl server thread

craftybones18:08:41

What’s a “canonical” way to deal with a problem where there [f1 f2 f3...] are functions that need to be applied to [v1 v2 v3] ? Assume that none of those f1… are in a list but you need the result of all those applications in order to compute a final result? Is it better to just stick a let [a (f1 v1) b (f2 v2)] or use a (map #(%1 %2) fs vs)

craftybones18:08:39

This happens a lot. juxt for instance applies multiple functions to one argument(s). What if I want a parallel set of functions applied to a parallel set of arguments If you know what I mean…?

noisesmith18:08:39

@srijayanth one approach is to use juxt

craftybones18:08:51

@noisesmith - juxt doesn’t solve this problem

noisesmith18:08:59

nothing stops you from mapping juxt across args

craftybones18:08:11

But I want each of those functions applied to different args

craftybones18:08:17

Meaning f1 v1, f2 v2 etc

craftybones18:08:22

Not (f1 v1 v2 v3) etc

noisesmith18:08:39

oh, then your map example is how I usually do that

craftybones18:08:08

@noisesmith - that’s what I resort to as well, however, is there a word we can give it? Its a pattern that occurs too commonly

noisesmith18:08:16

zip-apply might be a decent name, or zip-invoke

craftybones18:08:55

Yeah. I think some apply variant indeed. zip-apply sounds nice. I’ve dabbled with apply-against and apply-to

craftybones18:08:03

In fact, if we name the #(%1 %2) then we get a lot of benefits. The (map apply-to) would turn into a transducer that can be quite pliable

noisesmith18:08:26

except we don't have many transducing contexts that zip together multiple collections

noisesmith18:08:31

sequence is the only one I can think of

craftybones18:08:24

Which is still useful.

Jakub Holý (HolyJak)18:08:34

Any idea why I get an unbound Var when i def the result of a call?

(def _r (cci/concordion-run (org.concordion.api.Resource. "/UsageCharges.md") "./UsageCharges.md"))
=> #'charges-test/_r
charges-test/_r ;; eval the just created var
=> #object[clojure.lang.Var$Unbound ...
? Clojure calls work (e.g. (def _x (range 3))) Thank you! Update: This works:
(def _r (bean (cci/concordion-run (org.concordion.api.Resource. "/UsageCharges.md") "./UsageCharges.md")))
=> #'charges-test/_r
_r 
=> {:forExample false, ...}
If I replace bean with identity then evaluating _r fails as above.

noisesmith18:08:22

the only thing I can think is that cci/concordion-run might be returning the value of an unbound var, but that seems like a strange behavior

👎 4
noisesmith18:08:30

or that you are evaluating _r before the cci//concoridion-run returns

Jakub Holý (HolyJak)18:08:04

Thanks, I haven't even thought of that. But it doesn't seem to be the case, I now waited a while an _r does evaluate but calling e.g. bean on it fails:

_r 
=> #object[org.concordion.internal.SummarizingResultRecorder 0x6fc7e8e2 "org.concordion.internal.SummarizingResultRecorder@6fc7e8e2"]
(bean _r)
=> {:class clojure.lang.Var$Unbound}

noisesmith18:08:33

in order for it to change from a bound value to Var$Unbound something has to be explicitly setting it to that, or some auto-reload process is recreating the namespace in the background and you are catching it at an awkward moment

noisesmith18:08:42

those are the only explanations I can think of

noisesmith18:08:38

(eg. if using clojure via editor tooling)

athomasoriginal21:08:26

I have a coll like [1 2 3] and I want to verify that every element in the vector is a pos-int? so I am doing (every? pos-int? [1 2 3]). I noticed though that (every? pos-int? []) would also yield true. Is there a built in that does this, but would return false because the vector is empty? Just curious if I am missing something.

noisesmith21:08:33

according to logic, every member of [] is pos-int?, you can just use (and (seq %) (every (pos-int? %))

Alex Miller (Clojure team)21:08:43

oh sorry, I read that as a spec, that's fine

Alex Miller (Clojure team)21:08:14

(the every spec has built-in support for min-count)

Alex Miller (Clojure team)21:08:49

(s/every pos-int? :min-count 1)

jumpnbrownweasel00:08:57

that's pretty nice. it will be really interesting to see the new spec when it comes out.

Alex Miller (Clojure team)00:08:55

Well that’s just spec 1, nothing new there

noisesmith21:08:26

to be cute, it could be (and (pos-int? (count %)) (every? pos-int? %))

hiredman22:08:28

too many parens around pos-int?

chrisulloa22:08:58

the every? vacuous truth

nori22:08:26

What is the functionality that makes Boot and Leiningen look better than Maven?

noisesmith22:08:42

people don't like editing xml

noisesmith22:08:58

people want to define build tasks using clojure

nori22:08:43

What build tasks are commonly defined?

noisesmith22:08:52

compiling to aot, test running, compiling cljs to js, linting

dpsutton22:08:24

also compare how to specify a dependency in maven versus lein

noisesmith22:08:28

in boot or lein it's easy to pull in clojure code for those tasks (and that's how it's typically done via plugins or libraries)

noisesmith22:08:41

oh yeah, the clojure versions are more compact too

nori22:08:52

CLI seems to beat lein for dependency management.

noisesmith22:08:31

you can use deps.edn in lein via a plugin to replace it's regular dep management

noisesmith22:08:58

lein still defines a lot of tasks / plugins with no cli equivalent (but that's changing fast as more teams adopt cli)

dpsutton22:08:14

I wish there was a lein 3 that required the project.clj file not be evaluable but more static like deps.edn

dpsutton22:08:25

or, I wonder if this would kinda unify the lein and deps.edn workflows

noisesmith22:08:27

do that many projects end up using evaluated inline code?

nori22:08:31

via what plugin

dpsutton22:08:44

i doubt it but i thought that was the fundamental mismatch between the two

dpsutton22:08:59

i know cider-nrepl's does for simple things like def'ing the version

noisesmith22:08:04

but that only matters at your current project - code in your deps isn't run

noisesmith22:08:10

you just load their pom

dpsutton22:08:03

although it looks like cider-nrepl has had a "static" project.clj since january

noisesmith22:08:18

doesn't cider-nrepl's project.clj only matter if you are building cider-nrepl? it shouldn't effect a project using it as a dep or plugin? - or is it injecting executed code into the project.clj via the plugin?

dpsutton23:08:10

i meant if you wanted to target it with tools.deps

dpsutton23:08:30

i think that's what's preventing them from being targets like that. the description of the project must be evaled

dpsutton23:08:43

and that's a fundamental issue with any boot or lein project

noisesmith23:08:48

lein doesn't run code from your deps though - I must be misunderstanding you

dpsutton23:08:09

sorry i was being unclear. I mean why deps.edn can't target them as git deps etc.

noisesmith23:08:16

oh, right, it can only use it via the pom / deployed artifact

seancorfield23:08:41

You can always specify :deps/manifest :deps to "force" tools.deps to treat it as a deps.edn project with an empty list of dependencies, and then provide any transitive deps yourself manually.

seancorfield23:08:40

(but cider-nrepl has a deps.edn file -- so what repo are you talking about?)

dpsutton23:08:47

ah i forgot someone added that recently and the def inside the project.clj is gone. it was just an example. i think maybe my last job had some evaluation stuff in one of the project.cljs

seancorfield23:08:52

Yeah, we used to do some calculations in project.clj back when we still used Leiningen... coming up on four years ago 🙂