Fork me on GitHub
#clojure
<
2016-03-29
>
kingoftheknoll00:03:54

Don’t know how to phrase the question to ask google but I’m curious if Clojure has a way of stepping through the changes of a persistent data structure.

darwin00:03:27

persistent data structures do not change by definition, maybe that is why google does not understand you .-)

darwin00:03:13

do you want to track history of atom changes?

kingoftheknoll00:03:14

lol totally, but I’ve seen demos in clojurescript of ‘undo’ behavior because of the underlying data structure

darwin00:03:00

you can use (add-watch your-atom …) and record all changes into another data structure, thanks to structural sharing, that history recording will be cheap

kingoftheknoll00:03:50

ah so I could I could conj the value of every swap! of the atom

kingoftheknoll00:03:14

the swap! the index of whatever history I wanted into the current state

darwin00:03:13

no problem, google for “undo” and “add-watch” and you should get some relevant blog posts as a result

kingoftheknoll00:03:52

yup this is the right path 😃

Oliver George00:03:17

Is it possible to change things in the clojure.core namespace before running my code? (I was thinking about changing clojure.core/assert to have more information in the error message.)

Oliver George00:03:23

Perhaps a hook which would (load-file "patch-assert.clj")

Oliver George00:03:47

Okay, found lein :injections which seems about right.

sveri08:03:20

Hi, what is the preferred way to enable plumatic schema validation during tests?

sveri08:03:36

Just put (s/set-fn-validation! true) into every test?

sveri08:03:45

Or testfile

reborg09:03:58

Anyone can explain this?

user=> (loop [i (int 0) _ (println (type i))])
java.lang.Long
nil
user=> (let [i (int 0) _ (println (type i))])
java.lang.Integer
nil

mull09:03:41

I’m looking for articles dealing with JavaScript -> Clojure(Script). Specifically interested in hearing what worked/didn’t, rather than code conversion examples. Either my Monday morning Google-Fu isn’t up to the task or there articles that must be out there are poorly keyworded 😁 Would appreciate links/nudges!

bronsa10:03:41

@reborg: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L6270-L6282 this explains how that happens, as to why.. I could not figure it out. I suspect there's no good reason

reborg10:03:22

thanks @bronsa. So my int cast is thrown away in the loop. I have a code snippet I’m trying to optimise and that’s one of the thing I couldn’t understand.

reborg10:03:30

let me post the snippet

reborg10:03:38

(defn find-index-1 [f coll]
  (loop [n (int 0)
         s coll]
    (if (f (first s))
      n
      (recur (unchecked-inc-int n) (rest s)))))

(defn find-index-2 [f coll]
  (loop [i 0
         s coll]
    (if (f (first s))
      i
      (recur (inc i) (rest s)))))

reborg10:03:39

I was expecting find-index-1 to be faster, because of the (int) treatment. It’s not, it performs roughly the same as find-index-2

bronsa10:03:16

I think your best option is to use i 0 and unchecked-inc. Although I'd assume the seq traversal overhead will dominate those primitive ops

reborg10:03:25

so I’m unable to see it for the seq traversal?

bronsa10:03:17

I'd assume so

reborg11:03:06

@bronsa thanks for the hint. I was able to see the difference with the following:

(defn inc1 [n]
  (let [n (int n)]
    (loop [i 0]
      (if (< i n)
        (recur (inc i))
        i))))

(defn inc2 [n]
  (let [n (int n)]
    (loop [i 0]
      (if (< i n)
        (recur (unchecked-inc i))
        i))))

reborg11:03:51

unchecked-inc-int makes things worse in this benchmark. Also the (int 0) in the loop doesn’t make any difference.

bronsa12:03:37

yeah, the unchecked-inc-int was causing l2i and i2l to be emitted probably

bronsa14:03:07

@reborg: had another quick look on that over lunchtime, I'm reasonably convinced that should be considered a bug. Care to open a ticket?

reborg14:03:07

@bronsa sure. I cannot judge on the priority or impact, so expecting you to comment on that when I’m done

dominicm16:03:06

What was the lein-exec alternative that came out recently? It had a really bad pun for a name...

mping16:03:43

is there somethink like this: (take-nth [a b c d e f] 0 1 3) ==> [a b d] in the stdlib?

mping16:03:07

makes sense 😄

dominicm16:03:01

@bronsa: That won't work for a list will it?

bronsa16:03:46

no, lists are not IFns

bronsa16:03:20

that only works because ([foo] bar) is the same as (nth [foo] bar)

dominicm16:03:48

(map #(nth '(:a :b :c :d :e :f) %) [0 1 3]) Not as pretty, but more general.

dominicm16:03:05

No idea how to pawn that off to clojurebot

dominicm16:03:13

Worth considering if you're not sure about the data type.

reefersleep16:03:13

Not Clojure-specific, but I'm thinking someone here will know simple_smile What is the name of a position in x/y space along with an angle? Is there a term for it?

cky16:03:35

A vector describes a direction and length (or, alternatively, from-and-to coordinates). Is that what you meant?

wamaral17:03:16

I'd say an angle needs two points of reference at least

mping17:03:33

vectors and angles+length are interchangeable

dominicm17:03:38

@jakemcc: That's it! Thank you

tacosundae17:03:49

Does anyone know if it is possible to dispatch a multimethod on functions or funcallables? I come from CL and this is pretty straightforward but I have no idea how it might work in clj. Feels like CLOS with all of its limbs chopped off.

dominicm17:03:03

@tacosundae: When you defmulti you define a function for dispatching on. Or am I misunderstanding?

tacosundae17:03:27

Yes, but how to get at things like the arity of a function or the name if it is in a var and dispatch on that

pataprogramming18:03:20

@tacosundae: For functions defined with defn, you can inspect the metadata on the var.

user=> (defn foo [bar] (identity bar))
#'user/foo
user=> (meta #'foo)
{:arglists ([bar]), :line 1, :column 1, :file "/tmp/form-init8119837949873121332.clj", :name foo, :ns #object[clojure.lang.Namespace 0x444a7e76 "user"]}

tacosundae18:03:09

Thats perfect!

pataprogramming18:03:54

For a raw function, you can probably get something with Java reflection.

dominicm18:03:29

It might be worth specifying the interface as being via a symbol, instead of the function value.

dominicm18:03:43

But I'm not sure on your use-case.

tacosundae18:03:21

hehe, no real use case. I'm new to clojure and exploring the landscape. Thanks for the help.

abtv18:03:44

Can anyone explain when I should use lazy sequences with map/filter and when I should use transducers?

hiredman18:03:00

lazy sequences are an iteration interface, and transducers are a way of specifying reducing behaviors, so they are not entirely apples to apples

hiredman18:03:14

you can use transducers with lazy seqs

abtv18:03:30

I have an existing transformation of a lazy sequence with several maps and filters. What can I achieve with transducers?

abtv18:03:06

I'm not sure sequence call is faster

hiredman18:03:04

transducers help eliminate overhead

hiredman18:03:05

when you have a sequence of maps and filters, unless you build that using tranducers, you have intermediate lazy seqs between each operation

abtv18:03:50

not sure I understand it... I thought lazy-seq consumes elements one by one for all the processing steps

hiredman18:03:20

transducers, instead of transforming data, transform the function you reduce with, so you end up with a single function so no intermediate sequences between each step

hiredman18:03:02

if you call map, a collection goes in, a new lazy seq comes out

abtv18:03:26

ah, I see. so, no overhead in transducers for this transformation?

hiredman18:03:02

no intermediate lazy-seq overhead

hiredman18:03:02

the lazy seq library (map, filter, etc) are really single step transformations, plumbed together via intermediate lazy sequences, where as transducers are a nice way to specify what you want in a way that fuses those steps together in to a big step.

hiredman18:03:55

but you can build a transformation using transducers and still go lazy-seq in and lazy-seq out if you want

hiredman18:03:35

because the transducers work has sorted of elevated the usefulness of reduce there are now some interfaces and protocols that let you define "collections", "iterations", "views", or whatever you want to cal them, similar to lazy-seqs, but using reduce instead of first/rest

hiredman18:03:49

and those might be more of an apples to apples comparison to lazy seqs

abtv18:03:06

Interesting... And if I create core.async channels, I can pass a transducer to a channel creation function, right? But what to do with exceptions in transducer transformations?

hiredman18:03:49

channels can take an exception handler a long with a transform, but I would suggest making your transform total

mnespor18:03:02

Is there a name for the operation (zipmap coll (map f coll))? Like group-by, but the items in coll become the keys instead of the values

abtv18:03:53

@hiredman: what do you mean by total?

hiredman18:03:59

doesn't throw exceptions

hiredman18:03:47

actually it looks like the way the channel error handler works has the same effect as making your transform total

abtv18:03:27

What about core.async/pipeline function? As I see I can create a couple of channels and use pipeline to transform data with several processor cores. Is it work so?

hiredman18:03:18

yes, but there on several distinct flavors of pipeline depending on what you are doing, so be sure to read the docs

hiredman18:03:31

also don't use stateful transducers with pipeline

hiredman18:03:03

and pipeline-async is really odd, so if you do use that, be prepared for it to act in ways that you don't expect

abtv18:03:25

Like this: I have several producers and several consumers. I would like to parallelize processing (produce item -> process item -> consume item) and consuming.

abtv18:03:49

consume = save or send over network, etc

hiredman18:03:58

for io you'll want to use pipeline-blocking, for computation you'll want pipeline

abtv18:03:49

Am I right that producer and consumer doesn't know about transformation which I call with pipeline fn?

hiredman18:03:23

pipelines do need to produce output, so if you have a terminal stage that won't work as a pipeline

abtv18:03:24

Only pipeline fn knows about my transformation?

abtv18:03:52

Channels, producers and consumers are decoupled from a transformation in pipeline fn, right?

abtv18:03:14

btw, Is it good to build CPU consuming programs with core.async? I could use thread, <!!, >!! and pipeline for cpu bounded operations and go, pipeline-blocking for async operations.

hiredman18:03:16

no no, cpu bound operations go in go blocks and, blocking operations like io go in pipeline-blocking

abtv18:03:17

go uses thread pool, why use it for cpu bounded operations?

hiredman18:03:19

I, uh, what?

abtv18:03:07

I thought that thread is better than go for cpu tasks

hiredman18:03:11

if want the best performance for your cpu bound work, you cannot just launch hundred of thousands of threads to do it, you need a pool

hiredman18:03:18

you have that exactly opposite

abtv18:03:21

so, in general go is the best choice?

hiredman18:03:48

it depends on what you are doing

hiredman18:03:23

if you are doing io, don't use go (you can use thread, or future, or whatever else)

abtv18:03:10

but <!! parks its thread

hiredman18:03:31

never use the double bang variants in a go block

abtv18:03:44

oh, my mistake, I mean <!

hiredman18:03:02

it doesn't though

hiredman18:03:31

it logically parks the thread, but it doesn't actually block the thread

hiredman18:03:44

which is the whole point of the go macro

hiredman19:03:55

if you are familiar with continuations, inside a go macro, if you use <! you can think of it as meaning: take the continuation, give it to the channel, and the channel will schedule the continuation to be run if anything is put on the channel, and the current thread then is returned to the threadpool

abtv19:03:52

ok, I think I need to play with it and check. Thanks! @hiredman

alexisgallagher20:03:11

Does clojure itself provide any facilities for generating a valid edn representation of a user-defined record that was defined with defrecord? I had thought that pr and friends did that, but now that I'm looking at the matter it seems like they do not.

hiredman20:03:49

records don't exist in the edn spec

alexisgallagher20:03:06

Okay. But edn is extensible. So say I want to extend it to encode/decode a defrecord I have created. This seems quite likely and obvious. Is there an easy way to do this out of the box?

hiredman20:03:21

what exists in the edn spec are "tagged elements"

hiredman20:03:43

so you can change pr to print whatever you want, to print whatever you want, for any type

hiredman20:03:06

(not even restricted to any kind of well formed expressions)

hiredman20:03:28

so you can alter it to print the tagged element syntax

hiredman20:03:33

then, depending on how you are reading the edn, you'll need to define a reader for that tagged element

alexisgallagher20:03:19

This seems like something I will want to do a lot. I want to use clojure.edn/read just so that I'm reading forms safely. But that means I need to read/write edn, rather than clojure's ordinary serialization format (which read-string / pr provides). However, if I need to write custom encoder/decoders for every defrecord just in order to use edn, then that's a lot less convenient..

hiredman20:03:56

it depends of course

staypufd20:03:19

In the Clojure cookbook in chapter 4, recipe 16 they cover Emitting Records as edn Values. the next recipe also talks about how to handle unknown Tag values.

hiredman20:03:00

edn is itself rich (it has sets, maps, etc) so, at least in my experience, I don't end up tagging everything like that

hiredman20:03:45

how you serialize is sort of an extension of how you model data in general

hiredman20:03:08

do you use a lot of custom types, or do you use maps and lists and sets?

alexisgallagher20:03:10

@staypufd: Thanks. good to see.

staypufd20:03:18

Glad to help

alexisgallagher20:03:34

That seems to confirm my conclusion, alas. I had hoped edn was better integrated into the language so that I could handle defrecords without having to bring in yet another third-party library. 😕

hiredman20:03:41

tagging in edn is meant to be, sort of semantic, not representation specific

hiredman20:03:40

for example, the #inst means it is instance data in a certain serialized format, but that doesn't tell you want the deserialized format will be

alexisgallagher20:03:48

I really just want to round trip defrecords with safe reading. That's it for now. I had thought that clojure.edn/read was the recommended way to go about this but I'm guessing maybe it's not.

staypufd20:03:09

Well, if you take into account the reason Records are in the language then it makes sense that edn doesn’t have them. Records were added as a more efficient alternative to structs. They actually get made into Java class objects and have all the strengths and weaknesses of them. But b/c of that edn goes with the more general struct for it’s representation.

staypufd20:03:21

If all you want to do is round-trip then doing structs is better for you and then have a Record that you fill in from the struct (via same key names) for when you need to use the Record type for interop or speed

staypufd20:03:47

@schmir - that’s like 4.17 in the Clojure Cookbook

staypufd20:03:59

nice library though. No need to type in the code

alexisgallagher20:03:29

So you're suggesting at de/serialization time I go from MyRecord -> struct -> edn ?

hiredman20:03:03

he means maps

staypufd20:03:08

Not really. I mean if you don’t need Records for interop and the speed they give, then use structs.

hiredman20:03:17

he means maps

🙏 1
staypufd20:03:33

Clojure had defstruct too

alexisgallagher20:03:42

I believe I do need records for various other reasons.

staypufd20:03:44

I program in too many languages and get them mixed up

hiredman20:03:45

yeah, ignore that

hiredman20:03:14

if you care about exact type round tripping, edn is not for you

staypufd20:03:21

I only use Records when I need the interop or maps are not fast enough

staypufd20:03:51

the OReilly Clojure book explains it pretty well. Better than I am coming across

hiredman21:03:09

edn, as an interchange format, specifically does not do type round tripping

staypufd21:03:43

I know this is going to sound weird, but you could potentially use the Java RMI since your using Records

staypufd21:03:02

then the data passing forward and back may be hidden and done for you

alexisgallagher21:03:35

too much mental overhead. I've forgotten everything I once knew about Java RMI and would like to keep it that way. simple_smile

staypufd21:03:39

That code there doesn’t do it for a Record but since defRecord creates a class you’d be able to use the objects for RMI

alexisgallagher21:03:02

sounds like the easy options are to go with the miner library or just stick with pr / read-string and forget edn for now. Going to go with option 2.

alexisgallagher21:03:15

better unsafe reading than another dependency at the mo.

staypufd21:03:20

Simple wins again

alexisgallagher21:03:43

more easy than simple. I have unsafe reading now. c'est la vie.

reefersleep21:03:22

@cky @wamaral @mping : I'm thinking of a point and an angle, not a length and an angle.

cky21:03:08

@reefersleep: Yep, that can still be a vector. (Presumably angle is relative to x-axis.)

cky21:03:15

Oh wait.

cky21:03:18

No length?

reefersleep21:03:31

Just coordinates for an origin.

cky21:03:11

Yeah, I don’t have an answer off the top of my head then. 😞

reefersleep21:03:00

Thanks for the attempt! simple_smile

reefersleep21:03:11

I'm not sure there is a term for it.

reefersleep21:03:46

I'm just trying to name stuff in my code, so it's no biggie, just a minor annoyance... I do like to be precise.

wamaral21:03:51

angle relative to what?

wamaral21:03:45

could be relative to x, and length would be (x,y) - (0,0)

wamaral21:03:28

that's assuming a lot, though

josh.freckleton21:03:37

Is EDN still a popular format for passing around data from server to client? I read somewhere it's deprecated, and just curious what the best practice is for a REST API to communicate with CLJS on the front simple_smile

taylor.sando21:03:16

transit is more efficient

taylor.sando21:03:37

Though less readable, but more extensible.

josh.freckleton21:03:31

you mean less "human readable", right? sounds good. And is EDN in general still alive and well?

taylor.sando21:03:42

Human readable

taylor.sando21:03:02

I personally use transit. I think the options are Fressian, Edn or Transit. I'm not very familiar with fressian.

reefersleep21:03:20

@wamaral: relative to the x axis

josh.freckleton21:03:53

@taylor.sando: Is the best way to use transit with compojure ring-transit? : https://github.com/jalehman/ring-transit

reefersleep21:03:53

But the x/y position is meant as the "current" position, and the angle is the direction that a line is going to be drawn the next time. Only these two pieces of data are coupled - the length is unknown and not necessary at this point.

base69821:03:38

I am generating a lazy list from a db. The db driver is using korma raw.

(defn lazy-df-db
  ([] (lazy-df-db 20000))
  ([l]
   (map parse-rec (take-while (complement empty?)
                    (mapcat #(get-record-block %1 l)
                             (reductions + 0 (repeat l)))))))
 
This is so slow it never really returns. I can hook it into a text file with the same data and it returns in seconds. Any ideas, about how to speed up db operations? I'm having it query a materialized view in sorted order, the query runs super fast in psql, but again slow here.

wamaral21:03:21

@reefersleep: maybe you want the current position and an unit vector which you could scale at will

taylor.sando21:03:47

@josh.freckleton: If you're using ring it would work. I use pedestal which has its own middleware for handling it.

wamaral21:03:43

@reefersleep: an angle needs 3 points in space, just 2 (current position and x axis) is not enough to form one

taylor.sando21:03:21

@josh.freckleton: I mean compojure would work with it.

josh.freckleton21:03:43

@taylor.sando: gotcha, just trying to keep in touch with best practices. From a cursory look, Pedestal seems a little involved, and unless you'd suggest I learn it, I'll probably keep using compojure for now. Thanks for the help, there are a lot of descisions to navigate in this world of CLJ

taylor.sando21:03:05

Ya pedestal is pretty involved. I learned it awhile ago. Not sure it's significantly better than the other options.

hiredman21:03:38

base698: it is hard to tell from that what you are trying to do. your code + korma is almost certainly pulling all the data in your process's memory and operating on it there, without the benefit of any data layout and indexing optimizations that the sql database does

hiredman21:03:12

that whole function sort of strikes me as odd, like, reductions with + is going to result in a sequence of numbers, in which case why not use range or iterate

base69821:03:07

@hiredman I don't know how many records there and don't want to query with count to find out

hiredman21:03:23

I would double check your running against a file comparison to make sure it is doing the same thinig

hiredman21:03:42

base698: range and iterate both can generate infinite sequences

base69821:03:43

get-record-block get's the offset and limit worth of records.

base69821:03:08

@hiredman: Iterate would probably work

hiredman21:03:11

range can take a step as well

base69821:03:25

how could you do range if there is no end point?

base69821:03:31

if you want blocks of say size 100?

base69821:03:55

(range 0 100) ; 0 - 99 (range 0 100 2) ; range 0 2 4 ...

hiredman22:03:14

oh, right, so iterate

base69822:03:37

or reductions simple_smile

base69822:03:54

are people using korma? That's what I'm using

hiredman22:03:00

ok, so the issue your take-while

hiredman22:03:15

I've only used java.jdbc

base69822:03:21

i'm about to try that

hiredman22:03:41

what happens is because of your mapcat there no empty thing for take-while to see

hiredman22:03:10

so you have a seq, that isn't really infinite, but doesn't have an end

hiredman22:03:38

if you split mapcat in to map and apply concat, and put the take-while between you should be fine

base69822:03:26

holy shit and jdbc takes connection strings

base69822:03:35

so map and apply concat?

hiredman22:03:13

that would be the sort of minimal change to make it work

hiredman22:03:34

(apply concat (take-while ... (map ...)))

jrotenberg22:03:13

i just realized that if you use an empty vector with :keys in destructuring

hiredman22:03:34

you should google unfold which doesn't exist in clojure.core but is very useful for exactly this sort of thing

jrotenberg22:03:54

you can accept arbitrary (or no) elements

hiredman22:03:32

https://gist.github.com/hiredman/4d8bf007ba7897f11594 is unfold for CollReduce, but somewhere out there I know people have implemented it and things like it for lazy-seqs too

hiredman22:03:58

it makes working with sort of paginated apis much nicer

base69822:03:29

your help was amazing, thanks.

reefersleep22:03:41

@wamaral: When I say "angle", I really mean a radian. Maybe "angle" is not the proper term.

dg22:03:21

Has anyone ever seen a XML coercer for Schema?

base69822:03:38

korma wouldn't even return running a normal query "select *" looks like this is going to return. anyone know the defaults of Xmx for lein? and how to increase it.

base69822:03:54

can you do lein run -Xmx7g

ihercowitz22:03:44

@base698 i have used korma on a project, but i don't like much this kind of DB manipulation it uses (like rails, django, etc..) today I'm using yesql, i have wrote a few queries, and it looks like good for me :)

base69822:03:42

Is there a way to setup a build target to compile one clojure file into js via cljsbuild?

bostonaholic22:03:44

@base698: you might be able to accomplish that with modules