Fork me on GitHub
#clojure
<
2017-03-22
>
josh.freckleton00:03:00

so I have the most recent version locally, is there a way I can work on it (via local checkouts) at the same time that other libs are coming in remotely from s3 (ie :repositories)?

josh.freckleton00:03:35

>checkouts don't replace a dependency I just ran lein install, and things work, I'll have to check to make sure that solves the "checkouts" workflow though

mars0i00:03:57

@akjetma I wouldn't have any clue about the right way to do that. I get a lot of mileage from creative use of what's at hand: Monitor cpu from a script and send it a KILL when nothing's been happening. Definitely not the right way, but if it works ...

brycecovert00:03:06

Let’s say I have a set of 1,000 http requests I want to make, with up to 100 happening concurrently. My first thought would be that this is a great use case for core.async, but clj-http uses blocking io, and the default thread pool count for core.async is 8. I could increase this to 100, but that somehow feels wrong. Any advise?

brycecovert00:03:39

Or, perhaps, there’s a library that works similarly to r0man/cljs-http, where network io is non blocking, and plays well with core.async

jeff.terrell00:03:45

@josh.freckleton - I haven't seen lein checkouts before. I assume you've read [0] and [1]? If those don't answer your question, maybe try over in #leiningen. [0] https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#checkout-dependencies [1] http://jakemccrary.com/blog/2012/03/28/working-on-multiple-clojure-projects-at-once/

csm00:03:24

@brycecovert I’d reach for either http-kit or aleph; they’d both need some shim code to translate into core.async, but it’s not that hard

brycecovert00:03:19

@csm using this? http://www.http-kit.org/client.html I’m presuming that if http-kit parks the thread for io, it will be available for core.async, is that correct? It sounds like the “shimming” might be as simple as (<! (go (http/get …))), right?

brycecovert01:03:23

or, perhaps to be more specific, if parking a thread makes it available for other purposes, then simply putting responses onto a channel would be all that’s required

csm01:03:47

right, the “shim” would look more like (let [chan (async/promise-chan)] (http/get … (fn [r] (async/put! chan r))) chan)

csm01:03:43

i.e. since http-kit is callback-based, make the callback complete your channel

brycecovert01:03:44

Ah, got it. I misread the documentation. Makes perfect sense.

brycecovert01:03:51

I hadn’t noticed that it was callback based

brycecovert01:03:07

👍 Very straightforward, thanks!

noisesmith02:03:40

brycecovert the core.async thread pool size is (nprocs * 2)+42 - but they still shouldn't be used for blocking io

noisesmith02:03:58

there's also core.async/thread for things that should happen outside the thread pool

noisesmith02:03:54

@csm I'd argue for using >!! instead of put, since to get proper control of the task pool the http/get should be in a loop that reads a request from an in channel and writes the result to an out channel, and >!! can impose backpressure that prevents the request producers getting too far ahead of the request making

noisesmith02:03:02

I'm picturing a loop (go-loop [] (let [r (<! in)] (http/get ... (fn [r] (>!! out r))) (>! consumer (<! out)) (recur)) - the extra indirection of a consumer channel and out channel is so that the go-loop has back-pressure, to have N parallel requests, start N loops

seancorfield03:03:19

@noisesmith It used to be that size -- it got changed a while back I think?

noisesmith03:03:47

oh, I checked fairly recently... one moment

seancorfield03:03:06

Early/mid-2016... I'm going back through commits now...

noisesmith03:03:49

yeah, that's the same commit I linked

seancorfield03:03:33

Oh, sorry I didn't see you'd posted two links while I was still looking for it in GitHub

joshjones03:03:56

@brycecovert this may help out regarding core.async to generate http requests and guarantees a cap on concurrency http://stackoverflow.com/questions/42368332/easiest-way-to-use-a-i-o-callback-within-concurrent-http-kit-get-instances/

renewdoit04:03:46

any preferable way to define spec keys' relationship?

renewdoit04:03:12

e.g. I have a collection of seats, only one of them is active.

renewdoit04:03:22

it will generate struct like {::seats [] ::active-seat 2}, which actually is not legal

hiredman04:03:03

Write a predict that is only true when the seats count matches active seat

hiredman04:03:24

And it to the coll of

hiredman04:03:51

You most likely will need to write a custom generator

renewdoit04:03:42

@hiredman thanks. I modify it like s/def ::my-struct (s/and (s/keys [::seats ::active-seat]) (fn [{:keys [::seats ::active-seat]}] (< active-seats (count seats)))))

renewdoit04:03:02

but do not know how to write the generator

hiredman04:03:43

The spec guide on the clojure website has a section on generators

aengelberg04:03:58

user> (do (def arr (int-array 100)) (aset arr 0 (inc Integer/MAX_VALUE)))
-2147483648
user> (let [arr (int-array 100) (aset arr 0 (inc Integer/MAX_VALUE)))
IllegalArgumentException Value out of range for int: 2147483648  clojure.lang.RT.intCast (RT.java:1205)

mars0i04:03:23

I had been opening a file with , then storing the resulting Writer in a map in an atom that I pass around. I write to the file in a different namespace, and then close it in the original namespace when a condition is met. This worked fine. I wondered if I could use with-open instead. Now I call wrap the writer call in with-open, but again store the writer object the atom. Inside with-open, I call a function that calls a function in the other namespace that eventually calls something that pulls the writer out of the map in the atom and then tries to write to the writer. While this is happening, I have not left the scope of with-open in the first namespace. However, the write fails with "java.io.IOException: Stream closed". I'm not 100% surprised--I know it's possible to try to do something outside the scope of with-open, and I wasn't sure whether this would work. But I don't understand why it doesn't work, given that with-open is still "running" when the write happens. Must all of the uses of the writer from with-open be visible inside the parentheses surrounding with-open? Am I confused about something?

noisesmith04:03:21

are you sure the with-open form never exited?

noisesmith04:03:31

what keeps it waiting?

mars0i04:03:29

I don't see how it could. run-sim is the last call in the with-open form, and that's what calls the functions that write. But now I'm thinking that there's another way for run-sim to be executed, outside of the with-open. I should make sure that's not happening by accident.

jr04:03:47

.close is called after the clojure form is evaluated

noisesmith04:03:53

also, why use an atom, doesn't that just make it easier for something else to use the file handle after with-open exits? the usual pattern is to pass the file handle to a function inside the with-open call, so that with-open can't leave until the function call finishes

jr04:03:55

you can't store the writer

noisesmith04:03:35

well you can, if you ensure it's never used after the form exits, but that seems like a lot of unnecessary complexity compared to just, you know, not storing the writer

noisesmith04:03:21

also, be wary of any lazy ops inside with-open

qqq04:03:41

what is it about js vs python that we have clj -> js but no clj -> python ?

qqq04:03:49

(no, the answer is not Hy, it's too different from clj)

mars0i04:03:56

I could pass the writer rather than using an atom, but I'm storing a whole bunch of stuff that's needed in the other namespace anyway. It's the global config structure.

mars0i04:03:17

Good point about laziness. I don't think that's going to be relevant, but laziness bites when you're not looking.

noisesmith04:03:56

if I had a dime for every call to map inside a with open that did something unexpected (especially when hidden by eg. a database api...)

mars0i04:03:59

OK, maybe I'll try passing the writer. It means adding a parameter to several functions, but I shoudl try it.

noisesmith04:03:33

mars0i the thing is that adding an arg is much less complex than managing the validity of a temporary output resource manually

mars0i04:03:17

Thanks noisesmith and jr. There's a bit of work to setting things up to pass the writer explicitly. Will leave that for morning.

mars0i04:03:52

Yes, I see your point noisesmith. Seems like what I'm doing should be OK, but ... obviously it's not.

mars0i04:03:58

Anyway what you are telling me is that my understanding of with-open seems correct, so I have to look for other problems in my understanding of my code.

mars0i05:03:08

Or I could just go back to explicitly closing the file when I know I'm done with it. That worked. But sometday it might bite me.

mars0i05:03:30

now that I think abut it, what run-sim does is passes reifys to a scheduling function. this is from a java lib l didn't write. the write fns are called from the reify. this is starting to make sense. the scheduler probably runs the classes after with-open finishes.

mars0i05:03:33

with-open's not going to work if this correct.

beppu06:03:14

@qqq js gives you a lot of reach. Everyone is compiling to js so that they can get into the browser. Also, with React Native, js can also get you into mobile devices.

beppu07:03:22

clj -> python is possible, but is it worth the effort?

not-raspberry10:03:06

clj -> rpython may be worth it

not-raspberry10:03:44

pixie compiles to rpython but is not clj

not-raspberry10:03:08

clj -> llvm would be a killer (commandline tools!)

schmir14:03:19

How do I spec a map, which contains string keys instead of keyword (i.e. {"foo" "bar"} instead of {:foo "bar"}) ?

mpenet14:03:45

(s/map-of string? ...) or you convert the keys to kw (:() if you want k->v validation

mpenet14:03:03

or a custom predicate/validator

souenzzo14:03:15

:keywordize true 😛

schmir14:03:24

thanks. didn't know about map-of, but it looks like every value would need to conform to the same spec.

raspasov14:03:31

ok so what I’m going to ask might fall a little bit in the realm of some crazy dark magic 🎩 but… has anyone ever thought about “re-using” their destructuring? sometimes I have a pretty big data structure that I de-structure, often in a let

(let [;destructure props
          {{product-options                              :db.product/product-options
            product-id                                   :db.product/id
            {c-uuid :uuid :as selected-options} :client/selected-options
            :as                                          product} :product} props)

raspasov14:03:54

and often I want to use that in multiple places, in different functions; any better way than simply copy/paste that everywhere?

dpsutton14:03:08

i made an anaphoric macro that would destructure everything in the same way

raspasov14:03:12

(that’s an example from react/cljs, but it very well can happen on the server also)

dpsutton14:03:30

it ensured that the signature was consistent and that i correctly got everything

tbaldridge14:03:32

A lot of the complexity there ^^ is due to nested maps. When I get to needing something like that i normally flatten the map then use :keys [...]

raspasov14:03:56

very interesting indeed 🙂

raspasov14:03:02

what about… performance?

raspasov14:03:28

I guess if it works for your use-case… it works… just have to try it

tbaldridge14:03:46

a single level of map is probably going to be faster since you have 1 less lookup

raspasov14:03:56

right, I mean the transformation part

raspasov14:03:22

since I already have the nested map… but I see what you’re saying… pay the cost once, then access fast every time… you can cache that etc etc

raspasov14:03:36

because immutability rocks 🙂

tbaldridge14:03:40

ah, yeah, depends on where this information is from. Optimal method is to use namespaced keys and a single map, but you may not have control over that

bja14:03:28

@tbaldridge that's something that I've been slowly converting the edges of my APIs to use. the lack of nesting is refreshing in the parts of my codebase that have been adapted

raspasov14:03:39

@tbaldridge that’s a very interesting approach indeed… you suggested that in some other place as well in the google groups… I think I should use more of that

raspasov14:03:42

nesting is definitely a source of a lot of complexity… I’ve definitely cut the amount of it a lot, usually trying to stick to 3, max 4 levels, but in a way it’s infectious lol… once you have nesting, it only brings about more of itself haha

raspasov14:03:47

@tbaldridge so do you literally use a single level map everywhere, or you cut off at like, 2 levels maybe? 🙂

tbaldridge14:03:18

Unless my data is a tree, or I have a "many values" attribute, I use one level

raspasov14:03:15

yes… such a good tip, that should be a Clojure sticker 😜

raspasov14:03:29

USE SINGLE LEVEL {}

raspasov14:03:46

+ namespaces… that really makes that happen

raspasov14:03:09

allows that to happen in a very pleasant and elegant way

bja14:03:47

Of course, if this were go, our next version of map wouldn't support nesting other maps.

mpenet15:03:20

You'd probably use structs anyway for anything "speced"

raspasov15:03:31

@bja haha is that a fact? 🙂 (I barely know any Golang)

bja15:03:33

sorry, my joke was out of place. backs away from the kindling

tbaldridge15:03:10

we could continue in #off-topic

raspasov15:03:48

(don’t want to turn this into a golang discussion haha)

noisesmith16:03:17

@not-raspberry the problem is that llvm makes you bring your own gc, and clojure demands a lot from its gc. You can already do fast launch cli with cljs via lumo or planck.

noisesmith16:03:50

multiple “clojure on llvm” projects have started, none are usable yet

ddeaguiar16:03:06

@gungfoos, in #(when (vowels %2) (inc %1)) %1 is the first param, %2 is the second param to the anon fn.

gungfoos17:03:54

Sorry still can't see it. Read the link.

gungfoos17:03:39

when (vowels %2) is true do (inc %1) is content of anon func. Is word param #2?

rauh17:03:54

You can try (read-string "#(when (vowels %2) (inc %1))") and see how the reader expands the anon fn.

gungfoos17:03:56

Useful but I still don’t know what specifically goes into param 1 and param 2. I only see “word” as a possible parameter.

gungfoos17:03:59

Is (inc %1) the parameter into %2 and “word” is %1?

ddeaguiar17:03:55

so in this:

(def vowels #{\a \e \i \o \u})
;; => #'user/vowels
user> (keep-indexed #(when (vowels %2) (inc %1)) "foobar")
;; => (2 3 5)

ddeaguiar17:03:34

the word ”foobar” is treated as a seq of characters (\f \o \o \b \a \r)

ddeaguiar17:03:08

keep-indexed applies a 2-arity fn to the sequence

ddeaguiar17:03:15

the first arg will be the index

ddeaguiar17:03:13

I suspect the reason inc is called on the index because indexes are zero-based and they want to start counting from 1

ddeaguiar17:03:25

@gungfoos does this make sense?

ddeaguiar17:03:16

Sets can be used as functions

ddeaguiar17:03:32

hence the application of vowels in (vowels %2)

rauh17:03:38

Did you read the docstring of keep-indexed?

gungfoos17:03:44

Yes but struggled with the doc. But I think I’m getting your answer a bit better now. The first arg index was something I didn’t realize

gungfoos17:03:00

Yes, they wanted to start counting from 1

gungfoos17:03:14

Thanks to both of you for help. I will delve into docs again and play with REPL some more.

ddeaguiar17:03:50

When you read that doc-string, disregard this bit for now Returns a stateful transducer when no collection is provided.

ddeaguiar17:03:22

It’s not applicable to your problem and it’s a potential rabbit hole for you atm

gungfoos17:03:18

My problem was that I was trying to break it down like this in REPL:

gungfoos17:03:23

(def vowels #{\a \A \e \E \i \I \o \O \u \U \y \Y}) (def word "Apple") (#(when (vowels %2) (inc %1)) word)

gungfoos17:03:19

Which now I know doesn’t make sense without keep-indexed. I didn’t know how to read doc on keep-indexed and missed the (f index item) part plus didn’t realize the index was implicit param 1.

ddeaguiar17:03:40

ah yeah, I could see how one would do that

gungfoos17:03:53

Getting much clearer now. I don’t understand (vowel %2) still. %2 is the item and item is the word. I put (vowel “foobar”) in and get an error. vowel is a set and I don’t even understand how we pass it a param?

ddeaguiar17:03:00

I wouldn’t sweat it, seems like a reasonable mistake

ddeaguiar17:03:17

%2 is a char in the word

ddeaguiar17:03:33

remember strings are seq-able

gungfoos17:03:41

Tried (vowel “\a”) and it errors too

ddeaguiar17:03:53

(seq "foobar")
;; => (\f \o \o \b \a \r)

ddeaguiar17:03:24

it would be (vowel \a)

gungfoos17:03:33

Ah…thanks

ddeaguiar17:03:36

\a not ”\a”

ddeaguiar17:03:51

(class \a)
;; => java.lang.Character

gungfoos18:03:21

What makes it take a char at a time from the word? I don’t see mention of that in docstring for keep-indexed. Does “return a lazy sequence” imply that?

ddeaguiar18:03:09

Now, I’m not expecting anyone to look at that code

ddeaguiar18:03:18

if you read this guide https://clojure.org/reference/sequences, you’ll see that fns in the Seq library take seqs in and produce seqs

ddeaguiar18:03:45

I highly recommend spending time to grokk the Seq library. It’s fundamental and the api is very consistent

ddeaguiar18:03:06

Note that keep-indexed is present in the Shorter seq from a longer seq list of fns

gungfoos22:03:44

I’ll read up on Seq. I’m trying to go through the Clojure Programming book and at same time work on some coding challenges.

xiongtx17:03:47

Why does the following occur?

(require '[clj-time.core :as t])

(with-redefs [t/today-at (constantly 11)]
  (t/today-at 0 0))
;; => Unhandled java.lang.ClassCastException
;;    clojure.core$constantly$fn__4614 cannot be cast to clojure.lang.IFn$LLO

(with-redefs [t/today-at (constantly 11)]
  (#'t/today-at 0 0))
;; => 11
Something to do w/ Var roots? Working my way through @tbaldridge’s article now: http://blog.cognitect.com/blog/2016/9/15/works-on-my-machine-understanding-var-bindings-and-roots

tbaldridge17:03:53

heh, that's even more of a bizzare issue, it's not even timing related

tbaldridge17:03:52

So the problem is that today-at is type-hinted. So the compiler knows that it takes a long, and a long and returns an object. However, then you went and re-bound it to a function that takes an object an object and returns an object

bronsa17:03:58

you can't redef primitive type hinted functions

tbaldridge17:03:14

The second one works because you're calling through a var.

rauh17:03:56

@xiongtx Do you see the LL0 at the end? That means it's trying to invoke a primitive fn (optimization) because you pass in 0 0 (`L`ong, Long) which returns an object

tbaldridge17:03:35

so, a question for @xiongtx and anyone else, is there some source I'm missing where someone is recommending using with-redefs? There seem to be a lot of questions about it recently.

xiongtx17:03:40

@rauh I see, I knew about type hints but not how exactly they’re optimized

tbaldridge17:03:57

whereas I've always considered it one of those things-I-wish-someone-would-delete-from-clojure-core.

xiongtx17:03:40

@tbaldridge Well, with-redefs‘s own docstring says: > Useful for mocking out functions during testing. I guess we all thought it’s true (which it is, until you run into something like this)

tbaldridge17:03:05

eh, good point, maybe I'll go write a patch for that.

xiongtx17:03:41

If you put > [WARNING]: Avoid like the plague in the docstring, there’d be fewer Q’s like these 😛

xiongtx17:03:00

By the way, it’s LLO (letter O), not LL0

mobileink17:03:03

I have some defrecord values that seem to behave inconsistently wrt instance?. E.g. they’re all <#C4H71K9RQ|miraj>.co_dom.Element{…}, and both (type) and (class) return miraj.co_dom.Element. But some of them return true when I do (instance? miraj.co_dom.Element e) and some return false. What’s going on?

rplevy17:03:29

@xiongtx @bronsa @tbaldridge but if you redef it with a function that has the same type hints on its params, no problem

mobileink18:03:05

this is driving me nuts. (type e) prints miraj.co_dom.Element, but (= (type e) miraj.co_dom.Element) is false.

mobileink18:03:09

so I can’t make a distinction between a plain ol map and a defrecord value. 😞

xiongtx18:03:27

Do you get the same classloader?

thheller18:03:51

@mobileink if you are working on a REPL and reloaded the namespace that defines the record but still have remaining instances of the old one the instance? check will fail

mobileink18:03:11

@xiongtx: (log/debug "CL type" (type (.getClassLoader (class m)))) prints clojure.lang.DynamicClassLoader

mobileink18:03:46

@thheller: I’ll bet that it’s it. thanks!

vadlamak18:03:30

hello fellow clojurans - had a newbie question

vadlamak18:03:47

i am running a “lein run” command on my terminal

vadlamak18:03:00

and want to connect to it via emacs+cider

vadlamak18:03:25

for the life of me…i am not able to determine the default port to use for cider-connect command

vadlamak18:03:51

appreciate any pointers in this direction ^^

xiongtx18:03:54

Let’s move this over to the #cider channel

noisesmith18:03:03

it’s a lein issue

noisesmith18:03:16

@vadlamak when using lein run you won’t get an nrepl at all, unless your app starts nrepl manually

noisesmith18:03:25

lein repl will start nrepl

vadlamak18:03:18

@noisesmith thanks for pointing out my folly…use

lein repl
…i was doing
lein run

whilo18:03:34

can I circumvent the printing multimethod with pr-str somehow? i would like to use default record prints instead of the custom handlers that are active in one of the namespaces.

whilo18:03:54

for reading you can do clojure.edn/read-string with a read-handlers map

mobileink18:03:20

@thheller that was indeed the problem, a remove-ns did the trick. thanks bigly.

lxsameer19:03:37

hey guys, what is the best way to test this component ? http://dpaste.com/3FK1RJ0

hiredman19:03:55

(let [sys (component/start ....)] (try do http request (finally (component/stop sys))))

john20:03:22

hey yall

john20:03:33

so I'm getting this error: Uncaught Error: Could not find tag parser for error in ("inst" "uuid" "queue" "js" "datascript/Datom" "datascript/DB" "Agent")

john20:03:01

so specifically, the #error tag is not being found in the tag-table

john20:03:12

\tag-table\

john20:03:26

I'm trying to shuttle code/data between webworkers and on the return trip, I'm getting an error which won't be read by the reader, because #error appears not to be registered

john20:03:23

Is this intended behavior and, if so, does anyone know where the implementation of #error's reader function is, so I can replicate it?

hiredman20:03:55

#error doesn't have a reader function, if I recall

hiredman20:03:16

errors prn as data and it is up to you to interpret it

john20:03:55

the following: #error {:message "ERROR in file

hiredman20:03:06

that is a prn'ed error

john20:03:06

in mike's response

hiredman20:03:28

prn of a tagged literal does not imply a reader exists

hiredman20:03:42

I could be mistaken

noisesmith20:03:51

see also #object...

hiredman20:03:07

the easiest thing to do would be to set identity as the handler

hiredman20:03:18

(for #error)

john20:03:19

@hiredman so could it be that the error is being sent back over to the reader and the reader is mistakenly interpreting it as a tag?

hiredman20:03:43

you are doing something, prn'ing the result, the result is an error

hiredman20:03:24

#error .... is what you get when you call prn on an exception in clojure (not sure what kinds of objects do that on the clojurescript side)

john20:03:43

understood

hiredman20:03:07

it isn't a mistake to interpret it as a tag, it is a tag

hiredman20:03:19

just one without a defined handler

hiredman20:03:35

you can either set a catch all handler (if I recall) or set up a handler for the error tag

hiredman20:03:40

if you want to read it

john20:03:42

Should I add a handler for #error in a data_readers.cljc file or should I do so another way?

hiredman20:03:20

it depends how you are reading the data, I would look at the doc string on whatever read function you are calling

hiredman20:03:32

a number of them support passing in a map of handlers

noisesmith20:03:49

oh it ate my comment - sometimes I have wished that object had a readable prn, though I understand the argument for not having one

noisesmith20:03:05

@john I like using transit for round tripping clojure data between hosts, it makes it simple to set local handlers for tags / datatypes rather than needing to mess with clojure’s readers / printers

noisesmith20:03:11

it’s also somewhat smart about structural sharing within data, which the printer /reader won’t be smart with at all

john20:03:26

testing this out now: (cljs.reader/register-tag-parser! "#error" identity)

john20:03:54

@noisesmith you know, I thought butler, the webworker library I'm using, used transit to marshal data between contexts, but maybe not. I'll look into transit to see if I can use it on top of butler

noisesmith20:03:25

if so, you should be able to provide a map of data readers/writers, if it’s already using transit that’s the right way to handle it

john20:03:47

it sure doesn't

john20:03:57

So I'll look into that. thanks!

noisesmith20:03:34

yeah, they are using transit but aren’t letting you provide custom reader / writer maps - optional map arg would be a simple pr

john20:03:50

@jr nice catch! thanks. I just looked at the project.clj

john20:03:07

@jr @noisesmith okay, yeah, I see in the docs there

john20:03:09

(def w (t/writer :json-verbose {:handlers {Point (PointHandler.)}})) (t/write w (Point. 1.5 2.5)) ;; => "{\"~#point\":[1.5,2.5]}"

john20:03:19

good to go, thanks yall!