Fork me on GitHub
#clojure
<
2016-09-28
>
idiomancy00:09:39

why is nothing printed when I do this?:

(let [a (timeout 1000)
      b (chan)
      pipe (async/pipeline 1 a identity b)]
  (go-loop []
    (println (<! b))
    (recur))
  (go
    (>! a "hello")
    (>! a "world")
    (>! a "pipelines")
    ))

kurman00:09:04

@idiomancy you have a bug in your code

kurman00:09:19

(pipeline n *to* xf *from*)

idiomancy00:09:32

how about that

idiomancy00:09:36

damn it. thanks

shoxter01:09:31

Does anyone use Yesql here?

shoxter01:09:48

If so, it doesn't seem possible to mass update records using yesql (i.e. passing in a vector of records into a yesql function and having it execute the query on each item in the vector). Is there something I'm missing?

tom01:09:55

@shoxter No, YeSQL cannot do that. Look into HugSQL.

shoxter01:09:26

Thanks @tom -- Will do! I'm guessing it has the capabilities I'm looking for?

tom01:09:07

It does, it has minor differences from YeSQL otherwise

shoxter01:09:41

I'm seeing. I have a relatively large project and the conversion will take quite some time. It looks worth it though. Thanks for the suggestion. I'm going to evaluate it! 🙂

tom01:09:17

I just migrated a big project to it, was harmless with a few terminal commands and glue to sniff

tom01:09:29

Also, if you just need to insert multiple, you can just fall back to jdbc which works well except for some possible type conversion issues.

kardan06:09:07

+1 for Hugsql

kurt-yagram07:09:23

so, defprotocol doesn't really work well with optional arguments. Is using multimethods with class the best (clojure-y) solution to this?

metametadata07:09:11

defprotocol supports multi-arity methods:

(defprotocol Foo
  (method [this] [this x] [this x y]))

kurt-yagram07:09:31

Yeah, I know about multi-arity, but the optional arguments can vary a lot, so I would need a lot of possible arities, certainly if I want to embed all possibilities. Unless, of course, I take only 2 arities, and make the second arg one a map that can contain all options.

metametadata07:09:02

yeah, but I guess you can't really do it with multimethods/functions either?

metametadata07:09:55

there's defnk (https://github.com/plumatic/plumbing#bring-on-defnk) but it still requires you to pass a map into a function

kurt-yagram07:09:45

right! So, passing a map is the way to go, still using protocols. Thx.

metametadata07:09:58

yeah, I think it all boils down to the fact that clojure doesn't have a built-in way to pass named arguments

ajchemist07:09:11

Can I get a outputstream from a transit writer instance in jvm? Is there a interface for that?

grounded_sage08:09:20

So i've just spent literally 2 days creating an algorithm. I relied on a global Clojure Atom as I want to use the data captured like a database. This is the main engine of the code

(defn all-combinations [state]
  (let [{:keys [outside-number inside-number]} @constants
        last-combo (vec (last state))
        index (count last-combo)
        last-num (last last-combo)]
    (if (< last-num (+ (- outside-number inside-number) index))
      (do
        (swap! global-atom conj
               (inc-at-index (dec index) last-combo)
               )
        (println @global-atom)
        (all-combinations state)
        )
      (do
        (println "Time to step back" )
        (swap! global-atom conj (inc-reset last-combo))
                                         (println @global-atom)
                                            (all-combinations)))))
I wasn't quite sure how to turn this into a Loop so I just called
all-combinations
at the end of each if statement. I have the code successfully generating all the data I want in an atom and then I hit a
nil
and the whole thing blows up. How to I safely exit and continue doing transformations on the data I have generated?

grounded_sage08:09:30

Apologies for the messiness and random println's I'm SOOO close I just can't figure out how to safely exit the execution.

bcbradley09:09:46

i'm having an issue with core.async

yogsototh09:09:46

I don’t know what your editor is but to load core.async in the repl you need to load the namespace first

yogsototh09:09:55

and then you can launch the code

yogsototh09:09:33

Does it works with lein run in the console?

grounded_sage09:09:53

So nooby. I just fixed it all with a BOOL atom.

bcbradley10:09:04

nvm it appears nightcode's instarepl doesn't work with third party libraries right now

bcbradley10:09:13

it runs just fine though

dfan12:09:27

@bcbradley organizer is nice! It’s called a “topological sort” if you want to look up further discussion in the literature

bcbradley12:09:40

thanks i had no idea

hans12:09:54

is there a good reason why (nth nil 0) returns nil instead of throwing an exception?

bcbradley12:09:04

i can't think of one xd

bcbradley12:09:33

i guess since technically nth returns nil when the index is out of bounds

hans12:09:51

nope, it throws an exception.

bcbradley12:09:52

and the fact that nil can be interpreted as a container that cannot contain

bcbradley12:09:08

idk i'm just guessing

hans12:09:36

which could or could not be the better behavior, depending on how you look at it. but having it throw an exception except if the collection is nil seems to be inconsistent.

bcbradley12:09:01

i think i would prefer to have (get nil 0) -> nil and (nth nil 0) -> exception

hans12:09:55

right, that would be my preference as well. so i'd say it is a clojure bug.

pesterhazy12:09:13

@hans, definitely an oddity

pesterhazy12:09:43

I would have expected a NullPointerException

hans12:09:49

you can keep your euphemisms 🙂

pesterhazy12:09:57

It's only a bug if it doesn't conform to explicitly stated behavior, and the behavior for nil, which isn't a sequence, vector or string, is not defined.

pesterhazy12:09:25

(also it's probably a behavior that programs have come to rely on so hard to change at this point)

pesterhazy12:09:30

playing devil's advocate 🙂

hans12:09:18

well, the least thing then would be that the behavior is documented.

hans12:09:54

if it cannot be fixed because "too much code depends on the behavior", then how will bugs ever be fixed?

pesterhazy12:09:16

s/bug/surprising behavior/

dnolen12:09:13

but that’s just it, in most languages with lots of users, old “bugs” or underdefined behavior don’t ever get fixed (not in a consistent way anyway) 🙂

hans12:09:15

right. let's just call bugs "surprising behavior" from now on!

dnolen12:09:41

really popular languages even includes these “bugs” in their test suites

hans12:09:55

"the other guys don't do it, so let's just be like them"

dnolen12:09:48

or more likely - it’s a perfectly reasonable choice, just like the other option

pesterhazy12:09:03

my takeaway is to use get instead of nth as I don't like handling out of bounds conditions as exceptions

hans12:09:24

in a future docstring for nth, we'll read "If coll is passed as nil, nil is returned."

hans13:09:25

Which is, well, yes, a reasonable choice. there could also be a second part to this explanation ", for historic reasons".

hans13:09:34

@pesterhazy i like my bugs to be signalled as soon as possible. by using get, i'd just paper over the problem.

kokos13:09:45

user=> (nth nil 0)
(nth nil 0)
nil
user=> (get nil 0)
(get nil 0)
nil
user=> (count nil)
(count nil)
0
user=> (take 5 nil)
(take 5 nil)
()
user=> (first nil)
(first nil)
nil
user=> (rest nil)
(rest nil)
()
nil seems to be viewed as an empty collection when you use it with a function that receives a iseq, further more
user=>(map #(identity %) nil)
()
if there is a need for enforcing your symbol not to be nil a check is required before use. I personally find it usefully not to have an error thrown but a nil returned if my supposed to be iseq is actually nil.

pesterhazy13:09:10

I think amending the docstring would be a good change, @hans

hans13:09:55

@kokos the difference between nth and the other sequence functions is that nth has it in the contract that bounds checking is performed.

hans13:09:14

@kokos so it is my belief that making nil behave special is at least surprising. (nth [] 0) does throw an exception, so why does (nth '() 0) not do it?

pesterhazy13:09:50

(nth '() 0) does throw

hans13:09:10

oh, so nil is just the special case.

Lambda/Sierra13:09:19

nil is an empty sequence. (seq []) returns nil

pesterhazy13:09:58

@stuartsierra that doesn't explain this behavior though as (nth '() 0) doesn't match (nth nil)

bja13:09:07

'() is an empty sequence too. (seq '()) returns nil

bcbradley13:09:54

wow i didn't even know gist was a thing

bcbradley13:09:12

well here is my organizer function (renamed to topological-sort now that I know what it is)

bja13:09:13

it is a bit weird that (nth nil 0) and (nth '() 0) have different results

bja13:09:38

although they are only different on arity 2. if you pass the optional third arity, they work similarly (as I would expect anyway).

bja13:09:55

(= (nth nil 0 ::not-found) (nth '() 0 ::not-found))

bja13:09:59

I guess for consistency, you could always write (nth x i nil)

hans13:09:20

:face_with_rolling_eyes:

yonatanel13:09:13

What's an elegant way of tapping a manifold stream, preferably keeping the original stream usage intact?

dm314:09:20

@yonatanel if you manifold.stream/connect stream A to stream B (A -> B), B will receive every message that appears in A. However, if nothing is consuming B and it's not buffered (or goes over buffer size), the backpropagation will affect upstream (A). To avoid that you can specify a :timeout parameter to connect which will sever the connection in case a put! from A into B takes too long.

yonatanel14:09:37

@dm3 connect will remove from the original stream (A) which is a problem for me. I just want to "tap" and print debug messages while the stream behaves normally. I can connect two new streams A->B and A->C, B becoming the new source and C just for debugging, but for that I have to change the original usage of A, now needing to handle a split source and sink.

dm315:09:14

@yonatanel hm, this would only be a problem if you tap A while it's already running but nothing else is connected, so B becomes the only consumer

dm315:09:28

I can't think of a good way to tap if this is indeed the case.

yonatanel15:09:10

@dm3 This is interesting. If I use connect everywhere and then use it also for tapping, the original code remains as is

dm315:09:18

yes, if B is not the only consumer of A, the message is propagated to every consumer and B is effectively "tapping"

dm315:09:05

so you need to ensure that you have your main stream topology constructed by the time you tap

yonatanel15:09:20

It's a bit nuanced though. Sometimes I use consume so I need to check that it works with that too. And if I wanted to take! I couldn't

dm315:09:03

the only thing that won't receive a copy is take! - it will compete with the connected consumers

dm315:09:42

if you want to take!, you could create an intermediate stream and connect it to A, then take! from the intermediate stream

bcbradley15:09:46

i have a question about pipeline in clojure.core.async: if you make a pipeline out of an input channel and an output channel, does the pipeline survive as long as the input channel survives?

bcbradley15:09:16

In other words, if the pipeline falls out of scope but was connected to a channel that is still in scope, will it be garbage collected?

dm315:09:11

@yonatanel you can make use of manifold.bus for things like this, but you'd need to incorporate it into the design explicitly

dm315:09:26

which is how tapping works in core.async - you first need to create a mult - a kind of a bus

lukism17:09:05

(registry/add-module! :ping
                      :regex #"..."
                      :on-message (fn []))
what's the best way to format code like this?

bcbradley18:09:08

does anyone know how pipelines get garbage collected?

josh_tackett18:09:47

Is anyone having trouble with Heroku today?

josh_tackett18:09:55

looks like it is running very slowly

jfntn18:09:03

@bcbradley I’m not clear on the internal details but I’ve used pipelines before and never retained the values returned by the pipeline* invocations, just the input chan

codefinger18:09:08

@josh_tackett deploys or your runtime?

josh_tackett18:09:43

app is running fast locally, but slow on heroku

josh_tackett18:09:03

like 3 second response local

josh_tackett18:09:06

then 30 sec on heroku

josh_tackett18:09:11

and I am on fastest tier of heroku

josh_tackett18:09:58

@codefinger I’ve never had a ticket work haha

codefinger18:09:59

there’s not issues right now. but certainly seems not right https://status.heroku.com

bcbradley18:09:02

@jfntn so if a function takes an input channel as an argument, internally makes a bunch of other channels, and then hooks them all together in a chain with pipelines, but only returns the output channel at the end of the pipeline, the pipelines won't be kept alive as long as the output channel is?

bcbradley18:09:15

i'm just trying to figure out if its normal scope rules

bcbradley18:09:21

or whether something else is going on behind the scenes

codefinger18:09:41

@josh_tackett oh yea? i’m super sorry to hear that. I work at heroku and any of the advanced JVM/Clojure questions usually end up with me

codefinger18:09:48

let me know if there’s anything I can do to help

josh_tackett19:09:40

@codefinger Then you are just the person I need to talk to!

jfntn19:09:50

@bcbradley in my experience the pipeline will work as long as the input channel doesn’t close, not sure what happens if you lose ref of the pipeline’s input channel though

josh_tackett19:09:58

so what are some causes why it may run fast on local machine and slow on server?

codefinger19:09:17

moving to a back channel

bcbradley19:09:26

so if the input channel is kept alive, the pipeline is kept alive?

bcbradley19:09:36

ie. the output channel is irrelevant?

bcbradley19:09:09

is the scope "pipeline owns input and output channels" or is it "input owns pipeline owns output"

jfntn19:09:24

My guess is the former since it takes from input and puts on output, but you’d need to look at the source to make sure

bcbradley19:09:59

that was the first thing i did, but the source is a little too involved for me to parse

bcbradley19:09:17

i don't know what a lot of it means, i'm not that familiar with core async

bcbradley19:09:39

i mean i know what chans and go blocks are

bcbradley19:09:47

and pipelines and mults

bcbradley19:09:03

but the source of pipeline relies heavily on a bunch of private methods

jfntn19:09:34

@bcbradley not really, if you look at the private fn that powers all three pipeline variants, you’ll see that the last two go-loop respectively <! from and >! to so I’m pretty sure the pipeline will keep going as long as from or to don’t close

bcbradley19:09:04

oh ok i see!

bcbradley19:09:43

what happens if you make a pipeline out of two chans and keep the chans alive but let the pipeline fall out of scope?

bcbradley19:09:10

would the chans stop "seeing" eachother?

jfntn19:09:15

That’ll work, the go-loops in the pipeline* are not going anywhere until they stop recuring, and that’d only happen if in or out closes

bcbradley19:09:29

thats exactly what i wanted to know

bcbradley19:09:43

i didn't know whether or not there was something else going on

bcbradley19:09:56

i was tempted to guess that an exception was made for pipelines

bcbradley19:09:14

because the garbage collector won't instantly grab the pipeline when it falls out of scope

bcbradley19:09:26

which theoretically means it might continue to do its work even after its "gone"

jfntn19:09:39

the key intuition here is that go-loops don’t get gc’d until they stop recurring

bcbradley19:09:15

so basically, regardless of scope, pipelines are immortal as long as the chans you gave them are alive

jfntn19:09:26

I’m sure there are edge cases, but this has never really failed me

jfntn19:09:50

@bcbradley would like someone else to confirm but I’d say yes

bcbradley19:09:00

its kind of a strange behavior isn't it?

bcbradley19:09:15

i mean its useful, and probably the right thing to do, but its a bit exceptional

fabrao19:09:28

hello all, I trying to figure out how to get elements from java Enumeration interface, that has only hasMoreElements and nextElement, to clojure collection, is that poassible?

bfabry19:09:13

@bcbradley you should think of a go block as spawning a new thread, in which case it makes sense that it doesn't get gc'd unless that thread completes

bcbradley19:09:42

i do, but the pipeline is opaque

bcbradley19:09:47

it doesn't say anything about go blocks in the docs

bfabry19:09:53

(doesn't actually spawn a new thread, puts that go blocks work into a queue for a thread pool but whatever)

bcbradley19:09:19

i'd prefer if the documentation said something along the lines of "pipelines are immortal as long as both channels are alive"

bcbradley19:09:03

kind of like a lich with phylactories

bfabry19:09:06

I don't think pipelines are really the point here, go blocks are immortal until they return

bfabry19:09:19

*until their body returns

bcbradley19:09:27

*from the grave

fabrao19:09:49

nevermind, I see using while .hasMoreElements and .nextElement

bcbradley19:09:56

joking aside here is what i mean: the "immortalness" of pipelines is not in the interface or documentation. Technically, it is an implementation detail. I don't think it should be, I think this is an important (and potentially confusing!) effect that needs to be documented

bcbradley19:09:24

The fact it uses go blocks underneath is irrelevant to the interface it provides

bcbradley19:09:12

imagine someone picks up clojure core async and looks at a function like "pipeline" and assumes it behaves like any other function. Imagine this person makes a couple of new chans in the parameter list for pipeline, and assumes that the chans given to it will be garbage collected when the pipeline itself has no more references pointing to it-- in other words normal scope rules. And this person would witness the behavior expected. Now imagine instead that the chans were created somewhere else in the program, had plenty of references to them, and are in fact being referred to in the argument list for the pipeline. The person would expect that the chans wouldn't be going anywhere and that the transducer would exist only for as long as it is in scope. What actually happens is the transducer is totally immortal as long as the chans survive. That isn't normal scoping rules.

bfabry19:09:35

oh ok yeah you've got a point, pipeline has a similar caveat to go, it will continue until the source channel is exhausted. personally I think that's kind of obvious (it's spawning asynchronous workers) but I guess it could be in the docs

bcbradley19:09:18

i'm not saying i don't like that behavior

bcbradley19:09:22

its just not normal behavior

bfabry19:09:58

well, it's normal for any asynchronous programming surely? anything that spawns a new thread in any language causes a bunch of new independent references to the objects in that thread

bfabry19:09:29

like, I get the same behaviour in Java if I call Thread.new({...}).run

bfabry19:09:41

or whatever the syntax is, I forget

bcbradley19:09:49

idk, the whole point of a function is to give you hard scoping rules

bcbradley19:09:51

i think whats going on here is the pipeline isn't really a function, its more like a method that belongs to a thread pool, and the arguments you provide to this "method" are copied into the thread pool (through go blocks in the implementation), and that is what is producing this "odor"

bcbradley19:09:31

and that is perfectly ok.

bfabry19:09:48

I've occasionally seen people distinguish between pure functions and impure functions as functions and methods. if that's what you mean then yes pipeline (and most things in an async library) are methods, as in they're impure functions

bcbradley19:09:22

i would prefer to instead have something like (pipeline core.async/pool ...), where pool is a hard reference to "the" pool

bcbradley19:09:47

atleast that way it LOOKS impure

bcbradley19:09:24

or heck, just have (def core.async/pool (pipeline core.async/pool ...))

bcbradley19:09:37

and that way it is actually pure again

bcbradley19:09:46

the statefulness of what is going on becomes apparent too

bcbradley19:09:33

so basically if you want to "update" the pool you'd have to rebind it with another def

bcbradley19:09:29

i'm sure you could probably make this cleaner than constantly using def over and over, and I'm sure you could avoid needless copies by using one of clojure's many persistent data structures to represent the pool

bcbradley19:09:49

but the point is pipeline is kind of a wolf in sheep's clothing

tom19:09:08

For anyone using the JDBC, consider this:

user> (import '[java.sql Timestamp])
java.sql.Timestamp
user> (def time-test (t/now))
#'next.channels/time-test
user> time-test
#object[org.joda.time.DateTime 0x273e8ee3 "2016-09-28T19:41:55.415Z"]
user> (Timestamp. (c/to-long time-test))
#inst "2016-09-28T19:41:55.415000000-00:00" ;; Same as with (c/to-sql-time)
When inserted into a postgresql database using the jdbc/insert! then it seems the database records time with the local time offset for a`TIMESTMAP WITH OUT TIME ZONE` data type. Is there a jdbc setting that can easily be used, or does there need to be a JVM-level change to keep any mutation from occuring?

bfabry19:09:08

I disagree, as an asynchronous library and as a function that makes clear that it executes work in the background I think it's clear that normal scoping rules won't apply. that's just the nature

tom19:09:10

OK looks like setting -Duser.timezone=UTC as a jvm option in leiningen works

seancorfield21:09:49

@tom When dealing with JDBC and timestamps, you need to ensure everything is in the same timezone — and the general recommendation is set your database, your server, and your application to all be in UTC.

seancorfield21:09:14

This comes up repeatedly with JDBC with all databases and with all programming languages unfortunately.

seancorfield21:09:11

At World Singles, we have all our production servers set to run in UTC with NTP time sync. We set our (MySQL) databases to run in UTC (a separate setting from the server level!). And our applications run in UTC (because all our servers are set to UTC). It’s a giant pain in the ass. Especially since our data center is East Coast and every time we get a new server provisioned, the IT folks always deliver it set to Eastern time and we have to get it reset!

seancorfield21:09:06

Then you have to figure out end user timezone and do translation on every input / output.

tom21:09:28

I haven't worked on a project this datetime-intensive and how it relies on dates so pervasively, just keep running into corner cases and odd behaviors. Setting it all to UTC is helping. Datetimes are the least fun thing I've worked on this year.

lukism21:09:03

any way to get rid of clj-http warnings?

lukism21:09:38

I get a lot of WARNING: Cookie rejected which I can't do anything about

seancorfield22:09:45

Short of redirecting stderr, probably not. I believe those come from low-level libraries that clj-http uses. We see them a lot too with certain webservices (and just ignore them).

bcbradley23:09:14

i was wondering, can you make a temporary multimethod in a let?

seancorfield23:09:50

@bcbradley only at the top-level (defmulti et al create top-level vars)

fenak23:09:09

anyone using migrations can give me a tip on how to run them in production?

fenak23:09:55

i have a build process on jenkins that run my tests and generates a docker container with the uberjar for deployment..

fenak23:09:34

i’m not sure in which step should (and how) i run the migrations against the prod or staging db..

fenak23:09:40

i’m using duct