Fork me on GitHub
#clojure
<
2017-02-01
>
spieden00:02:38

my employer is sponsoring spinning out some of our code into OSS libs — first one: https://github.com/uwcpdx/bean-dip

richiardiandrea02:02:32

Looks very good thanks

Pablo Fernandez06:02:49

How do I manually install a library, a clojar, on a computer? I created it on one and I need it on another.

seancorfield06:02:16

@pupeno When you say you "created it on one", did you deploy it somewhere? (clojars, nexus, etc)

seancorfield06:02:17

If you didn't deploy it somewhere (where you can just specify a dependency and have Leiningen or Boot fetch it for you), then you'll need to build it on the new computer the same way you did on the previous one.

Pablo Fernandez07:02:42

@seancorfield: no, it hasn’t been deployed anywhere. Are you claiming it’s impossible to copy a library from one computer to another?

seancorfield07:02:16

No, but I assumed you know how to just copy a file and needed some Clojure-specific advice?

Pablo Fernandez07:02:48

@seancorfield yes, how to install it in the .m2 repo. Technically, it would me maven specific advice.

seancorfield07:02:34

Ah, then look for lein localrepo -- I'm on my phone but that should get you started in Google. That lets you take any JAR and install it onto the maven repo at specific coordinates.

artur07:02:44

Anyone going to the :clojureD conference in Berlin this month?

pesterhazy08:02:17

Artur, I will

pesterhazy08:02:40

Join us in #clojure-berlin !

eslachance08:02:32

Would anyone that's around right now know aleph and manifold and could help troubleshoot an issue? This line never seems to run - the session connects but never receives any message https://github.com/eslachance/dithcord/blob/master/src/dithcord/core.clj#L121 . However, if I do the following in the REPL, no problem, I get the packets I'm expecting:

(def conn @(http/websocket-client "")))
(s/consume (fn [msg] (prn msg)) conn)

danielgrosse08:02:06

I'm actually trying the vase library. I follow the first-api document. How can I reload the service, after I changed the service.edn? Currently I restart the REPL to see the changes.

dominicm10:02:17

I'm looking at writing custom reporters for clojure.test. I'm trying to do something when we begin a testing section, but there doesn't seem to be an event for that. Is there any information on why / open jiras / how to work around it.

lmergen10:02:14

is there something comparable to midje’s tabular for vanilla core.test ?

lmergen10:02:37

that is, to parametrize different inputs / outputs

thegeez10:02:09

@lmergen what does clojure.test/are lack that midje's tabular has?

thegeez10:02:20

(clojure.test/are [f a b] (f a b)
        = 3 3
        not= 3 4)

lmergen10:02:19

i think that might actually do what i want

lmergen10:02:54

why did i not know about this function..

pesterhazy10:02:31

are is weird and wonderful

eslachance11:02:50

I'm thinking... is there any functional difference between using let and def ?

eslachance11:02:56

like at all?

lmergen11:02:04

what do you mean functional difference ?

eslachance11:02:12

I mean any difference in behaviour

eslachance11:02:18

I'm asking this because of my above issue

eslachance11:02:24

works with def, doesn't work in a function with let

lmergen11:02:58

looks like it might be an initialization issue

lmergen11:02:04

the behaviour is different in the moment that the code is ran

karol.adamiec11:02:26

@lmergen let is strictly lexically scoped

dominicm11:02:53

There's no functional difference between:

(def x 10)
(println x)
and
(let [x 10]
  (println x))

eslachance11:02:24

I mean I know I'm new to clojure but I simply can't understand why

(def conn @(http/websocket-client "")))
(s/consume (fn [msg] (prn msg)) conn)
works but
(let [ws @(http/websocket-client "")]
    (s/consume #(on-message %1 session) ws) 
doesn't.

eslachance11:02:44

Something I'm missing somewhere.

dominicm11:02:42

@eslachance does it work if you (s/consume prn ws)?

dominicm11:02:37

I think that the websocket might be being closed as it goes out of scope.

dominicm11:02:33

(def block (promise))
(let [ws @(http/websocket-client "")]
  (s/consume #(on-message %1 session) ws)
  @block)
;; do some testing

;; do this once you're done testing (deliver block nil)

neena11:02:34

@eslachance It also looks like your second snippet is missing a closing parenthesis. Is that a mis-paste?

eslachance11:02:27

oh I have like 6 other lines in that function, didn't want to put in the whole thing

eslachance11:02:00

Now, about the websocket closing, actually it doesn't close right away - it takes some seconds before it times out

eslachance11:02:32

which, btw, I also have (s/on-closed ws #(on-ws-close session)) which sends to a function that prints out "Websocket connection closed". Which does trigger

eslachance12:02:09

never... nevermind

dominicm12:02:12

Does s/consume take it's input arguments in a different order?

eslachance12:02:46

just... look at that URL. just closely look at the URL

dominicm12:02:16

Oh, it should be ws:// 😆 It happens to everyone 🙂

eslachance12:02:23

the arguments at the end

eslachance12:02:38

And I hung... my head... and cried...

dominicm12:02:49

What's wrong with using JSON? 😕

eslachance12:02:03

It's a URL parameter

eslachance12:02:18

I had a typo in my URL and lost the whole night

eslachance12:02:37

I'd be done by now. WELL then. Throw that in the "Massive Derp" folder, and move on!

dominicm12:02:40

ah, I was reading it as /v the url, heh.

dominicm12:02:10

I mean, if the library doesn't get a websocket from the URL, I'm surprised it doesn't throw an exception.

dominicm12:02:46

@eslachance is there an on-error or something?

eslachance12:02:03

not as far as I know

eslachance12:02:32

which was actually surprising. I'll have to react on that but for now... that's a side-trip I'm not going to take 😉

dominicm12:02:27

is that what you needed maybe?

dominicm12:02:35

You'll want to take time & figure it out. It'll save you pain.

eslachance12:02:43

hmm. will have to look at that

eslachance12:02:51

Would anyone know how to decompress a zlib packet, #object["[B" 0x7282913c "[B@7282913c"] is how it shows up in the console?

bronsa12:02:09

that's a byte array

bronsa12:02:39

you can invoke seq on it to look at the individual bytes

tankthinks12:02:42

speaking of byte arrays ... is there a fn to concatenate two arrays?

tankthinks12:02:37

nm ... stupid question ... concat of course works ... i had different bug

tankthinks13:02:25

actually ... maybe not as stupid ... is this the best way to do it? (byte-array (map byte (concat b1 b2)))

rauh13:02:27

@tankthinks You could do (into-array Byte/TYPE (concat a b))

tankthinks13:02:16

thanks @rauh ... i was worried about

;; if you assign a type, you still have to coerce values
user=> (into-array Byte/TYPE (range 4))
;; Evaluation aborted.

user=> (into-array Byte/TYPE (map byte (range 4)))
#<byte[] [B@68ffefc9>
but it should be fine since I know the input is bytes

joshjones13:02:14

byte-array seems to be marginally faster and is more succinct

joshjones13:02:02

@tankthinks .. with both into-array and byte-array, I get no warnings on coercion , seems to work:

(->> (range 65 69)
     byte-array
     String.)

joshjones13:02:14

=> “ABCD”

eslachance13:02:18

Does a readable byte array (that I can convert to string) look like (123 34 116 34 58 34 71 85 73 76 68 95 ...) (`...` meaning a bunch of other numbers) and if so, how DO I convert it to string?

eslachance13:02:33

I've never done byte arrays before I'm thoroughly confused

eslachance13:02:08

if I do (byte-array blah) on that list of number I get back that #object["[B" 0x5985d02e "[B@5985d02e"] and that's not helpful 😞

joshjones13:02:43

(String. (byte-array [65 66 67 68])) is one way — java’s String constructor can take a java byte array

luxbock13:02:01

would anyone happen to know if *assert* is by default set to true or false when generating an uberjar via lein?

eslachance13:02:31

yeah ok I'm just going to put this all aside and do something else, my head is spinning with numbers and strings of weird characters "xœ?•MoÛ0\f†ÿJ¡ëÒ@Ÿ¶ìÓÖ®Øì4d§¡0…NÔÊVÉ-Š\"ÿ}´óÑ´k¶µ'[Iñ!¥W?$` ugh

eslachance13:02:27

This is what I have, just... attempting to figure out how to deal with this

(defn on-message [msg session]
  (do
    (if (= (type msg) "java.lang.String")
      (prn (json/parse-string msg true))
      (try (do (prn (bs/to-string msg))
               (prn (String. msg))
               (prn (String. (z/inflate msg))))
           (catch Exception e (str "caught exception: " (.getMessage e)))
          )
    ))) 

joshjones13:02:39

luckyevie: no need for the first do, it’s implied, and also no need for the do under try … also, prefer (instance? String msg) to your type check for string

eslachance13:02:16

I always get the weirdest results when I google for stuff about clojure. instance? was not part of the results for my search for "get type of variable in clojure". Gotta learn the right terminology!

eslachance13:02:05

Ok this, very simplified, gives me

(defn on-message [msg session]
  (if
    (instance? String msg)
    (prn (json/parse-string msg true))
    (prn (String. (byte-array (z/inflate msg))))
    )) 

joshjones13:02:15

using type is fine, but not the way you did: (= (type “blah”) String)

eslachance14:02:12

now, I don't want to print them, I want to do something with msg, after it's been unpacket if it is. I'm guessing I need a "ternary assignment" or something, as we say in javascript?

joshjones14:02:13

no, if you remove the try then you need the do, otherwise the second prn will always be executed

eslachance14:02:48

isn't that the else of the condition though

joshjones14:02:21

oh sorry, read yours wrong

joshjones14:02:42

i was thrown off by the condition being on the line after the if .. my mistake. put your if condition on the same line as if — in clojure it’s how it’s written, but what you had is correct.

eslachance14:02:57

Alright I think I got this

(defn on-message [msg session]
  (let [msg (if
              (instance? String msg)
              (json/parse-string msg true)
              (String. (byte-array (z/inflate msg))))]
    (prn msg))
  ) 

eslachance14:02:36

This works great, I got my 5 different packets I was expecting, both the compressed and decompressed ones.

joshjones14:02:15

yes that is the general idea — (just put your (instance? …) one line higher ;)

eslachance14:02:47

Yes! Absolutely! Whew.

joshjones14:02:35

generally, you will simply return msg and choose to print it in the caller. that way the function is pure and has no side effects

eslachance14:02:14

This was, of course, just debugging without being bogged down by the rest of the code. My actual function is this:

(defn on-message [msg session]
  (let [m (if (instance? String msg)
            (json/parse-string msg true)
            (String. (byte-array (z/inflate msg))))
        debug-handler (get-in @session [:handlers :debug])]
    (spit "event.log" (str m "\r\n") :append true)
    (if-not (nil? debug-handler)
      (debug-handler session m))
    (doall (map #(apply % [session m]) (:internal-handlers @session)))
    )) 

eslachance14:02:18

now time to test!

joshjones14:02:42

👌:skin-tone-2:

joshjones13:02:49

ASCII letters A-Z are 65-90, etc .. you’ll want to use ascii chars most likely http://www.asciitable.com/

pesterhazy13:02:53

@luxbock I think it always defaults to true

eslachance13:02:47

First one prints out

"x???Mo?0\f??J???@???????4d??0?N??V`? 
the other prints out
"xœ?•MoÛ0\f†ÿJ¡ëÒ@Ÿ¶ìÓÖ®Øì4d§¡0…NÔÊV 

eslachance13:02:51

neither is recognizable to me

eslachance13:02:32

sorry actually (prn (z/inflate msg) is what prints out (123 34 116 34 58 34 71 85 73 76 68 95 ...)

eslachance13:02:54

I can't seriously have to use an ascii table to process these - I know clojure enough to know there's a magic way to do this.

eslachance13:02:16

this is made difficult by the fact that I have no idea how to "save" any of those in memory, they're triggered by a websocket event

joshjones13:02:24

(String. (byte-array ....)) converts those ascii numbers to a string

eslachance13:02:49

isn't that already a byte array though?

joshjones13:02:20

no, it’s a list of numbers

joshjones13:02:47

(instance? Byte 42) is false

eslachance13:02:53

wow this work. (prn (String. (byte-array (z/inflate msg))))

eslachance13:02:40

One day I will understand these things. Thank you very much for your patience and help, Josh.

joshjones13:02:12

no problem mate, we are all learning new things every day :thumbsup::skin-tone-2:

eslachance13:02:37

The roller coaster ride isn't over but I'm glad to be on it 😄

schmee14:02:11

does anyone have any resources on when to use core.async vs when to use java.util.concurrent stuff?

pesterhazy15:02:06

@schmee, for what use case?

schmee15:02:39

in general I guess, but the case I have in mind is high performance analytics systems

schmee15:02:19

I’m trying to understand the pros and cons of performance, expressivity and so on

schmee15:02:33

like, is there ever any reason to use j.u.c over core.async?

sgerguri15:02:24

CSP implementation in an external dependency that has not reached v1.0 vs battle-tested low-level concurrency management through the JVM. Pick your poison.

pesterhazy15:02:07

well if you get get away with futures/pmap, use those

pesterhazy15:02:35

if you have more complicated needs, it really depends

pesterhazy15:02:25

personally I tend to think that the complexity cost of core.async is not worth it in many cases, but I'm sure there are situations where core.async simplifies things

sgerguri15:02:18

^ This. Clojure gives you first-class tools for managing (or rather, not having to manage) shared memory access, which is typically the biggest pain point of using shared memory/threads.

schmee15:02:05

if I read this correctly: start with j.u.c and if things get too complicated move to core.async?

mpenet15:02:26

depends, both are useful and they are very different. hard to know with the (little) details you give about potential usage.

mpenet15:02:19

I wouldn't advise to use pmap tho

schmee15:02:24

I’m talking about building a highly concurrent system processing 10000s of events / sec

schmee15:02:13

can core.async handle those kinds of workloads?

mpenet15:02:16

still kinda vague. but riemann for instance does this kind of things without c.async

mpenet15:02:35

yes very likely, but again, it depends how/what you use it for

mpenet15:02:53

and your constraints

schmee15:02:21

yeah, I’m just trying to get some intuition for where the boundaries of core.asyncs applicability are

schmee15:02:31

compared to using the low-level stuff

mpenet15:02:35

it's quite low level, just like juc

schmee15:02:41

I’ll check out Riemann for some inspiration

joshjones15:02:35

when you say “concurrent” — what specifically is concurrent? simultaneous processing of many requests? or each request needs synchronization in the computation of the result?

schmee15:02:45

joshjones primarily the first one

joshjones15:02:01

is the nature of your work primarily cpu-bound, or i/o-bound (you’ll be doing network requests yourself in order to complete the request, reading files, etc.)?

schmee15:02:16

I’ll be reading events from Kafka, but I’m not really sure where I will end up on that spectrum

schmee15:02:40

as you might have noticed, I have a very vague idea of what I’m trying to do 😄

pesterhazy15:02:02

my advice is, start with the simplest possible version, don't overthink it, only parallelize when you absolutely need to

joshjones15:02:02

ok — well, your needs will take you in the direction best for you, but if you’re just needing to service many requests, your server (http?) will handle that part of the concurrency equation. whether or not you need to actually make concurrent computations, or whether to use async methods or not, is another matter

pesterhazy15:02:45

don't fall in the trap of "everything needs to be async because I need performance but now my architecture is 10x more complicated"

pesterhazy15:02:53

a.k.a. nodejs

schmee15:02:42

good stuff, thanks everyone for your input!

conormcd15:02:47

Also, if you're working with Kafka via something like gregor then there's a bunch of async-ish stuff going on already. You don't want layers of it.

conormcd15:02:53

Debugging is a nightmare if you do that.

donaldball16:02:56

Also make sure you understand and choose the correct offset commit behavior or you will wind up dropping events on the floor at some point

ghadi16:02:24

best to build for idempotent processing

ghadi16:02:48

and use the default offset commit behavior (every 60s)

nickbauman16:02:35

@schmee I would not reach for a “concurrency framework” as a first step. The thing is if you program it in Clojure your need to managed shared state should be reduced considerably because of the emphasis on immutability in the first place. A large, complex and highly concurrent system often boils down to having only one or two atoms in Clojure and that’s all you need.

nickbauman16:02:23

(or a single agent, or a ref or two or whatnot)

mccraigmccraig16:02:59

@pesterhazy all-async doesn't need a complicated architecture - and it ends up being simpler when you need to control resource usage with things like backpressure. i would say that getting started with all-async is conceptually harder in clojure though

tbaldridge16:02:26

Async code is always harder than non-async code. You're adding another dimension of complexity.

mccraigmccraig16:02:28

@tbaldridge true, for single threaded tasks it's harder, but not much harder, depending on how you approach it - perhaps no more so than lazy vs strict evaluation - and once you get around to dealing with concurrent resource consumption then it's simpler

jeff.terrell16:02:18

Seems like an important consideration for whether to use async is whether backpressure is needed. (Like, do you want to block the client/producer of work from adding more work when your pipeline is full?) If so, async might be helpful. If not, maybe not.

nickbauman17:02:18

@jeff.terrell isn’t the backpressure question all about being careful to not expose an unbounded resource? Or eliminate unbounded resources? You can achieve this with or without async, no?

nickbauman17:02:07

(question may be a little too open-ended, sorry)

jeff.terrell17:02:16

Yes, you can do backpressure with a thread-safe blocking queue (like the ones in java.util.concurrent). I was just thinking that, if that's a concern, async might be better since the whole thing is built on such queues and you can easily propagate backpressure through multiple steps of analysis. But you're right, it's not a black-and-white thing. 🙂

nickbauman17:02:02

Sure. Those queues are bounded by design, hence the assertion. For me I’d stay away from Java-only-based solutions because they have a much different design imprimatur. Every time I suck in a Java library, I always find the ugliest, less maintainable code tends to be around Java interop to support it.

jeff.terrell17:02:37

Agreed. Though last I checked, queues were just about the only basic type of data structure that Clojure didn't have a built-in version of. I think I saw in a book somewhere (Joy of Clojure maybe) that the authors used a Java queue since a Clojure one wasn't available. (But it's been a while since I checked on all that, so I may be out of date here.)

nickbauman17:02:30

Me too. Does seem odd. I don’t even know the underlying impl in core.async

nickbauman17:02:22

Ha! so it’s just Java (j.u.c) at the bottom.

nickbauman17:02:49

(Hmm. I haven’t done the warn-on-reflection thing and added the type hints to my code in a long time. I guess I don’t want to give up the concision and dynamism…)

bbktsk17:02:29

I have a quite elementary questions, but it keeps bugging me and I was not able to find an answer: why is seq? false for vectors? Or, asked in another way, why vectors do not implement ISeq? What is it that I can do with a ISeq that I can not do with a vector?

hiredman18:02:04

a seq is like an iterator

hiredman18:02:42

you can use an iterator(a seq) to traverse the values in a vector, but a vector is not an iterator or a seq

hiredman18:02:55

same with other types of collectors

hiredman18:02:59

collections

hiredman18:02:59

it is kind of confusing because lots of functions that work with seqs will automatically get the seq on a collection that is not seq when it is passed in

jeff.terrell18:02:52

Good question @bbktsk. If that were on SO I'd 🔼 it.

bbktsk18:02:34

@hiredman Thank you. Is there something ISeq can do that vector can not, ie. is there some function that works with ISeq but not with vector? I have not checked all the functions in the ISeq hierarchy, but so far I found none. Well, there’s ISeq more(); defined in ISeq.java as part of the interface, but that does not seem to be visible from clojure.

hiredman18:02:49

seqs and vectors are constructed differently

nickbauman18:02:14

subvec :face_with_rolling_eyes:

hiredman18:02:18

the interface seqs support is much smaller

hiredman18:02:13

seqs are not counted, and can be constructed lazily, vectors support fast indexed access

hiredman18:02:59

seqs are a generic functional iteration protocol, vectors are ordered indexed collections

hiredman18:02:56

you could likely fiddle with vectors to get them to implement ISeq in such a way that would work (although cons would turn the vector in to something else) but to what end? What would it simplify?

hiredman18:02:11

and simple here has performance implications for the class hierarchy and object allocation

nickbauman18:02:22

☝️ very well said, especially after my puerile joke.

neverfox19:02:59

I’m was getting unacceptably slow times from (sort-by sort-fn coll), so just to verify that my sort-fn wasn’t the problem, I timed (sort (mapv sort-fn coll)) and found it to be 20x faster. Obviously, the latter form doesn’t give me what I need, but suggests the issue isn’t the sort or sort-fn. Why is sort-by so much slower?

hiredman19:02:52

what version of clojure? I know there were some jira issues about sort-by calling the sort-fn too many times

neverfox19:02:59

But that’s very interesting to hear

hiredman19:02:50

still open I guess

neverfox19:02:05

but thanks that’s very helpful

neverfox19:02:02

I’ll try implementing the patch

neverfox19:02:17

@hiredman That did the trick

Alex Miller (Clojure team)19:02:58

@bbktsk sequence is a logical list abstraction. vectors (and sets and other things) do not directly implement this abstraction but can provide an implementation when asked (via seq). Thus these collections are seqable, not seqs themselves. Lists are an exception - they implement the sequence abstraction directly so seq? on a list returns true. Also see more detail at: http://insideclojure.org/2015/01/02/sequences/

Alex Miller (Clojure team)19:02:20

@neverfox dropping a real world use case and timings on that ticket would be great

Alex Miller (Clojure team)19:02:32

that stuff counts for a lot with Rich

Alex Miller (Clojure team)19:02:46

as I last commented on that ticket, it’s lacking enough information to be properly screened. if someone did that (demonstrated the performance improvements), I could move it along.

Alex Miller (Clojure team)19:02:02

there’s a language (and other technology) poll at https://jaxenter.com/jaxenter-survey-preliminary-results-programming-131480.html if anyone’s interested in demonstrating your love for Clojure, btw :)

mobileink19:02:32

if i aot everything, say for a servlet app, do i still need the clojure jar? if so, is there any way to eliminate code that is not needed?

Alex Miller (Clojure team)19:02:14

yes, you still need it for the Java classes that make up the runtime implementation (and potentially for other things as well)

Alex Miller (Clojure team)19:02:29

there is nothing provided in Clojure to slim that down

Alex Miller (Clojure team)19:02:49

maybe back up - what’s your goal?

mobileink19:02:16

just wondering, at the moment. well, fast startup - the context here is boot-gae, just occurred to me it might be a good idea to make gae/ build --prod aot everything, since (i speculate) there usually will bot be much need for ad hoc dynamic stuff in the prod cloud env.

mobileink19:02:54

currently only the servlet stubs get the aot treatment.

danielcompton19:02:41

Speaking of polls, is there a date for the state of clojure survey coming out?

mobileink19:02:03

i know faster startup time is a perennial topic, just not sure what the state of the art is. any refs (blogs etc.) stand out?

mobileink19:02:01

i wonder if anybody has performance numbers on minimal v. maximal aot.

Alex Miller (Clojure team)19:02:37

@danielcompton soon soon … there have been some external forces that have delayed the release (that is, people being busy with non-work stuff)

bfabry19:02:55

I benchmarked this for us the other day, it came out as not worth it. try this: jar your app with no aot, launch clojure.main on that jar, time requiring a namespace that requires every other namespace. now jar your app with aot all, and time doing the same thing

Alex Miller (Clojure team)19:02:21

it really depends a lot on what you’re doing

bfabry19:02:45

yeah I mean you could have some crazy macroexpansion in there that takes 10 minutes, at that point probably worth it

bfabry20:02:52

oh yeah I always forget core.async is a lot of compile macroexpand time magic, its api just integrates so well

Alex Miller (Clojure team)20:02:03

I have a page with some past research at http://dev.clojure.org/display/design/Improving+Clojure+Start+Time but that data is likely worthless in relation to your particular problem

mobileink20:02:43

heh, no core.async in google app engine. 😢

Alex Miller (Clojure team)20:02:10

I guess that makes sense :)

mobileink20:02:56

i guess i'll give it a try and see what happens.

mobileink20:02:28

there was some talk about adapting core.async to gae's thread stuff a while back, anything ever come of that? sounds like a fun project.

mobileink20:02:43

@alexmiller nice article, thanks. irrespective of the data it's a nice survey of the issues involved.

Alex Miller (Clojure team)20:02:19

@mobileink I’m not familiar with anyone doing the core.async thing for gae

mobileink20:02:50

up to me, i guess, ha.

mike_ananev20:02:15

how to get source code of my own fn like data? (source _) works for clojure.core fns but not for my ns

hiredman20:02:47

source doesn't work for fns in fact

hiredman20:02:04

it works on vars, sometimes

hiredman20:02:02

for example, when you pass a function object to source it throws an error`(source (fn []))`

hiredman20:02:43

just never use it, then you won't care if it only works sometimes

hiredman21:02:48

the way source works is it takes a symbol, resolves it to a var, then looks at the var metadata to find file and line information, then opens the file, seeks to the line, then reads the next clojure form

hiredman21:02:06

if any of those steps fail, which they can in all kinds of ways, then source won't work

mike_ananev21:02:16

i want to transfer some short source code via network as string

hiredman21:02:38

source is not a way to do that

hiredman21:02:51

have your code as a string and send it

mike_ananev21:02:03

so, it means that i should open the src file and lookup for given name

hiredman21:02:31

that sounds terrible, why are you doing that?

mike_ananev21:02:59

@hiredman i do global registry of my corporate data based on clojure spec

mike_ananev21:02:22

i want that it would be like webservice

hiredman21:02:38

there is a spec registry to

mike_ananev21:02:55

yes s/form is correct but sometime in specs i use helper functions

hiredman21:02:59

none of that requires monkeying with source or sending code around

hiredman21:02:09

you don't want to do that

mike_ananev21:02:13

and i want to pass them via network

hiredman21:02:47

leave the helper functions in spec forms has symbols, and allow consumers of your specs to interpret them as they wish

mike_ananev21:02:27

@hiredman may be you're right.

tbaldridge21:02:40

Sending functions over the internet is all fun and games until you send (fn [] my-data) where my-data is a 3TB database

grav21:02:34

Anyone with experience with clojure.data.xml?

mike_ananev21:02:07

the idea is to allow remote etl tools request data registry webservice to resolve spec

tbaldridge21:02:00

@grav, some yes, what's your issue?

grav21:02:15

@tbaldridge: Is it possible to ignore newlines when using parse-str?

grav21:02:34

An example:

(xml/parse-str "<foo>\n<bar>42</bar></foo>")
=> #xml/element{:tag :foo, :content ["\n" #xml/element{:tag :bar, :content ["42"]}]}

grav21:02:52

In the above, I’d like to avoid the \n from the :content vector, since it’s between tags

tbaldridge21:02:53

sadly no, I don't know the answer to that

grav21:02:37

okay, no problem 🙂

jr21:02:39

@grav: the beta version of 0.1 seems to have fixed that issue

jr21:02:52

(xml/parse-str "<foo>\n<bar>42</bar></foo>")
<#C03S1KBA2|clojure>.data.xml.node.Element{:tag :foo, :attrs {}, :content (<#C03S1KBA2|clojure>.data.xml.node.Element{:tag :bar, :attrs {}, :content ("42")})}

jr21:02:16

[org.clojure/data.xml "0.1.0-beta3"]

jr21:02:39

if you don't mind using beta software

juhoteperi21:02:48

@piotr2b Instead of requiring your namespaces in ns user, load them on runtime when functions in user need them, them when you run the function in runtime, classes are probably already compiled by Lein:

(defn fixtures! []
  (require 'backend.users)
  ((ns-resolve 'backend.users 'fixture) (ctx)))

piotr-yuxuan21:02:39

@juhoteperi thx for your answer. I’m just wondering — is it a good pattern to have unpure functions?

seylerius22:02:11

Where's that site with the categorized collection of third-party clojure libs?

grav22:02:59

@jr I’m actually using v0.2.0-alpha, which if I remember correctly is the first to handle namespaces

grav22:02:12

Might be a regression

grav22:02:14

But thanks for the hint, might be worth looking at the earlier version

jr22:02:42

(xml/parse-str "<foo>\n<bar>42</bar></foo>" :skip-whitespace true)
#xml/element{:tag :foo, :content [#xml/element{:tag :bar, :content ["42"]}]}

jr22:02:51

undocumented option

seylerius22:02:13

Yes, @geoffs, that's exactly it!

seylerius22:02:33

Actually, I need to submit one I just read about last night to the toolbox, while I'm here.

seylerius22:02:15

Oh, there's already a PR for it.

jr22:02:28

@grav did you try :skip-whitespace true ?

grav22:02:32

@jr Did not! Works like a charm - thanks! Google didn’t help me - did you search through the code on Github, or how did you find it?

jr22:02:55

yeah took a peep at the parser

jr22:02:55

I posted a link to the code above

grav22:02:34

Great! It seems skipping white space per default isn’t allowed as per the spec (http://stackoverflow.com/a/2084909/202538). So I searched high and low for an option, but couldn’t find it.

seylerius22:02:15

Can most clojure libraries (the ones that don't depend on a java lib or a non-shared feature) be used in clojurescript apps?

seylerius22:02:37

Also, is there a tool to see the dependency tree of a clojure lib?

seylerius22:02:10

(Basically trace down the direct and indirect deps until you hit bottom?)

hiredman22:02:13

I would say most clojure libraries do depend on java

seylerius22:02:42

Oh, I just noticed that toolbox lists whether different libs work with clojure or clojurescript.

hiredman22:02:02

most clojure projects specify their deps as maven artifacts, and most libraries live in maven respositories, so anything that can consume and spit out information from maven repos

hiredman22:02:05

if you have a project that uses a particular build tool, most build tools have a build tool specific command to spit that out

eslachance22:02:40

Would anyone happen to have a link to a project that uses aleph for http? I've said this before and it's still true: I can't read "API-speak" or "devspeak" at all, so I'm looking for examples. Specifically, something doing http/post (with aleph)

shayanjm22:02:43

I'm trying to optimize a pretty basic core.async implementation. I'm seeing some pretty rough bottlenecking happening and I have a feeling it's because of my implementation

shayanjm22:02:53

essentially, I have a channel that data is being thrown into via >!!, and pulled out of via <!!. I've parallelized a bit through having multiple futures

shayanjm22:02:01

but at the end of the day they're all throwing data into and pulling data out of the same channel (many producers, many consumers)

shayanjm22:02:10

I have a feeling that I'm probably not implementing it correctly (i.e: probably should be using go + >! instead of >!!), but would love some pointers on common design patterns

shayanjm22:02:18

with core.async to maximize multi-producer multi-consumer on a single channel

spieden22:02:54

@piotr2b i ended up creating a separate ns dev-repl instead of user for various reasons. then in my :dev lein profile i make it the :main

spieden22:02:20

seems like user is loaded early by clojure which can cause issues

hiredman22:02:06

@shayanjm I would avoid consumers and producers feeding in to the same channel if possible

hiredman22:02:52

you'll want to have the buffer of the channel to be at least the same size as the number of producers/consumers

bfabry22:02:17

yeah I think one channel per producer is probably going to work better

shayanjm22:02:30

What about single producer, multiple consumer?

bfabry22:02:33

then consumers use alts!! rather than <!!

shayanjm22:02:34

what's the best design pattern for that?

hiredman22:02:48

you may want to look at pipeline-blocking

hiredman22:02:45

but pipelines provide things above and beyond parallel execution, the most notable being an in order stream of results out the other end

shayanjm22:02:19

@hiredman: ultimately I'm trying to optimize for throughput. Basically the "consumers" will be grabbing from the pipeline/channel and throwing the message into a database

shayanjm22:02:24

the faster that happens, the better

hiredman22:02:08

a single channel to consume and a single channel to produce to I think would be fine, it is just a single process both consuming and publishing to the same channel can result in deadlocks

hiredman22:02:17

yeah, for that kind of thing pipeline isn't the best, because pipeline expects actions to have results that you care about consuming

hiredman22:02:54

I would consider using an executor directly

spieden22:02:08

sometimes success/failure is a decent enough thing to care about

spieden22:02:15

for retries, etc.

shayanjm22:02:37

Basically, messages will be thrown into this channel as quickly as they come (consumed from Kafka), and then consumers will be pulling from the channel to throw directly into a db

shayanjm22:02:23

that's basically the extent of this implementation

shayanjm22:02:26

but throughput is super important

shayanjm22:02:30

messages might be coming in very quickly

hiredman22:02:31

yeah, the problem I tend to have with executors on their own is they don't have a great feedback story, by default they have unbounded queues, which could be a problem there

spieden22:02:43

yeah channel might not be buying much — maybe just dispatch to core.async/thread

spieden23:02:00

ah hmm, yeah backpressure

hiredman23:02:46

what you might do is run both the fetching from kafka and the db writing in the same threadpool, so they will naturally throttle each other

shayanjm23:02:13

That makes sense

seylerius23:02:20

I've decided that the planck REPL looks like the best way to get clojure-tastic lispiness for my shell scripting.

hiredman23:02:10

(let [exec (Executors/newFixedThreadPool 2)]
  (.execute exec (fn fetch-from-kafka []
                   (let [msg (pull-from-kafka)]
                     (.execute exec (fn [] (do-whatever msg)))
                     (.execute exec fetch-from-kafka)))))

hiredman23:02:12

or something

spieden23:02:07

take and put right next to eachother always helps keep things straight.. nice way to add the backpressure to that ^

shayanjm23:02:49

@hiredman What's the difference between executor threadpooling and just managing a vec of futures or something?

hiredman23:02:18

futures are built on top of the executors framework

hiredman23:02:41

futures don't themselves have a workqueue, so you have to write that

hiredman23:02:25

the default threadpool for futures is unbounded