This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-01
Channels
- # aleph (71)
- # aws (1)
- # bangalore-clj (4)
- # beginners (36)
- # boot (153)
- # cider (23)
- # clara (9)
- # cljs-dev (67)
- # cljsjs (2)
- # cljsrn (22)
- # clojure (348)
- # clojure-argentina (4)
- # clojure-austin (12)
- # clojure-berlin (9)
- # clojure-dusseldorf (6)
- # clojure-france (4)
- # clojure-italy (4)
- # clojure-russia (358)
- # clojure-spain (2)
- # clojure-spec (28)
- # clojure-uk (109)
- # clojurescript (130)
- # core-typed (1)
- # cursive (35)
- # datascript (6)
- # datomic (18)
- # emacs (12)
- # hoplon (4)
- # klipse (64)
- # lein-figwheel (13)
- # leiningen (3)
- # luminus (4)
- # lumo (51)
- # mount (22)
- # off-topic (83)
- # om (22)
- # om-next (8)
- # onyx (3)
- # pedestal (8)
- # perun (6)
- # portland-or (2)
- # re-frame (50)
- # ring (8)
- # ring-swagger (5)
- # untangled (10)
- # yada (9)
my employer is sponsoring spinning out some of our code into OSS libs — first one: https://github.com/uwcpdx/bean-dip
Looks very good thanks
How do I manually install a library, a clojar, on a computer? I created it on one and I need it on another.
@pupeno When you say you "created it on one", did you deploy it somewhere? (clojars, nexus, etc)
If you didn't deploy it somewhere (where you can just specify a dependency and have Leiningen or Boot fetch it for you), then you'll need to build it on the new computer the same way you did on the previous one.
@seancorfield: no, it hasn’t been deployed anywhere. Are you claiming it’s impossible to copy a library from one computer to another?
No, but I assumed you know how to just copy a file and needed some Clojure-specific advice?
@seancorfield yes, how to install it in the .m2 repo. Technically, it would me maven specific advice.
Ah, then look for lein localrepo
-- I'm on my phone but that should get you started in Google. That lets you take any JAR and install it onto the maven repo at specific coordinates.
Thanks.
Artur, I will
Join us in #clojure-berlin !
Would anyone that's around right now know aleph
and manifold
and could help troubleshoot an issue? This line never seems to run - the session connects but never receives any message https://github.com/eslachance/dithcord/blob/master/src/dithcord/core.clj#L121 . However, if I do the following in the REPL, no problem, I get the packets I'm expecting:
(def conn @(http/websocket-client "")))
(s/consume (fn [msg] (prn msg)) conn)
I'm actually trying the vase library. I follow the first-api document. How can I reload the service, after I changed the service.edn? Currently I restart the REPL to see the changes.
I'm looking at writing custom reporters for clojure.test. I'm trying to do something when we begin a testing
section, but there doesn't seem to be an event for that. Is there any information on why / open jiras / how to work around it.
i’m seeing a discussion from 2009 here what i want, kind of http://stackoverflow.com/questions/1867760/how-can-you-parameterize-clojure-contribs-test-is
are
is weird and wonderful
I'm thinking... is there any functional difference between using let
and def
?
like at all?
I mean any difference in behaviour
I'm asking this because of my above issue
works with def, doesn't work in a function with let
@lmergen let is strictly lexically scoped
There's no functional difference between:
(def x 10)
(println x)
and
(let [x 10]
(println x))
I mean I know I'm new to clojure but I simply can't understand why
(def conn @(http/websocket-client "")))
(s/consume (fn [msg] (prn msg)) conn)
works but (let [ws @(http/websocket-client "")]
(s/consume #(on-message %1 session) ws)
doesn't.Something I'm missing somewhere.
@eslachance does it work if you (s/consume prn ws)
?
(def block (promise))
(let [ws @(http/websocket-client "")]
(s/consume #(on-message %1 session) ws)
@block)
;; do some testing
;; do this once you're done testing (deliver block nil)
@eslachance It also looks like your second snippet is missing a closing parenthesis. Is that a mis-paste?
oh I have like 6 other lines in that function, didn't want to put in the whole thing
Now, about the websocket closing, actually it doesn't close right away - it takes some seconds before it times out
which, btw, I also have (s/on-closed ws #(on-ws-close session))
which sends to a function that prints out "Websocket connection closed". Which does trigger
oh god
never... nevermind
I'm ashamed
just... look at that URL. just closely look at the URL
the arguments at the end
And I hung... my head... and cried...
It's a URL parameter
I had a typo in my URL and lost the whole night
I'd be done by now. WELL then. Throw that in the "Massive Derp" folder, and move on!
I mean, if the library doesn't get a websocket from the URL, I'm surprised it doesn't throw an exception.
@eslachance is there an on-error or something?
not as far as I know
which was actually surprising. I'll have to react on that but for now... that's a side-trip I'm not going to take 😉
hmm. will have to look at that
Would anyone know how to decompress a zlib packet, #object["[B" 0x7282913c "[B@7282913c"]
is how it shows up in the console?
speaking of byte arrays ... is there a fn to concatenate two arrays?
nm ... stupid question ... concat
of course works ... i had different bug
actually ... maybe not as stupid ... is this the best way to do it? (byte-array (map byte (concat b1 b2)))
@tankthinks You could do (into-array Byte/TYPE (concat a b))
thanks @rauh ... i was worried about
;; if you assign a type, you still have to coerce values
user=> (into-array Byte/TYPE (range 4))
;; Evaluation aborted.
user=> (into-array Byte/TYPE (map byte (range 4)))
#<byte[] [B@68ffefc9>
but it should be fine since I know the input is bytes@tankthinks .. with both into-array
and byte-array
, I get no warnings on coercion , seems to work:
(->> (range 65 69)
byte-array
String.)
Does a readable byte array (that I can convert to string) look like (123 34 116 34 58 34 71 85 73 76 68 95 ...)
(`...` meaning a bunch of other numbers) and if so, how DO I convert it to string?
I've never done byte arrays before I'm thoroughly confused
if I do (byte-array blah)
on that list of number I get back that #object["[B" 0x5985d02e "[B@5985d02e"]
and that's not helpful 😞
(String. (byte-array [65 66 67 68]))
is one way — java’s String
constructor can take a java byte
array
would anyone happen to know if *assert*
is by default set to true or false when generating an uberjar via lein?
yeah ok I'm just going to put this all aside and do something else, my head is spinning with numbers and strings of weird characters "xœ?•MoÛ0\f†ÿJ¡ëÒ@Ÿ¶ìÓÖ®Øì4d§¡0…NÔÊV
É-Š\"ÿ}´óÑ´k¶µ'[Iñ!¥W?$` ugh
This is what I have, just... attempting to figure out how to deal with this
(defn on-message [msg session]
(do
(if (= (type msg) "java.lang.String")
(prn (json/parse-string msg true))
(try (do (prn (bs/to-string msg))
(prn (String. msg))
(prn (String. (z/inflate msg))))
(catch Exception e (str "caught exception: " (.getMessage e)))
)
)))
luckyevie: no need for the first do
, it’s implied, and also no need for the do
under try
… also, prefer (instance? String msg)
to your type check for string
I always get the weirdest results when I google for stuff about clojure. instance?
was not part of the results for my search for "get type of variable in clojure". Gotta learn the right terminology!
Ok this, very simplified, gives me
(defn on-message [msg session]
(if
(instance? String msg)
(prn (json/parse-string msg true))
(prn (String. (byte-array (z/inflate msg))))
))
now, I don't want to print them, I want to do something with msg
, after it's been unpacket if it is. I'm guessing I need a "ternary assignment" or something, as we say in javascript?
no, if you remove the try
then you need the do
, otherwise the second prn
will always be executed
isn't that the else
of the condition though
i was thrown off by the condition being on the line after the if
.. my mistake. put your if
condition on the same line as if
— in clojure it’s how it’s written, but what you had is correct.
Alright I think I got this
(defn on-message [msg session]
(let [msg (if
(instance? String msg)
(json/parse-string msg true)
(String. (byte-array (z/inflate msg))))]
(prn msg))
)
This works great, I got my 5 different packets I was expecting, both the compressed and decompressed ones.
Yes! Absolutely! Whew.
generally, you will simply return msg
and choose to print it in the caller. that way the function is pure and has no side effects
This was, of course, just debugging without being bogged down by the rest of the code. My actual function is this:
(defn on-message [msg session]
(let [m (if (instance? String msg)
(json/parse-string msg true)
(String. (byte-array (z/inflate msg))))
debug-handler (get-in @session [:handlers :debug])]
(spit "event.log" (str m "\r\n") :append true)
(if-not (nil? debug-handler)
(debug-handler session m))
(doall (map #(apply % [session m]) (:internal-handlers @session)))
))
now time to test!
ASCII letters A-Z are 65-90, etc .. you’ll want to use ascii chars most likely http://www.asciitable.com/
@luxbock I think it always defaults to true
Thing is this is zlib compressed. I'm trying to use this: https://github.com/amatus/GNUnet-in-Clojure/blob/master/src/main/clojure/org/gnu/clojure/gnunet/zip.clj
First one prints out
"x???Mo?0\f??J???@???????4d??0?N??V`?
the other prints out "xœ?•MoÛ0\f†ÿJ¡ëÒ@Ÿ¶ìÓÖ®Øì4d§¡0…NÔÊV
neither is recognizable to me
@pesterhazy thanks
sorry actually (prn (z/inflate msg)
is what prints out (123 34 116 34 58 34 71 85 73 76 68 95 ...)
I can't seriously have to use an ascii table to process these - I know clojure enough to know there's a magic way to do this.
this is made difficult by the fact that I have no idea how to "save" any of those in memory, they're triggered by a websocket event
isn't that already a byte array though?
wow this work. (prn (String. (byte-array (z/inflate msg))))
finally >.<
One day I will understand these things. Thank you very much for your patience and help, Josh.
The roller coaster ride isn't over but I'm glad to be on it 😄
does anyone have any resources on when to use core.async vs when to use java.util.concurrent stuff?
@schmee, for what use case?
CSP implementation in an external dependency that has not reached v1.0 vs battle-tested low-level concurrency management through the JVM. Pick your poison.
well if you get get away with futures/pmap, use those
if you have more complicated needs, it really depends
personally I tend to think that the complexity cost of core.async is not worth it in many cases, but I'm sure there are situations where core.async simplifies things
^ This. Clojure gives you first-class tools for managing (or rather, not having to manage) shared memory access, which is typically the biggest pain point of using shared memory/threads.
if I read this correctly: start with j.u.c and if things get too complicated move to core.async?
depends, both are useful and they are very different. hard to know with the (little) details you give about potential usage.
I’m talking about building a highly concurrent system processing 10000s of events / sec
yeah, I’m just trying to get some intuition for where the boundaries of core.asyncs applicability are
when you say “concurrent” — what specifically is concurrent? simultaneous processing of many requests? or each request needs synchronization in the computation of the result?
is the nature of your work primarily cpu-bound, or i/o-bound (you’ll be doing network requests yourself in order to complete the request, reading files, etc.)?
I’ll be reading events from Kafka, but I’m not really sure where I will end up on that spectrum
my advice is, start with the simplest possible version, don't overthink it, only parallelize when you absolutely need to
ok — well, your needs will take you in the direction best for you, but if you’re just needing to service many requests, your server (http?) will handle that part of the concurrency equation. whether or not you need to actually make concurrent computations, or whether to use async methods or not, is another matter
don't fall in the trap of "everything needs to be async because I need performance but now my architecture is 10x more complicated"
a.k.a. nodejs
Also, if you're working with Kafka via something like gregor then there's a bunch of async-ish stuff going on already. You don't want layers of it.
Also make sure you understand and choose the correct offset commit behavior or you will wind up dropping events on the floor at some point
@schmee I would not reach for a “concurrency framework” as a first step. The thing is if you program it in Clojure your need to managed shared state should be reduced considerably because of the emphasis on immutability in the first place. A large, complex and highly concurrent system often boils down to having only one or two atoms in Clojure and that’s all you need.
(or a single agent, or a ref or two or whatnot)
@pesterhazy all-async doesn't need a complicated architecture - and it ends up being simpler when you need to control resource usage with things like backpressure. i would say that getting started with all-async is conceptually harder in clojure though
Async code is always harder than non-async code. You're adding another dimension of complexity.
@tbaldridge true, for single threaded tasks it's harder, but not much harder, depending on how you approach it - perhaps no more so than lazy vs strict evaluation - and once you get around to dealing with concurrent resource consumption then it's simpler
Seems like an important consideration for whether to use async is whether backpressure is needed. (Like, do you want to block the client/producer of work from adding more work when your pipeline is full?) If so, async might be helpful. If not, maybe not.
@jeff.terrell isn’t the backpressure question all about being careful to not expose an unbounded resource? Or eliminate unbounded resources? You can achieve this with or without async, no?
(question may be a little too open-ended, sorry)
Yes, you can do backpressure with a thread-safe blocking queue (like the ones in java.util.concurrent
). I was just thinking that, if that's a concern, async might be better since the whole thing is built on such queues and you can easily propagate backpressure through multiple steps of analysis. But you're right, it's not a black-and-white thing. 🙂
Sure. Those queues are bounded by design, hence the assertion. For me I’d stay away from Java-only-based solutions because they have a much different design imprimatur. Every time I suck in a Java library, I always find the ugliest, less maintainable code tends to be around Java interop to support it.
Agreed. Though last I checked, queues were just about the only basic type of data structure that Clojure didn't have a built-in version of. I think I saw in a book somewhere (Joy of Clojure maybe) that the authors used a Java queue since a Clojure one wasn't available. (But it's been a while since I checked on all that, so I may be out of date here.)
Me too. Does seem odd. I don’t even know the underlying impl in core.async
@nickbauman: https://github.com/clojure/core.async/blob/f8e87e1625b1660b7f3b0aea044aad1327441741/src/main/clojure/clojure/core/async/impl/channels.clj#L32
Ha! so it’s just Java (j.u.c) at the bottom.
(Hmm. I haven’t done the warn-on-reflection thing and added the type hints to my code in a long time. I guess I don’t want to give up the concision and dynamism…)
I have a quite elementary questions, but it keeps bugging me and I was not able to find an answer: why is seq?
false for vectors? Or, asked in another way, why vectors do not implement ISeq
? What is it that I can do with a ISeq
that I can not do with a vector?
you can use an iterator(a seq) to traverse the values in a vector, but a vector is not an iterator or a seq
it is kind of confusing because lots of functions that work with seqs will automatically get the seq on a collection that is not seq when it is passed in
Good question @bbktsk. If that were on SO I'd 🔼 it.
@hiredman Thank you. Is there something ISeq can do that vector can not, ie. is there some function that works with ISeq
but not with vector? I have not checked all the functions in the ISeq hierarchy, but so far I found none. Well, there’s ISeq more();
defined in ISeq.java
as part of the interface, but that does not seem to be visible from clojure.
subvec
:face_with_rolling_eyes:
seqs are not counted, and can be constructed lazily, vectors support fast indexed access
seqs are a generic functional iteration protocol, vectors are ordered indexed collections
you could likely fiddle with vectors to get them to implement ISeq in such a way that would work (although cons would turn the vector in to something else) but to what end? What would it simplify?
and simple here has performance implications for the class hierarchy and object allocation
☝️ very well said, especially after my puerile joke.
I’m was getting unacceptably slow times from (sort-by sort-fn coll)
, so just to verify that my sort-fn wasn’t the problem, I timed (sort (mapv sort-fn coll))
and found it to be 20x faster. Obviously, the latter form doesn’t give me what I need, but suggests the issue isn’t the sort or sort-fn. Why is sort-by
so much slower?
what version of clojure? I know there were some jira issues about sort-by calling the sort-fn too many times
@bbktsk sequence is a logical list abstraction. vectors (and sets and other things) do not directly implement this abstraction but can provide an implementation when asked (via seq
). Thus these collections are seqable
, not seqs themselves. Lists are an exception - they implement the sequence abstraction directly so seq?
on a list returns true. Also see more detail at: http://insideclojure.org/2015/01/02/sequences/
@neverfox dropping a real world use case and timings on that ticket would be great
that stuff counts for a lot with Rich
@alexmiller Will do
as I last commented on that ticket, it’s lacking enough information to be properly screened. if someone did that (demonstrated the performance improvements), I could move it along.
there’s a language (and other technology) poll at https://jaxenter.com/jaxenter-survey-preliminary-results-programming-131480.html if anyone’s interested in demonstrating your love for Clojure, btw :)
if i aot everything, say for a servlet app, do i still need the clojure jar? if so, is there any way to eliminate code that is not needed?
yes, you still need it for the Java classes that make up the runtime implementation (and potentially for other things as well)
there is nothing provided in Clojure to slim that down
maybe back up - what’s your goal?
just wondering, at the moment. well, fast startup - the context here is boot-gae, just occurred to me it might be a good idea to make gae/ build --prod aot everything, since (i speculate) there usually will bot be much need for ad hoc dynamic stuff in the prod cloud env.
Speaking of polls, is there a date for the state of clojure survey coming out?
i know faster startup time is a perennial topic, just not sure what the state of the art is. any refs (blogs etc.) stand out?
@danielcompton soon soon … there have been some external forces that have delayed the release (that is, people being busy with non-work stuff)
I benchmarked this for us the other day, it came out as not worth it. try this: jar your app with no aot, launch clojure.main on that jar, time requiring a namespace that requires every other namespace. now jar your app with aot all, and time doing the same thing
it really depends a lot on what you’re doing
yeah I mean you could have some crazy macroexpansion in there that takes 10 minutes, at that point probably worth it
like core.async :)
oh yeah I always forget core.async is a lot of compile macroexpand time magic, its api just integrates so well
I have a page with some past research at http://dev.clojure.org/display/design/Improving+Clojure+Start+Time but that data is likely worthless in relation to your particular problem
I guess that makes sense :)
there was some talk about adapting core.async to gae's thread stuff a while back, anything ever come of that? sounds like a fun project.
@alexmiller nice article, thanks. irrespective of the data it's a nice survey of the issues involved.
@mobileink I’m not familiar with anyone doing the core.async thing for gae
how to get source code of my own fn like data? (source _) works for clojure.core fns but not for my ns
for example, when you pass a function object to source it throws an error`(source (fn []))`
the way source works is it takes a symbol, resolves it to a var, then looks at the var metadata to find file and line information, then opens the file, seeks to the line, then reads the next clojure form
if any of those steps fail, which they can in all kinds of ways, then source won't work
i want to transfer some short source code via network as string
so, it means that i should open the src file and lookup for given name
@hiredman i do global registry of my corporate data based on clojure spec
i want that it would be like webservice
yes s/form is correct but sometime in specs i use helper functions
and i want to pass them via network
leave the helper functions in spec forms has symbols, and allow consumers of your specs to interpret them as they wish
@hiredman may be you're right.
Sending functions over the internet is all fun and games until you send (fn [] my-data)
where my-data
is a 3TB database
the idea is to allow remote etl tools request data registry webservice to resolve spec
@grav, some yes, what's your issue?
@tbaldridge: Is it possible to ignore newlines when using parse-str
?
An example:
(xml/parse-str "<foo>\n<bar>42</bar></foo>")
=> #xml/element{:tag :foo, :content ["\n" #xml/element{:tag :bar, :content ["42"]}]}
In the above, I’d like to avoid the \n
from the :content
vector, since it’s between tags
sadly no, I don't know the answer to that
(xml/parse-str "<foo>\n<bar>42</bar></foo>")
<#C03S1KBA2|clojure>.data.xml.node.Element{:tag :foo, :attrs {}, :content (<#C03S1KBA2|clojure>.data.xml.node.Element{:tag :bar, :attrs {}, :content ("42")})}
@piotr2b Instead of requiring your namespaces in ns user
, load them on runtime when functions in user
need them, them when you run the function in runtime, classes are probably already compiled by Lein:
(defn fixtures! []
(require 'backend.users)
((ns-resolve 'backend.users 'fixture) (ctx)))
@juhoteperi thx for your answer. I’m just wondering — is it a good pattern to have unpure functions?
@jr I’m actually using v0.2.0-alpha, which if I remember correctly is the first to handle namespaces
https://github.com/clojure/data.xml/blob/master/src/main/clojure/clojure/data/xml/jvm/parse.clj#L96
(xml/parse-str "<foo>\n<bar>42</bar></foo>" :skip-whitespace true)
#xml/element{:tag :foo, :content [#xml/element{:tag :bar, :content ["42"]}]}
@seylerius is this what you're looking for? http://www.clojure-toolbox.com/
Actually, I need to submit one I just read about last night to the toolbox, while I'm here.
@jr Did not! Works like a charm - thanks! Google didn’t help me - did you search through the code on Github, or how did you find it?
Great! It seems skipping white space per default isn’t allowed as per the spec (http://stackoverflow.com/a/2084909/202538). So I searched high and low for an option, but couldn’t find it.
Can most clojure libraries (the ones that don't depend on a java lib or a non-shared feature) be used in clojurescript apps?
Oh, I just noticed that toolbox lists whether different libs work with clojure or clojurescript.
most clojure projects specify their deps as maven artifacts, and most libraries live in maven respositories, so anything that can consume and spit out information from maven repos
if you have a project that uses a particular build tool, most build tools have a build tool specific command to spit that out
Would anyone happen to have a link to a project that uses aleph for http? I've said this before and it's still true: I can't read "API-speak" or "devspeak" at all, so I'm looking for examples. Specifically, something doing http/post (with aleph)
I'm trying to optimize a pretty basic core.async implementation. I'm seeing some pretty rough bottlenecking happening and I have a feeling it's because of my implementation
essentially, I have a channel that data is being thrown into via >!!, and pulled out of via <!!. I've parallelized a bit through having multiple futures
but at the end of the day they're all throwing data into and pulling data out of the same channel (many producers, many consumers)
I have a feeling that I'm probably not implementing it correctly (i.e: probably should be using go + >! instead of >!!), but would love some pointers on common design patterns
@piotr2b i ended up creating a separate ns dev-repl
instead of user for various reasons. then in my :dev
lein profile i make it the :main
@shayanjm I would avoid consumers and producers feeding in to the same channel if possible
you'll want to have the buffer of the channel to be at least the same size as the number of producers/consumers
but pipelines provide things above and beyond parallel execution, the most notable being an in order stream of results out the other end
@hiredman: ultimately I'm trying to optimize for throughput. Basically the "consumers" will be grabbing from the pipeline/channel and throwing the message into a database
a single channel to consume and a single channel to produce to I think would be fine, it is just a single process both consuming and publishing to the same channel can result in deadlocks
yeah, for that kind of thing pipeline isn't the best, because pipeline expects actions to have results that you care about consuming
Basically, messages will be thrown into this channel as quickly as they come (consumed from Kafka), and then consumers will be pulling from the channel to throw directly into a db
yeah, the problem I tend to have with executors on their own is they don't have a great feedback story, by default they have unbounded queues, which could be a problem there
what you might do is run both the fetching from kafka and the db writing in the same threadpool, so they will naturally throttle each other
I've decided that the planck REPL looks like the best way to get clojure-tastic lispiness for my shell scripting.
(let [exec (Executors/newFixedThreadPool 2)]
(.execute exec (fn fetch-from-kafka []
(let [msg (pull-from-kafka)]
(.execute exec (fn [] (do-whatever msg)))
(.execute exec fetch-from-kafka)))))
take and put right next to eachother always helps keep things straight.. nice way to add the backpressure to that ^