Can anyone recommend a job queue? Similar to Resque in Ruby or Celery in Python.

I think quartzite is probably the answer, but I've not used it

Looks like Quartzite handles scheduling. I'm more looking to be able to queue up some background work from a web request, for example.

Preferably, one that supplies its own worker process, so all you have to do is supply job implementations.

ah sorry, yeah wrong thing

if it were me, I'd probably look up whatever is most popular for java and use that, and if there's a clojure wrapper then yay

@carocad I don’t think we’re going to make any changes for that

Strangely enough, I'm not aware of a popular job queue for Java. Closest is probably Jesque, and I don't think that's too popular. Maybe most people in Java land use different workflows.

yeah it's looking to me like the most popular is a combination of spring technologies

Also, the idiomatic Java ones will likely expect you to gen-class your tasks since they look them up by reflection. (I believe. Not certain.)

oh they will, but that's just a matter of creating a java class that calls the clojure code you want

I'd say it depends on how big this project is, if I was pretty sure I'd need to scale to millions of jobs I would just do the work to start out with rabbitmq+spring or something else big like that

@mingp in theory, one can use celery with clojure, but probably not in practice. "Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there’s node-celery for Node.js, and a PHP client."

immutant should have something that should do what you want.

@mingp Immutant provides a messaging lib that may do what you want, w/o having to bring in rabbitmq+spring: http://immutant.org/documentation/current/apidoc/guide-messaging.html

Thanks. I'll look into those.

if you have any questions, feel free to come see us in #immutant

Hi! I have a rather CS101 concurrency question. I've got a vec of 20 strings (wrapped in a cycle , and let's say 100 workers (manged by an ExecutorService, that part's fine). I want each worker to take the next available string, and release it when done. If there's no strings available, then there's no new work. I know my way around atom/refs/swap, but I feel like I'm missing the coordination piece, and tracking which worker has which string. Any guides out there for an intro on this?

so many great choices, I would reach for core.async myself, not atoms or refs

or a java.util.concurrent queue

ya, another good option

one of the pipeline variants from core.async may be exactly what you want

depending on what you mean by "release" the string

yeah, if you want the worker management taken care of for you pipeline-* in core.async will basically give you a single function that does your whole thing

if you are using an executor service you could also just the executor services own queue, again depending on what you are want

the issue with the default queue you get with an executorservice is it is unbounded

and if you use a bounded queue, you get an exception when the queue is full instead of a blocking

so not super when you have an infinite seq you want to feed through it, but workable, and you could do something with a executorcompletionservice to make it a one in, one out sort of thing

ooo I always want an excuse to use core.async, I built a stream processing simulator in it

@hiredman: > the issue with the default queue you get with an executorservice is it is unbounded > and if you use a bounded queue, you get an exception when the queue is full instead of a blocking or blocking if you need it: https://github.com/tolitius/lasync

very cool. tjx!

Oh, by "release" the string, I mean that I basically have x profile directories for web browsers ( for crawling -- that's about my CPU's limit). So each worker gets its own profile, and needs to give it up when it's done crawling for the next browser.

@bradford Alternatively, have you considered having some sort of single mediator process that keeps track of the available pool and hands the items out accordingly?

Yeah, that's what I'd do in java...

@carocad: You are welcome to file an issue on eastwood's Github project page if you like, but I can't promise if/when I will take a look

hi, how do i import this and use it in clojure? (import 'jnr.ffi.Runtime)

i get java.lang.IllegalStateException: Runtime already refers to: class java.lang.Runtime

You could just use it fully qualified in your code, and not import it.

@zentrope thanks, just found that upon further searching..

Ah, good!

@nux Ideally, I'd like to be able to import a Java name renamed. But, according to StackOverflow (http://stackoverflow.com/questions/18389788/clojure-import-java-rename), seems like that's not supported.

Put it in another namespace (oy.clj) and wrap it?

@mingp correct, that’s not supported

is there any way to use compojure.coercions for optional query params ?

I want to run a binary, which runs in a shell. The program is located in another directory. When I execute the command in the shell it executes. But when I use shfrom clojure.java.shell it doesn't find the file.

can you print out resuots of pwd and env variables?

that sounds like two places where this could be going wrong

Ah, i passed the argument for the program in the first argument of sh. So I found the failure

@bradford alternatively you can take a look at https://github.com/ztellman/manifold, which lets you process items in a stream on an executor service you can provide

How can I run sh asynchrounus?

@danielgrosse Perhaps you could try the low-level API provided by https://github.com/Raynes/conch

I have created a huge vector in REPL, then took a small subvector, then disposed of the large one (def largevector nil), and then ran GC. I don't see the memory being returned to the system -- is that a byproduct of the way memory management in JVM works, does subvector retain reference to the original one, or is there some reference back to this large vector (maybe in REPL?).

Nevermind, I have figured it out: I have created another large vector, and the memory usage hadn't doubled to 8G, so it's a JVM holding onto unused memory.

I am researching what the best options for native GUIs (cross platform desktop, mac, win, linux) with Clojure are now (I am aware of the browser/html option, but am explicitly looking for alternatives). seesaw seems to be not too active anymore but there is no obvious, active lib for JavaFX instead. Maybe @mtnygard has something to point at from the experience developing the new Defold IDE?!

@nblumoe when you say browser/html, have you also considered stuff like http://electron.atom.io/ ?

react native also comes to mind

yes sure @val_waeselynck

specifically but not exclusively electron with cljs 😉

@mpenet for Linux, macOs & (Desktop) Windows?

no clue really, I just remember seeing articles about people doing this on windows/osx at least, I guess it's probably feasible on linux too

What do people think of the pros/cons of using custom hierarchies rather than the global hierarchy with multi-method’s? I’d like the hierarchy to be extensible as well as the multi-method… so something like this works: https://gist.github.com/RickMoynihan/0a1ce7ef9b96ff3df2e07607dc8f00a1 But it does basically mean swap!ing an atom at the top-level… Which I guess is basically what multi-methods do underneath with the global hierarchy anyway. The other requirement is that I need to basically get the hierarchy as data… In particular I need to represent a “super-type-chain” as an ordered list of types… With the global hierarchy this is also achievable through code like this: I’ve written some code already that can do this with the global-hierarchy, with something like this: https://gist.github.com/RickMoynihan/96a0556116106e4e3c4356f99198a815

(You can also hackily access the global hierarchy through @#‘clojure.core/global-hierarchy but I’d rather not do that)

@nblumoe https://github.com/halgari/fn-fx perhaps?

Heh just wrote pure-print that returns the haskell unit value.

(defmacro pure-print
  [expression]
  (list 'let ['result expression]
        (list 'println 'result)
        ()))

Does anyone know of a library/code snippet that has a variation of pr-str that will always return the same string for identical datastructures, e.g. not giving different results for {:a 1 :b 2} and {:b 2 :a 1}

last time I had that problem I enforced using sorted-maps

similarly, (= (list 1 2 3) [1 2 3])

so you’d pretty much have to define which specific types you’re going to use anyway

@jeroenvandijk sorting all the maps is going to be the only way to do this, but what are you trying to accomplish?

@tbaldridge I want to create a unique signature for a datastructure (in order to achieve content-based addressing). So I’m trying to find a reliable way of doing (sign data). If the signature method doesn’t always return the same for the same data, it means I’ll have multiple signatures for the same data. If you have structures with signatures of other structures this can easily blow up

I thought of sorting sets and maps, but there must be some more edge cases so I hoped someone else solved this for me 🙂

E.g. choosing the wrong signature algorithm will also have an effect at a certain scale. I’m looking at RIPEMD-160 right now https://github.com/xsc/pandect

@jeroenvandijk have you seen https://github.com/weavejester/whorl

cool, thanks I hadn’t seen it. Looks like a solution to this problem. I think i need a bit more but this is a start

@jeroenvandijk I also worked a bit with Luke on the thought process behind this library: https://github.com/arachne-framework/valuehash

I think that’s pretty close to what I am after. Thanks! I’ll give it a try

Did you consider other signature algorithms than md5 and sha1 too? I’m looking at ripemd160 because it gives me the practical guarantee of unique hashes

Maybe I am paranoid but I don’t want to debug hash collisions

md5 and sha1 and any other cryptographic hash function are certainly designed with collision avoidance as a primary goal

@jeroenvandijk: you can plug in any hash algo you like, readme describes how

I've never heard any concern about collisions for md5 and sha1 in a nonadversarial context, but I Am Not A Crptographer®™©

@gfredericks I am attracted to ripemd160 because somewhere in (the/some) paper (i believe, don’t have it hand) they say the first collision still has to be found.

@luke thanks, sounds good

@jeroenvandijk I think that's true for sha1 too

@jeroenvandijk this table suggests collisions found for md5 but not sha1: https://en.wikipedia.org/wiki/SHA-1#Comparison_of_SHA_functions

@jeroenvandijk: that was probably talking about maliciously creating hash collisions to break crypto. If you're not in a security context, you should think of a hash function as a "random oracle" that is evenly distributed over the output space. Collisions then will depend mostly on bit size... sha256 might be your thing

Ok maybe I should reconsider those other options. It’s not for security purpose, i just need a unique id without needing to check a db basically

Though for most applications the odds of a random collision for md5 are still in "odds that an asteroid destroys the earth " territory

Aerospike uses it too so I thought they did for some reason

haha yeah probably true. Maybe paranoid then 🙂

git makes quite a lot of sha1 hashes and I haven't noticed anybody nervous about accidental collisions

I wonder how many hashes github has stored

that’s true, but they don’t have global hashes externally (i.e. hashes are per repo). Who knows what they use internally

i think per repo the number of commits/blobs (not sure until what level the hashing works) are never that big I suppose. I’m expecting a lot more hashes than the average git repository

@jeroenvandijk do the math then...

@karol.adamiec thanks i will

@nblumoe For Defold, we just used JavaFX directly. The API is very simple and regular, so it is easy to use via interop. There were only a few things that caused us some headaches: 1. JavaFX really, really wants to be the thing to launch your application. If you don't use its launcher, then you must take care to initialize JavaFX before even loading any of its classes. There's a hack to do this by instantiating a JFXPanel, but you must be very careful with your namespace loading and require sequence. 2. We found it useful to create macros and wrapper functions for executing code on the GUI thread. Like most UI libraries, JavaFX is effectively singlethreaded. Other than that, it was quite a pleasant experience!

Wondering what the opinions are on my multi-method/hierarchy question above are? https://clojurians.slack.com/archives/clojure/p1483523954009169

@nblumoe and in contrast, fn-fx provides a ReactJS-like interface to JavaFX, you define your interface in a declarative way, and diff it against the previous state of the UI. fn-fx then modifies all the JavaFX components to match the new state executing minimal changes. It's not going to be as fast as going directly against JavaFX components, but you do get a nice declarative API.

I’m favouring a custom hierarchy - rather than a global one

@mtnygard @tbaldridge thanks a lot. Will give JavaFX a spin then and do a spike with bare interop as well as fn-fx

Is there a way to see the request generated by http-kit?

@pupeno ngrep works 🙂

How?

what bit are you looking to spy on, might be able to help you.

I need the whole request. Over SSL, otherwise yes, a packet sniffer works.

What are you looking to peek at, the TLS request?

The http request.

(Spent ages debugging httpkit's TLS implementation a while ago)

Oh? Where there issues with it?

Somewhat, we were using it to push a few thousand events/second to a HTTP API, once it dropped a TLS connection it tried to reconnect pretty aggressively.

To the point of us DDoSing endpoints with TLS requests.

Convincing http-kit to use a proxy would also be helpful.

For your question, you could try setting up a dummy endpoint (nginx/whatever) to dump request info is probably easiest.

Decoding TLSed packets is tedious at best.

@val_waeselynck supdate looks great

quick question, is it possible to pass a protocol function as an argument to another function? like

(defprotocol Foo
foo [bar])

(do-something 2 foo) 

yes, protocol functions are just functions

well I thought that but the compiler is complaining of not knowing the foo symbol when stating that a protocol function should be used as default on another function call. like

(defn gaps
  ([]
   (gaps MAX-GAP foo))
.....

looks like you need to require 'foo' from another namespace?

also, can you copy/paste your defprotocol? The original one you sent isn't the correct syntax

should be (defprotocol Foo (foo [bar]))

@tbaldridge thanks for your feedback, requiring the 'foo' function was the correct thing. I didnt realize that since I thought that requiring the Protocol itself would provide me with the functions vars aswell

forgive my new arrival and newbie question, but what is the consensus regarding the traction that Clojure is gaining in the (economic) world? In other words, if my desire is Clojure but my need is also income, is Clojure a reasonable path? (compared to Ruby, Python, and many other unmentionable languages)

Sorry, this should have been off-topic I guess

If you already know other languages, adding clojure is a win. If you don’t, and need income (who doesn’t?), learn something else first, then add clojure later. Look at the number of job descriptions requiring Java/C#/javascript (most of them), compared with clojure (hardly any of them) — you need to know a couple of mainstream languages to be marketable. Even in my job where I do 100% of it in clojure, I still need to know Java, which is a totally separate paradigm.

an examination of google search trends for clojure should tell you that it hasn’t gained much mainstream popularity in the last few years overall. https://www.google.com/trends/explore?q=%2Fm%2F03yb8hb Also understand this is somewhat regional, so in your country or region, or particular corner of the software industry, it may be more or less popular.

@michael_teter As Josh indicates, much will depend on where you live (or where you can get remote work) since Clojure usage varies dramatically by location.

Hi Josh. Thanks. I have used Clojure for small projects, and I have used about 10 other languages in my lengthy career. So far, though, I've only really seen Clojure listed in the heavy Java world. To me, Clojure is the antithesis to Java, and I am loathe to go back to the Java world. Maybe I am tired of typing? 😛

Netherlands, US citizen, prefer to stay in NL or preferably some wealthy island

hehe

I'm also a former Ruby guy, if that helps

borkdude, a strong Dutch Clojure guy, directed me here. However, he mentioned a jobs area that I don't see... sorry for bothering this channel

@michael_teter : #jobs

Since you’re new-ish to Clojure, your best bet is probably to find a company that has an opening in one of the languages you’re strong in, but also does Clojure work, so you can join that company and then switch to Clojure internally.

Also #jobs-discuss I suspect...

@borkdude sorry! I'm not used to Slack. #jobs didn't appear automatically, but other channels did

thanks

@michael_teter np 🙂

@michael_teter Like @seancorfield suggested, #jobs-discuss is more for discussing these kinds of questions. #jobs is for jobs, and #remote-jobs is for what is says

Yes, thank you. I'll add those

I appreciate the patience and advice 🙂

Sometimes, working alone (remote or being the guy forging new paths at a larger company) results in not knowing the team systems

(I just joined #jobs-discuss — happy to continue the discussion there)

@joshjones: > an examination of google search trends for clojure should tell you that it hasn’t gained much mainstream popularity in the last few years overall. https://www.google.com/trends/explore?q=%2Fm%2F03yb8hb while I share the sentiment, I don't think this graph is accurately visualizing the true "popularity". these are google searches, but Clojure community is a bit different from the mainstream languages. I would think that a large number of Clojure questions / searches bypass google and go directly to irc/slack/other more people focused mediums.

There can be some value to choosing well trending language. When I chose Ruby, it was just luck. Then it became a curse, really. You could have almost any job, but you knew that it wouldn't satisfy you. Now I'm more discerning (not that Ruby is terrible, mind you)

Well @tolitius , I use the word “mainstream” popularity to differentiate between people active in the clojure community and those more in the mainstream, who might be hiring, researching tools for company projects, etc., but who don’t actively use it already. It’s impossible to put a true popularity metric on it, but I think google search is a good tool for this. There are trend tools on job sites as well (monster, indeed, etc), that might give a more jobs-focused indication.

@joshjones the results would probably be pretty close. Ultimately we choose to be followers or leaders I guess

(or contrarians perhaps)

@joshjones don't get me wrong, I agree that we can do much better: https://www.reddit.com/r/Clojure/comments/5bw4jh/selling_clojure_to_businesses/ but I would say that most of the searches about the programming language come from developers, and they are not well represented in google trends

Hmm, last commit to master branch in Clojure git repository was on 28 October. Is it a hammock time for core team? :)

or they're touring to promote the latest release like a rock band. They'll be back in the studio soon to work on the next fire release 😆

@gdeer81 A lot of good albums are written in private studios 😉

@michael_teter clojure may feel very different from java, but in many cases java is the reason clojure gets used where otherwise they might try out racket, or common lisp, or even ocaml or haskell (depending on what features they wanted)

@noisesmith Forgive me, but coming originally from C and then C++, I didn't find Java so greatly different from C++... just an evolution. Clojure, however, seems utterly different. And with Clojurescript, it exists in the Javascipt world. And isn't there a .Net implementation?

it's about the ecosystem - they are a java shop and they can bring in clojure without having to replace infrastructure

It could be my lack of Clojure experience then 🙂

just swap in a clojure jar where they had a purely java jar before, and use the same libs containers and tooling

I thought the java/jdk integration was just an easy way to sneak it in 😉

in that case in all I have seen that "sneaking it in" accounts for the vast majority of usage

Yes, it’s essentially how we became a Clojure shop. We’d originally tried a bit of Scala for a particularly “hard” problem and, whilst it worked, it introduced its own problems, so we tried Clojure instead and that worked better — and for quite a while, Clojure was just something we wrote little libraries in and used them from our primary code base.

Over time, we wrote more and more Clojure (and less and less of our “legacy” code) until in 2015 we declared Clojure our primary language and now we have standalone Clojure apps and we’re actively retiring nearly all our legacy code.

All because we were already on the JVM and could try other JVM languages.

also, there are so many existing libraries and frameworks out there for the jvm that I consider that at least as big a feature as being able to sneak it into existing infrastructure

ie, "ah crap, we need to connect to teradata, better go find the libs for doing that" <- I invite you to compare that situation in clojure vs ruby 😆

actually don't, in a previous life I did actually have to connect to teradata with ruby, I wouldn't wish that experience on my worst enemy

@bfabry including....don't use a CLJ lib, just use the vendor supported Java lib. That's what I do 90% of the time, it cuts out questions of "where's the bug" when it comes time to ask the vendor for help

ya, unless it adds something or is very very widely adopted I'm not into wrapper libs. clojure interop is plenty good. though honestly there's a general point there that unless it's seriously adding value you shouldn't be adding yet another dependency to your project in the first place

regardless of language