Fork me on GitHub
#clojure
<
2018-03-12
>
eggsyntax01:03:02

Anyone around at the moment who knows the details of how clojars versioning works? I've got a lib up that's at a stable production version, say 1.0.0. I'd like to put a snapshot of 2.0.0 up so that folks can kick the tires, but I don't want it to get displayed as the current version. Does anyone happen to recall what gets shown as the default if the newer version ends in -SNAPSHOT?

eggsyntax02:03:01

(I don't want to let it get shown as the default version, because I don't want folks to start using it when it may not be fully ready -- but I'd like to give beta testers easy access)

ghadi02:03:15

Not really answering your question @eggsyntax but if you use the new clj tool, you can use git deps. Point to a commit sha directly, no need to "release"

noisesmith02:03:33

@eggsyntax if it ends in -SNAPSHOT anyone who switches to it deserves whatever they get

eggsyntax02:03:45

Yeah, but in the interest of being kind to users, it'd be nice to have the default version be the production version...

noisesmith02:03:15

wait whose defining a default?

eggsyntax02:03:17

Clojars, visually. If I got to a project page on clojars (eg https://clojars.org/adgoji/cascalog-graph ), I tend to add a dep to whatever version is shown on the main part of the screen (not the sidebar). If it's a snapshot, I try to look at the sidebar, but it would be easy to forget. So by "default," I just mean whatever version is shown on the main part of the screen.

noisesmith02:03:21

I don't know - I would consider it a clojars problem if pushing a SNAPSHOT made it the version clojars suggests, but others might disagree

eggsyntax02:03:39

Yeah, I can see an argument either way, so I can't guess what the actual behavior will be. I'll find out, though 😉

dpsutton02:03:40

Ghadi can your lib be extended to make a jar not just an uberjar? Or are there better tools for that

dpsutton02:03:36

Was looking at modifying it but it produced invalid jars with my changes. It looks like it relies on the dependencies manifest files?

ghadi02:03:07

Dunno, hadn't considered it. Right now we exclude all jar manifests

ghadi02:03:25

What kind of invalid jars?

dpsutton02:03:27

Oh I thought you concated them

qqq02:03:56

What is the simplest way to convert an int-array (INT_ARGB data from an image) to a byte-array ?

dpsutton02:03:17

Not sure. I don't know jvm that well so I don't know what should be in there. Just modified it so I could specify the class path from command line rather than asking for the full class path

noisesmith02:03:00

@qqq do you want one byte per int, or bytes representing the same bytes that would represent the ints?

qqq02:03:21

@noisesmith: well, each int is 4 bytes packed into it, representing ARGB

qqq02:03:30

so I want each int pulled apart into four bytes

noisesmith02:03:48

you can push ints into a ByteBuffer, then read bytes out

qqq02:03:34

in C, given memory layout, I can just cast the int* to an byte*

qqq02:03:45

does JVM safety prevent me from doing such hacks ?

noisesmith02:03:51

right, that's what ByteBuffer does for you

noisesmith02:03:44

I don't know of a method to cast the pointers, but with ByteBuffer writes and reads you can get the same result

qqq02:03:11

I don't get it, are you telling me to call putInt on ebery element of the int-array ? https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

noisesmith02:03:02

that's exactly what I'm saying

noisesmith02:03:10

then ask for the byte-array that's in it when you are done

qqq02:03:37

It's the (doseq [i ...] (.putInt byteBufferObj i)) part that looks inefficient.

noisesmith02:03:14

if you find a way to do it in java via a cast, let me know, I'd be fascinated

noisesmith02:03:19

as far as I know there's no such thing

noisesmith02:03:17

@qqq alternately you could just bit-mask and bit-shift the ints

dpsutton02:03:35

ah i meant the META/INF stuff

(re-find #"^META-INF/services/" filename)
    :concat-lines

qqq02:03:33

@noisesmith: an unrelated issue that is baffling me, PNG appears to be stored 0 - 255, java only has signed bytes, so "averaging" has interesting results.

ghadi02:03:31

@dpsutton that's only for java ServiceLoader / SPI business

ghadi02:03:52

most other META-INF stuff is ignored

noisesmith02:03:22

@qqq there's a simple trick to get an unsigned value from the byte... let's see who finds it first

qqq02:03:22

from what I have read, jvm does not ahveunsigned byte

qqq02:03:30

so if you convert it, there would have to be some intermediate type

noisesmith02:03:34

user=> (bit-and -1 0xFF) => 255

noisesmith02:03:58

bit-and returns an int

qqq02:03:08

type says Long, but sure, int

noisesmith02:03:14

right, right

noisesmith02:03:27

the original java trick uses & which returns an int 😄

dpsutton02:03:40

oh ok. i need to look into what needs to be in a jar

qqq02:03:48

so my XY problem is to start with a bufferedImage and read out a float[]

dpsutton02:03:05

i'm porting chas emerick's URL lib into the new stuff. the current version is 5 years without a commit and in cljx land

dpsutton02:03:17

wanted to use clj and coming up on needing artifacts

dpsutton02:03:23

thanks so much

ghadi02:03:28

@dpsutton if there already is a maven artifact, you can just pull it with deps.edn

ghadi02:03:10

yeah, loads right up. but yes it would be nice to get rid of the cljx stuff

dpsutton02:03:18

i mean create an artifact. one of the dependencies of URL is just as old. pathetic. i've got it converted and need to figure out how to make an artifact. but for sure you're right the rewrite could depend on a commit of it

eggsyntax03:03:13

For anyone who's curious about the question I brought up earlier re clojars: deploying a newer snapshot version to clojars does not make it the "default" version, ie the one shown on the main part of the page. It only appears in the sidebar. I'll paste a screenshot as a self-comment. @noisesmith @ghadi

eggsyntax03:03:56

I think that's probably the sanest choice on their part. Can't post images in a thread, I forgot; I'll imgur it up.

eggsyntax03:03:51

0.1.12 is the version that was already there. 0.1.17-SNAPSHOT is the version I just deployed.

qqq03:03:38

https://docs.oracle.com/javase/8/docs/api/java/lang/String.html endsWith will do string -> string -> boolean is there a function which does string -> string (returning the actual suffix/extension/ part after the last .)

qqq03:03:58

alright, I guess I can use:

(last (clojure.string/split "blah.txt" #"\."))
instead

mpenet05:03:13

It's a simple regex, if you just want to get the extension

qqq06:03:07

In Clojure+JVM, how do I do 'back pressure' ? I have one queue who's behaviour is: (java.util.concurrent.Executors/newFixedThreadPool 48) which basically says: create a queue where there are 48 workers popping stuff off now, when I submit things to this queue, I want to say: if there's < 1000 items, submit to this queue if there is >=1000 items, pause current thread until < 1000 items, then resubmit

alexk13:03:44

Is that also not exactly what (chan 1000) would do? Channels with a fixed size will never accept more than that number of values

leonoel13:03:11

yes, a chan with >!! and <!! is basically a blocking queue

mbjarland08:03:34

is there a way in clojure to do keys-destructuring on something like:

(my-fn :arg-a "val-a" :arg-b "val-b" "non-map-value")
i.e. I would like a specific set of keys to get destructured into arg-a, arg-b etc and then have a & rest or similar match the remaining args? Note the uneven number of args in the call. I suspect this is not possible....

sundarj10:03:09

yeah not possible i think. you have to do it manually; something like

=> (defn foo [& args]
     (let [{:keys [a b]} (apply hash-map (butlast args))
          c (last args)]
       [a b c]))
#'boot.user/foo
boot.user=> (foo :a 2 :b 3 "42")
[2 3 "42"]

mbjarland10:03:19

Ok kind of expected that one, thanks for the verification

mbjarland11:03:27

Was hoping there was some way of getting editor intellisense help with params for functions like this...guess we are out of luck there

sundarj11:03:18

maybe you could manually hack the :arglists metadata?

mbjarland11:03:12

ha! is that what the editors pull out? will definitely look into this. The functions in question are generated code so adding things to the metadata could definitely be an option

sundarj11:03:19

it's what doc uses, so i would imagine so

mbjarland11:03:00

indeed it does work, then again, that makes the arglists a "lie" in that you can not do this in clojure

mbjarland11:03:22

but still, might opt for convenience over correctness on this one

sundarj11:03:33

fair enough!

mbjarland11:03:36

thanks again!

kurt-o-sys11:03:26

So, would this diagram (based on https://es.slideshare.net/adorepump/clojure-an-introduction-for-java-programmers ) be correct? (tried to add tagged literals)

Alex Miller (Clojure team)13:03:08

I don’t think there are any lines between the data reader and the evaluator/compiler

kurt-o-sys14:03:32

doesn't a tag literal invoke a function, returning the result of that function to replace the original tagged literal? Invoking a function is evaluating something?

Alex Miller (Clojure team)14:03:11

yes, but I took the lines you already have from the reader as being that. it uses read-time evaluation for that

kurt-o-sys14:03:39

ok, right, I just tried to do something similar as 'macro': the compiler is just doing it's job, but in that diagram (I took from Rich), it's just a different block in the diagram. But yeah, I need to adapt it a little. No lines going back to the reader and replacing 'data reader' with 'tagged literal/function' may be more in line with the original diagram? (just to emphasis what happens with tagged literals)

Alex Miller (Clojure team)14:03:56

tagged literal is a kind of value, data reader is the code evaluated to construct one

Alex Miller (Clojure team)14:03:28

macro expansion is also evaluation (compile-time rather than read-time)

Alex Miller (Clojure team)13:03:38

And I don’t know what “program” is there at all

kurt-o-sys14:03:20

just copied it from a talk from Rich Hickey 🙂. I just added the blue stuff.

Alex Miller (Clojure team)13:03:24

Where you have tagged elements those are also data structures

kurt-o-sys14:03:56

right... just data structures which happen to be tagged literals. get it.

anler13:03:36

Hi, I was experimenting with a macro similar in concept to this one: (defmacro defdyn [name & body] '(def ^:dynamic ~name ~@body)) and I noticed that defining dynamic vars like this doesn’t seem to work, am I doing something wrong?

moxaj13:03:48

@inbox you are applying the metadata to the wrong symbol, try ~(vary-meta name assoc :dynamic true) instead of ^:dynamic ~name

anler13:03:05

yoo-hoo! thanks, that did it 🙂

anler13:03:34

now that you told me I see what was wrong, thanks!

qqq14:03:34

1. Does the JVM use SSE instructions? 2. If so, if there anyway to control it? (I don't see it in the JVM bytecode).

Alex Miller (Clojure team)14:03:42

look up -XX:+UseSuperWord option

qqq14:03:25

I need a function where: input: namespace, symbol name output: string, which can be a valid Java classname I also need the function to be deterministic and one-to-one Suggestions ?

Alex Miller (Clojure team)14:03:09

The Clojure compiler has a munge function to do this but it’s n to 1

Alex Miller (Clojure team)14:03:37

As things like - and _ both map to _

qqq14:03:27

1-1 is not that important; munge is good enough; thanks!

Alex Miller (Clojure team)15:03:04

there’s a demunge too

qqq15:03:37

(let [my-queue (java.util.concurrent.ArrayBlockingQueue. 5)
      my-pool (java.util.concurrent.ThreadPoolExecutor.
               24 24 0
               java.util.concurrent.TimeUnit/MILLISECONDS
               my-queue)]
  (doseq [i (range 10)]
    (println "before put queue " i)
    (.put my-queue
          (fn []
            (println "line " i)))
    (println "after put queue " i)))
this code enqueues 0, 1, 2, 3, 4 but nothing is printed (i.e. executed) why? what am I doing wrong I'm expecting the ThreadPoolExecutor to be draining the ArrayBlockingQueue

gonewest81815:03:13

How is the thread pool executor involved? I see it created but not used.

gonewest81815:03:47

Nvm. I see it now.

leonoel15:03:27

that's not how ThreadPoolExecutor is supposed to be used

leonoel15:03:15

don't feed the queue directly, use ExecutorService methods

qqq16:03:31

so the issue is that, if I use thre threadpool executor's .execute, if the queue is full, it rhwos an exception

qqq16:03:37

however, if the queue is full, I want it to block

leonoel16:03:52

why do you need a ThreadPoolExecutor ? start N threads, make them poll the queue, then feed the queue

qqq16:03:58

I don't have a good reason for using ThreadPoolExecutor. I saw that it was available, it seemed to proviide the right abstractions (managing tthe threadcpool for me), so I tried it. It's entirely possible the right solution is to manage threads manually. Is that what you suggest ?

bfabry16:03:42

@qqq it sounds like you want Executors/newFixedThreadPool

bfabry16:03:24

java.util.concurrent.Executors to be specific

qqq16:03:29

@bfabry: when the "queue" hits a certai nsize, I want all enqueues to block -- newFixedThreadPool only provides: 1. unlimited queue or 2. throw exception on full

bfabry16:03:55

oh sorry you want to block the caller. yeah I don't think anything provides that out of the box

leonoel16:03:58

there's nothing wrong using threads directly if you know in advance what they're dedicated for

bfabry16:03:03

that's not normal thread pool behaviour

qqq16:03:55

Let me ask another question. Do thread pools provide any mechanism for back pressure ?

bfabry16:03:19

kind of the same question isn't it? not unless you consider an exception back pressure

bfabry16:03:55

core.async has a mechanism for back pressure. otherwise if your producer is only one thread it's easy enough to do yourself

ghadi16:03:00

section Queueing and RejectedTasks is what you want

qqq16:03:35

@ghadi: yes, I'm aware of the possibility of throwing exception on queue being full; is there a way to turn this into a blocking op? the best I can do is turn this into a delay + retry

noisesmith16:03:33

"It is possible to define and use other kinds of RejectedExecutionHandler classes"

ghadi16:03:13

using an unbounded queue and blocking is a bad idea. delay+retry sounds great to me

noisesmith16:03:24

@qqq what about using CallerRunsPolicy? that is effectively blocking

noisesmith16:03:53

and it's a form of backpressure surely

qqq16:03:38

That's an interesting suggestion.

qqq16:03:51

One way to write a ByteArrayOutputSream to a file is:

(let [baos (java.io.ByteArrayOutputStream.)
      os (java.io.FileOutputStream. out-fname)] 
  (.writeTo baos os))
is there a way to do this async, so the .writeTo does not block the calling thread ?

Bravi17:03:25

hello everyone. I’m trying to generate a namespaced keyword from a string. "hello" -> ::hello is it possible to do this?

(keyword "hello") ;; => :hello
(keyword ":" "hello") ;; => ::/hello

noisesmith17:03:35

@bravilogy ::hello is a shorthand for auto-namespacing to the current ns - one way to do it programatically would be (keyword (name (.name *ns*)) "hello")

tanzoniteblack17:03:45

(keyword (str *ns*) "foo") will product :currentNs/foo; or you can manually specify the ns with: (keyword "user" "foo")

noisesmith17:03:19

oh I forgot (str *ns*) worked

logistark17:03:34

Hi, when is expected to have Java 8 interop?

tanzoniteblack17:03:21

we already have java 8 interop?

tanzoniteblack17:03:47

can you explain what you're trying to do, your build tool settings, and any error messages you might be seeing?

logistark17:03:00

I mean, functional interfaces

alexk17:03:52

hmm, *ns* is not reliable, as in it doesn’t do what you’d expect

noisesmith17:03:14

nothing stops you from interacting with functional interfaces

noisesmith17:03:44

I wouldn't expect clojure core to ever have a syntax for them like java does of course

noisesmith17:03:28

@jimenezsaezjoseantoni more specifically you can use reify or proxy to make an instance of Function or Producer etc.

logistark17:03:36

but i think it would be helpful ifn interface match functional interfaces

logistark17:03:54

so, there is no need to convert from one type of function to other

noisesmith17:03:14

that's a separate question, even if we targetting minimum java 8 I'm not sure IFn would ever implement those interfacs

noisesmith17:03:40

you don't need to convert - use reify or proxy instead of fn

tanzoniteblack17:03:34

also known as, I've never attempted to make use of Java's function stuff even within java, let alone Clojure

Bravi17:03:33

@noisesmith @tanzoniteblack thanks, that works! 🙂

logistark17:03:55

It would be helpful for instance

logistark17:03:58

(-> (Stream/of 1) (.map #(+ % 1)))

logistark17:03:11

that does not work without making a reify

qqq17:03:40

when writing jvm bytecode, is there a way to say: I highly recommend you inline this :invokestatic ?

bronsa17:03:30

you can give hints to the jvm at startup tho

bronsa17:03:04

actually it might be possible with some jvms through annotations

bronsa17:03:09

but likely to be internal details

bronsa17:03:07

but it doesn’t sound like a good idea anyway

anler17:03:28

Hi all, do you know of an alternative of with-redefs that is thread-safe so I can use it in a concurrent test-suite scenario?

noisesmith17:03:22

@anler the two best ways to avoid with-redefs are to take stateful or out of scope elements as an argument so they can easily be substituted in a test (eg. stuartsierra/component simplifies doing this) or using a dynamic var so that it can have a per-thread binding

anler17:03:15

yes, I’m aware, ironically we are using component in the project I’m working on, but the business logic code is all coupled and I want to unit test a function with lots of deps to other functions

anler17:03:11

so I was using with-redefs but in my mind I had the idea it was binding, so it was a bit of a surprise when I saw my test suite failing 😛

tanzoniteblack17:03:24

I generally just dynamic variables in this case, @inbox.

anler17:03:03

but if what I want to mock is another function, those function vars aren’t dynamic right?

tanzoniteblack17:03:43

(defn ^:dynamic my-fn ...) works just fine

anler17:03:03

hmm, that might be a solution 🙂 thanks!

noisesmith17:03:59

@anler I actually use component to decouple function logic (since I'm already paying for it to manage stateful things), and it makes testing much simpler

noisesmith17:03:19

just pass in a hash map with stubs of the things you need, customized for testing

anler17:03:21

yeah, but our codebase needs a lot of changes before we are able to do that, I would do it but moving things around make my team nervous ¯\(ツ)/¯

noisesmith17:03:41

you can also do this via a wrapper (not using component)

noisesmith17:03:02

refactor the function to be tested to use a new function that takes an implementation as an argument

noisesmith17:03:19

then instead of testing the function, test the implementation, passing in a stub

anler17:03:09

for example, one of the dependencies for our functions is a db, but that db is passed directly to jdbc functions, so I guess we should first create a DB protocol or something that decouples our code from jdbc itself, but doing such changes right now seems unlikely

noisesmith17:03:34

you can just do this with a higher order function, no protocol needed

anler17:03:45

found this though, it seems it works, it might be the quick and dirty solution: https://gist.github.com/gfredericks/7143494

anler17:03:24

can you post an example?

noisesmith18:03:09

from

(defn foo [db x] (blah db (bar x)))
to
(defn foo-impl [instance x] (instance (bar x))
 (defn foo [db x] (foo-impl (partial blah db) x))

tbaldridge18:03:14

@anler yeah, that gist does a few things that I'd be wary of. It's pretty much a hack

noisesmith18:03:08

@anler the idea with the above is that you don't need to test (partial blah db) - it's some database function provided by your adaptor

noisesmith18:03:23

so you can test foo-impl with a different stub replacign the db stuff

anler18:03:36

oh I see, I’ll take note and see how I can introduce such changes without too much drama 😄 thanks!

noisesmith18:03:36

that pattern shouldn’t require more than splitting one function and making the second function a thin wrapper - definitely add a doc string describing why foo-impl exists though, because in my experience people will ask

noisesmith18:03:28

and really the above could be a protocol with reify implementing it, but that tends to raise even more confusion

noisesmith18:03:02

where a first class function makes it more “humble” and doesn’t make people assume some bigger design that isn’t there

anler18:03:55

yeah, I like it, to put you in context the function I’m trying to test is 57 lines long and depends on ~13 other functions 🙈

noisesmith18:03:19

haha, that's an argument for refactor in itself

Ilya19:03:33

Can I use with-redefs to redefine a form within a let that's within a function?

dpsutton19:03:12

usually about this point i refactor so that the thing i'm trying to reder is an argument

Ilya19:03:02

ok, so sounds like no 🙂

Ilya19:03:10

And is with-redefs not meant to be used in production?

Ilya19:03:19

I see the docstring...

Ilya19:03:23

>quote These temporary changes will be visible in all threads. Useful for mocking out functions during testing.

Ilya19:03:33

But wanted to double check

dpsutton19:03:37

oh this isn't testing. that's a code smell to me. pretty big 🙂

Ilya19:03:52

this is testing actually

Ilya19:03:02

but I can see uses in production

Ilya19:03:08

aside from a code smell, any reason not to?

hiredman20:03:57

with-redef redefs named vars, which are by definition global, so by using it you immediately have global mutable state

hiredman20:03:24

even using it in tests is not great

hiredman20:03:58

I say that not as a dogmatic thing, but as someone who has worked on a large clojure code base with a large test suite that used with-redefs a lot, it sucks

hiredman20:03:34

if you are using with-redefs, whatever it is should be passed as an argument instead of def'ed as a global value

didibus21:03:23

I have a macro where I need the arguments to be evaluated first. I tried making it a function instead, but then the code I return doesn't get evaluated. So my only solutions right now are either to call eval on the args inside the macro, or call eval inside the function.

greglook21:03:48

can you not do:

(defmacro foo
  [x y & body]
  `(let [x# ~x
         y# ~y]
     ; do stuff with body
     ,,,)

didibus21:03:11

Except my arguments are inside & body

greglook21:03:52

then you’ll need to parse the structure - this is one of the use-cases that clojure.spec was built for, so you could write a spec for the body and then use conform to pull out the pieces you need

greglook22:03:24

you may want to rethink the design of this though if that turns out to be too complex; to me that points to a macro that is going to be hard to understand

didibus22:03:17

Its the challenge of trying to wrap a spec macro inside another

qqq22:03:29

can you write a sample "input to amcro" and "output to macro" pair so we can see what you have in mind?

didibus22:03:57

combine-keys being the macro

didibus22:03:30

and/or the function, since I need a and b in normal evaluation order, I don't really need a macro, but then I need to wrap combine-keys in an eval if I make it a function

xiongtx22:03:20

Anyone have experience writing Cucumber tests in clj, then generating a class to be called in Java? Not sure if cucumber-jvm-clojure would help w/ that; it seems intended for writing Cucumber tests in clj and testing via a leiningen plugin. https://github.com/cucumber/cucumber-jvm-clojure

gklijs04:03:05

@U2J7JRTDX should be possible, bit it might be easier to write macro's for #etaoin wich van connectie to a webbrowser driver.

greglook22:03:38

@didibus for that specifically, why not use s/merge?

qqq22:03:40

I'm looking at: https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html https://docs.oracle.com/javase/8/docs/api/java/lang/Float.html and I see an exp(double) but no exp(float) Does exp only operate on doubles, and not floats ?

didibus22:03:05

@greg316 Well, that's where things are a bit more complicated, I use a modified keys macro which prevents additonal keys, so it doesn't work with merge properly

didibus22:03:21

Also, the return of describe which I use is different for a merge spec

didibus23:03:30

Overall, I find its a situation I often get into. I want to control when the arguments would be evaluated, and maybe also when the return would be evaluated. What's the edge cases to watch out for when using eval or resolve in a macro?

noisesmith23:03:47

resolve leads to spooky runtime errors with little clue at how to solve it, eval leads to bad performance

didibus23:03:45

So, appart from performance and safety if used with untrusted input, eval executes in the same environment?

didibus23:03:47

Would (defn foo [] (+ 10 global-symbol)) be equal to (defn bar [] (eval '(+ 10 global-symbol)))

noisesmith23:03:18

you'd have to try and see, I forget some of the details of eval's environment resolution (I know it does not capture locals)

didibus23:03:02

I guess my issue is I've always found it difficult to be sure of the context of eval, so I always worry there will be a scenario it fails

didibus23:03:19

My macro is

didibus23:03:59

(defn combine-keys [& kv] (eval `(su/keys ~@(mapcat identity (apply merge-with into kv)))))