Fork me on GitHub
#clojure
<
2015-11-21
>
gordonatheroku01:11:20

Hey Gang - ouch - ~/.lein/profiles.clj :java-cmd - didn't see that coming. The value was a removed version of Java. Once updated the path was found.

nowprovision06:11:03

what is the :scope "test" thing I see in some projects and for example om next tutorial, is this a leinengen feature?

dm307:11:28

syntax is pomegranate's, functionality is native to Maven - test dependencies are only pulled during testing. See https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html (Dependency Scopes) for more info

jaen10:11:26

Is there any way in Clojure to do something akin to a Haskell where you can alias a type so the type system thinks they are two disparate types. Use case - I have some coercion using schema, but I have to coerce some values differently depending on the context. The simplest way I can think of is to take (in my example) DateTime and somehow alias it to PayloadDateTime that it behaves exactly like DateTime, but I can match on a different class name.

jaen10:11:07

Now that I have written it out loud it sounds like I could just inheritance or something, but I'm not aware of how to inherit a Java class in Clojure.

jaen10:11:51

Welp, DateTime in joda time is final : |

rauh11:11:40

@jaen: I usually just create a new defschema with a pred for that kind of functionality

gtrak15:11:31

I'm getting some really weird AOT behavior with 1.8, seems specific to my project

gtrak15:11:18

(pprint/pprint 4)
ClassCastException clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush  clojure.pprint/pretty-writer/fn--5686 (pretty_writer.clj:391)
> (isa? clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 clojure.pprint.PrettyFlush)
false
> (ancestors clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848)
#{java.lang.Appendable java.lang.Object java.io.Flushable clojure.lang.IProxy java.lang.AutoCloseable java.io.Closeable java.io.Writer clojure.lang.IDeref clojure.pprint.PrettyFlush}

gtrak15:11:39

both those classes are shipped with the clojure jar

gtrak15:11:59

but, then:

> (every? (partial isa? clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848) (ancestors clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848))
true

gtrak15:11:36

seems there's two copies of PrettyFlush in different classloaders

ragge15:11:23

@gtrak: you've probably already done this, but if not wiping target would be my first place to start

gtrak15:11:21

more evidence:

gtrak15:11:42

> (.getClassLoader clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848)
#object[sun.misc.Launcher$AppClassLoader 0x14dad5dc "sun.misc.Launcher$AppClassLoader@14dad5dc"]

> (.getClassLoader clojure.pprint.PrettyFlush)
#object[clojure.lang.DynamicClassLoader 0x68859ee7 "clojure.lang.DynamicClassLoader@68859ee7"]

gtrak15:11:08

and PrettyFlush comes from a definterface call

gtrak15:11:59

this was the case for an earlier beta, I'd hoped I could ignore it simple_smile

gtrak15:11:31

this is unsettling: https://github.com/clojure/clojure/blob/clojure-1.7.0/src/clj/clojure/genclass.clj#L722

(defmacro gen-interface
  [& options]
    (let [options-map (apply hash-map options)
          [cname bytecode] (generate-interface options-map)]
      (when *compile-files*
        (clojure.lang.Compiler/writeClassFile cname bytecode))
      (.defineClass ^DynamicClassLoader (deref clojure.lang.Compiler/LOADER)
                    (str (:name options-map)) bytecode options)))

ragge15:11:30

@gtrak: why is that unsettling? do you have a minimal repro for your issue?

gtrak15:11:35

I would expect it to look for an existing PrettyFlush interface in another classloader. I don't think I can make a minimal repro case without dropwizard, maven, etc..

gtrak15:11:01

eg, the definterface call in pprint always gens a class

gtrak15:11:54

so I'm leaning towards playing with clojure.lang.Compiler.LOADER and seeing if that affects anything

gtrak16:11:25

huh, but in a regular lein repl, PrettyFlush uses the appclassloader. anyways...

bronsa16:11:24

@gtrak: can you try and see if the issue is there for 1.7?

gtrak16:11:43

it's not in 1.7 because the structure of pprint itself has changed

bronsa16:11:28

the last commit to pprint is 2 years ago

gtrak16:11:55

I think that's what introduced the issue

gtrak16:11:03

in pprint_base

gtrak16:11:46

I can give you a full stacktrace

gtrak16:11:14

#error {
 :cause "clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush"
 :via
 [{:type java.lang.ClassCastException
   :message "clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush"
   :at [clojure.pprint$pretty_writer$fn__5686 invoke "pretty_writer.clj" 391]}]
 :trace
 [[clojure.pprint$pretty_writer$fn__5686 invoke "pretty_writer.clj" 391]
  [clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 flush nil -1]
  [clojure.core$flush invokeStatic "core.clj" 3611]
  [clojure.core$flush invoke "core.clj" -1]
  [clojure.core$prn invokeStatic "core.clj" 3622]
  [clojure.core$prn doInvoke "core.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.pprint$pprint invokeStatic "pprint_base.clj" 252]
  [clojure.pprint$pprint invoke "pprint_base.clj" -1]
  [clojure.pprint$pprint invokeStatic "pprint_base.clj" 245]
  [clojure.pprint$pprint invoke "pprint_base.clj" -1]
  [com.redowlanalytics.query.elasticsearch.mds$eval27851 invokeStatic "NO_SOURCE_FILE" 1]
  [com.redowlanalytics.query.elasticsearch.mds$eval27851 invoke "NO_SOURCE_FILE" -1]
  [clojure.lang.Compiler eval "Compiler.java" 6913]
  [clojure.lang.Compiler eval "Compiler.java" 6876]
  [clojure.core$eval invokeStatic "core.clj" 3107]
  [clojure.core$eval invoke "core.clj" -1]
  [clojure.main$repl$read_eval_print__4176$fn__4179 invoke "main.clj" 240]
  [clojure.main$repl$read_eval_print__4176 invoke "main.clj" 240]
  [clojure.main$repl$fn__4185 invoke "main.clj" 258]
  [clojure.main$repl invokeStatic "main.clj" 258]
  [clojure.main$repl doInvoke "main.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__20491 invoke "interruptible_eval.clj" 58]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 645]
  [clojure.core$apply invoke "core.clj" -1]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1892]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 56]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" -1]
  [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__20533$fn__20536 invoke "interruptible_eval.clj" 191]
  [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__20528 invoke "interruptible_eval.clj" 159]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}

gtrak16:11:06

but it's not really the type-hint's fault afaict

gtrak16:11:23

utilities is where PrettyFlush is defined, but that's loaded before pretty-writer

bronsa16:11:58

how can I reproduce this?

gtrak16:11:44

I have no idea. Unfortunately, it's inside a multi-module dropwizard project. I could try to work a minimal repro test-case, but I'm confused about the classloading.

gtrak16:11:12

It seems like if I can figure that out, then it'll work itself out.

bronsa16:11:13

classloading changed significantly in 1.7

gtrak16:11:21

we were on 1.7 before

bronsa16:11:17

what seems to happen is the definterface is being reloaded while the type-hint refers to the first version

gtrak16:11:02

I was interpreting it as the proxy is in a parent and its super is in a child classloader.

gtrak16:11:39

but for some reason that doesn't happen in a lein repl, PrettyFlush is in the app classloader instead of the dynamicclassloader

gtrak16:11:51

looks relevant: 'it changes gen-interface to always emit an in-memory interface along with the [optional] on-disk interface so that the in-memory class is always updated.'

bronsa16:11:06

yes, that was intentional

bronsa16:11:30

unfortunately knowing what you're doing to repro this I don't really have any way of trying to understand what's going on

gtrak16:11:16

right, I can work on that. basically, it's a dropwizard app that spins up a REPL during initialization.

gtrak16:11:22

using clojure.java.api

bronsa16:11:57

@gtrak: too many variables at play there, could be a user/tooling/config bug

gtrak16:11:42

I agree in principle, but I've been poking around at checksumming classfiles in the uberjar etc and it all checks out

jaen16:11:43

@rauh: yeah, I figured that is what I'll do for now, so thanks for confirmation people do that. It would have been nice to be able to do some contextual validation/coercion with schema, but I would have to write more code since I don't think schema's walker supports that (or I'm missing something).

gtrak16:11:22

eg, what's in the uberjar is what's in clojure.jar

gtrak16:11:43

if I'm loading things a different way than I should be, then that's where the problem is

bronsa16:11:47

@gtrak: doesn't matter. if anything in there messes with classloading, it might get clojure into a inconsistent state

bronsa16:11:03

i.e. you have parts of pprint loaded AOT and parts reloaded JIT

gtrak16:11:52

yea, so my next experiment is going to be to load everything via reflection simple_smile.

gtrak16:11:22

I wonder if class initialization is part of the problem

gtrak16:11:35

s/initialization/initializers

cigitia19:11:09

I’m looking for a data structure that lets me remove things from its front, add things to its end, and access its items by indices efficiently. Regular vectors only let you efficiently remove things from its end, of course. clojure.core.PersistentQueue has linear-time nth, unfortunately. Is core.rrb-vector (https://github.com/clojure/core.rrb-vector) my best bet?

arohner19:11:32

using Schema, is there a schema for truthy, without using s/pred?

ghadi20:11:49

@cigitia data.finger-tree

ghadi20:11:53

Is Persistentqueue really linear lookup?

cigitia20:11:50

@ghadi: Do you mean data.finger-tree/counted-double-list?

cigitia20:11:10

I took a look at that too, but compared to rrb-vector, counted-double-list has worse lookup—`nth` has logarithmic time—and I don’t need to add things to the front end; rather, I need to remove them from the front end.

cigitia20:11:13

Unfortunately, rrb-vector’s front-end removal appears to be in logarithmic time.

cigitia20:11:46

Assuming that I have to use slicing to remove things from its front end.

cigitia20:11:36

I’m wondering if there are any better alternatives: something sequential with fast popping from the front end yet still constant-time lookup…but maybe there isn’t.

ghadi20:11:32

Make yourself a banker's queue with deftype & two vectors

ghadi20:11:52

Or don't pop off the front of a vec, instead keep an int as the front position

cigitia20:11:08

@ghadi: Ah, that would work too, except I want to free the memory of many items at the start of the queue efficiently.

cigitia20:11:10

To explain more: I’m looking for a true queue to serve as part of the state of a long-lasting stateful transducer (specifically one for a backtracking parser).

cigitia20:11:59

Because the parser transducer needs to backtrack, it needs to hold onto previously consumed input items in a buffer inside its state.

cigitia20:11:07

However, oftentimes, after passing certain points in the input, the parser transducer is guaranteed to not use the front parts of its buffer of saved input items; it doesn’t need them anymore.

cigitia20:11:50

I want to be able to free up the memory taken up by the now-unnecessary part of the buffer—

cigitia20:11:43

—essentially, I need a data structure that can release the items taking up its front (i.e., popping them) while being able to randomly access any particular items inside.

cigitia20:11:41

I could just use one vector for the transducer’s internal buffer, yes, but the vector will just keep growing and growing as the transducer consumes more input.

ghadi22:11:57

@cigitia sounds like a job for a mutable ArrayList

andy.fingerhut22:11:55

@cigitia: I'm not doing my homework before saying this, but I thought Clojure's PersistentQueue was implemented internally via two vectors? If so, it seems copying and pasting that implementation and adding an O(1) nth would be possible.

ghadi22:11:22

Shameless plug, I just wrote a PEG parser that deterministic backtracking...

ghadi22:11:37

Implemented with a virtual machine

bronsa22:11:07

@andy.fingerhut: a seq and a vector IIRC

ghadi23:11:48

@bronsa that sounds right...

ghadi23:11:53

I wonder why not two vecs

bronsa23:11:45

you can't pop from the head of a vector and keep it a vector

bronsa23:11:23

so IIRC it's always popping from a sequence and just using a vector when you push into the queue, eventually using that vector as a sequence

ghadi23:11:11

A straightforward impl is to invert the direction sense of the front vector

ghadi23:11:36

"Middle out"

ghadi23:11:03

Heh. Really "outside in"