Clojurians Log v2

@threepointone has joined the channel

@rockyj has joined the channel

Is there any way in Clojure to do something akin to a Haskell where you can alias a type so the type system thinks they are two disparate types. Use case - I have some coercion using schema, but I have to coerce some values differently depending on the context. The simplest way I can think of is to take (in my example) DateTime and somehow alias it to PayloadDateTime that it behaves exactly like DateTime, but I can match on a different class name.

Now that I have written it out loud it sounds like I could just inheritance or something, but I'm not aware of how to inherit a Java class in Clojure.

Welp, DateTime in joda time is final : |

@jaen: I usually just create a new defschema with a pred for that kind of functionality

@tbayer has joined the channel

I'm getting some really weird AOT behavior with 1.8, seems specific to my project

(pprint/pprint 4)
ClassCastException clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush  clojure.pprint/pretty-writer/fn--5686 (pretty_writer.clj:391)
> (isa? clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 clojure.pprint.PrettyFlush)
false
> (ancestors clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848)
#{java.lang.Appendable java.lang.Object java.io.Flushable clojure.lang.IProxy java.lang.AutoCloseable java.io.Closeable java.io.Writer clojure.lang.IDeref clojure.pprint.PrettyFlush}

both those classes are shipped with the clojure jar

but, then:

> (every? (partial isa? clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848) (ancestors clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848))
true

seems there's two copies of PrettyFlush in different classloaders

@gtrak: you've probably already done this, but if not wiping target would be my first place to start

more evidence:

> (.getClassLoader clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848)
#object[sun.misc.Launcher$AppClassLoader 0x14dad5dc "sun.misc.Launcher$AppClassLoader@14dad5dc"]

> (.getClassLoader clojure.pprint.PrettyFlush)
#object[clojure.lang.DynamicClassLoader 0x68859ee7 "clojure.lang.DynamicClassLoader@68859ee7"]

and PrettyFlush comes from a definterface call

this was the case for an earlier beta, I'd hoped I could ignore it simple_smile

this is unsettling: https://github.com/clojure/clojure/blob/clojure-1.7.0/src/clj/clojure/genclass.clj#L722

(defmacro gen-interface
  [& options]
    (let [options-map (apply hash-map options)
          [cname bytecode] (generate-interface options-map)]
      (when *compile-files*
        (clojure.lang.Compiler/writeClassFile cname bytecode))
      (.defineClass ^DynamicClassLoader (deref clojure.lang.Compiler/LOADER)
                    (str (:name options-map)) bytecode options)))

@gtrak: why is that unsettling? do you have a minimal repro for your issue?

I would expect it to look for an existing PrettyFlush interface in another classloader. I don't think I can make a minimal repro case without dropwizard, maven, etc..

eg, the definterface call in pprint always gens a class

so I'm leaning towards playing with clojure.lang.Compiler.LOADER and seeing if that affects anything

@erwanb has joined the channel

@ragge: for instance, get-proxy-class tries a RT/loadClassForName first: https://github.com/clojure/clojure/blob/clojure-1.7.0/src/clj/clojure/core_proxy.clj#L250

huh, but in a regular lein repl, PrettyFlush uses the appclassloader. anyways...

@gtrak: can you try and see if the issue is there for 1.7?

it's not in 1.7 because the structure of pprint itself has changed

the last commit to pprint is 2 years ago

look at the typehint for prettyflush: https://github.com/clojure/clojure/commit/0a6810ab3484b5be0afe4f505cd724eb5c974a09

I think that's what introduced the issue

in pprint_base

et al

I can give you a full stacktrace

ah I see

#error {
 :cause "clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush"
 :via
 [{:type java.lang.ClassCastException
   :message "clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 cannot be cast to clojure.pprint.PrettyFlush"
   :at [clojure.pprint$pretty_writer$fn__5686 invoke "pretty_writer.clj" 391]}]
 :trace
 [[clojure.pprint$pretty_writer$fn__5686 invoke "pretty_writer.clj" 391]
  [clojure.pprint.proxy$java.io.Writer$IDeref$PrettyFlush$4923d848 flush nil -1]
  [clojure.core$flush invokeStatic "core.clj" 3611]
  [clojure.core$flush invoke "core.clj" -1]
  [clojure.core$prn invokeStatic "core.clj" 3622]
  [clojure.core$prn doInvoke "core.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 397]
  [clojure.pprint$pprint invokeStatic "pprint_base.clj" 252]
  [clojure.pprint$pprint invoke "pprint_base.clj" -1]
  [clojure.pprint$pprint invokeStatic "pprint_base.clj" 245]
  [clojure.pprint$pprint invoke "pprint_base.clj" -1]
  [com.redowlanalytics.query.elasticsearch.mds$eval27851 invokeStatic "NO_SOURCE_FILE" 1]
  [com.redowlanalytics.query.elasticsearch.mds$eval27851 invoke "NO_SOURCE_FILE" -1]
  [clojure.lang.Compiler eval "Compiler.java" 6913]
  [clojure.lang.Compiler eval "Compiler.java" 6876]
  [clojure.core$eval invokeStatic "core.clj" 3107]
  [clojure.core$eval invoke "core.clj" -1]
  [clojure.main$repl$read_eval_print__4176$fn__4179 invoke "main.clj" 240]
  [clojure.main$repl$read_eval_print__4176 invoke "main.clj" 240]
  [clojure.main$repl$fn__4185 invoke "main.clj" 258]
  [clojure.main$repl invokeStatic "main.clj" 258]
  [clojure.main$repl doInvoke "main.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__20491 invoke "interruptible_eval.clj" 58]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 645]
  [clojure.core$apply invoke "core.clj" -1]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1892]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" -1]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 56]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" -1]
  [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__20533$fn__20536 invoke "interruptible_eval.clj" 191]
  [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__20528 invoke "interruptible_eval.clj" 159]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
  [java.lang.Thread run "Thread.java" 745]]}

but it's not really the type-hint's fault afaict

utilities is where PrettyFlush is defined, but that's loaded before pretty-writer

how can I reproduce this?

I have no idea. Unfortunately, it's inside a multi-module dropwizard project. I could try to work a minimal repro test-case, but I'm confused about the classloading.

It seems like if I can figure that out, then it'll work itself out.

classloading changed significantly in 1.7

we were on 1.7 before

http://dev.clojure.org/jira/browse/CLJ-979

what seems to happen is the definterface is being reloaded while the type-hint refers to the first version

@laurikin has joined the channel

I was interpreting it as the proxy is in a parent and its super is in a child classloader.

but for some reason that doesn't happen in a lein repl, PrettyFlush is in the app classloader instead of the dynamicclassloader

looks relevant: 'it changes gen-interface to always emit an in-memory interface along with the [optional] on-disk interface so that the in-memory class is always updated.'

yes, that was intentional

unfortunately knowing what you're doing to repro this I don't really have any way of trying to understand what's going on

right, I can work on that. basically, it's a dropwizard app that spins up a REPL during initialization.

using clojure.java.api

@gtrak: too many variables at play there, could be a user/tooling/config bug

@v.kolesnikov has joined the channel

I agree in principle, but I've been poking around at checksumming classfiles in the uberjar etc and it all checks out

@rauh: yeah, I figured that is what I'll do for now, so thanks for confirmation people do that. It would have been nice to be able to do some contextual validation/coercion with schema, but I would have to write more code since I don't think schema's walker supports that (or I'm missing something).

eg, what's in the uberjar is what's in clojure.jar

if I'm loading things a different way than I should be, then that's where the problem is

@gtrak: doesn't matter. if anything in there messes with classloading, it might get clojure into a inconsistent state

i.e. you have parts of pprint loaded AOT and parts reloaded JIT

@v.kolesnikov has left the channel

yea, so my next experiment is going to be to load everything via reflection simple_smile.

I wonder if class initialization is part of the problem

s/initialization/initializers

@j-po has joined the channel

I’m looking for a data structure that lets me remove things from its front, add things to its end, and access its items by indices efficiently. Regular vectors only let you efficiently remove things from its end, of course. clojure.core.PersistentQueue has linear-time nth, unfortunately. Is core.rrb-vector (https://github.com/clojure/core.rrb-vector) my best bet?

using Schema, is there a schema for truthy, without using s/pred?

@cigitia data.finger-tree

Is Persistentqueue really linear lookup?

It is, according to https://www.innoq.com/blog/st/2010/04/clojure-performance-guarantees/.

@ghadi: Do you mean data.finger-tree/counted-double-list?

Yeah

I took a look at that too, but compared to rrb-vector, counted-double-list has worse lookup—`nth` has logarithmic time—and I don’t need to add things to the front end; rather, I need to remove them from the front end.

Unfortunately, rrb-vector’s front-end removal appears to be in logarithmic time.

Assuming that I have to use slicing to remove things from its front end.

I’m wondering if there are any better alternatives: something sequential with fast popping from the front end yet still constant-time lookup…but maybe there isn’t.

Make yourself a banker's queue with deftype & two vectors

Or don't pop off the front of a vec, instead keep an int as the front position

@ghadi: Ah, that would work too, except I want to free the memory of many items at the start of the queue efficiently.

To explain more: I’m looking for a true queue to serve as part of the state of a long-lasting stateful transducer (specifically one for a backtracking parser).

Because the parser transducer needs to backtrack, it needs to hold onto previously consumed input items in a buffer inside its state.

However, oftentimes, after passing certain points in the input, the parser transducer is guaranteed to not use the front parts of its buffer of saved input items; it doesn’t need them anymore.

I want to be able to free up the memory taken up by the now-unnecessary part of the buffer—

—essentially, I need a data structure that can release the items taking up its front (i.e., popping them) while being able to randomly access any particular items inside.

I could just use one vector for the transducer’s internal buffer, yes, but the vector will just keep growing and growing as the transducer consumes more input.

@cigitia sounds like a job for a mutable ArrayList

@cigitia: I'm not doing my homework before saying this, but I thought Clojure's PersistentQueue was implemented internally via two vectors? If so, it seems copying and pasting that implementation and adding an O(1) nth would be possible.

Shameless plug, I just wrote a PEG parser that deterministic backtracking...

Implemented with a virtual machine

https://groups.google.com/forum/m/#!topic/clojure/2ph-6o_Zydc

@andy.fingerhut: a seq and a vector IIRC

@bronsa that sounds right...

I wonder why not two vecs

you can't pop from the head of a vector and keep it a vector

so IIRC it's always popping from a sequence and just using a vector when you push into the queue, eventually using that vector as a sequence

A straightforward impl is to invert the direction sense of the front vector

"Middle out"

Heh. Really "outside in"

@kephale has joined the channel

Hey Gang - ouch - ~/.lein/profiles.clj :java-cmd - didn't see that coming. The value was a removed version of Java. Once updated the path was found.

@martinparedes has joined the channel

what is the :scope "test" thing I see in some projects and for example om next tutorial, is this a leinengen feature?

syntax is pomegranate's, functionality is native to Maven - test dependencies are only pulled during testing. See https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html (Dependency Scopes) for more info