This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-05-19
Channels
- # ai (3)
- # aws (1)
- # beginners (94)
- # boot (26)
- # cider (3)
- # cljs-dev (99)
- # cljsrn (86)
- # clojure (263)
- # clojure-dusseldorf (4)
- # clojure-greece (22)
- # clojure-italy (2)
- # clojure-quebec (1)
- # clojure-russia (12)
- # clojure-spec (71)
- # clojure-uk (123)
- # clojurescript (92)
- # core-async (4)
- # cursive (13)
- # data-science (2)
- # datomic (123)
- # docker (2)
- # emacs (15)
- # events (1)
- # graphql (2)
- # hoplon (71)
- # jobs-discuss (7)
- # lumo (5)
- # off-topic (12)
- # om (6)
- # onyx (97)
- # other-languages (4)
- # overtone (2)
- # pedestal (1)
- # re-frame (20)
- # reagent (33)
- # remote-jobs (1)
- # ring-swagger (1)
- # rum (5)
- # slack-help (6)
- # uncomplicate (1)
- # unrepl (33)
- # untangled (48)
- # vim (23)
- # yada (21)
Huh interesting with the #db/fn
I found other stuff related to db/fn out there when I was googling datomic and delays
so maybe a db/fn was installed and we’re trying to pass it down and we can’t serialize it
Maybe something strange with the #db/fn
reader literal. Maybe there’s a problem with the tag reader?
#db/fn
?
Oh yea I’m talking about the reader function for all #db/fn
types
*data-readers*
So that #db/fn
datom gets parsed by datomic.function/construct
it looks like.
The callstack here seems to indicate that function gets wrapped in a delay https://groups.google.com/forum/#!topic/datomic/ba0o-G8CPZE
Good to have figured out what’s going on there. I could never reproduce it.
That's probably the simplest fix. If you could come up with a predicate that detects it in a segment I'll put it in onyx-datomic
Or a reproducing test case as part of onyx-datomic
@lucasbradstreet what's the code you're using to force serialization?
Sure. Gimme 10 mins. Just out and haven't pushed it anywhere
@devth https://github.com/onyx-platform/onyx-datomic/blob/90c2be2d4bea5f1ba843e4dc936ec042f53bd514/src/onyx/plugin/datomic.clj#L311
Hi guys, what's the correct command to generate an uberjar to use media-driver from lib-onyx project? Im trying lein uberjar lib-onyx.media-driver
but i have dependency errors.
I think it’ll need an update to the project.clj
@lellis, lib-onyx hasn’t been updated for a while. This might work for you.
:media-driver {:main onyx.messaging.aeron_media_driver
:aot [onyx.messaging.aeron_media_driver]
:prep-tasks ^:replace ["clean" "compile"]
:uberjar-name "mediadriver.jar"}
Let us know if that works. We might put it in lib-onyx
I wouldn’t mind dropping it out of lib-onyx and keeping it in Onyx core. One less dep to track for something where version matching Aeron is pretty important.
That’s a good point
I think it belongs in Onyx anyway.
As a new profile
Then lein with-profile media-driver uberjar I think
Now i got
lein with-profile media-driver uberjar
Compiling onyx.messaging.aeron_media_driver
java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver', compiling:(/private/var/folders/rm/__tkcysn7s530fwvy96c4npr0000gn/T/form-init2491997162488158336.clj:1:1060)
Exception in thread "main" java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver', compiling:(/private/var/folders/rm/__tkcysn7s530fwvy96c4npr0000gn/T/form-init2491997162488158336.clj:1:1060)
at clojure.core$throw_if.invokeStatic(core.clj:5656)
at clojure.core$load_one.invokeStatic(core.clj:5698)
at clojure.core$compile$fn__5682.invoke(core.clj:5903)
at clojure.core$compile.invokeStatic(core.clj:5903)
at clojure.core$compile.invoke(core.clj:5895)
at user$eval20$fn__29.invoke(form-init2491997162488158336.clj:1)
at user$eval20.invokeStatic(form-init2491997162488158336.clj:1)
at user$eval20.invoke(form-init2491997162488158336.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6927)
at clojure.lang.Compiler.eval(Compiler.java:6917)
at clojure.lang.Compiler.eval(Compiler.java:6917)
at clojure.lang.Compiler.load(Compiler.java:7379)
at clojure.lang.Compiler.loadFile(Compiler.java:7317)
at clojure.main$load_script.invokeStatic(main.clj:275)
at clojure.main$init_opt.invokeStatic(main.clj:277)
at clojure.main$init_opt.invoke(main.clj:277)
at clojure.main$initialize.invokeStatic(main.clj:308)
at clojure.main$null_opt.invokeStatic(main.clj:342)
at clojure.main$null_opt.invoke(main.clj:339)
at clojure.main$main.invokeStatic(main.clj:421)
at clojure.main$main.doInvoke(main.clj:384)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver'
... 4 more
Compilation failed: Subprocess failed
@lucasbradstreet haven't quite been able to get a repro test. tried this: https://github.com/devth/onyx-datomic/commit/ae66733807eef73b243b1cd43a247b721f0f264f
but take-segments!
is currently blocking forever. i've tried a bunch of variants of transacting people mixed in with the db-fn-tx
but i can never get the #db/fn
to come out the other side from take-segments!
.
actually, here's a minimal repro without going through an onyx job:
(messaging-compress
(d/function
'{:lang :clojure
:params [db]
:code (and 1)}))
Cool that's all I need
That's alright. I can work from here
@devth was it in the value key of the tuple?
@devth if you could run 0.9.15.1-20170519.065305-4
that’d be good. I added some assertions to tell us whether it’s in e, a, v, or t. I assume it’s v or somewhere else weird, but I want to confirm first
lucasbradstreet: looks like v
. here's the tx:
[17592186045419 52 #db/fn{:lang :clojure, :imports [], :requires [], :params [db norm-attr norm index-attr index tx], :code (when-not (seq (q (quote [:find ?tx :in $ ?na ?nv ?ia ?iv :where [?tx ?na ?nv ?tx] [?tx ?ia ?iv ?tx]]) db norm-attr norm index-attr index)) (cons {:db/id (d/tempid :db.part/tx), norm-attr norm, index-attr index} tx))} 13194139534314 true]
i didn't see any output from assertions that indicated which position it was in though. just the usual:
[idx-3769151588-2n9wl idx] ERROR COMPRESSING TXRANGE ENTRIES
[idx-3769151588-2n9wl idx] [{:id #uuid "591a05cc-9a78-4005-84f3-fad2cf9694da", :data ([131
...
Hi guys i need some help. I have some jobs killed by this exception
:status :success, :result #error {
:cause "Insufficient usable storage for new log of length=50332096 in /dev/shm (tmpfs)"
:data {:original-exception :io.aeron.exceptions.RegistrationException}
:via
[{:type clojure.lang.ExceptionInfo
:message "Insufficient usable storage for new log of length=50332096 in /dev/shm (tmpfs)"
:data {:original-exception :io.aeron.exceptions.RegistrationException}
:at [io.aeron.ClientConductor onError "ClientConductor.java" 264]}]
:trace
Its a aeron media problem? Disk espace? Any tip? I using :onyx.messaging.aeron/embedded-driver? trueThe nearest I got was having a split in the workflow going to two separate output channels.
only way I can think of is to do it via a normal function make the request and add the results to the segment to send to the output
@jasonbell well yes, but onyx-http already does retries and other nice stuff, so it would be nice to re-use it instead of doing all that on our own
@asolovyov agreed
Hi, i have a simple problem. I’m new to onyx and have created a small app that is trying to grab segments from datomic transaction log. The issue is that I’m not getting any logging to the console. my app starts up peers then submits my simple job. In trying to figure it out, I happened upon the datomic console, which has allowed me to see why onyx wasn’t happy about my job. But, i’d like to get it dumping to the console as well
also, now that i can see what’s going on via the console, I got past some configuration, etc issues, but now i’m seeing an issue whose cause is a little less obvious to me
#error {
:cause "Unfreezable type: class clojure.lang.Delay"
:data {:type clojure.lang.Delay, :as-str "#object[clojure.lang.Delay 0x19a380aa {:status :pending, :val nil}]", :original-exception :clojure.lang.ExceptionInfo}
:via
[{:type clojure.lang.ExceptionInfo
:message "Unfreezable type: class clojure.lang.Delay"
:data {:type clojure.lang.Delay, :as-str "#object[clojure.lang.Delay 0x19a380aa {:status :pending, :val nil}]", :original-exception :clojure.lang.ExceptionInfo}
:at [clojure.core$ex_info invokeStatic "core.clj" 4617]}]
:trace
[[clojure.core$ex_info invokeStatic "core.clj" 4617]
Any idea what’s might cause this?@eoliphant we're working through that very issue! the problem is when there's a Database Function in Datomic, the reader literal #db/fn
somehow gets encoded as a Delay
. @lucasbradstreet is working on a fix to skip over db fns when reading from the log
@eoliphant we discovered it yesterday, see thread around https://clojurians.slack.com/archives/C051WKSP3/p1495152432012567
@devth Did you bring this up in #datomic ?
Awesome
It’s not part of their contract, I couldn’t find anywhere that said you need to complete delays or maps returned from d/datoms
No, I think there’s a problem with the reader tag for #db/fn
Or the reader tag function rather
@asolovyov @jasonbell @camechis We discussed doing this a few months ago. It’s a little weird in that I didn’t intend for workflows to be structured with non-terminal outputs, but missing out on reusability of a plugin really sucks.
There aren’t many changes required to make it happen because function and output type tasks have the same lifecycle methods be invoked on them.
If someone wants to send in a patch for it, we can get it it merged.
I’ve not looked at the code base for a while but I’ll be back on some new Onyx jobs very soon. Will see what I can do around the edges as usual.
Looking back at my notes, the main things to do are:
- drop the validation requiring outputs sit in the leaf position of a workflow
- move the portion of code that writes to the output medium outside of the protocol and into a function
- invoke that function with :onyx/fn
Lifecycle uniformity takes care of most of the challenges.
If i have media-driver standalone the peer-config is this?
:onyx.messaging/bind-addr "10.0.1.251"
:onyx.messaging/peer-port 40200
:onyx.messaging.aeron/embedded-driver? false
As long as you have the stand alone driver running on the same machine
It's kind of like a TCP stack for each machine
@lucasbradstreet I made some changes in lib-onyx and now i can make lein uberjar ...
. If u want i can push these changes.
If you could push them up that'd be good. I'll move it over the onyx core though