Fork me on GitHub
#onyx
<
2017-05-19
>
devth00:05:46

ok back to the original Delay exception

devth00:05:00

I see a ton of datoms printed out but they don't contain a Delay

devth00:05:48

there is a #db/fn in one of the datoms

gardnervickers00:05:12

Huh interesting with the #db/fn

lucasbradstreet00:05:26

I found other stuff related to db/fn out there when I was googling datomic and delays

lucasbradstreet00:05:36

so maybe a db/fn was installed and we’re trying to pass it down and we can’t serialize it

gardnervickers00:05:56

Maybe something strange with the #db/fn reader literal. Maybe there’s a problem with the tag reader?

devth00:05:01

it comes from conformity

devth00:05:15

yep. it installs a fn

gardnervickers00:05:55

Oh yea I’m talking about the reader function for all #db/fn types

gardnervickers00:05:49

So that #db/fn datom gets parsed by datomic.function/construct it looks like.

gardnervickers00:05:48

The callstack here seems to indicate that function gets wrapped in a delay https://groups.google.com/forum/#!topic/datomic/ba0o-G8CPZE

lucasbradstreet00:05:46

Good to have figured out what’s going on there. I could never reproduce it.

devth00:05:21

sorry had to go afk for a bit. should onyx-datomic filter these out?

lucasbradstreet00:05:46

That's probably the simplest fix. If you could come up with a predicate that detects it in a segment I'll put it in onyx-datomic

lucasbradstreet00:05:13

Or a reproducing test case as part of onyx-datomic

devth00:05:47

sure, let me try to repro in a test

devth01:05:11

@lucasbradstreet what's the code you're using to force serialization?

lucasbradstreet01:05:10

Sure. Gimme 10 mins. Just out and haven't pushed it anywhere

lellis01:05:36

Hi guys, what's the correct command to generate an uberjar to use media-driver from lib-onyx project? Im trying lein uberjar lib-onyx.media-driverbut i have dependency errors.

lucasbradstreet01:05:29

I think it’ll need an update to the project.clj

gardnervickers01:05:29

@lellis, lib-onyx hasn’t been updated for a while. This might work for you.

:media-driver {:main onyx.messaging.aeron_media_driver
                        :aot [onyx.messaging.aeron_media_driver]
                        :prep-tasks ^:replace ["clean" "compile"]
                        :uberjar-name "mediadriver.jar"}

lucasbradstreet01:05:50

Let us know if that works. We might put it in lib-onyx

gardnervickers01:05:41

I wouldn’t mind dropping it out of lib-onyx and keeping it in Onyx core. One less dep to track for something where version matching Aeron is pretty important.

lucasbradstreet01:05:52

That’s a good point

lucasbradstreet01:05:59

I think it belongs in Onyx anyway.

lellis02:05:58

Where i have to put these info? As a profile? Inside my dev profile? at root?

lucasbradstreet02:05:22

As a new profile

lucasbradstreet02:05:49

Then lein with-profile media-driver uberjar I think

lellis02:05:00

Now i got

lein with-profile media-driver uberjar
Compiling onyx.messaging.aeron_media_driver
java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver', compiling:(/private/var/folders/rm/__tkcysn7s530fwvy96c4npr0000gn/T/form-init2491997162488158336.clj:1:1060)
Exception in thread "main" java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver', compiling:(/private/var/folders/rm/__tkcysn7s530fwvy96c4npr0000gn/T/form-init2491997162488158336.clj:1:1060)
        at clojure.core$throw_if.invokeStatic(core.clj:5656)
        at clojure.core$load_one.invokeStatic(core.clj:5698)
        at clojure.core$compile$fn__5682.invoke(core.clj:5903)
        at clojure.core$compile.invokeStatic(core.clj:5903)
        at clojure.core$compile.invoke(core.clj:5895)
        at user$eval20$fn__29.invoke(form-init2491997162488158336.clj:1)
        at user$eval20.invokeStatic(form-init2491997162488158336.clj:1)
        at user$eval20.invoke(form-init2491997162488158336.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6927)
        at clojure.lang.Compiler.eval(Compiler.java:6917)
        at clojure.lang.Compiler.eval(Compiler.java:6917)
        at clojure.lang.Compiler.load(Compiler.java:7379)
        at clojure.lang.Compiler.loadFile(Compiler.java:7317)
        at clojure.main$load_script.invokeStatic(main.clj:275)
        at clojure.main$init_opt.invokeStatic(main.clj:277)
        at clojure.main$init_opt.invoke(main.clj:277)
        at clojure.main$initialize.invokeStatic(main.clj:308)
        at clojure.main$null_opt.invokeStatic(main.clj:342)
        at clojure.main$null_opt.invoke(main.clj:339)
        at clojure.main$main.invokeStatic(main.clj:421)
        at clojure.main$main.doInvoke(main.clj:384)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at clojure.lang.Var.invoke(Var.java:383)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.Var.applyTo(Var.java:700)
        at clojure.main.main(main.java:37)
Caused by: java.lang.Exception: namespace 'onyx.messaging.aeron_media_driver' not found after loading '/onyx/messaging/aeron_media_driver'
        ... 4 more
Compilation failed: Subprocess failed

devth02:05:21

@lucasbradstreet haven't quite been able to get a repro test. tried this: https://github.com/devth/onyx-datomic/commit/ae66733807eef73b243b1cd43a247b721f0f264f but take-segments! is currently blocking forever. i've tried a bunch of variants of transacting people mixed in with the db-fn-tx but i can never get the #db/fn to come out the other side from take-segments!.

devth03:05:02

actually, here's a minimal repro without going through an onyx job:

(messaging-compress
  (d/function
    '{:lang :clojure
      :params [db]
      :code (and 1)}))

lucasbradstreet03:05:10

Cool that's all I need

devth03:05:28

ok, i'll submit that

lucasbradstreet03:05:05

That's alright. I can work from here

lucasbradstreet03:05:42

@devth was it in the value key of the tuple?

lucasbradstreet06:05:30

@devth if you could run 0.9.15.1-20170519.065305-4 that’d be good. I added some assertions to tell us whether it’s in e, a, v, or t. I assume it’s v or somewhere else weird, but I want to confirm first

devth13:05:20

lucasbradstreet: looks like v. here's the tx:

[17592186045419 52 #db/fn{:lang :clojure, :imports [], :requires [], :params [db norm-attr norm index-attr index tx], :code (when-not (seq (q (quote [:find ?tx :in $ ?na ?nv ?ia ?iv :where [?tx ?na ?nv ?tx] [?tx ?ia ?iv ?tx]]) db norm-attr norm index-attr index)) (cons {:db/id (d/tempid :db.part/tx), norm-attr norm, index-attr index} tx))} 13194139534314 true]

devth13:05:01

i didn't see any output from assertions that indicated which position it was in though. just the usual:

[idx-3769151588-2n9wl idx] ERROR COMPRESSING TXRANGE ENTRIES
[idx-3769151588-2n9wl idx] [{:id #uuid "591a05cc-9a78-4005-84f3-fad2cf9694da", :data ([131
...

lellis14:05:13

Hi guys i need some help. I have some jobs killed by this exception

:status :success, :result #error {
 :cause "Insufficient usable storage for new log of length=50332096 in /dev/shm (tmpfs)"
 :data {:original-exception :io.aeron.exceptions.RegistrationException}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Insufficient usable storage for new log of length=50332096 in /dev/shm (tmpfs)"
   :data {:original-exception :io.aeron.exceptions.RegistrationException}
   :at [io.aeron.ClientConductor onError "ClientConductor.java" 264]}]
 :trace
Its a aeron media problem? Disk espace? Any tip? I using :onyx.messaging.aeron/embedded-driver? true

jasonbell14:05:41

@lellis I’d give the shm-size at least 1GB if you are using Onyx 10.x

asolovyov14:05:22

why can't you have output after output? 🙂

asolovyov14:05:38

I'd really want to write something to Kafka after sending an http request 🙂

jasonbell14:05:45

I’ve wanted that as well, never got round to figuring a way out.

jasonbell14:05:21

The nearest I got was having a split in the workflow going to two separate output channels.

jasonbell14:05:40

but not -> :out-kafka -> :out-http

asolovyov14:05:44

eh, but I need http response in order to generate kafka message

jasonbell14:05:15

So that’s just part of the workflow then

Travis14:05:19

only way I can think of is to do it via a normal function make the request and add the results to the segment to send to the output

asolovyov14:05:58

@jasonbell well yes, but onyx-http already does retries and other nice stuff, so it would be nice to re-use it instead of doing all that on our own

asolovyov14:05:18

@camechis I guess that's what we're going to do in the end

eoliphant15:05:19

Hi, i have a simple problem. I’m new to onyx and have created a small app that is trying to grab segments from datomic transaction log. The issue is that I’m not getting any logging to the console. my app starts up peers then submits my simple job. In trying to figure it out, I happened upon the datomic console, which has allowed me to see why onyx wasn’t happy about my job. But, i’d like to get it dumping to the console as well

eoliphant15:05:47

also, now that i can see what’s going on via the console, I got past some configuration, etc issues, but now i’m seeing an issue whose cause is a little less obvious to me

#error {
 :cause "Unfreezable type: class clojure.lang.Delay"
 :data {:type clojure.lang.Delay, :as-str "#object[clojure.lang.Delay 0x19a380aa {:status :pending, :val nil}]", :original-exception :clojure.lang.ExceptionInfo}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Unfreezable type: class clojure.lang.Delay"
   :data {:type clojure.lang.Delay, :as-str "#object[clojure.lang.Delay 0x19a380aa {:status :pending, :val nil}]", :original-exception :clojure.lang.ExceptionInfo}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4617]}]
 :trace
 [[clojure.core$ex_info invokeStatic "core.clj" 4617]
Any idea what’s might cause this?

devth15:05:54

@eoliphant we're working through that very issue! the problem is when there's a Database Function in Datomic, the reader literal #db/fn somehow gets encoded as a Delay. @lucasbradstreet is working on a fix to skip over db fns when reading from the log

eoliphant15:05:15

lol #devth Yeah I’d just searched and found you guys going back and forth about it

gardnervickers15:05:50

@devth Did you bring this up in #datomic ?

devth15:05:07

i also opened an issue with cognitect

devth15:05:23

but not sure if it's something they would "fix"

devth15:05:29

i.e. is it an actual problem?

gardnervickers15:05:04

It’s not part of their contract, I couldn’t find anywhere that said you need to complete delays or maps returned from d/datoms

devth15:05:32

i think it's related to the encoding of #db/fn itself right?

devth15:05:37

not the outer datom that contains it

devth15:05:57

normally it'd be completely transparent to the user

devth15:05:01

until you try to serialize it 😂

gardnervickers15:05:11

No, I think there’s a problem with the reader tag for #db/fn

gardnervickers15:05:21

Or the reader tag function rather

devth15:05:37

ok, i'll mention that in my open issue

michaeldrogalis15:05:05

@asolovyov @jasonbell @camechis We discussed doing this a few months ago. It’s a little weird in that I didn’t intend for workflows to be structured with non-terminal outputs, but missing out on reusability of a plugin really sucks.

michaeldrogalis15:05:32

There aren’t many changes required to make it happen because function and output type tasks have the same lifecycle methods be invoked on them.

michaeldrogalis15:05:55

If someone wants to send in a patch for it, we can get it it merged.

jasonbell15:05:42

I’ve not looked at the code base for a while but I’ll be back on some new Onyx jobs very soon. Will see what I can do around the edges as usual.

michaeldrogalis16:05:14

Looking back at my notes, the main things to do are: - drop the validation requiring outputs sit in the leaf position of a workflow - move the portion of code that writes to the output medium outside of the protocol and into a function - invoke that function with :onyx/fn

michaeldrogalis16:05:28

Lifecycle uniformity takes care of most of the challenges.

lellis19:05:55

If i have media-driver standalone the peer-config is this?

:onyx.messaging/bind-addr              "10.0.1.251"
                                                                              :onyx.messaging/peer-port              40200
                                                                              :onyx.messaging.aeron/embedded-driver? false

lellis19:05:56

and others peers will connect to this media-driver machine right?

lucasbradstreet19:05:39

As long as you have the stand alone driver running on the same machine

lellis19:05:00

Its not on the same machine.

lellis19:05:26

I need one media running in each machine?

lucasbradstreet19:05:12

It's kind of like a TCP stack for each machine

lellis19:05:15

Roger That!

lellis19:05:53

@lucasbradstreet I made some changes in lib-onyx and now i can make lein uberjar .... If u want i can push these changes.

lucasbradstreet19:05:58

If you could push them up that'd be good. I'll move it over the onyx core though

eoliphant22:05:12

@devth so just for my edification, this issue is pretty much a show stopper or is there some sort of workaround?