Fork me on GitHub
#datomic
<
2019-07-05
>
eoliphant22:07:51

Hi, I’m running into a situation on Cloud where we’re persistently getting busy indexing anomalies. upgraded to the latest rev, and have killed the transactors, but the problem hasn’t gone away

marshall23:07:55

@eoliphant Can you look in your CloudWatch logs for any Alerts

marshall23:07:06

if there are some, can you please share the text of the alerts

eoliphant23:07:25

yeah there are some. trying to pick out stuff that might be relevant, vs our apps messages

eoliphant23:07:48

nothing is really jumping out. that reminds me, lol, i meant to submit a feature request. it would be nice to keep the datomic system stuff in a separate log group. Ok, i’ve updated my filter, still nothing jumping out from datomic itself, it’s almost entirely our alerts that we’re logging when we retry/fail etc

marshall23:07:07

Are there any alerts at all that are not from your own use of cast?

eoliphant23:07:52

think i have something

{
    "Msg": "IndexerJobException",
    "Ex": {
        "Via": [
            {
                "Type": "clojure.lang.ArityException",
                "Message": "Wrong number of args (2) passed to: datomic.excise/pred",
                "At": [
                    "clojure.lang.AFn",
                    "throwArity",
                    "AFn.java",
                    429
                ]
            }
        ],
        "Trace": [
            [
                "clojure.lang.AFn",
                "throwArity",
                "AFn.java",
                429
            ],
            [
                "clojure.lang.AFn",
                "invoke",
                "AFn.java",
                36
            ],
            [
                "clojure.core$partial$fn__5824",
                "invoke",
                "core.clj",
                2624
            ],
            [
                "clojure.core$map$fn__5851",
                "invoke",
                "core.clj",
                2755
            ],
            [
                "clojure.lang.LazySeq",
                "sval",
                "LazySeq.java",
                42
            ],
            [
                "clojure.lang.LazySeq",
                "seq",
                "LazySeq.java",
                51
            ],
            [
                "clojure.lang.RT",
                "seq",
                "RT.java",
                531
            ],
            [
                "clojure.core$seq__5387",
                "invokeStatic",
                "core.clj",
                137
            ],
            [
                "clojure.core$seq__5387",
                "invoke",
                "core.clj",
                137
            ],
            [
                "datomic.index$merge_db$fn__21535",
                "invoke",
                "index.clj",
                1635
            ],
            [
                "datomic.index$merge_db",
                "invokeStatic",
                "index.clj",
                1621
            ],
            [
                "datomic.index$merge_db",
                "invoke",
                "index.clj",
                1615
            ],
            [
                "datomic.indexer$merge_db",
                "invokeStatic",
                "indexer.clj",
                185
            ],
            [
                "datomic.indexer$merge_db",
                "invoke",
                "indexer.clj",
                181
            ],
            [
                "datomic.indexer$maybe_queue_index_job$fn__28554",
                "invoke",
                "indexer.clj",
                250
            ],
            [
                "clojure.core$binding_conveyor_fn$fn__5739",
                "invoke",
                "core.clj",
                2030
            ],
            [
                "datomic.async$daemon$fn__10439",
                "invoke",
                "async.clj",
                146
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ],
        "Cause": "Wrong number of args (2) passed to: datomic.excise/pred"
    },
    "DatomicIndexerDbId": "5f06733b-f7c1-4a6f-9aab-3c665b7d498d",
    "Type": "Alert",
    "Tid": 595,
    "Timestamp": 1562369932970
}

eoliphant23:07:18

I’ve a lambda pulling stuff off of kinesis, so I turned that back on to generate some activity, these are popping up pretty frequently now

jaret23:07:22

@eoliphant I am going to open a ticket up in your name and copy this info over

eoliphant23:07:22

FYI the storage and compute are on 480-8770

jaret23:07:38

Production templates?