Clojurians Log v2

I've added separate functions registered to trigger clean up:

(defn clear-logs [& _] (reset! tap-list '()) filtered-logs)
(portal/register! #'clear-logs)

(defn get-logs
  "Returns logs after applying all filters in order."
  []
  (reduce (fn [acc f] ((:f f) acc)) @tap-list @filters))

(defonce filtered-logs (atom (get-logs)))

(defn- id-filter-fn
  "Returns a function that removes logs with the given log-id."
  [log-id]
  (fn [logs] (remove #(= (:result %) log-id) logs)))

(defn id-filter
  "Creates a filter that removes logs with the given log-id(s).
   Accepts a map with :id, a collection of maps with :id, or a single id."
  {:shortcuts [["i"]]}
  [& args]
  (when (= 1 (count args))
    (let [arg (first args)
          ids (cond
                (map? arg) [(:id arg)]
                (coll? arg) (mapv :id arg)
                :else [arg])
          ids (filter some? ids)]
      (doseq [log-id ids]
        (let [filter-id (keyword (gensym "id-filter-"))]
          (add-filter {:id filter-id
                       :f (id-filter-fn log-id)
                       :log-id log-id})))))
  filtered-logs)

I just doubled checked, reducing it to 100 has no impact, I get the same runtime disconnected etc.

Ohh that's interesting, thanks for explaining. How big would you say each log entry is? I wonder if it's an issue with the size of the data, or if Portal doesn't like something in particular with the log data. 🤔

My guess as to where Portal might be running into issues is https://github.com/djblue/portal/blob/master/src/portal/runtime.cljc#L119 which was added to help better distinguish different collection types and metadata. You could try patching it locally with the following to see if it helps:

(in-ns 'portal.runtime)
(def hash+ hash)

Initially I was doing separate both atom and window for logs. Where logs were added without tap> with just conj to the atom and portal was as a viewer. But with high volume it was breaking whole app. Now I've moved to triggering tap> on log (with custom submit function that conj to the atom). But we are still having that issue when there is to many logs or to big data on them. I think we will need to just keep portal separate from logs because it's not performant enough and connection to portal breaks. And when it not breaks but is high volume it just comes very slowly so it's a bit annoying.

(declare portal-logs)

(defn on-load [] (p/eval-str portal-logs (slurp (io/resource "custom_viewer.cljs"))))

(defn open-portal-logs
  "Call this to get logs open in portal view example usage in start-portal below"
  ([] (open-portal-logs nil))
  ([params]
   (def portal-logs
     (p/open (merge {:window-title "Logs Viewer"
                     :on-load on-load
                     :value log/tap-list} (when (and params (map? params)) params))))))

(defn start-portal!
  [LPORTAL-env]
  (let [portal-configuration (or
                                (some-> LPORTAL-env
                                 edn/read-string)
                                {})]
      (open-portal-logs portal-configuration)
      (add-tap #'log/submit)
      (tap> [:connected-repl (java.util.Date.)])))

And than in my logs namespace:

(defn signal->portal
  [t]
  (if (map? t)
    (let [{:keys [id uid level error msg_ data inst ns coords kind run-nsecs parent root]
           :as log}
          t]
      (with-meta
        (merge {:level level
                :kind (if (= :trace kind) :spy kind)
                :ns (symbol (or ns "?"))
                :line (first coords)
                :column (second coords)
                :result id} (when parent {:parent-id (:id parent)})
         (when (and root (not= (:id parent) (:id root)) (not= id (:id root))) {:root-id (:id root)})
         (when-let [msg (force msg_)] {:msg msg}) (when error {:error (d/datafy error)})
         (when-let [ts (force inst)]
           {:time #?(:clj (java.util.Date/from inst)
                     :cljs (js/Date.))}) (when run-nsecs {:run-nsecs run-nsecs}) (when data
                                                                                   {:data data})
         (when parent {:parent parent}) (when root {:root root}))
        {:portal.viewer/default :lekta.viewer/log-viewer
         :portal.viewer/for {:form :portal.viewer/pprint
                             :time :portal.viewer/relative-time}}))
    t))

(def tap-list
  (atom (with-meta '() {:portal.viewer/default :lekta.viewer/log-viewer
                        :portal.viewer/for {:form :portal.viewer/pprint
                                            :time :portal.viewer/relative-time}})))

(defn submit [value] (swap! tap-list (fn [taps] (conj taps (signal->portal value)))))

(defn portal-log
  ([] (reset! tap-list []))
  ([log] (tap> log)))

We are using telemere for logs, but it's not problem with telemere. Because other handlers do not break, console and file saving works without problem with this amount of logs we have.

It was just very pleasing idea to have logs in portal, and see everything that happens in the app, saving time on initial tap> and finding out what happened. But I don't think it's a portal problem as it was not designed as a log viewer, so I think we need to figure out where to view our logs and let tap> be just manually written debug tap>

Ohh, it looks like you are doing almost everything in the guide aside from capping the list itself. Is there a reason you want your entire log history available in Portal? I not only cap the number of entries to the 100 most recent, but also filter out noisy periodic notifications. Also, how big is your log list getting?

Hi, sorry for responding so late. So the problem happens at the newest portal version 0.61.0 I've nailed it down, the problem is casued by two things either it's big data on tap> or a lot of tap> happening. It's because I've connected our logs to portal (so each time we got a log I would do tap>), and it's just seem to be to much.

Something that might help is putting logs in their own atom and having an upper limit to the number of logs you keep. You can start with 100 and increase the number to see how portal behaves. https://cljdoc.org/d/djblue/portal/0.61.0/doc/guides/logging/timbre is how you can set that up.

The https://github.com/djblue/portal/blob/master/src/portal/ui/connection_status.cljs#L19-L37 works by pinging the runtime periodically on a 5 second interval. So I would check the web socket connection first. If that is working, there may be in issue with how the message is being https://github.com/djblue/portal/blob/master/src/portal/runtime.cljc#L455

If the web socket connection is failing to connect, the issue could be that the port isn't being communicated correctly to the client.

Also, what version are you experiencing this issue with?