Fork me on GitHub
#beginners
<
2023-05-26
>
Abhi K05:05:41

Hi All, Below code calculates the status correctly but runs into NPE, Exception in thread "async-thread-macro-2" Exception in thread "async-thread-macro-4" java.lang.NullPointerException at clojure.core.async.impl.channels$chan$fn__5906.invoke(channels.clj:300) Can you please help me understand if the syntax is correct and pipeline has been called correctly:

(defn get-all-records-with-status [context filtered-port-data target-study-data
                                   actual-record-map]
  (let [split-list (reduce #(let [c-s-id (or (-> (get actual-record-map
                                                      [(:dbid %2) (:sid %2)])
                                                 :c_s_id)
                                             "NA")]
                              (if (contains? target-study-data [(:dbid %2) (:sid %2)])
                                (assoc %1 :with-target
                                          (conj (:with-target %1)
                                                (let [chan (async/chan 1)
                                                      parallelism 5
                                                      data (assoc %2 :csid c-s-id)
                                                      data-processor
                                                      (fn []
                                                        (let [status (try (get-status-message-match context (:uuid data)
                                                                                                    (:dbid data) (:sid data))
                                                                          (catch Exception e
                                                                            (log/error e (.getMessage e)
                                                                                       (format "for record: db-id %d, study-id %d" (:dbid data) (:sid data)))
                                                                            (log/error (Throwable->map e))
                                                                            "ERROR"))]
                                                          (assoc data :actualvstarget status)))]
                                                  (async/pipeline parallelism chan (data-processor) (async/to-chan! data)))))
                                (assoc %1 :without-target (conj (:without-target %1) (assoc %2 :actualvstarget "NA" :csid c-s-id)))))
                           {:without-target [] :with-target []} filtered-port-data)]
    (concat (:without-target split-list)
            (->> (:with-target split-list)
                 (async/merge)
                 (async/reduce conj [])
                 (async/<!!)))))
TIA!!

hiredman05:05:53

That's a big blob of code and I am on my phone, but the usage of pipeline-async is nonsense, what you are doing with data-processor there is entirely made up, I would check the docstring for pipeline-async to see what argument is pass there

hiredman05:05:04

You also seem to be expecting the return value of pipeline-async to be some transformation of the data your poured into it, which it is not

Abhi K13:05:09

pipeline is supposed to make the call to data-processor n number of times in parallel, during the run of data-processor the channel buffer somehow becomes null, I am not closing the channel explicitly, thanks for your response.

hiredman15:05:08

The channel buffer does not become nil

hiredman15:05:42

You are passing the result of calling the data-processor function to pipeline-async (looks like a map?) instead of a function

hiredman15:05:25

The return value of pipeline-async is a channel that closes when the go loop running the pipeline exits

hiredman15:05:47

The to chan, where pipeline writes its output you are just ignoring and doing nothing with

hiredman15:05:36

I think you need to stop, be clear about what you are implementing first, maybe write a simple single threaded version, then revisit core.async

hiredman15:05:20

pipeline-async is almost certainly not what you want here anyway, pipeline-async is typically for when you want to do some core.async stuff in parallel but you have something like a specific executor you want to shift the execution to

👍 2
hiredman15:05:54

But you might want to use pipeline-async here anyway so you can execute blocking io in a good context for it, so you may need to introduce an executor for that, but this is already such a mess adding more complicated things (an executor) is just going to make it more so

Abhi K16:05:46

Initial code had executor :

(.execute @thread-pool-executor
                                                            (fn []
                                                              (try
                                                                (async/put! chan (data-processor))
                                                                (finally
                                                                  (async/close! chan)))))
I was trying to replace it with pipeline, to see if the performance improves

jwind10:05:35

Cursive is formatting defn like this out of the box in CLJS 😞 how would i fix this?

jwind10:05:26

I suspect it's related to the fact that none of my autocompletions are working... defn can't be unresolved lol. I am an absolute beginner. Working on a kit-clj projeect w/cljs frontend

kennytilton10:05:49

I hate it when this happens. 🙂 And it always waits until I have forgotten the fix. I think it is in the Cursive FAQ, even. Have you tried File/Invalidate caches..., taking all the defaults? And are you getting any warnings from Intelli? You can also try the #C0744GXCJ channel.

leifericf13:05:00

Is this considered a reasonable way of implementing functions with optional parameters that have default values? :thinking_face:

(defn get-projects
  ([] (get-projects ""))
  ([org] (shell (str "az devops project list --org " org))))

Alex Miller (Clojure team)13:05:01

it is, but you might also want to consider an options map or kwargs - if you have one option now, you might have more later...

💡 2
👍 2
mister_m20:05:19

Is there anything like a python PEP8 for clojure outlining a thing such as "best practices" for documenting functions? If there are any community resources with such recs I'd be happy to see them

Bob B20:05:06

can't say I'm super-familiar with 'PEP8', but there's a clojure style guide at <https://guide.clojure.style/>

mister_m20:05:33

ah I just found that !