Fork me on GitHub
#clojure
<
2023-11-20
>
Ben Lieberman17:11:37

Hello. Question re: clojure.core/iteration: I am using it to interact with a Java API. I don't have much to go on so I closely followed the example available on ClojureDocs. But I get a ClassCastException . Code in thread

Ben Lieberman17:11:45

(def paged-queries (atom {"a" {:query (SoqlQuery. nil nil nil nil nil nil (int 0) (int 14)) :next "b"}
                          "b" {:query (SoqlQuery. nil nil nil nil nil nil (int 14) (int 14)) :next "c"}
                          "c" {:query (SoqlQuery. nil nil nil nil nil nil (int 28) (int 14))}}))

(defn get-paginated-data [init]
  (let [query (get @paged-queries init)
        resp (.query ^Soda2Consumer consumer "aws9-xwqm.json" HttpLowLevel/JSON_TYPE ^SoqlQuery query)]
    (.readEntity resp java.util.ArrayList)))

(iteration get-paginated-data :initk "a" :vf :query :kf :next)

Ben Lieberman17:11:52

It says I can't cast a PersistentArrayMap to a SoqlQuery which makes sense but shouldn't :initk be accessing a which then looks in :query and gives me the instance of SoqlQuery

hiredman18:11:35

What is the exception

hiredman18:11:16

A place to start would be the doc string for iteration talks about stuff being passed to initk, so initk should be a function

hiredman18:11:58

Misread that

hiredman18:11:27

Ah you are type hinting query to be a String, which it clearly is not

Ben Lieberman20:11:14

yeah that was a mistake, I changed the hint back to SoqlQuery but the error remains:

; Execution error (ClassCastException) at scratch/get-paginated-data (scratch.clj:22).
; class clojure.lang.PersistentArrayMap cannot be cast to class com.socrata.model.soql.SoqlQuery (clojure.lang.PersistentArrayMap and com.socrata.model.soql.SoqlQuery are in unnamed module of loader 'app')

scratch/get-paginated-data (scratch.clj:22)
scratch/get-paginated-data (scratch.clj:20)
clojure.core/iteration (core.clj:7877)
clojure.lang.RT/seqFrom (RT.java:543)
clojure.lang.RT/seq (RT.java:537)
clojure.lang.RT/iter (RT.java:613)
clojure.core/sequence (core.clj:2680)
clojure.core/sequence (core.clj:2664)
scratch/eval7810 (NO_SOURCE_FILE:28)
scratch/eval7810 (NO_SOURCE_FILE:27)
clojure.lang.Compiler/eval (Compiler.java:7177)
clojure.core/eval (core.clj:3221)
clojure.core/eval (core.clj:3217)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.core/apply (core.clj:667)
clojure.core/with-bindings* (core.clj:1990)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.main/repl (main.clj:438)
clojure.main/repl (main.clj:459)
clojure.main/repl (main.clj:369)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:84)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:56)
nrepl.middleware.interruptible-eval/interruptible-eval (interruptible_eval.clj:152)
nrepl.middleware.session/session-exec (session.clj:218)
nrepl.middleware.session/session-exec (session.clj:217)
java.lang.Thread/run (Thread.java:833)
I was able to get the code to execute by doing (:query (get @paged-data init))but it returns nothing. Should each step be passed a version of the paged-data atom? That seems like the only thing that makes sense. So on each iteration I could update the atom with the return from .query?

hiredman20:11:44

No, the error has nothing to do with iteration, it is bad data getting passed to your method call, the error says you are passing a map where it should be whatever query object

hiredman20:11:52

It is because you are calling the map you look up query, when the query object is actually under the :query key in the map

hiredman20:11:33

You'll get the same error if you just call get-paginated-data with "a"

hiredman20:11:55

But once you get that sorted out, you will hit your error with iteration

hiredman20:11:21

Iteration is reduce in reverse, so for reduce the reducing function gets an accumulator and an item from the input, the iteration step function takes an accumulator and needs to return something that is a a composite of an accumulator and an item for the output

hiredman20:11:38

"a" is obviously not an accumulator, and the result of your step function also doesn't include an accumulator (something that can be fed in to the next call to the step function)

hiredman21:11:42

More concretely with a paginated api, the iteration step function takes a page of results and returns the next page of results

hiredman21:11:02

Where a page of results is some combination of results and a pointer to the next page

hiredman21:11:00

The other values you pass in (vf, kf, etc) let you tweak that contract a little, but that is basically it, and I would start with not passing in the other stuff, get the basic contract working

👀 1
Ben Lieberman21:11:07

that makes sense. In the example online, each of the values returned by :vf is a vector, so that satisfies the contract you are talking about. And I would need something similar I think.

hiredman21:11:29

Not exactly, the input to vf is a page (the return value from the step)

hiredman21:11:51

So the input to vf is logically a pair of the data and the pointer to the next page, the point of vf is to only include the data in the output

hiredman21:11:28

If you think of iteration as producing a sequence of outputs of step, vf like it is wrapped in (map vf (iteration ...))

1
Ben Lieberman21:11:32

That hint about the pair of (page, pointer) was helpful, thanks. I got it figured out