Fork me on GitHub
#clojure
<
2023-03-13
>
James04:03:00

I am trying to build some functions to upload and download images to mongodb using monger and grid-fs, and I am using reitit and swagger-ui for the routes. I am able to upload the images to the mongo/gridfs database, that works fine. I am running into problems when trying to download the images in swagger. In a gridfs namespace I have the following functions:

(ns guestbook.db.grid-fs
  (:require
   [monger.core :as mg :refer
    [connect connect-via-uri disconnect get-db get-gridfs]]
   [monger.gridfs :as mgfs :refer
    [store-file make-input-file filename content-type metadata]]
   [monger.collection :as collection]
   [monger.conversion :as conv]
   [basevan.config :refer [env]]
   [ :as io]
   [ring.util.http-response :as response])
  (:import
   [java.nio.file Paths]))


(defn connect-grid-fs []
  (connect {:uri (env :database-url)}))

(defn upload [file body]
  (let [conn (connect-grid-fs)
        db   (get-db conn "base")
        fs   (get-gridfs conn "base")
        {:keys [tempfile size filename content-type]} file]
    (store-file (make-input-file fs body)
                (mgfs/filename filename)
                (mgfs/metadata {:format (get-extension filename)})
                (mgfs/content-type content-type))))

(defn download [file]
  (let [conn (connect-grid-fs)
        db   (get-db conn "base")
        fs   (get-gridfs conn "base")]
    (mgfs/find-by-filename fs file)))
And for the routes I have imported ring, reitit, swagger, muuntaja etc, now the upload works fine but the download does not, the relevant snippets for the routes are as follows:
(ns guestbook.routes.services
  (:require
   [guestbook.auth :as auth]
   [guestbook.db.core :as db]
   [guestbook.db.grid-fs :as grid-fs]
   [guestbook.listings.handlers :as listings]
   [guestbook.auth.ring :refer
    [wrap-authorized get-roles-from-match]]
   [guestbook.media :as media]
   [guestbook.middleware :as middleware]
   [guestbook.middleware.formats :as formats]
   [guestbook.responses :as responses]
   [ :as io]
   [clojure.tools.logging :as log]
   [clojure.spec.alpha :as s]
   [clojure.string :as string]
   [reitit.swagger :as swagger]
   [reitit.swagger-ui :as swagger-ui]
   [reitit.ring.coercion :as coercion]
   [reitit.coercion.spec :as spec-coercion]
   [reitit.ring.middleware.muuntaja :as muuntaja]
   [reitit.ring.middleware.exception :as exception]
   [reitit.ring.middleware.multipart :as multipart]
   [reitit.ring.middleware.parameters :as parameters]
   [spec-tools.data-spec :as ds]
   [ring.middleware.cors :refer [wrap-cors]]
   [ring.util.response :as rr]
   [ring.util.http-response :as response]
   [ring.middleware.multipart-params :as multipart-params]))

;; At the top of the routes I have this:

{:middleware [;; query-params & form-params
                 parameters/parameters-middleware
                 ;; content-negotiation
                 muuntaja/format-negotiate-middleware
                 ;; encoding response body
                 muuntaja/format-response-middleware
                 ;; exception handling
                 (exception/create-exception-middleware
                  (assoc exception/default-handlers ::exception/wrap
                         (fn [handler e request]
                           (.printStackTrace e)
                           (handler e request))))
                 ;; cors handling TODO double check this
                 [wrap-cors
                  :access-control-allow-origin [#"" #""] ; <-- order matters
                  :access-control-allow-methods [:get :post :put :patch :delete]
                  :Access-Control-Allow-Credentials "true"]
                 exception/exception-middleware
                 ;; decoding request body
                 muuntaja/format-request-middleware
                 ;; coercing response bodys
                 coercion/coerce-response-middleware
                 ;; coercing request parameters
                 coercion/coerce-request-middleware
                 ;; multipart params
                 multipart/multipart-middleware
                 ;;Our auth middleware
                 (fn [handler]
                   (wrap-authorized
                    handler
                    (fn handle-unauthorized [req]
                      (let [route-roles (get-roles-from-match req)]
                        (log/debug
                         "Roles for route: "
                         (:uri req)
                         route-roles)
                        (log/debug "User is unauthorized!"
                                   (-> req
                                       :session
                                       :identity
                                       :roles))
                        (response/forbidden
                         {:message
                          (str "User must have one of the following roles: "
                               route-roles)})))))]
    :muuntaja formats/instance
    :coercion spec-coercion/coercion
    :swagger {:id ::api}}

;; and the two routes I have for the upload and download are below, and like I said the upload works fine, it is the download that is giving me problems:

["/media-name/:name"
    {::auth/roles (auth/roles :media/get)
     :get {:parameters
           {:path {:name string?}}
           :handler (fn [{{{:keys [name]} :path} :parameters}]
                      (if-let [{:keys [data type]}
                               (grid-fs/download {:name name})]
                        (-> (io/input-stream data)
                            (response/ok)
                            (response/content-type type))
                        (response/not-found)))}}]

["/media/upload"
    {::auth/roles (auth/roles :media/upload)
     ;; :conflicting true
     :post
     {:summary "upload a file"
      :parameters {:multipart {:file multipart/temp-file-part}}
      :handler (fn [{:keys [params body]}]
                 (let [file (:file params)]
                   (grid-fs/upload file body)))}}]
but when I try and retrieve the image in the browser using swagger-ui I get this error in the repl: --- :response--- {:body {:class "java.lang.IllegalArgumentException", :type "exception"}, :status 500} --- :response :reitit.ring.middleware.exception/exception --- {:body {:class "java.lang.IllegalArgumentException", :type "exception"}, :status 500} --- :response :reitit.ring.middleware.muuntaja/format-response --- {:body {:class "java.lang.IllegalArgumentException", :type "exception"}, :status 500} --- :response :reitit.ring.middleware.muuntaja/format-negotiate --- {:body -{:class "java.lang.IllegalArgumentException", :type "exception"} +#<http://java.io.ByteArrayInputStream@b346a63>, :status 500, +:headers {"Content-Type" "application/json; charset=utf-8"}} --- :response :reitit.ring.middleware.parameters/parameters --- {:body #<http://java.io.ByteArrayInputStream@b346a63>, :headers {"Content-Type" "application/json; charset=utf-8"}, :status 500} --- :response--- {:body #<http://java.io.ByteArrayInputStream@b346a63>, :headers {"Content-Type" "application/json; charset=utf-8"}, :status 500} 2023-03-13 03:49:23,081 [cluster-ClusterId{value='640d9b7b69d2d064fa4b2c43', description='null'}-66.88.197.45:20017] DEBUG org.mongodb.driver.protocol.command - Sending command '{ "ismaster" : 1, "$db" : "admin" }' with request id 25466 to database admin on connection [connectionId{localValue:36, serverValue:1159}] to server 66.88.197.45:20017 And the following error in the response in the browser: Server response Code Details 500 Error: Internal Server Error Response body Download { "type": "exception", "class": "java.lang.IllegalArgumentException" } Response headers content-length: 65 content-type: application/json; charset=utf-8 date: Mon,13 Mar 2023 03:49:22 GMT server: http-kit x-content-type-options: nosniff x-frame-options: SAMEORIGIN Please advise

rolt09:03:21

the server side strack trace should have more info. But from what i'm seeing i don't think you want to set the response content type if you're streaming the result: it should be application/octet-stream

p-himik11:03:42

Note that such long questions should be put in a thread or at least have their long code blocks turned into attachments so that the question itself doesn't span a few screens.

👍 2
James11:03:02

Noted, Apologies, new to this forum 🙂

James11:03:09

THanks for the reply also

kwladyka17:03:06

write *e in REPL to see stacktrace

kwladyka17:03:29

{:body {:class "java.lang.IllegalArgumentException",
          :type "exception"},
   :status 500}
doesn’t tell too much

kwladyka17:03:02

unless you catch this somewhere, then log it with full stacktrace

emilaasa10:03:22

I'd like to find out if I have unused functions in a library that has quite a large API, and several other internal projects that using it. They are all leinigen projects, and reside in under different folders/repos. Something like this, where both app1 and app2 use lib1:

emilaasa10:03:59

A couple of ideas I'm going to try: • configure LSP to handle the multiple projects • look at lein yagni

emilaasa11:03:08

Ah, how do I feed it the entry points from the apps?

rolt11:03:08

you pass all the source folders to the tool

rolt12:03:40

which reminds me: when I was doing the same task I used https://github.com/borkdude/carve in report mode with emacs

borkdude13:03:36

carve can do this: just give it the paths to everything that should be in scope and then it figures out the unused vars within that collection of code, it doesn't care about lein, just throw the source code at it

emilaasa13:03:19

Ah interesting. I found this in @U05254DQM’s awesome pro-files!

;; Carve - search through code for unused vars and remove them
  ;; clojure -M:search/unused-vars --opts '{:paths ["src" "test"]}'
  :search/unused-vars
  {:extra-deps {borkdude/carve
                {:git/url ""
                 :sha     "f45dc3cb35a8b9c6c11d4681f8c673fa347d54be"}}
   :main-opts  ["-m" "carve.main"]}

❤️ 2
emilaasa13:03:18

If I only want to carve stuff out of lib1, can I do that somehow?

emilaasa13:03:59

I might be misunderstanding how it works a bit, we'll see 🙂

javahippie12:03:54

If I require a library via Git dependency in my Clojure project and the GitHub repo moves to a different organization, will the redirect GitHub creates for the repository also work for Clojure tools, or would builds break?

javahippie12:03:07

(I’m aware that moving a repo to a different organization has a lot of other pitfalls, just trying to understand the technical ramifications)

dgb2312:03:49

I would assume that it depends on how git works rather than deps?

javahippie12:03:52

That’s… a very good point 😅 Somehow I did not think that far

😁 2
dgb2312:03:51

Got me interested into how the cli does stuff. It seems like it simply shells out to git: https://github.com/clojure/tools.gitlibs/blob/master/src/main/clojure/clojure/tools/gitlibs/impl.clj So if there's any issues with the git integration you could maybe look there for clues.

Alex Miller (Clojure team)13:03:00

the CLI determines uses the lib name (by default) to locate the git lib, but you can specify a :git/url to override that. I don't think any of it would work automatically, but can't say I've tried it. It is also possible to use insteadOf git config - that does work.

Alex Miller (Clojure team)13:03:37

if you want to see what the CLI is doing wrt git, you can export GITLIBS_DEBUG=true in the env to see all the commands being run

👍 2
👀 2
Noah Bogart15:03:21

Is there a way to debug reify not being able to find a method in an interface that definitely has the method defined?

2
hiredman15:03:27

it involves looking at the javadoc for the interface, making sure the javadoc you are looking at is for the version of the library you are using, and then realizing the interface inherits the method from another non-public interface

👍 2
😅 2
Noah Bogart16:03:42

After poking around at the javadocs, i realized I forgot that I need the "this" initial arg for clojure.

🙂 4
pooriaTaj16:03:56

Hi , I am trying to work with on Hazelcast jet with clojure , I was wondering if there is a way lambda expressions in java can be expressed in clojure?

hiredman16:03:57

java lambdas will be adapted by the java compiler to implement some SAM(Single Abstract Method) interface/class, so you look at the docs and reify that interface

seancorfield16:03:31

Please don't cross-post questions in multiple channels. You already asked this -- and provided more details -- in #C053AK3F9

stopa22:03:36

(def node {:value 1
           :children [{:value 2
                       :children [{:value 3
                                   :children []}]}
                      {:value 4
                       :children []}]})

(acc-values node) 
; => 
(1 2 3 4)
Hey team, how would you accumulate a list of all the values in this tree? One idea:
(map
 :value
 (tree-seq
  (fn [x] (contains? x :value))
  :children
  node))
Just discovered tree-seq to do this, but I wonder if there's something better

phronmophobic22:03:16

I would probably just use tree-seq, but I'm also a big fan of zippers for more complex tree walking. https://github.com/redplanetlabs/specter/wiki/Using-Specter-Recursively#a-basic-example is also a good option.

reefersleep22:03:17

I was thinking postwalk or zippers, but tree-seq seems great for this

borkdude22:03:46

For a regular structure like this I would just write a bespoke function:

(defn acc-values [m] (list* (:value m) (mapcat acc-values (:children m))))

phronmophobic22:03:17

I've stopped using postwalk for treewalking. The other options are typically a similar amount of code and postwalk is less flexible.

2
borkdude22:03:41

and the postwalk + metadata issue

phronmophobic22:03:49

The benefit of tree-seq over acc-values is that you don't have to worry about stackoverflows

(defn nested-map
  ([i]
   (nested-map {} i))
  ([m i]
   (if (pos? i)
     (recur {:children [m]
             :value i}
            (dec i))
     m)))

(->> (tree-seq :children
               :children
               (nested-map 10000))
     (map :value)
     count)
;; 10001

(defn acc-values [m]
  (list* (:value m) (mapcat acc-values (:children m))))

(count (acc-values (nested-map 10000)))
;; Execution error (StackOverflowError)

borkdude22:03:28

yeah. that's something to keep it mind, it depends if you have deeply nested data, 10k levels deep might not be that common

borkdude22:03:48

I guess this would be the stack-safe version:

(defn acc-values [m] (lazy-seq (cons (:value m) (mapcat acc-values (:children m)))))

👍 2
stopa14:03:42

Thank you team!

pppaul21:03:45

tree-seq looks neat, I'll have to try that out some more. I use walking for stuff like this, along with core.match