Anyone have an example of a file upload using hoplon/castra? Like (form … (input :type “file” …))

my naive attempt seems to just be passing the filename:

(let [file (cell nil)]
    (form :submit #(rpc/upload-file @file)
          (p (input :type "file" :value file :change #(reset! file @%)))
          (p (button :type "submit" "submit"))))

@flyboarder i was just reading your GCL patch yesterday. it's an awesome example of how to use the GCL lib and learn about its various namespaces 🙂 i feel i must thank you sharing all this great work!

@flyboarder regarding your question about "what's the number 1 issue", i can't name just a single one. there are a couple of small issues which all add up. without any serverness order: - version numbers are marketing tools too and alpha16 communicates lack of trust. i think we should change that to something less fearsome, since quite a few of us are using it in production for quite awhile 🙂 - starter kit with solid version number combinations and practical dependencies (eg devtools & dirac, something like this https://github.com/onetom/hoplon-layouts) - formula cell usage for side effects is very tempting to do but leads to a mess since the commands and queries are not separated anymore - lack of summarised examples for the various of static and cell values as elem parameters - IDE integration examples. I think I figured out a great Cursive setup already, but just on this chat someone was asking about cider again. - wait times must be documented because they are non-trivial and unexpected, hence already raise skepticisim from the get go

@parkeristyping file upload is not trivial, thats why there are full businesses built around this topic, like https://www.filestack.com/ (which seems a bit broken atm, but has cljsjs support https://github.com/cljsjs/packages/tree/master/filestack) or https://transloadit.com

@parkeristyping BUT filestack for example forces you to do and store things in certain ways and u have to subscribe to a paid plan if u want to upload into your own s3 bucket so we have ended up rolling our own solution too

i would like to release an example of it but im not sure yet how to package it up in a minimal form or as a library

but here are some useful parts of it for you:

(defn file->clj [f]
  {:name               (.-name f)
   :size               (.-size f)
   :type               (.-type f)
   :last-modified      (.-lastModified f)
   :last-modified-date (.-lastModifiedDate f)})

(defn form-data [m]
  (let [fd (new js/FormData)]
    (doseq [[k v] m]
      (.append fd (name k) v))
    fd))

(defn progress->percent [progress-event]
  (when (and progress-event (.-lengthComputable progress-event))
    (-> (.-loaded progress-event)
        (* 100)
        (/ (.-total progress-event))
        js/Math.round)))

(defn xhr-with-progress [progress]
  ; The "load" event makes sure that the 100% upload state is detected
  ; explicitly.
  ; Source: 
  ;             #Handling_the_upload_process_for_a_file
  (fn []
    (let [xhr (new js/XMLHttpRequest)]
      (when progress
        (doto (.-upload xhr)
          (.addEventListener "progress" progress false)
          (.addEventListener "load" progress false)))
      xhr)))

(defn upload [data to progress]
  (let [reset-progress #(reset! progress (progress->percent %))]
    (-> {:method      "POST"
         :url         to
         :data        data
         :xhr         (xhr-with-progress reset-progress)
         :cache       false
         :contentType false
         :processData false}
        clj->js
        js/jQuery.ajax)))

(def generate-presigned-url
  (-> 'app.docs/generate-presigned-url
      (app.state/remote (cell nil))))

(defn open
  [key]
  (-> (generate-presigned-url key)
      (.done #(let [a (h/a :href % :target "_blank")] (.click a)))
      (.fail (partial js/console.error :app.docs/open))))

@onetom: I'm taking notes, thanks!

this is how u use it (with hoplon/ui):

(defelem upload-button [{:keys [title purpose progress] :as attrs} _]
  (let [file-list (cell nil)
        file-list-changed #(do (js/console.debug :file-changed)
                               (reset! file-list (.. % -target -files)))
        file-input (doto (h/input :type "file")
                     (.addEventListener "change" file-list-changed false))
        files (cell= (for [i (range (and file-list (.-length file-list)))]
                       (.item file-list i)))
        file (cell= (first files))
        progress (or progress (cell nil))
        upload% (cell= (when progress (str "(" progress "%)")))
        upload-file (fn [file {upload-url :action :as upload-req}]
                      (let [sig-fields [:key
                                        :Content-Type
                                        :success_action_status
                                        :policy
                                        :AWSAccessKeyId
                                        :signature
                                        :acl]
                            signature (select-keys upload-req sig-fields)]
                        (js/console.debug "Uploading" (app.docs/file->clj file) upload-req)
                        (-> (app.docs/upload
                              (app.docs/form-data (assoc signature :file file))
                              upload-url progress)
                            (.done #(js/console.debug "Done uploading" %))
                            (.done #(reset! progress nil))
                            (.done #(app.docs/record-upload!
                                     :file/key (:key signature)
                                     :file/name (.-name file)
                                     :file/purpose purpose)))))
        sign-upload-req (s/remote 'app.docs/sign-upload-req (cell nil))]
    (cell= (when file
             (-> (sign-upload-req
                   {:file-name (str (app.docs/doc-folder s/current-user)
                                    "/" (.-name file))
                    :mime-type (.-type file)})
                 (.done #(upload-file file %)))))
    (cell= (when progress (js/console.debug "Progress" progress)))
    (elem
      (if-tpl (cell= (and file progress))
        (elem (cell= (str (.-name file) " " upload%)))
        (mui/flat-button :ah :mid (dissoc attrs :title :purpose)
                         title
                         :click #(do (js/console.debug :upload-clicked)
                                     (.click file-input)
                                     (.stopPropagation %)))))))

@onetom thanks so much! this is more than I could’ve hoped for. I’ll be sure to let you know if I can successfully adapt this to my use case! (though from the looks of it I expect I will, once digesting)

this is the backend:

(ns app.docs
  (:require
    [system.repl :refer [system]]
    [amazonica.core :refer :all]
    [amazonica.aws.s3 :as s3]
    [environ.core :refer [env]]
    [castra.core :refer [defrpc ex *session*]]
    [clojure.string :as str]
    [clj-time.core :as t]
    [org.httpkit.client :as http]
    [datomic.api :as d]
    [s3.client])
  (:import (java.util Date)
           (com.amazonaws.auth DefaultAWSCredentialsProviderChain)))

(def b (or (env :docs-bucket) "addplus-test"))

; Contrary to 
; we don't need path style access anymore because the domain
; style access has a valid SSL certificate automatically.
; Using path style access results in a HTTP/301 Permanent Redirect
; without a Location header so the browser has nothing to follow.
;
(s3/set-s3client-options {:path-style-access false})

(defn url-expiry-time []
  (-> 60 t/seconds t/from-now))

(defrpc generate-presigned-url [key & [method]]
  (.toString (s3/generate-presigned-url b key (url-expiry-time) (or method "GET"))))

(defrpc sign-upload-req [file]
  (let [creds (.getCredentials (new DefaultAWSCredentialsProviderChain))
        opts* {:region     (env :docs-bucket-region)
               :bucket     (env :docs-bucket)
               :access-key (.getAWSAccessKeyId creds)
               :secret-key (.getAWSSecretKey creds)}]
    (s3.client/sign-upload file opts*)))

where s3.client is basically the great, super minimalist https://github.com/martinklepsch/s3-beam/blob/master/src/clj/s3_beam/handler.clj minus the last function (`s3-sign`) which is a ring handler but we need a castra endpoint instead (`sign-upload-req`)

ah, and s/remote is just a little convenience fn:

(defn log-rpc-error [endpoint args error]
  (let [msg (str "Error in " (apply list endpoint args))]
    (js/console.groupCollapsed msg)
    (js/console.error (.-serverStack error))
    (js/console.groupEnd)))

(defn remote [& [endpoint state error loading opts]]
  (let [url (str (env :backend-url) "/" (str endpoint))
        rpc (mkremote endpoint state
                      (or error (cell nil))
                      (or loading (cell []))
                      (merge {:url url} opts))]
    (fn [& args]
      (doto (apply rpc args)
        (.fail #(log-rpc-error endpoint args %))))))

(defn when-done
  "Attach a success callback to an RPC function
   (which returns a promise when called)"
  [remote-fn done-fn]
  (comp #(.done % done-fn) remote-fn))

maybe there should be an example for this in https://github.com/hoplon/demos

@flyboarder I haven't used this new pull request review feature before, but i've accepted the changes on the review. @micha @jumblerg @alandipert @ul @mynomoto now what's next? who is supposed to accept the pull request? im not an expert on this at all, so the only formal process description im aware of is C4, the Collective Code Construction Contract (latest revision is https://rfc.zeromq.org/spec:42/C4/) in 2.4.13 it says: > Maintainers SHOULD NOT merge their own patches except in exceptional cases, such as non-responsiveness from other Maintainers for an extended period (more than 1-2 days). which raises the question: who do we consider maintainers? according to https://github.com/orgs/hoplon/teams/hoplites we have a bunch of Owners (in github terms) who can be Administrators of the project (in C4 terms, see 2.7.1) it also means (as per 2.7.2) that we should manage a Maintainer list. if i understand well, then admins are automatically maintainers too and whoever is on the Hoplites github team without an Owner flag are considered C4 Maintainers.

it's slightly confusing because - i think - the Owner badge on the team page is shown when u click the "Promote to team maintainer" menu next to a member

C4 doesn't dictate a versioning scheme, so we should still agree on that separately, but it recommends to represent versions simply as tags on the master branch, which is already how we are doing it now.

If C4 is an agreeable governance model, then we should add a link to the bottom of https://github.com/hoplon/hoplon/blob/master/README.md

fwiw, i think any policies we need fall out of the existing permissions system in github already.

we can take care of the rest on a case by case basis by communicating here

@jumblerg and C4 also mentions in its 1st sentence: > The Collective Code Construction Contract (C4) is an evolution of the http://github.com Fork + Pull Model, aimed at providing an optimal collaboration model for free software projects. but how does it answer my original question: > Who is supposed to accept the pull request? The permissions only describe all possibilities; they don't prescribe a process

btw, im looking into your https://github.com/addplus/ui/commit/c0773c63181f3d1b6fcba512b5b18f71c3bb99a9 patch. since there is no clear problem statement anywhere, im not sure what is it supposed to solve.

i was hoping this would work:

(hui/elem
        (h/for-tpl [x [1 2 3]]
          (mui/row (cell= (str x))))
        (h/for-tpl [x [10 20 30]]
          (mui/row (cell= (str x)))))

then i thought this one, but no:

(hui/elem
        (h/for-tpl [x ["a" "b" "c"]]
          (hui/elem :sh (r 1 1) x))
        (hui/elem :sh (r 1 1) "Static elem"))

then i thought maybe it helps with for-tpl nesting:

(hui/elem
        (h/for-tpl [x ["a" "b" "c"]]
          (h/for-tpl [y ["1" "2" "3"]]
            (hui/elem :sh (r 1 1) (cell= (str x y))))))

(hui/elem
        (h/for-tpl [x ["a" "b" "c"]]
          (hui/elem
            (h/for-tpl [y ["1" "2" "3"]]
             (hui/elem :sh (r 1 1) (cell= (str x y)))))))

so im not sure what is a typical situation when the children as just something sequential...

you'll always need an elem between the tpls, per the hoplon docs

the commit you referenced ensures elems can accept sequential things that are not vectors, eg seqs

(elem (cell= (map identity some-cell)))

in this case the "children" are just 1 cell, which is not sequential?

so the previous line of the cond would apply

no?

before the commit, that example broke bc it was dispatching on a vector instead

ah, just noticed that swap-elems! is a recursive function...

one i'll be junking entirely soon

okay, pointless to discuss further then 🙂

lot of legacy nonsense going on up there from previous experiments

I think we can cut down on these conversations if we could follow the 2.4.1 - 2.4.6 points of the C4 Development Process

for example u shouldnt have committed the cross origin attribute handling of the objectable middleware into the same patch

i understand while you are in experimentation mode you don't really want to be so meticulous about commit history

i'm shifting to a new model where the number of elements in an elem is variable, where wrapping elements are introduced only by the middleware corresponding to the attributes that require them.

but on the other hand it also helps with thinking clearer maybe... no?

yeah, that approach sounds quite exciting

i would be very curious to see early releases of it

my git hygiene isn't the best right now, but the immediate focus is shipping my projects.

same here but the though clarity which would come with being more hygienic would probably worth those few extra minutes of admin overhead

i definitely plan on doing things in a more disciplined way once we hit a release.

but in that case you shouldn't be afraid to release your dev branch either, so we can think together with you

no matter how dirty is it

i have a few, happy to share

although i think i've settled on a more incremental approach, am going to refactor master in pieces instead

@jumblerg is there a huge master refactor pending? 🙂

@flyboarder for your talk - starter kit with Hoplon/ui - I’ve seen some examples but still - maybe a template - clone this and start here type example

so many good things happening in here

great channel, right? 🙂

the IRC !m bot would be going insane

(that was the bot that said "you're doing great work, X!")

@onetom i like the 0mq collaboration model, what do you think @alandipert ?

probably about time we go through the repo settings and organize the collaborator settings

I pushed an example Hoplon project with • the latest Clojure(Script) 1.9.x dependencies • Dirac setup aaand • IntelliJ+Cursive+Leiningen compatibility instrumentation. It's a good off-the-shelf project for starting hacking on Hoplon. It shows a (checkout) example too, so it's good for newcomer contributors (in C4 speech 🙂) https://github.com/onetom/hoplon-layouts/blob/master/build.boot

@tbrooke: thanks for the suggestion!

@onetom: awesome!

@flyboarder: when is your talk? i have a documentation project for hoplon/ui that might also serve this purpose well.

The talk is on Nov 15th, however it is mostly an intro to hoplon I dont know enough about UI to really speak on it’s behalf

ah. i only spotted tbrooke's comment upstream, happy to help in any way you find useful.

i'm working on the readme as i have time; you may find some useful info there. will also check in the doc project soon.

@jumblerg what I am really interested in is converting existing css to UI

that could be a pretty cool compare and contrast sort of exercise, before and after.

yeah and would work with existing libraries

some kind of boot task which could generate a hl file from a css library would be cool, and probably bring in more adopters

that would be pretty challenging, i think, because there are so many ways of doing things in css

@jumblerg you would need to know lots about the library I think

i dont think you could make a generic task for such a thing

i'm also skeptical of third party libs that have dom access

if they do, there needs to be a way to sandbox them

they're unsafe

yeah It starts to be a problem with things like semantic-ui which modifies the dom as part of it’s “styling"

yeah. can't have those things tinkering around in there doing whatever. but maybe inside a frame.