You could say HTTP rest api is an RPC too. But what exactly do you mean by without buffering it into memory? You mean without loading all the data into memory at once?

yeah

I mean, I suppose HTTP works, but I was ideally looking for something a little less hands-on (with HTTP I'd have to set up a whole API; really I just want a hassle-free but performant way to call a bunch of functions)

ideally I'd like to be able to have bidirectional communication as well, which would require something like long polling with a REST API

I'd probably go with gRPC, but it seems like that would require me to maintain a bunch of boilerplate

I haven't used it, remember reading about it somewhere- https://github.com/sunng87/slacker

If it's a trusted execution context you could just wrap some streaming around the capabilities of the core.server namespace, which seems much lighter weight than gRPC or HTTP https://clojuredocs.org/clojure.core.server/start-server

Send the function you want to eval (as data), get a streaming result back

https://clojuredocs.org/clojure.java.io/input-stream The core.server instance is intended to operate over a socket connection, so you could potentially turn that connection into an InputStream

@U0A5B1LJU Slacker looks really cool, but AFAICT you can't really stream data with it

@UFTRLDZEW cool, I'll take a look at that; thanks

@U15RYEQPJ for something simple and minimal, I've had success sending EDN (or Transit) over a websocket. I did have the luxury of having a clojure backend and clojurescript front end, but you can just as easily send JSON messages too, or anything really, the "socket" is agnostic about the content of the messages. It's bi-directional message passing. If you need request/response semantics, you'll need an additional layer to manage that. Streaming is also supported but I've never done it in clojure.

@UHL84CDTP good idea. With websockets, what I'd worry about is that it's hard to guarantee at-least-once delivery. You'd need to manually add a mechanism to track if messages are getting through or not

In case you're not familiar with https://github.com/ptaoussanis/sente - just throwing it out there. Not sure if it would work well with arbitrary streaming data, but perhaps if you can chunk/batch it correctly? At the very least, it gives you an idea on dealing with websocket reconnects, syn/ack delivery, etc.

Is prod-config.edn in your uberjar?

from project.clj:

:profiles
  {:uberjar {:omit-source true
             :aot :all
             :jvm-opts ["-Dconf=prod-config.edn"]
             :uberjar-name "clj-blog.jar"
             ;source paths and resource paths are very lightweight & specific
             ;prod/clj does not include support for starting and stopping the server within REPL for example
             :source-paths ["env/prod/clj" "env/prod/cljc" "env/prod/cljs"]
             :resource-paths ["env/prod/resources"]
             :prep-tasks ["compile" ["run" "-m" "shadow.cljs.devtools.cli" "release" "app"]]}

If you open your uberjar, is the prod-config.edn file in it?

I don’t see it.

I don't know what library you are using for reading configs, but there is a very good chance it loads configs as resources from a classloader, the most common way to make things available like is in jarfiles or directories on the classpath

The only thing on your classpath is your uberjar, so if the config file is not in there, there is a good chance whatever you are using to load it can't find it

I think luminus uses something called cprop

I added a log & it looks like it’s grabbing the url correctly which is weird:

(defstate ^:dynamic *db*
  :start (if-let [jdbc-url (env :database-url)]
           (do
             (log/warn (str "database url is the following I guess:" (env :database-url)))
             (conman/connect! {:jdbc-url jdbc-url}))
           (do
             (log/warn "database connection URL was not found, please set :database-url in your config, e.g: dev-config.edn")
             *db*))
  :stop (conman/disconnect! *db*))

Have you asked on the #luminus channel also?

Will do, thank you

It’s working now - thanks everybody