Fork me on GitHub
#beginners
<
2024-06-11
>
lyall08:06:58

I hope this isn't too controversial of a topic (and sorry if it gets asked all the time), but is there any consensus on what's considered the most standard, boring, go-to option of library/framework for building a basic web app that talks to a DB and serves HTML pages? It feels like there are a ton of options out there but they all seem to boil down to "roll your own framework", "project that nobody has touched in five years", and "new thing that nobody actually uses in prod yet". I'm sure there are many choices out there but it's hard to weigh the pros and cons when you're new to the ecosystem

Adham Omran08:06:37

Hey Lyall, as for the controversial status, I don't think it is that controversial so it's fine As for the "standard" as you might have noticed there isn't one per se but some libraries are more in use and popular than others. For a web app that talks to a db and serves HTML, rolling your own is not as hard as you might initially think. This is what we use at work and we prioritize active and complete libraries • Routes for the API: https://github.com/metosin/reitit • Server: Jetty • Talking to DB: For SQL we used https://github.com/seancorfield/honeysql but moved to Datomic last year • Serving HTML pages depends on how you generate the HTML pages, you can do anything from slurping the HTML file to generating it with https://github.com/weavejester/hiccup That's it really

delaguardo08:06:21

afaik, non of popular clojure frameworks are as simple as you want. I, personally, prefer to stack up a few libraries to get me there: aleph as a webserver, reitit as a router, next-jdbc to talk to DB and then selmer to render html from the template. Redmes of those libs are complete enough to end up with more or less "good to go" app in an hour.

Adham Omran08:06:19

One thing I'd recommend is setting requirements for a MVP and testing several libraries to see which best suite your style of development

dawdler08:06:51

I think :man-shrugging: the most common combination of libraries used for full stack web development are (at least that's what I often see in job ads, and what I've often used myself): • Component/Integrant for global state on backend (system components and shared state, e.g. db connections) • next-jdbc for DB layer on backend • re-frame/reagent for frontend app-db/react wrapper • httpkit, aleph, et al. for http server • reitit/ring for routing++ • Plus a whole lot of smaller libraries for data formats/serialization/validation etc. But you are right, there are many more alternatives, some of which might be a better fit for you than the above. The best way is to try different libs, and it's fairly easy in Clojure, given its REPL-driven (interactive) development style. Note that even if libraries are "old", they are not necessarily deprecated or not functioning. This isn't like in the Python or JS ecosystems :)

👍 2
Adham Omran08:06:03

I second dwalder on this > Note that even if libraries are "old", they are not necessarily deprecated or not functioning. This isn't like in the Python or JS ecosystems :) Libraries can just be complete

1
lyall08:06:20

So it sounds like “roll your own framework” is the standard 🙂 while I understand that may be the best approach in the long run (composability!), it unfortunately adds to the already relatively high startup cost of learning clojure

lyall08:06:15

And I mostly agree that “libraries can be complete”, it makes it even harder to evaluate what the right choice is. Is this library “complete”? Or has the community moved on to something else? It’s hard to tell

Ed08:06:02

There is also #C013Y4VG20J which is a "framework" that may get you started quickly.

👍 2
dawdler08:06:24

Sounds like what you need is a "starter kit", there are a few, iirc. Luminus could be used for that, for instance: https://luminusweb.com/

lyall08:06:05

I’ve looked into Biff pretty extensively already, and I like a lot of it, but enough seems to be not quite what I want (eg using XTDB) that it might be more work to adapt it to my needs than it would be to just use something else to begin with

delaguardo08:06:54

> And I mostly agree that “libraries can be complete”, it makes it even harder to evaluate what the right choice is. Is this library “complete”? Or has the community moved on to something else? It’s hard to tell what is the right choice? 🙂 if it works — you are ok, if not — switch to something different.

lyall08:06:20

And re: luminus, they’ve moved onto kit! So do I use the older project that’s been declared out of date, or use the new project that hasn’t proven itself yet? Though fwiw kit has been the thing I’ve most liked so far. It seems mostly straightforward with reasonable defaults but adaptable. It just seems like it’s still in the earlier stages of development though

lyall09:06:34

> if not — switch to something different I’d like to choose the right thing from the start and avoid wasting the extra time and effort down the road!

lyall09:06:58

so just trying to do my research now 🙂

lyall09:06:38

And thank you everyone for the input so far!

dawdler09:06:27

You could build your knowledge/experience by not starting with a set up for a complete website/app (it tends to be a lot to manage in one go), e.g. start with just firing up your db service of choice, then fire up a REPL, and use next-jdbc to connect to it and create tables, populate, and query. Then move on to some system component stuff on a backend, and so on and so forth. In any case, one is usually better off not relying on a "full stack framework" in the long run, imo. ymmv and all that. Btw, there's also https://clojure.stream/courses, albeit not free.

Mario Trost09:06:36

@U076H6ZE91A From the luminus creator: https://kit-clj.github.io/ Haven’t used it, but the docs seem comprehensive enough to give a good start

lyall09:06:30

thanks, I think that’s good advice re: iterating from the REPL. My only problem is the more time spent learning is the less time spent actually building the thing I’m trying to build. I’m worried at some point my patience will run out and I’ll just say “fuck it I’m building a Django app” (I do not want to build a Django app)

😁 1
Mario Trost09:06:25

#C02T4GSBSJ1 👈 kit channel

dawdler09:06:19

@U076H6ZE91A The community here is very helpful, that's important to keep in mind when learning, even if peeps may not have the time/resources to walk you through everything. Point being: do not hesitate to ask when you are stuck. :)

❤️ 1
lyall09:06:57

@ULA8H51LP you may notice that I was the last person to post in the kit channel 😅

😄 1
Mario Trost09:06:02

Haha, then sorry for stating out things you already 😉 Next try 😄 Yesterday I skimmed a bit of the http://pedestal.io/pedestal/0.6/index.html and saw among other things the baked in open telemetry. Could be that it comes with more batteries included than other frameworks. But here too: I haven’t really used it myself.

lyall09:06:57

no worries, thanks for trying to answer my original question 😂

lyall09:06:47

yeah I’ve also looked into pedestal and it seems promising also maybe has a bit of a learning curve? The docs did look pretty good at first/second glance though

jpmonettas12:06:47

IME unless you have pretty specific needs, all the libraries mentioned here will work, doesn't matter if you use compojure or reitit, http-kit or jetty, etc. I wouldn't spend too much time deciding between them for a normal web app since there will not be a 10x difference between them and I don't think you will find yourself rewriting it because you hit a wall. Now if you think you have some specific needs, then the libraries approach is even better than the framework one because you get to choose all the parts to support your needs.

👍 2
lyall12:06:55

yeah I'm just trying to build a CRUD app, so shouldn't be breaking any new ground at all

Nim Sadeh13:06:01

Clojure doesn't have the Python "all batteries included" ecosystem. Pretty much every project I've done required selecting libraries. Often some libraries were incomplete, the biggest example is needing e.g., a separate CORS library for you routing/webserver lib. My current stack for writing HTTP backends: • Reitit for routing, also has a nice integration with OpenAPI • Aleph for web server/request handlers, streaming • HoneySQL for SQL • Ragtime for SQL migrations as code • Environ for env variables management • Mulog for logging Here's a project.clj dependencies section for a current project I am running:

:dependencies [[org.clojure/clojure "1.11.1"]
                 [com.brunobonacci/mulog "0.9.0"]
                 [com.brunobonacci/mulog-adv-console "0.9.0"]
                 [environ "1.2.0"]
                 [ragtime "0.8.0"] ;; 
                 [honeysql "1.0.461"] ;; 
                 [ring-cors "0.1.13"]
                 [aleph "0.7.1"]
                 [cheshire "5.13.0"]
                 [ring/ring-jetty-adapter "1.12.1"]
                 [metosin/reitit "0.7.0"]
                 [metosin/ring-swagger-ui "5.9.0"]
                 [net.clojars.wkok/openai-clojure "0.17.0"]
                 [com.google.cloud/google-cloud-storage "2.39.0"]
                 [com.google.cloud/google-cloud-texttospeech "2.45.0"]
                 [com.github.seancorfield/next.jdbc "1.3.939"]
                 [org.postgresql/postgresql "42.7.3"]]
LMK if you need code samples or help on any of these

🙏 1
Nim Sadeh13:06:28

On a more high-level note: Clojure (at least to me) seems built and maintained by engineers who understand that the last 5% of making an application productionworthy takes 95% of the work, and also understand the simple stuff on a deeper, more fundamental level. That's why the easy stuff like "just setup a CRUD app" app is more difficult than something like Python, which is built and maintained for and by engineers who want to stand up a prototype really fast. So what takes 5 minutes and just pip install fastapi in Python might take much longer in Clojure, because it wasn't designed with that goal, but things like debugging concurrency issues (the cause of my last port Python->Clojure) and other pro player moves are much easier on Clojure

lyall14:06:10

I appreciate the pros of the "roll your own framework" approach, but one worry I have is that even if I choose solid libraries to build with that I'll mess up or forget about some subtle but important security config related to CORS, CSRF, authn/authz, etc. The vast majority of my software engineering experience is as a deep server side engineer where I don't have to think about any of that stuff, so knowing there are guardrails and sane defaults would make me feel better

seancorfield17:06:15

@U076H6ZE91A As a starting point, look at https://github.com/seancorfield/usermanager-example -- the README has a link to a version using reitit instead of Compojure and Integrant instead of Component, so you can see the differences.

👀 1
James Amberger11:06:48

I’ll pile on here; my current boring pretty-much-CRUD business app is just ring+compojure+hiccup+htmx, oh and next.jdbc and HoneySQL to talk to the db. Very little code, zero duplication over the server/client boundary, very REPL-friendly, etc etc, way to work. My first serious stint as an app developer was back in PL/SQL+Spring+AngularJS world and I now thionk people are just kidding themselves calling that world “easier” because “YAFjs” has a sharp-looking splash page and a button that will copy the yarn command to your clipboard for you. Not saying anything that hasn’t been posted here a thousand times but just another datapoint—I would stop write boring business apps entirely if I couldn’t do it this way and sneak the resulting JAR files past the admins at work. @U076H6ZE91A if you came so far as to post here it’s time to jump in the pool. Water’s great.

🙏 1
lyall12:06:39

yeah I'm already 100% on board for htmx! and "jump in the pool", I think I'm already in too deep to get out 😅

didibus17:06:32

One thing I want to point out is that, rolling your own is way easier in Clojure I feel. And you don't need to build your own, just combine existing pieces together with the libraries available. As for the CQRS, auth and all that. I agree, that's a fair point. But most of it is handled by just adding the right ring middleware. In fact, ring is kind of the default "framework" for Clojure really. The choices are not that many I think: HTTP handling: Ring Web Server: Jetty or http-kit or aleph Routing: Compojure or Reitit HTML templating: Hiccup or Selmer SQL DB: next-jdbc SQL builder: HoneySQL or HugSQL SQL DB migration: migratus or ragtime NoSQL: Use java clients or aws-api or call rest endpoints directly Schema/Validation: malli or spec or schema Logging: clojure.tools.logging (with logback or log4j2) or timbre (and newer telemere) Singleton State management: Integrant or Component or plain delays or Redelay Those I feel are the most common options. You'll find people using other things than those for sure, but I'd say can't go wrong with any combination of the above. Here's a very simple example of rolling your own stack: https://github.com/didibus/simple-website-with-posts it really doesn't take much to get a working CRUD app.

👍 1
James Amberger21:06:23

Lemme throw this in, would credit whoever showed it me if I could remember: https://mccue.dev/pages/12-7-22-clojure-web-primer

1
🙏 1
didibus22:06:01

It's personal preference, but I think it's okay for the top level entry points to grab delayed stateful resources. It's not as clean, but it can often be simpler. I like the article though, it does a good job at explaining it. See this thread for rationale: https://clojureverse.org/t/how-to-manage-database-connection-in-clojure/5067

stopa14:06:26

Hey team, noob question: Consider a structure like this:

test
  foo_test.clj 
  test_helpers.clj
(ns foo-test) 
...
I have a dev REPL running. I don't include 'test' files in the classpath. I do like to evaluate tests in the repl though. Right now as it is, I could evaluate foo-test. But, If I require 'test-helpers':
(ns foo-test
 (:require [test-helpers :as th]) 
...
I will no longer be able to evaluate this file in my repl, as test-helpers won't be in my classpath anymore. I can solve this in two ways: 1. Add test to extra-paths in deps.edn 2. Move test-helpers into src/ What is the more idiomatic solution? Looking around I haven't noticed projects define helper namespaces inside the test directory, but it feels weird to move test-only code into source.

dpsutton14:06:47

add "test" as a classpath root to an alias you use for testing or local dev

👍 2
2
stopa14:06:17

Fair -- thanks @U090DFC3A!

Nim Sadeh20:06:17

I know I shouldn't use blocking I/O in go-blocks, but are java semaphores fine to use? I am trying to coordinate a resource sharing limit between go-blocks

hiredman20:06:00

complicated, but best to assume no

hiredman20:06:16

blocking operations, things that would stop a thread from running are bad

hiredman20:06:00

so no-blocking operations on semaphores (like tryLock on lock, I forget what semaphore exposes) can be fine, but easy to get wrong

Nim Sadeh20:06:39

I think that would be acquire Does dressing sync/blocking code in async/parking code help? Is there a difference between

(let [sem (->semaphore 8)]
  (go (acquire sem)
      (do-something-else))) 
vs
(let [sem (-> semaphore 8)
      c (chan)]
  (thread (acquire sem)
          (>! c :go))
  (go (<! c)
      (do-something-else)))

hiredman20:06:15

yes, similar to the way a lot runtimes that natively support some kind of light weight threads will spin up a new thread when dealing with a blocking operation

Nim Sadeh20:06:24

So basically I am throwing a bit more memory on the problem by spinning a new thread on each acquisition but the code is still async?

hiredman20:06:09

likely whatever you are trying to do with a semaphore can be done using channels

Nim Sadeh20:06:48

What I am trying to do is code block X can be run only N times in parallel

hiredman20:06:31

e.g. a pair of channels for acquires and releases and a go block that takes N from acquires and then stops reading from acquires until it can take N from releases

Nim Sadeh20:06:33

Can I just do a single channel (chan N) ? Before a function enters the code blocks that's being guarded, it needs to take from the chan When it's done, it puts on the chan Am I missing something?

hiredman20:06:53

no, needs 2

hiredman20:06:25

oh, I guess one can work if you know in advance how many tokens you need and that never changes

Nim Sadeh20:06:13

Yea it's a static parameter protecting a resource

Nim Sadeh23:06:42

OK I found a reason why you can't use chans as semaphores, I think:

java.lang.AssertionError: Assert failed: No more than 1024 pending takes are allowed on a single channel.
The whole point of it is to protect a resource from a lot of accesses

nikolavojicic09:06:57

Maybe you can use promise-chan or timeout.

didibus17:06:58

By code block you mean a go block? Or what do you mean exactly?

Nim Sadeh17:06:03

No, I do mean "code block." In this situation, there's a certain function/piece of code that should only have N ongoing executions at any given time. This is common with e.g., database connections

didibus17:06:07

Where does the go-block come in the picture though? Do you call that piece of code from a go-block?

hiredman17:06:39

We have a custom redis connection pool at work, and it is internally a go loop around a "command" channel and some of the commands are borrowing connections from the pool and returning them

hiredman17:06:09

The borrowing command, if I recall, sends in a channel it expects the borrowed connection to come back on

hiredman17:06:39

Which would avoid having too many waiting on a single channel

👀 1
Nim Sadeh17:06:08

That’s helpful, thanks!

didibus17:06:28

If the go-blocks are making IO, you need to use non-blocking IO. In which case, the non-blocking IO is what should enqueue async requests to the DB. DB IO is blocking though in Java (unless using some experimental jdbc drivers). So what I would do is that one thread-loop takes N from a channel at a time, makes the DB calls, and returns the result to an injected promise-chan. Now your go-blocks, when they want to make a DB call, they put on the db-chan a request to do so which includes a promise-chan, and then they take from the promise-chan. Your thread is blocked on new elements on the db-chan, since one was just added, it'll wake up, take N or like take up too, and then process it, and when the DB result is back it'll put it on the promise-chan that was part of the request.

didibus18:06:03

So maybe something like (untested, wrote it on phone):

(def db-chan (chan 10)) ;; decide on the buffer size you want

(defn take-up-to [n ch]
  (go-loop [collected []]
    (if (>= (count collected) n)
      collected
      (let [item (poll! ch)]
        (if (nil? item)
          collected
          (recur (conj collected item)))))))

(defn db-worker-loop [db-spec]
  (thread
    (loop []
      (let [requests (!! result-chan {:status :success :result result}))
            (catch Exception e
              (>!! result-chan {:status :error :error e}))))
        (recur)))))

(go
  (let [result-chan (promise-chan) )
        query ["SELECT * FROM users WHERE id = ?" 1]
        _ (>! db-chan [["SELECT * FROM users WHERE id = ?" 1] result-chan)
        result (

didibus18:06:23

Or more simply, just have the db-chan-worker take 1 at a time, and start N number of workers to get your max parrallel db calls