Fork me on GitHub
#biff
<
2023-02-16
>
Jacob O'Bryant03:02:45

I'm close to having some new authentication stuff released. I already have an additional paste-the-6-digit-code working. I've set up the default project so that it uses email links for signup and email codes for signin, which I think hits the right combination of low friction + resilient to embedded browsers on mobile devices. the big additional thing I'm trying is to put all the backend auth code in the biff library instead of including it in the default project like how it works now. it's basically a plugin--there'll be a com.biffweb.auth/features function which takes some optional config and returns a features map, which you include with the rest of your feature maps. the plugin provides reitit routes for the backend functions (all the ui is still done in your own project) and some schema for storing the 6-digit codes in XT. the whole plugin thing is pretty experimental. I'll release it on a dev branch tomorrow and then try using the plugin for yakread. if anyone else wants to give it a spin too, that'd be great. if this plugin architecture feels good, it could open the doors for some more interesting things. it's a way to package up full-stack functionality into a reusable library. e.g. a few of yakread's features might be amenable to releasing as plugins, like a plugin for receiving emails, or for syncing rss feeds... it'd be analogous to firebase extensions: https://firebase.google.com/products/extensions. I'm also planning to write a how-to guide for using firebase authentication. so by default you get biff's passwordless email link/code auth, and if you want e.g. passwords or social sign on, you can use firebase auth (or auth0, or whatever--but firebase auth will at least provide an example)

🔥 12
🚀 6
👍 4
Jacob O'Bryant03:02:58

The release candidate might have to wait until tomorrow, but I have finished writing the reference documentation: https://gist.github.com/jacobobryant/527ce48a66a5138ea907c0307793e140 (The default project will be updated to use the plugin, so you won't necessarily need to read all that.) (If I ever try to get a normal job again I should put all this on my resume... "look, I actually write docs! (Pay no attention to the lack of tests...)")

Jacob O'Bryant03:02:07

With a lot of Biff fns I try to expose a public API not only for the top-level function, but also for some of the lower-level functions. So if you want custom behavior you don't have to copy 500 lines of code into your project just so you can change one thing. I'm on the fence about doing that for this plugin, though. Perhaps just because the whole thing feels so experimental, I feel uneasy about committing to not breaking any of the implementation functions. I've added options for the things I know I'll need to change in some of my own projects, and for the rest (e.g. redirect endpoints) I'm leaning on convention-over-configuration. So the only addition to Biff's public interface for now will be this authentication-plugin function. I may revisit that after the plugin has been released and used for a bit.

Jacob O'Bryant03:02:41

Other changes: I'm adding a :schema key to feature maps and refactoring schema.clj to use that. I also might rename "feature maps" to "plugins" in general. So you write your app as a collection of Biff plugins, which get inserted into your Biff context map, which gets passed through all your Biff components.

Jacob O'Bryant03:02:44

Got it on dev: • https://github.com/jacobobryant/biff/blob/dev/example/src/com/example.clj#L16https://github.com/jacobobryant/biff/blob/dev/example/src/com/example/feat/home.cljhttps://github.com/jacobobryant/biff/blob/dev/example/src/com/example/email.cljhttps://github.com/jacobobryant/biff/blob/dev/example/src/com/example.clj#L64 • See also changes to schema.clj and stuff • Also notice that there's no more auth.clj file Tomorrow I'll start updating Yakread to use that commit, and if that all goes smoothly, I'll probably merge to master and release soon afterward. If anyone wants to play around with it, you can do this to create a new project based on the dev branch:

wget 
bb new-project.clj dev
Or set your biff dependency to com.biffweb/biff {:git/url "", :sha "883ac1d23769b1521829196d8dae6cde788a61c3"} to use the auth plugin in an existing project.

🎉 4
m.q.warnock18:02:47

File uploads are working for me now, except that slightly-larger files fail with a 417, and I haven't been able to figure out how to increase the limit. After a lot of flailing about based on search results (I tried adding :request-header-size to the config passed to run-jetty in use-jetty, as well as passing a 'configurator' that calls .setAcceptedReceiveBufferSize ), I think what should work is adding -J-Dorg.eclipse.jetty.server.Request.maxFormContentSize=31457280 to my clj-args, but it doesn't. I tried updating my local biff to the latest sunng/ring-jetty9-adapter (which uses jetty11 confusingly). Grepping through all the jetty11 source it downloads doesn't turn up maxForm though. Any suggestions?

Jacob O'Bryant18:02:06

Are you running into this problem in production or locally? I have in the past set an nginx option that increases the max upload size (which is 1mb by default): https://www.tecmint.com/limit-file-upload-size-in-nginx/. You'd want to put that change in http://setup.sh/server-setup.sh (I changed the name recently) as well as editing /etc/nginx/sites-available/app. Other than that, I haven't messed with any jetty settings. But for yakread, I have used this huge-uploader script to let people upload ebooks: https://github.com/Buzut/huge-uploader. It chops the file into chunks on the client side and then uploads them a piece at a time. You set up an endpoint that saves all the chunks and stitches them back together once the last one has been received. If that sounds useful I can paste some code; but if you just need to e.g. receive images up to 10mb or something, then just increasing the upload size would probably be better.

m.q.warnock18:02:53

ahhh! I hadn't thought of nginx being the culprit (running locally, but through nginx for... reasons). I bet that's it; thanks yet again!

m.q.warnock18:02:09

yep; that was it! doh 😛

Jacob O'Bryant19:02:25

heh, excellent!

pavlosmelissinos19:02:36

So, I'm trying to add websockets to a biff application and I wonder if I got the idea right. 1. It's more idiomatic to use :on-tx and monitor db transactions, rather than do :ws-send with htmx, right? 2. If I have multiple events to fire in the same screen (different parts of the DOM need updates at different intervals), I should still create one socket per client (browser tab), so the :on-tx function should broadcast the updated html to all the clients that need an update, based on the event. edit: The "realtime updates" lesson in the eelchat example was nice but I haven't grokked it yet completely. I believe it kind of assumes that the reader knows how the pieces fit in the puzzle (e.g. it tells you to add :on-tx and then try it out but why does it work?), so I have a hand wavy feeling that there's some room for improvement.

👀 2
Jacob O'Bryant20:02:01

Hm, might be a good subject for some more documentation. (I guess I also need to update the default project so it uses the new htmx websocket extension like eelchat does, instead of using the old built-in websocket thing). As for your questions: :ws-send is for sending websocket messages from the client to the server. You can either use that, or you can use plain http requests instead. That's what eelchat does--when you create a chat message, it gets POSTed to the server, and websockets are only used for sending new chat messages from the server to all the clients. :ws-send will be a little bit faster, which may or may not matter for your use case. On the server, you need some way to know when there's new data to send to clients. :on-tx is the recommended way to do that; so you can say e.g. "when there's a new chat message on the transaction log, send it to all the clients in the relevant chat room." The other method is you can do it from the request/web socket message handler. e.g. as I mentioned in eelchat's case, there's an http handler that gets POSTed to whenever a client has a new message. Instead of using :on-tx, we could just add some code there that sends the message to all the clients in the relevant chat room. The advantage of using :on-tx is that it works even if you have multiple web servers--each web server will see the message on the transaction log, and they'll handle any clients that have an open websocket connection to them. If you put the client-notification code in the http handler, then you'd only notify clients that happen to be connected to that particular web server. Re: multiple events. Yeah, I would use a single socket per client as you've described.

🙏 2
Jacob O'Bryant20:02:56

Note that the default project also has some websocket example stuff, though the client-side portion needs to be updated slightly as I mentioned. If you do want to use :ws-send instead of having clients submit regular http requests, there's an example of how to receive websocket messages on the server here: https://github.com/jacobobryant/biff/blob/master/example/src/com/example/feat/app.clj#L129

🙏 2
pavlosmelissinos21:02:22

Riiight. :on-text was the piece I was missing... Also the server is "subscribed" to the transaction log, so the :on-tx function gets triggered whenever there's a new transaction. So, in the case of :ws-send & :on-tx: 1. the client sends a websocket message to the server, 2. the server writes the transaction to the db and 3. all the web servers (including the one that made the transaction) pick up the event from the db, process it and send the response via websocket to all the connected clients. The advantage over using the HTTP handler is clear! Thanks for the explanation, I think I get it now 🙂

👌 2
Jacob O'Bryant21:02:23

Yep, that's how it works!

Jacob O'Bryant21:02:55

I'll make a note to go over that somewhere in the docs, probably the tutorial at least

😍 2
Jacob O'Bryant23:02:02

Me, rewriting my consulting landing page "I can't believe I actually called this thing 'Biff'"

😅 10
😆 4
Lyn Headley00:02:01

The name is awesome.

🙌 2
2
Jacob O'Bryant00:02:33

I think so too 😉

Epidiah Ravachol16:02:06

I don't know if this helps or hurts, but you've become known as "Jacob Biff" in this household. My wife will ask, "How'd your day go?" and I'll say something like "Great! I was stuck on this problem, but then I asked about it on the Slack and Jacob, the guy who create Biff, solved it—" and my wife would say something like, "Good Old Jacob Biff."

Jacob O'Bryant18:02:03

That's great 🙂

Jacob O'Bryant03:02:58
replied to a thread:I'm close to having some new authentication stuff released. I already have an additional paste-the-6-digit-code working. I've set up the default project so that it uses email links for signup and email codes for signin, which I think hits the right combination of low friction + resilient to embedded browsers on mobile devices. the big additional thing I'm trying is to put all the backend auth code in the biff library instead of including it in the default project like how it works now. it's basically a plugin--there'll be a com.biffweb.auth/features function which takes some optional config and returns a features map, which you include with the rest of your feature maps. the plugin provides reitit routes for the backend functions (all the ui is still done in your own project) and some schema for storing the 6-digit codes in XT. the whole plugin thing is pretty experimental. I'll release it on a dev branch tomorrow and then try using the plugin for yakread. if anyone else wants to give it a spin too, that'd be great. if this plugin architecture feels good, it could open the doors for some more interesting things. it's a way to package up full-stack functionality into a reusable library. e.g. a few of yakread's features might be amenable to releasing as plugins, like a plugin for receiving emails, or for syncing rss feeds... it'd be analogous to firebase extensions: https://firebase.google.com/products/extensions. I'm also planning to write a how-to guide for using firebase authentication. so by default you get biff's passwordless email link/code auth, and if you want e.g. passwords or social sign on, you can use firebase auth (or auth0, or whatever--but firebase auth will at least provide an example)

The release candidate might have to wait until tomorrow, but I have finished writing the reference documentation: https://gist.github.com/jacobobryant/527ce48a66a5138ea907c0307793e140 (The default project will be updated to use the plugin, so you won't necessarily need to read all that.) (If I ever try to get a normal job again I should put all this on my resume... "look, I actually write docs! (Pay no attention to the lack of tests...)")

Jacob O'Bryant03:02:44
replied to a thread:I'm close to having some new authentication stuff released. I already have an additional paste-the-6-digit-code working. I've set up the default project so that it uses email links for signup and email codes for signin, which I think hits the right combination of low friction + resilient to embedded browsers on mobile devices. the big additional thing I'm trying is to put all the backend auth code in the biff library instead of including it in the default project like how it works now. it's basically a plugin--there'll be a com.biffweb.auth/features function which takes some optional config and returns a features map, which you include with the rest of your feature maps. the plugin provides reitit routes for the backend functions (all the ui is still done in your own project) and some schema for storing the 6-digit codes in XT. the whole plugin thing is pretty experimental. I'll release it on a dev branch tomorrow and then try using the plugin for yakread. if anyone else wants to give it a spin too, that'd be great. if this plugin architecture feels good, it could open the doors for some more interesting things. it's a way to package up full-stack functionality into a reusable library. e.g. a few of yakread's features might be amenable to releasing as plugins, like a plugin for receiving emails, or for syncing rss feeds... it'd be analogous to firebase extensions: https://firebase.google.com/products/extensions. I'm also planning to write a how-to guide for using firebase authentication. so by default you get biff's passwordless email link/code auth, and if you want e.g. passwords or social sign on, you can use firebase auth (or auth0, or whatever--but firebase auth will at least provide an example)

Got it on dev: • https://github.com/jacobobryant/biff/blob/dev/example/src/com/example.clj#L16https://github.com/jacobobryant/biff/blob/dev/example/src/com/example/feat/home.cljhttps://github.com/jacobobryant/biff/blob/dev/example/src/com/example/email.cljhttps://github.com/jacobobryant/biff/blob/dev/example/src/com/example.clj#L64 • See also changes to schema.clj and stuff • Also notice that there's no more auth.clj file Tomorrow I'll start updating Yakread to use that commit, and if that all goes smoothly, I'll probably merge to master and release soon afterward. If anyone wants to play around with it, you can do this to create a new project based on the dev branch:

wget 
bb new-project.clj dev
Or set your biff dependency to com.biffweb/biff {:git/url "", :sha "883ac1d23769b1521829196d8dae6cde788a61c3"} to use the auth plugin in an existing project.

🎉 4