Fork me on GitHub

Hi, I'm experimenting with some stuff in Clojure and would like to ask for thoughts and insights. I have a state atom with some nested structures and I need to perform some queries on it. Some queries are cacheable, like fetching a value of some immutable entry by id. I'm wondering if core.cache (or core.memoize) could be the correct tool for this. The data structure isn't large, some thousands of entries. The same queries are performed very frequently. There aren't many different types of queries, less than 10. Optimizing for latency is a priority. I guess I'm curious of the performance characteristics of core.cache regarding overhead. Would such a simple case be better solved with some custom lookups I maintain myself? The suggested use cases in the docs don't seem as lightweight as mine. Obviously I should profile this in the end, but I thought I'd learn more by asking for input.


if you are using the right data structures, most queries should be cheap - what kind of calculation is the query doing? what is the structure of the data?


core.cache has the memory overhead of storing each function input and result, and the perf overhead of the equality check of the arg list and atomic update of the cache (atoms are relatively expensive if altered frequently)


My queries are mostly ”find element with identifier”, so it’s basically O(n) searches. The elements have more than one identifier so I can’t simply use a map. In a way I’m trying to achieve a very light indexing for the data. Since the data rarely changes caching seemed like a good fit. But the application is very latency sensitive (a part of a realtime multiplayer server) so I’m wondering if for example maintaining multiple maps as lookup tables would perform better than core.cache.


> The elements have more than one identifier so I can’t simply use a map. I don't understand why this would be a problem? the same map can be the value for multiple keys in a map


core.cache is itself an immutable hash-map lookup, so I don't see how it could speed up what is already a hash-map lookup


if multiple elements have one identifier, then you can use group-by to build the lookup


also I would only use core.cache for side effecting operations (the kind where multiple calls with the same args might over time have different results) - otherwise massaging the data into a structure that allows fast lookup is simpler and also more efficient


Ah, using the same map for multiple different keys is a thought I never had. I think that will help me forward, thanks a lot!


So my -main function takes a map and I call it with cli tools like clj -X:ip-weather :ip '""' but if I create an uberjar and try to start the app with java -jar foo.jar ... how do I feed it those arguments correctly?

Alex Miller (Clojure team)14:01:52

-jar will only call the static void main(String[] arg) method of the main class, which it sounds like you don't have

Alex Miller (Clojure team)14:01:55

we have a long-going discussion about how to expose the exec stub code but not decided on that yet (but you'd still need to call that instead of your main)


Ahh, ok. So could I change the arguments given to -main so that it would work for both? The cli tools require key value pairs for arguments doesn't it?


Not that I actually need this for a real world case right now. I was just curious. When using the cli tool, it took 17-20 seconds to get it running but when I hardcoded the values in order to test the timing on the jar it only took 5 seconds.

Alex Miller (Clojure team)15:01:13

well, -X assumes a function that takes a map and -main assumes an array of strings so they aren't compatible

Alex Miller (Clojure team)15:01:02

certainly if you have a lot of deps, precompiling with aot will save you a lot of startup time


Yeah it was mostly pathom which I imagine is a big dep. The official tutorial on it is really well done and fun btw. Really cool piece of software.


So then I should change my -main to take an array of strings and then parse that into a map to feed to a run function or something you think? Or just stop this tinkering. haha

Alex Miller (Clojure team)15:01:44

yeah, better to make your map entry point something else and then wrap with -main


One more clarification and I'll let you go. So "an array of strings" should be passed in as java -jar foo.jar ":ip" "" or [":ip" ""] or how would that be given to the command? And then my -main should take [args] or [& args]?

Alex Miller (Clojure team)15:01:43

when you call java -jar :a b :c d that will invoke -main with an array of [":a" "b" ":c" "d"]

Alex Miller (Clojure team)15:01:37

so mostly what the -X exec stub does is clojure.edn/read-string the args, then assoc them into a map


Ok, got it. Thanks so much, Alex.

Alex Miller (Clojure team)15:01:25

if you have deeper questions on that, probably best to take it to #tools-deps

John Bradens17:01:31

Hi, I'm having trouble understanding, what an API is and what it means in the context of Clojure & ClojureScript web app?

John Bradens17:01:52

I see thinks online, like "how to build an API" but I don't see this for web development with clojure and clojurescript

John Bradens17:01:24

Can I make a web app without an API? Is it something that's already included in a lot of lein templates like luminus?


The "what is an API" question is almost a philosophical one. The key is that for a program meant to be used by other programs (eg. a web server, used by the front end js code and the browser) API as distinct from eg. CLI or GUI tells us that the consumer is code written by a human, rather than a human directly using the program, and that guides us to a different style of humane interface design.


so simple intuitiveness and consistency with other applications guides a GUI, ease of programming (which is a different kind of intuitiveness, refined for application programmers) guides an API


@bradj4333 hope this helps, forgot to tag you above


none of this is in any way different in clojure than it would be in another language


(except the details of what we find intuitive are slightly different, if the API is meant to only be used from clojure?)

John Bradens17:01:16

@noisesmith Thanks, that does help a lot. I appreciate the comparison to a GUI, because I am familiar with using those at least


another aspect: if code works it works, of course, but what tells you an implementation is elegant or high quality? with a GUI that means easy to use when pointing and clicking, with an API that means easy to access from your code

John Bradens17:01:38

I think one thing I'm having trouble understanding, is... if I generate an app using lein, like the luminus template, then have I built an API? One of my programming friends asked me "have you built your API yet?" And I don't know if I did or not? I just have code, and then you can type things in the browser etc, like for example, I followed a to-do list tutorial. Did I make an API when I did the tutorial? Or not necessarily? Or is it hard to say?


typically the luminus template creates two programs - one clojure program that runs under java and acts as a web server, the other a clojurescript program that runs in the browser the clj program is an API for the cljs program


you can tell them that your template generates the API and the client code in one codebase using the same language

John Bradens17:01:11

@noisesmith Ok interesting. So is my API the collection of clj files that work together as a program? Is the API basically encompasses the whole backend including the database, or is it just the clj part of the program?


from the point of view of the front end code (the usual point of view here) the API is the part that the front end is expected to access


from the point of view of the back end code, the db has an API (via sql and the libraries that provide programmatic access)


the API isn't the clj code, it's the INTERFACE to the running program that other code sees and uses


you can change the code radically and it's considered the same API if it reliably gives the expected answers to any given query


analogy: LEGO, the bumps and crevices form interfaces, it's the same interface if the same pieces can lock into it, no matter the shape of the actual object

John Bradens17:01:32

Ohhhhhhhh ok wow that's cool. So the code sees and uses the interface, but I don't see the interface? I only see lines of code in the back end that generate the interface for the front end? And the specific lines of code don't matter, as long as the front end gets the necessary information/connections?

John Bradens17:01:58

The lego analogy helps a lot, thanks. I'm just now trying to put together how this connects to the lines of code that are written, and how they generate this?


Every "thing" in a stack uses some sort of API. Have you programmed something before?


he said above that he had not


Ah missed that 🙂

John Bradens17:01:18

Right now it seems a little like magic that this all happens

John Bradens17:01:32

No, I'm a total beginner that has just been following clojure & clojurescript tutorials

John Bradens17:01:45

Trying to learn everything as I go. It's like drinking from a fire hose

John Bradens17:01:32

Ah, so the API is the lego bumps that connect all the pieces of the stack together


the relationship between the API and the code is usually kind of abstract, though in recent years there have been projects to programmatically define API (and make the promises that this programmatic description implies testable)


@bradj4333 for example, something I've done in multiple jobs is to take a working ruby codebase, and make a clojure replacement that can replace it for an existing client (idea being that clojure would perform better or be more maintainable) - so it would provide the same API, reducing the work compared to a full rewrite


When your friend asked you if you'd built your API yet he could mean any number of things, like have you managed to get a server going, or have you thought about and written down what your whole app is supposed to do and so on. When you write a Clojure hello world program you'd be using the API:s that the programming language authors defined for you like println and so on, and the programming language authors are using the underlying Java APIs to represent the strings.


But yeah - thinking about it as lego bumps or a contract that makes it so things fit together isn't too bad

John Bradens17:01:57

@noisesmith Ok, so the API is the outcome of the clojure replacement? The code itself provides the same API as ruby, but the way it is done in clojure, is a better way of provding the same API?


the API was created by the original program, and the clojure replacement's job is to recreate it

John Bradens17:01:39

Ok I think I see

John Bradens17:01:07

@emilaasa Ok, thanks for the println example and connecting it with the Java APIs

John Bradens17:01:21

I might watch some more videos about it


You can go even higher level and talk about things like a weather API which implies weather info that can be consumed by a computer program. Or go lower level until you end up reading some of Intels instruction set specification so you can write machine instructions. APIs all the way down. Point is an API is something thought up by a human, so humans building the next / adjacent lego piece can use their thing.

John Bradens17:01:50

Let's say I wanted to create an api from scratch, that was going to connect backend and frontend. I'd write code in clj right so that the cljs would read it?


to be precise the cljs wouldn't read your code, there would be a query / response over http between the two


I mean it's usually http

John Bradens18:01:15

Ok, in one of my tutorials I use the http-kit library. How does that relate to the API?


a web server and web client use http to communicate, it's a protocol

John Bradens18:01:55

Is the http another lego block between clj and cljs? Or it's just used for communication?


http-kit translates between the API of the protocol itself and clojure code


It's pretty interesting trying to explain this shit from first principles.


so you can see http library as a lego block - it makes calls to communicate with the other machine look more like your other call


I can't belive how complicated it sounds when I try to write out how it works 😄


and http itself is designed for gluing programs together over the network

John Bradens18:01:40

Haha I appreciate you both helping me! I am very confused but this is helping a lot

John Bradens18:01:49

I hope to someday just intuitively understand all this


noisesmith I thikn your explanations are spot on - I'm going to go drink whisky instead

John Bradens18:01:30

The analogies have helps a lot, thanks

emilaasa18:01:49 - understanding browsers also help a lot when doing web application programming. This was just the first google hit that seemed OK, but something like this

John Bradens18:01:44

Thanks!!! I'll read this

John Bradens18:01:19

I'm so impatient about just making stuff right away, but then I have all these issues when I don't understand the underlying concepts. So I keep going back and forth, coding, then learning the basics


I think that's what learning always looks like

John Bradens18:01:20

Awesome, thanks for all your help


I don't think it helps things that these days what is considered basic / normal is already a multi client distributed application. I took a long digression into the world of CLI programming on a single user computer and the simplifications (your interface with other programs will be text, files, and return values, one program runs at a time in most use cases...) really make it easier to build the individual concepts I think

👆 1

of course you can make CLI programs in clojure (if you are willing to wait for startup, or use a tool like babashka instead of clojure itself)


but that's just what worked for me, YMMV

John Bradens18:01:36

Wow that sounds interesting. And yes, I think it's all very complicated to try to jump into, even the beginner stuff! I have no idea what's going on under the hood. So, I'm able to do tutorials that are these web apps, but if I want to add a feature of my own, I am totally lost

John Bradens18:01:56

I don't know if I'll have time to make a CLI in Clojure and look into all that, but it sounds liek it would be really useful, so I'll keep it in the back of my mind


CLI is the least effort version of UI possible (or nearly so) - all you need is strings in, strings out

John Bradens18:01:09

Ohhhhh ok good to know. Yea I like that, being able to build the individual concepts


also, every programmer ends up doing a lot of CLI usage (git, the shell, build tools like lein or clj...) so it's a world you'll need to be somewhat familiar with

John Bradens18:01:29

Wow ok I didn't even know that those things are CLIs! Good to know


you use it by typing in a string and it prints something back right?


anyway, best of luck

John Bradens18:01:27

Yes, got it. Thanks!


One thing that might be a fun hands-on experience for you is to run through this Pathom (a clojure library) tutorial that actually has you reach out to a couple of api's for some data and display that information. It's pretty straight forward (ignore trying to understand some of the high level stuff like resolvers and what they are). I did it last night with little difficulty.

John Bradens18:01:50

Cool @chase-lambert I'lm going to try that! I want to wrap my head around APIs in as many ways as possible


One of the links in that tutorial is to a github repo with a bunch of different public api's. What those do is allow you to "call out" to them with your clojure code and then they give you back some data (a lot of times as json). Then you take that data, parse it, and display it or whatever you want with it.

John Bradens18:01:07

Oh cool, ok. I notice the language is Python, does the clojure code not care about interacting with Python api's? Does the language of an API not really matter to use it? (I realize this question might totally not make sense because I'm still a little confused on how codes are related to the api)


So an api used in that tutorial is Metaweather. These docs show you how to ask them for data and what that data looks like when it comes back to you. So for example you can do (slurp "") right now in a running repl and it will return some json: "[{\"title\":\"London\",\"location_type\":\"City\",\"woeid\":44418,\"latt_long\":\"51.506321,-0.12714\"}]"

John Bradens18:01:25

Ok, so the language behind the api doesn't matter, because you just get data from it in json format?


Yeah a lot of API's will give you instructions on how to ask for data using various programming languages (usually the popular ones like js, python, java so you have to learn how to translate that to clojure).

John Bradens18:01:19

Ok, this makes tons of sense now!! Thanks!

John Bradens18:01:57

Is there a resource for how to do the clojure translations? Or do you figure it out on your own?


For example, Supabase is a cloud db provider (or something like that) and their api reference shows you how to use it with various languages:

John Bradens18:01:56

I'm not sure how I'd use something like supabase for a personal project. If i create a web app using the luminus template, and I start adding features, let's say I want to make a facebook clone, when would I decide to use a public api? I'm assuming there's also api's I would pay for, to solve a problem instead of coding it myself?

John Bradens18:01:20

This is a question I have, because my friend has sent me a couple links to api's that he's used for his startup, and I don't know what to do with them!


So in this same way, your Clojure backend is like an API for your frontend. Your frontend will ask your backend for info ("ok the user has clicked save, please put their info in the database, etc and give me back this data to show them") and then your frontend will display that data it got back from the backend "api"

👍 1
John Bradens18:01:27

Ok cool, thanks. That makes sense to me, I like understanding this all in terms of the data a lot


Same. Caveat is I'm pretty new to this stuff too so might not be explaining it too accurately. This is just how I think of it so far.

John Bradens18:01:29

You're definitely a few steps ahead of me, so I appreciate your explanations

John Bradens18:01:52

Once agian though, if you don't mind @chase-lambert, can you explain a situation where I would use something like supabase? Like let's say I'm creating a facebook clone and using libraries and templates in clojure, what problem would I have, or situation, where I would want to use a public api? I don't think I fully understand how to take advantage of that stuff right now & I'd love to know more


Well Supabase is basically a cloud based postgresql database from my current understanding. Most folks are using their own db's hosted on their servers or whatnot. But Supabase also handles user authentication which is the main reason I want to use it. I'm not confident enough right now to want to do something so security critical by myself. So I want to use their api to help me do that.


One thing I am looking to have in an app I'm building is location based parity discounts (so if you are in the US, you get one price, if you are in Colombia or whatever you might get offered a discount to better reflect economic parity) and so I might need to get a user's location. I can try and write all that code myself or maybe there is a good api that gives me a user's location and I can just reach out to that api with whatever data it needs from what I have and it sends me back the user's country.

John Bradens18:01:08

Ohhhhhhh I see.

John Bradens18:01:04

So if I identify that I need some data for something, I can use an api for that. Or, for handling certain things that I don't feel comfortable with yet

John Bradens18:01:59

I think CLojure has a library called "buddy" for user authentication. This might be a stupid question, but is that basically not "good enough" to use for production level apps? Like good for tutorial purposes only?


Yep. Just like pulling in a library as a dependency. In some ways, those libraries/dependencies are their own API and you have to figure out how to "call in" to them by using things like (:require [ :as json) and then using it json/read-str data-I-got-back-from-api :key-fn keyword) for example that would change that data we got back from the metaweather api into a Clojure map that I can then use in my code.


Not sure about buddy . I have seen it mentioned a lot but it might be in maintenance mode. But yeah, I'm not the right person to ask for auth. It intimidates the heck out of me and I haven't wrapped my head around it.


Hence my desire to see if I can use an outside source to do it for me. haha

John Bradens18:01:12

Wow ok, that is very helpful for me to think about api's like dependencies. And then it makes a lot of sense, why for auth, you have more options to choose from and can go with something like supabase!

John Bradens18:01:47

I feel like a whole world of possibilities just opened up for me lol, tons of more tools that I can use! I thought I was basically limited to clojure libraries


Another service provider that can provide auth services is Google Firebase and some clojure members have written their own Clojure libraries to help you use the Firebase api: (looks unmaintained, probably wouldn't use this) while others just reach out to the Firebase api directly using clojure(script)'s great interop:


Well with interop Clojure gets access to the whole Java ecosystem while CLJS gets all of JS. (and there are now bindings for Python, Erlang, Dart maybe?, etc) so technically Clojure might have the largest ecosystem in the game right now.


The flipside is you have to figure out how to do all that yourself most of the time because finding good clojure docs for it can be tough (especially for beginnners)


That's why this slack community is essential if you ask me.

John Bradens18:01:28

Wowwwww ok that's very interesting and good to know! I just have to spend a little more time learning on how to use all of this

John Bradens18:01:36

Yes, I need to post here more, I'm always very grateful when I do

John Bradens18:01:43

And learn tons

John Bradens19:01:21

Why do you think there aren't much good clojure docs? I think there's a big whole in the ecospace for something like khan academy, but for clojure web stuff. Do you think people would use that?

John Bradens19:01:07

(I'm suddenly thinking of a potential project for some day after I become a master of all this..)


Well, I mean, good documentation is hard labor and most are providing it for free. Plus, the community is maybe 25,000 members or so I would guess while I'm betting the javascript, java, and python communities are measured in the millions for each of them. The sheer volume of docs available for those will mean a lot more beginner friendly docs there (and a lot of bad)

John Bradens19:01:19

Yea, that makes tons of sense. Anyway, I'm really grateful for slack! Thanks again for all your help 🙂

John Bradens18:01:15

@hiskennyness WOW this is very helpful!!! Thanks, I'll look into this!!


Watch out for the bit rot! 🙂 Ping me if you get stuck.

👍 1

Is it possible to get the arity of an anonymous function defined as (fn [a b] (list a b)) or #(list %1 %2)? I'm trying to write a general "flip arguments" function, since it would make threading macros, partials, and some other code-segments more readable and convenient, instead of needing to wrap them in a fn form with swapped arguments.


getting arity from a function isn't supported, the invoke is overloaded with multiple arities but there's no reflective trick to know which are implemented aside from calling them and checking for a method not implemented error the java compiler enforces implementing every method on an interface, no reflection trick for checking implentation exists that I know of (since clojure directly generates byte code it doesn't have to meet that constraint)

👍 1

What would a general "flip arguments" function do if it could figure out the arity? Maybe there's a variadic solution...


For example, we have a flip function in our utility library that's inspired by Haskell's flip function -- but extended to more arities.

👍 1

@U02QCB2FHU6 ☝️:skin-tone-2: Is that anything like what you're thinking?

👍 1

Yes, it is. Alternatively I'm guessing you could 'hack' it by eval-ing a backquoted list, something like (fn [f a b & args] (eval `(f b a @args))) I was hoping to have it actually check the arity, however, so as to avoid the use of a naked eval or having to turn it into a macro rather than a function.


you could try/catch for each arity and check if the error thrown indicates that arity doesn't exist :D

👍 1

I mean, that's expensive but likely cheaper than eval

👍 1

maybe reflect on the methods implemented?


oh actually I think this would work!

)user=> (-> (fn []) (reflect/reflect))
{:bases #{clojure.lang.AFunction}, :flags #{:public :final}, :members #{#clojure.reflect.Constructor{:name user$eval148$fn__149, :declaring-class user$eval148$fn__149, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Method{:name invoke, :return-type java.lang.Object, :declaring-class user$eval148$fn__149, :parameter-types [], :exception-types [], :flags #{:public}}}}
(ins)user=> (-> (fn ([]) ([x])) (reflect/reflect))
{:bases #{clojure.lang.AFunction}, :flags #{:public :final}, :members #{#clojure.reflect.Method{:name invoke, :return-type java.lang.Object, :declaring-class user$eval152$fn__153, :parameter-types [java.lang.Object], :exception-types [], :flags #{:public}} #clojure.reflect.Constructor{:name user$eval152$fn__153, :declaring-class user$eval152$fn__153, :parameter-types [], :exception-types [], :flags #{:public}} #clojure.reflect.Method{:name invoke, :return-type java.lang.Object, :declaring-class user$eval152$fn__153, :parameter-types [], :exception-types [], :flags #{:public}}}}
that's using clojure.reflect - notice that each invoke method has a :parameter-types you can inspect for length

👍 1

@U02QCB2FHU6 I think this does what you want

user=> (->> (fn ([]) ([x]))
                 (filter (comp #{'invoke} :name))
                 (map (comp count :parameter-types)))
(1 0)


it breaks for varargs though

ins)user=> (defn arities
               (->> f
                    (filter (comp #{'invoke} :name))
                    (map (comp count :parameter-types))))
(ins)user=> (arities (fn []))
(ins)user=> (arities (fn [& args]))

👍 1
😄 1

It seems to work for any combination of standard (explicit) arities. Thanks, I wasn't aware clojure.reflect existed!


reflection isn't cheap (but it's cheaper than the other weirder options discussed prior)


my hunch is it could be fixed by checking for applyTo as well as invoke


nope - but there's doInvoke and getRequiredArity


Actually, I wonder how well just using apply would do on this problem, since f would probably have to be a function...


right, most versions of flip I've seen rely on apply mixed with reordering the arglist


but instead of saying that I jumped into the reflection thing since it's a harder problem 😆

❤️ 1

Yup, apparently apply with the last element as an empty collection just ignores it, so apply could actually work for the 2+ arity option! Reflection is still useful, though, since we can add a check for single-arity in the input function and do nothing in that case, making flip resilient to invalid inputs. That resiliency could be useful if you were creating a DSL or a tightly-integrated library framework (and likely is also useful for things like dipping into typed-clojure, though I'm not sure how practically applicable that is).


if all you care about is the first two args: (defn flip [f] (fn [x y & args] (apply f y x args))) just works unless it's a function of arity 0 or 1, in which case you get an arity error where it's called

👍 1

Mhm. That's where the reflection code you found comes in 🙂


I'm failing to think of a case where you want to flip an arbitrary function but a zero arg or one arg function would be valid

👍 1

It'd be pretty rare, but for instance if you want to write a macro for converting a list of functions from a ->> macro format to a -> macro format (e.g. for library compatibility reasons), then you could be lazy and map "safe-flip" to every function in the list, rather than having to write the code yourself to check the arity. A convenience function as part of a larger library, perhaps.


I didn’t read this whole thread, sorry if I missed this, but for theading pipeline operations, I’ve used

(<- & body)
Converts a ->> to a ->

(->> (range 10) (map inc) (<- (doto prn)) (reduce +))


which is just

(defmacro <-
  [& body]
  `(-> ~(last body) ~@(butlast body))))


@U02QCB2FHU6 perhaps thrush is close to what you seek?

(defn thrush
  "Thrush combinator. Txy -> yx. Or, the reverse of `comp`.

  Implementation, due to Chris Houser, via Michael Fogus, via Combinatory Logic
  (i.e. logic with Combinators).

  Raganwald explains nicely:
  [& args]
  (reduce #(%2 %1) args)) it can be used.