Fork me on GitHub
#clojure
<
2022-09-12
>
eskos08:09:22

I hit an issue last week in regards with clj/tools.build/kaocha that might be just me not understanding the intended use pattern, but since I couldn’t figure this out with a colleague and couldn’t really pinpoint if it’s a bug, us not knowing something essential, a misunderstanding or something else, I let the issue simmer for a bit. Enough about confusing preface; We have a deps.edn based project, with basic tools.build setup (with Corfield’s build-clj, as we find it neat and useful ❤️). We want to run our tests with kaocha, and provide an unified interface through commandline to run all the things; one part is self-documenting code (“just look at which tasks are defined in build.clj), another is to ensure that we have full programmatic control over all the tasks and the context they’re being executed in. So, to run tests we do clj -T:build run-tests and we’d like to parameterize that so that we can easily define which kaocha suite gets run, optimally just clj -T:build run-tests :integration , but somehow we couldn’t figure out a way to carry the extra params all the way through the toolchain to kaocha runner. Running kaocha directly isn’t a problem so that’s what we opted on, but we are suspecting that the tests not going through the build tooling might cause issues in mid-to-long run, since we might need to create some more complex build scripts due to the environment our project lives in (two words: health care) which is also why we’d want to sort of defensively make sure the build context is always there, even if it doesn’t actually end up being utilized. So…tl;dr could be “How do we pass kaocha suite selection parameter through deps to tools.build or are we thinking this entirely the wrong way around?” 🙂

seancorfield11:09:06

I'm not entirely sure what you're asking but I will point out that -T (and -X) invocation expect key/value pairs after the function name so you'd need a command line like:

clojure -T:build run-tests :selector :integration
and then your run-tests task function will be invoked with {:selector :integration} as its argument. How exactly you pass that to Kaocha will depend on exactly how you are invoking the test runner in your build.clj file.

eskos11:09:32

@U04V70XH6 Somewhat. We’re currently not doing much at all beyond defaults, in fact our kaocha-within-build is directly grabbed from build-clj’s readme, so our understanding stopped at passing the argument in correct form within build.clj to bb/run-tests .

seancorfield11:09:23

OK, so run-tests accepts all the same arguments that run-task accepts, so you can pass :main-opts as a vector of strings, containing additional values to be added to the :main-opts from the alias.

seancorfield11:09:25

So you could just do:

clojure -T:build run-tests :main-opts '[":integration"]'
but you could use my example above (`:selector :integration`) and then update your run-tests task to conditionally pass the :selector value as a string in a :main-opts vector, as an option to bb/run-tests

seancorfield11:09:45

LMK if you need more information than that @U8SFC8HLP

eskos13:09:30

@U04V70XH6 Ah, those work! 👍 Now I just need to fiddle around a bit since our full setup doesn’t work but that’s on our side. Also a bit shame that :main-opts has to be a string (vector of strings seems to work as well), but that we can live with.

eskos14:09:05

Thinking back, we probably did hit that serialization step issue but without further context/reference just didn’t realize what’s going on.

seancorfield14:09:03

:main-opts is a vector of strings.

seancorfield14:09:14

Not sure what you mean about "serialization"...?

eskos14:09:19

Well, can’t do [:foo] , has to be [(str selector)] . I’m in Clojure code, so it feels weird I need to do this.

seancorfield15:09:07

Because -main functions only accept strings (in a vector).

vemv08:09:15

can I tell Timbre "for ERROR log calls of ns foo , log those as WARN instead"?

vemv09:09:48

....maybe with middleware

javahippie12:09:16

I’d like some pointers or directions on writing a HTTP-API which communicates via JSON. Until now we used transit+json, because both server and client used Clojure. We are using reitit and clojure spec to validate and coerce the payloads to Clojure datastructures. The same works for a JSON payload. We are creating a Swagger / OpenAPI endpoint for the payload, which is also based on our spec. Our specs are namespaced and named in kebap-case, so an example JSON for the payload looks like this (very roughly):

{
    "event/duration-in-minutes": 60,
    "event/description": "string",
    "event/timezone": "string",
    "event/name": "string",
    "event/start-date-time": "string",
    "event/organizer": "string",
    "event/invitation": "string"
  }
This causes problems in the clients, as this format is very uncommon for JSON – usually, the keys would not be namespaced and represented in lower camel case. Even if we considered that we have a spec with lowerCamelCase names for the data types that are received in the API, we still have the namespaces. The only way out I am seeing right now is to accept an un-specced payload, create the OpenAPI files by hand, convert the data into our “core” data model (which is specced) and perform the validation afterwards. Any thoughts or experience on this? Are we missing something, or are we ignoring a tool that solves this?

p-himik12:09:47

> This causes problems in the clients Sounds like those are new clients. So why not just stick to the obj['field'] accessing notation instead of the slightly more convenient obj.field?

javahippie12:09:33

In OOP languages (in this case: Swift, Kotlin, maybe Typescript later) it is common to map the requests and responses to objects. I’d like to provide a standard API that does not close this door for others

p-himik12:09:54

Using POJOs is not closing the door to others - on the contrary. But, assuming the decision to use namespaced keywords was the correct one - what happens when your payload has a/b and x/b? It will invariably be trading correctness for convenience. And later on, if the time comes when you knee deep in mud with sudden x_b and xB that came out of the blue, it will be trading correctness for "just one more crutch so it works for one more week." Been there, done that. Not fun.

p-himik12:09:33

Even if the assumption is wrong and using namespaced keywords is unnecessary in your case, you can get rid of them and solve 95% of your problems. However, there will still be potential cases of a-b, a_b, and aB - still mud. Now, if you have and will have total control over your API and it will never have anything beyond your control - that might be fine and that 5% might never pop up. You can see better from there.

javahippie12:09:04

It’s not necessarily trading for correctness. We can still validate the payloads before mapping them to our “domain” model (for a better word), and could even apply a spec on this – for a stable API, it is necessary to have a dedicated, decoupled data structure, anyways. I agree with you that the namespaced keywords in Clojure are an approach that works very well, but I’d also like to provide an API which is standardized and does not contain any surprises for developers who don’t know that there is Clojure running on the server. Edit: But although I don’t agree in all points, thank you for your answers, that’s a good additional perspective to have!

p-himik12:09:46

By "correctness" I don't mean the data correctness but rather the model one. When your data is a/b-c, it's not a-b-c or a_b_c or aBC or whatever else the common JSON API notation happen to be. It's still a/b-c. > I’d also like to provide an API which is standardized There are no standards here - just conventions based on convenience, that's it. But a mapping is indeed possible. My point is that, such a mapping is useful only as far as it gets you. As soon as you need something that's not entirely within the boundaries of that mapping, it immediately becomes a hindrance with a great potential of you never wishing you ever went that way. Pretty much same deal with ORMs, conceptually - such a great thing, on the surface and when you stay within its area of convenience. > does not contain any surprises for developers who don’t know that there is Clojure running on the server. The only way to guarantee it is to just serve a plain JSON data without any promises about what a key will not have. There are plenty of APIs that don't serve data that can be immediately accessed as obj.field after parsing - and that's totally fine.

javahippie12:09:57

Just to be sure we are talking about the same model, would you expect the API to have its own model, or are we talking about our applications internal model? Just because the key in our internal model is a/b, does not necessarily mean that the b in the model for the HTTP call is the same, does it?

Daniel Tan12:09:03

It causes alot of issues when using stuff like C#, usually I convert it to something like underscore for C# to denote name spaced items, then use CamelCase for the rest

👍 1
p-himik12:09:14

I'm talking about the model of the existing transit+json API that you already have. The internal model does not matter here. It's the simplest solution, to just not do anything about it. And it works great in so many cases. There are plenty of answers on how to parse JSON data in Clojure when the keys can have all sorts of symbols and not just the ones that can go into keywords. The simplest answer is always "just don't keywordize your keys". Same with DBs - if there's a doubt, just don't keywordize column/table names, or at least don't transform them in any way. Extra transformations incur extra assumptions incur extra work incur extra problems. Ones that you get stuck with, unless you make a v2 API.

👍 1
Daniel Tan12:09:32

Event_DurationInMinutes

javahippie12:09:19

Thanks for the input, really appreciate it! I guess I have to take a walk and think about some stuff a little more. I guess the declarative approach we took with reitit + spec + swagger + coercion made us mash up some things in general :thinking_face:

pithyless14:09:26

There are several competing views and trade-offs, so there is no clear cut answer; but maybe that gives you some more food for thought for your walk :)

pithyless14:09:13

I, for one, welcome our new Namespace Overlords. But if I were looking for examples outside of Clojure, perhaps someone has shared what really large federated GraphQL schemas look like; I would be surprised if it wasn't a bunch of "namespaced" types via LongCamelCaseNameAlaNSCocoaJava

javahippie14:09:37

Thanks for the links! I found the vvvvalvalval blog some minutes ago and think it describes my issues quite well, but I also saw a lot of great arguments in the comments

didibus15:09:27

Just use unqualified keys no?

didibus15:09:25

Also, in general, I have two spec. I've got a spec for my API designed around clients. And I have a spec for my internal models. And sometimes I'll even have another spec for my database.

javahippie15:09:35

@U0K064KQV that’s what I am planning. “External” spec with :req-un, and internal, namespaced spec

👍 2
didibus15:09:58

Even in OO, people almost never use their API objects internally, it's not considered a good practice to couple yourself to API concerns internally. And even more so, in OO, normally the generation goes the other way, your classes/objects are generated from the API spec itself, so if the generation ever changes even your internal code would be broken.

Ed09:09:22

What I've seen in a lot of JSON APIs written in JS/TS is the grouping of values behind a single key to represent namespaces. As in

{
    "event": {
      "duration-in-minutes": 60,
      "description": "string",
      "timezone": "string",
      "name": "string",
      "start-date-time": "string",
      "organizer": "string",
      "invitation": "string"
    }
  }
So in the JS code there could be an Event object that is used to group these properties together, but they really refer to the same entity and in the past I've worked on a codebase where some JS was passing information to some Clojure code with several of these groupings in the same message. The people working on the Clojure had no real input into the structure of the data being processed, but it quite naturally fit into a structure like:
{
  "namespace": {
    "key": "value"
  }
}
This might be a useful strategy for serialising your data to something that makes sense to developers in other languages @U0N9SJHCH?

javahippie17:09:03

Thanks for the idea, @U0P0TMEFJ! It’s a good approach for structuring data, but I’m not sure how I would handle maps which contain keys from different namespaces, that could end in confusing hierarchies in the json :thinking_face:

👍 1
borkdude15:09:38

This is currently more a toy/experimental project to see what I could do with SCI in postgres

Carlo15:09:39

Were you able to run more complex functions? How does it work? The clojure code is compiled via SCI to what?

borkdude15:09:40

SCI is an interpreter. you can give it access to any clojure library you want, as long as you include it

borkdude15:09:57

This project is compiled to a native binary which runs within postgres

Carlo15:09:04

I see, thank you: so it's the SCI interpreter itself that's compiled and integrated, and the clojure evaluation happens "on the fly"

Carlo15:09:07

Mostly, I was wondering if this could be use to do some light transformation on a postgres TRIGGER

borkdude15:09:01

@UA7E6DU04 Feel free to try it out and bug me with questions

emccue17:09:57

Started looking at wrapping the new helidon nima server in ring...why does their http server complect itself with routing? ugh code so far in thread

😍 4
👀 1
Ben Sless18:09:13

Anything special about it?

lispyclouds18:09:33

It is based fully on Loom/Virtual Threads, quite exciting though!

didibus01:09:14

Built by Oracle itself :thinking_face:

didibus01:09:56

> I will introduce Helidon Nima - new microservices framework which is built on top of a server designed for Loom with fully synchronous routing that can block as needed > I would guess the blocking synchronous routing is a big deal, maybe that's why it's part of it.

didibus01:09:58

So did you try to Thread/sleep or something inside the handler?

emccue15:09:16

No, i kinda moved on to the rest of my day after posting it here

emccue15:09:39

they still haven't released sources or docs so its more frustrating than I want to handle right now

Cora (she/her)19:09:21

does anyone offhand know of a built-in way in java to parse a query string? I can get the query string using http://java.net.URI/URL but it would be nice to be able to parse it into a map

p-himik19:09:04

Nothing built-in. Gotta do it either manually with a few extra lines or use some library for it.

dpsutton19:09:59

couple lines copied over with attribution from cemerick’s url package: https://github.com/cemerick/url/blob/master/src/cemerick/url.cljx#L48

p-himik19:09:09

Which doesn't support repeated params. But query strings are highly domain-specific (in the broad sense of the word "domain"), so that might be fine.

dpsutton20:09:02

good point. I’m also a big fan of the ruby style for indicating params are collections with multiple elements

p-himik20:09:57

Yeah, but that whole area is pretty much unregulated. Even when it comes to plain kv-pairs, the query format is not standardized in any way in the context or generic URLs. Might even have a base64-encoded string of msgpack'ed transit data in there as the whole query. :)

Cora (she/her)00:09:14

thanks everyone!

Cora (she/her)00:09:44

thinking more about it, an elixir keyword list like object would be just about perfect for parsing query strings

Cora (she/her)00:09:13

oh well, I just pulled in lambdaisland/uri since it makes what I'm trying to do more convenient in other ways too

skylize20:09:00

Why is this

(let [foo (fn [] (foo))] (foo))
an unresolved symbol foo, but this
(let [foo (fn foo [] (foo))] (foo))
is just a bad idea.

p-himik20:09:09

Because within let, the names are bound to the values only after the values are evaluated. You might wanna use letfn, BTW. But depends on your use case. The bottom option is also fine, especially if you mix regular values with functions within the same bindings vector.

✔️ 1
borkdude20:09:34

you could also use recur

1
skylize21:09:39

The syntax for letfn feels weird next to the rest of the ways of uses of let or fn. And it doesn't provide for non-function bindings. Gets totally unreadable quickly when outside the very specific case of multiple functions but no other bindings.

(let [x 1]
  (letfn [(foo [y]
               (+ x y))]
    (let [z (foo 2)]
      (letfn [(bar [q] (* z 4))]
        (let [r 6]
          (bar r))))))

(let [x 1
      foo (fn [y] (+ x y))
      z (foo 2)
      bar (fn [q] (* z 4))
      r 6]
  (bar r))

skylize21:09:22

Yeah. I like the recur option. 👍 Although not all recursive functions can reasonably be made tail recursive. Question really was about the how/why of binding situation: Naming an otherwise anonymous function gives it a binding available from within itself. While naming it in the let does not.

didibus01:09:57

The naming for recursion inside fn is unrelated to the let binding, you could use different names for each for example.

didibus01:09:43

You can also wrap it yourself in an atom (or maybe a promise as well).

(let [foo (atom nil)
      _   (reset! foo
           (fn[a] 
            (println a)
            (when (true? a)
             (@foo false))))]
 (@foo true))
Let bindings are supposed to be bindings, so effectively immutable final local variables, it wouldnt make sense to have them exist before the binding ends. Think about the non-function use-case:
(let [i (+ i 0)] ...)
It would be quite difficult for let to be smart and somehow try to recognize when the bound value is delayed somehow versus when it isn't and something very weird would happen like here. Like it could work for fn, and maybe delay, but what about custom macros/functions that also delay things?

didibus01:09:55

That's why there's letfn, it solves this exactly for fns only, but it's more for like accross multiple fns, for mutually recursive calls. > All of the names are available > in all of the definitions of the functions, as well as the body. > For just single recursive function I would use recur or give the fn a name.

skylize02:09:16

Lol. Definitely no code... smell there. 🤢

(let [foo (atom nil)
      _   (reset! foo
           (fn[a] 
            (println a)
            (when (true? a)
             (@foo false))))]
 (@foo true))

didibus05:09:52

Ya, that was just for being completionist about all options. Having a name in fn is the normal way, and letfn is the way for mutual recursion

Al Z. Heymer10:09:15

For the sake of ease: The unresolved foo error is not by calling the function you let'd, but rather calling foo before it exists in the symbol table. To make that clearer, replace the foos with f1, f2, f3 and observe the error. The recursion in the second option works because you scoped the foo for recursion to itself.

👍 1