That's not what I'm seeing, I get:

$ bb /tmp/dude.clj
./invalid.html (No such file or directory)

@U04V15CAJ how about this code without slurping

(require '[babashka.pods :as pods])
(pods/load-pod 'retrogradeorbit/bootleg "0.1.9")
(require '[pod.retrogradeorbit.bootleg.utils :as utils])

(try (-> "<div id=\"wrong attr\"\"></div>" (utils/convert-to :hickory))
     (catch Exception e (println (-> e Throwable->map :cause))))

I'll try later today, thanks for the clarification

I looked into it. I think what we should do is convert the read error into an exception response in the pod loop. I'll make an issue for this

I found a solution

so quick! I saw your commits in the repo, thank you very much @U04V15CAJ! though I can’t try it immediately but I’ll let you know after I try it again

@U042NRFKHQE which os / are are you using?

I am using MacOS Monterey (v12.4)

ok, you can install and test it with:

bash <(curl ) --dev-build --dir /tmp

@U04V15CAJ I just tried it, it seems like it still hangs like previously, I made sure I’m using this version

Babashka v1.0.169-SNAPSHOT

Are you not using m1? This might explain it since the intel build didn’t run

Yep, it's not M1

If this build succeeds, you can try again: https://app.circleci.com/pipelines/github/babashka/babashka/6411/workflows/3b5192a1-367f-464c-9dc6-fdf725ab8ff7/jobs/33604

yep, just repeat the --dev-build install once again

just did, it’s working now! printed this successfully and script terminates

Cannot read EDN:  "{:type :element, :attrs {:id \"wrong attr\", :\" \"\"}, :tag :div, :content nil}"
Invalid token: :

awesome

About 1: it seems the question here is drop-in compatibility or not About 2: most headers, like 99% of them are single values, but with set-cookie for example, you can have multiple... Most existing clojure clients use variant 2...

re 1: the URIs that aren't also URLs would be invalid for an http client right?

Don't know, let me try..

Message:  invalid URI scheme file

(I've yet to see a working example of this)

To me it seems like following what the existing clj http libraries do makes sense, since the alternative isn't obviously better in every way

TBH the vector thing: I went with the "dynamic type" if you will in babashka.curl, because I only discovered later that the same header can occur twice and then to not break backwards compatibility, you're kind of forced to do it this way. I can imagine the same thing happened to those other libs earlier on.

Ah, interesting

I wonder if there are any that would be incorrect to send multiple times

Like Content-Length maybe? If so I think that is an argument for the dynamic approach too

Yeah, one could argue that the Java result always has a List because dynamic typing is much harder in Java

Look at this interface.. https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpHeaders.html

Interesting, not super statically typed out, like plucking out well known properties

well, the structure is statically typed as Map<String,List<String>> but they have convenience functions for getting the Optional<String> first element ;)

I would second the approach of header values as list or values or a single value. im also fine with it always taking a list too. this should simplify some contajners code 😄

always being a list of values seems more consistent

also this happens in query parameters too

yes, I'm sensitive to this "consistency" argument, but there's also the "drop-in" argument... Perhaps we could offer an API function to get the first header: (header response "Content-Length") or so

not sure whats the perspective behind the drop-in argument is, is it something the other http libs offer?

no, but having it "single value or multiple values" will make babashka.http-client work better as a drop-in for babashka.curl, clj-http, etc, etc

Note that the ring spec also allows for this: https://github.com/ring-clojure/ring/blob/8af4ab93190dfe5b4827c14b416a4cb92e18cdaf/SPEC#L117

the ring spec applies to http servers though

right, makes sense! im happy with supporting both. being able to take in a vec is great, everything else is a cherry on top 🙂

I was referring to the response, but I didn't mention this yet - sorry

so if the response does something else than clj-http, babashka.curl, etc you can't use it as a drop-in

ahh got it

yeah i got confused, makes more sense now

but since it came up, would be nice to have vecs in requests too? 😉

yes, it's easy to support more flexible input, but not the other way around, so that's not the issue

agreed

as for the convenience fn, is that something people do often? most of my experience is to get the whole thing out and deal with it myself

@U7ERLH6JX Yes, but if you're going to deal with it yourself, you have to know if the format is dynamic (list or single) or not

yeah i mean we can define the spec as value or vec of values and not have the convenience fn. not sure of the value benefit of it?

if the server decides to send content-length twice, and you expect a single value, your calls could break, unless you explicitly check for vector or single value

I think I'm just going to re-implement babashka.curl in babashka.http-client since it's the API that follows the conventions that already exist

> you expect a single value, your calls could break the way im thinking here is that maintaining the convenience fn is more work than just setting the expectation of a value | [value] but also agree to the breakage argument

more work vs having the flexibility to make breaking changes by saying: always use the fn to get the header rather than relying on the raw response

it's not a lot of work though

yep, its quite easy to write it. my feelings are more from we are adding fns over data and somehow down the line people would go more and more around it as they have use case for multi headers. the data > fn clojure thoughts 🙂

maybe the first version doesnt have this fn and when people ask for it we can add it? people who are using the raw data can continue using it

the idea of the fn would be to protect people from breaking changes to the data, so that doesn't really make sense

anyway, got the first tests running, I'll just copy paste bb.curl's test and will make it work

https://github.com/babashka/http-client

it would be helpful if someone could look into running a non-third party service to run http tests as they are the single most reason for flakiness on bb CI

we could run http://httpbin.org locally?

it has a docker image

https://github.com/postmanlabs/httpbin

See for example https://ci.appveyor.com/project/borkdude/babashka/builds/45797186 ERROR in (test-basic-response) (C:\projects\babashka\.\test-resources\lib_tests\hato\client_test.clj:94) verbs exist expected: (= 200 (:status (head "https://httpbin.org/status/200"))) actual: clojure.lang.ExceptionInfo: status: 502 This happens routinely

Sure, if you know how to run this in circleci / github actions / appveyor...?

should work. they all should have docker

i'll look into it, this should be in the http-client repo's CI right?

also in bb's own CI (see broken build above)

yep

another less complex option could be to poll the service before we run the test, if it returns a 502, we just skip the test or so

would be good to think about a couple alternatives to not make things too complex. e.g. I don't like to run docker locally to run these tests

another alternative: run this service ourselves and use a token or so

yeah or some Go thing

or maybe I can just disable these tests on master, while still running them in branches, this should be enough to make sure they keep working (if we don't commit directly to master)

also it won't break builds during releases, which is the most annoying thing about all of this

yeah this seems to be the first thing to do, since all of the breakages are external. i can find a simpler server thing til then

The tests are mainly there to ensure that Java interop for http client libraries is working

and that the SCI mapping for built-in libs is correct

ok, let's just disable on master

let's use master / main since I might rename to main in the near future

we could use metadata ^:flaky and disable all flaky tests on the default branch

I'd say it's quite annoying that the default JDK client doesn't follow redirects. E.g. now you have to do:

(client/get ""
                               {:client (client/client {:follow-redirects :always})})

I think I'll make bb http-client's default client always follow redirects

Now ported most of what's in babashka.curl, except for throwing on exceptional status codes, will do that later today https://github.com/babashka/http-client

It says its a drop in replacement for bb curl, but we cant do unix sockets with this right?

it says mostly

right

bb curl won't disappear

but the unix domain thing will be fixed in 5 years probably ;)

yeah one can be hopeful

What was wrong with babaska.curl ? How do I choose between it and the soon-to-be built-in babashka.http-client ? Will babashka.http-client also run on JVM clojure? Perhaps the trade-off/rational could be in the babashka.http-client readme? THANKS!

Http client will be the recommended one. Bb curl will still be included. Trade-off is that it doesn’t call out to curl so no OS process per call. Also http client supports async.

Also since http kit is included and its client also has these features, would that be marked as deprecated?

Httpkit client has another set of problems. E.g. the entire response has to be kept in memory at once. Yes, it will also be deprecated

In practice it’s fine for most scripting but bb client solves all of these (and eventually the Unix domain too)

yeah i was almost always using https://github.com/schmee/java-http-clj for this. good to have an official solution 😄

@U04V15CAJ URLis notorious in the java community as having side effects (ie DNS resolution when you create an object), versus URI which enforces the IETF structure, but has no side effects. re: should you support multiple schemes, you could make it open, so http and https resolve, but I could say use tel: handler that I then choose to map that scheme to Twilio and things like that