Httpc/httpd improvements

Erlang/OTP Proposals Proposals: Ideas
, , ,
1

Short story: I wonder if the OTP team and community would be interested in refactorings of inets, specifically httpc and httpd. I think it could be done in stages and an initial pass would be on the API’s to be more user friendly.

I won’t have time to take part in something like this for many many months but it has been on my mind off and on for so long I thought I’d better get it out.

I think it is great and important to have the http client/servers available in OTP, and httpc in particular has improved a lot over the years, and know they will never be able to completely remove need for third party alternatives, but I think a little love would go a long ways.

Below is just what was on top of my mind and is not a deep dive in any sense.

httpc

Awkward API:

inets starts httpc profiles but then any options have to be added with httpc:set_options.

request(Method, Request, HttpOptions, Options, Profile) – Why is Request a tuple instead of something like httpc:request(Method, URL, Headers) or httpc:get(Url, Headers)

Responses are currently large tuples:

{{http_version(), status_code(), reason_phrase()}, headers(), body()}

A map would be nicer:

#{status_code := status_code(), 
   headers := headers(),

Could also have a Request record or map that holds the middlewares to call, base url, headers, options and makes re-usability easier:

BasicAuth = fun(Request, Next) -> Next(httpc_request:add_header(Request, ...)) end,
Request = httpc_request:new(#{base_url => "http://google.com", middlewares => [BasicAuth]}),
Response = httpc:get(Request, #{url => "/search"}),

Instrumentation: As shown in the above example, I’d like to see middlewares of some sort supported so things like distributed tracing and metrics collection could be more easily supported.

httpd

I’m less familiar with httpd but recall my uses have always felt frustrating to get setup. This may mean it is either just me or the need for documentation geared more towards usage similar to other Erlang web servers.

But questions to answer first are:

  • Is there interest?
  • How would backwards compatibility be handled?
    • Have none?
    • All new modules/applications?
    • Maybe it would be lower level than httpc/httpd and those made to use the new modules/applications?
    • I specifically skipped questions like pooling to start with and maybe this would be the answer to that: There would be no pooling support, the new client (maybe server too?) would be single connection based and pooling have to happen by a third party library. So inets profile based process/connection handling would remain.

In that order :slight_smile:

18 Likes
2

Where do you plan to stop before discovering you have re-implemented Cowboy/Gun? HTTP/2, HTTP/3, websockets, …? There are so many edge cases dealing with HTTP, even major implementations (AWS and Azure serverless, Gunicorn, …) fail to support sanely HTTP Expect headers.

Having a less creaky HTTP client/server would be nice but I also get the impression that the OTP team probably wish there never was HTTP or FTP functionality in there otherwise they would have seen a lot more love than they currently receive.

I think crucially, the output of all this work and no matter how good it would be might end up just being compared to that “well the ISC License of Cowboy really is not an obstacle for me…”. I cannot think of where it would be?

Maybe propose the opposite? Lets remove the HTTP and FTP functionality from OTP.

That should heat up the discussion!

4 Likes
3

Nothing is being re-implemented, httpd and httpc already exist.

HTTP/2, HTTP/3, websockets, …?

Separate concerns.

1 Like
4

I much support this idea for httpc. Python does something similar where you can either directly call urlopen("http://google.com") or supply it with a Request object that has headers, body etc. I always found this relatively intuitive (aside from the documentation naming it url). I think accepting a request map / record in addition to URLs in request/1 be a good first step.

I’m less familiar with httpd but recall my uses have always felt frustrating to get setup. This may mean it is either just me or the need for documentation geared more towards usage similar to other Erlang web servers.

I think httpd has some excellent functionality but the one that’s probably most interesting to devs, implementing request handlers in Erlang via ESI, could probably benefit from some documentation polishing :slight_smile:

4 Likes
5

We use httpc and httpd extensively. Why?, for the same reason I don’t drive an SUV to the corner store. I don’t want for much however yeah, the documentation for httpd is very sparse. It’s missing type specifications and most of the functionality is a guessing game. Today I was reading the source to satisfy myself that #mod_data.entity_body was something like [byte()] and never iolist() (I think so).

4 Likes
6

I love the proposed changes @tsloughter ! Bravo.

Another aspect that always confused me with httpc was how to correctly verify TLS. I was never confident that I was doing it correctly, and I never knew where to find guidance. For me it would be ideal if this was all handled for the programmer automatically, and they only had to ensure there were cacerts on the host machine for httpc to load.

3 Likes
7

Hi @tsloughter :smiley:

I am generally of the opinion of having fewer applications as part of Erlang/OTP. With build tools and package managers available, I would leave concerns such as httpd to the community. :slight_smile:

For httpc, I would focus on lower-level process-based abstractions. Think gen_http. This solves long-standing problems in other HTTP clients, which do impose an architecture and often end-up incurring additional performance costs (and more binary references). This is a design we explored with Mint. You can still provide high-level functions that open up the connection and perform a request but without the pooling bits (which the community could build on top of).

I am almost sure this is the default and correctly done on Erlang/OTP 26.

6 Likes
8

httpc ergonomics improvements would be very welcome. Just returning a map response would go a long way.

Right, exactly. I’d leave these to the community and it would be enough to support for HTTP/1 so that, you know, package managers could use it to download packages for these other things. :slight_smile:

Although a low-level gen_http would be fantastic too. It would be enough for package managers and also all BEAM languages could build on top too.

I agree in principle. But man, it would be so nice if we could just erl -s httpd serve . that one time we need to quickly serve files (because we test some html/js) as opposed to this or reaching for python.

7 Likes
9

I have also been missing something like this. Looking at the way -s works I thought this wouldn’t be too much work to implement and made a draft PR, if the OTP team would be interested to integrate something like this: https://github.com/erlang/otp/pull/7299

4 Likes
10

I love how short the implementation is!

3 Likes
11
2 Likes
12

Oh how delightful! Great to hear.

1 Like
13

Yea, I agree at first the more I think about it. But more because of scope. Limiting to httpc and then if there is stomach for it, httpd.

Yup, that is an option as well. httpc and inets profiles could then be built on this new http client that is even lower level. I think we’ll still want changes to httpc interface tho, and then the backwards compatiblity question and if it should be a new module remains.

1 Like
14

Oh yea, remembered where I mostly recently hit what I thought was odd in the httpd API:

I have to include those (plus server_name) to start a server that just calls do/1.

Plus the binary/list conversions both httpd and httpc require.

I’d be less likely to have to dig up an example every time I need to start an httpd server if it were something like

httpd:start_link("localhost:5000", #{modules => [?MODULE]})

Or (dropping the need for the store and remove callbacks):

httpd:start_link("localhost:5000", #{callbacks => [?MODULE:do/1]})

:slight_smile:

3 Likes
15

Actually we have broken all ftp functionality out of inets into their own applications (some releases ago)! We do not consider ftp a very modern technology and think there are better options such as sftp, it is in maintenances mode. I always thought the inets application to be a strange construct, which I believe was constructed on premise I do not agree with, but this was all a long time ago and water under the bridge. Now inets only have HTTP functionality. httpd was done in the 80’s and heavily inspired by apache, and very sparsely documented and tested (by people that left the OTP team a long time ago) . I would say mod_esi is what is mostly used and personally I think we should strive to make httpd a light-wight embedded server and remove as much legacy features as possible instead of trying to figure them out. When it comes to the client we have thought of such ideas as proposed many times, the challenge is getting it prioritized enough. A big part of the somewhat flaky API is the lacking routines we had for user contributions and internal handling too back in the days. Also this was before maps existed.

6 Likes
16

I figured that was the case. I think the community could take it up with the right guidance on what would be accepted by the project.

2 Likes
17

We of course are interested in such help, thereby not saying it will be easy. There are many thing needs considering one of them being what to do with the code we have. We can not just throw it out, it will be need to be phased out. And it is really hard to stop supporting some functionality that is already in place without providing some substitute. In the long term it would be nice to totally phase out inets as an application and replace it with other applications. I am thinking some “gen_http” library as suggest by @josevalim that could be used as building blocks. It would also be really nice if this could provide enough functionality to let applications such as ssl, xmerl, ldap and possible other do simple http request without creating external (outside of OTP) or circular dependencies (as in case of ssl and a possible CRL handling scenario that uses plain HTTP, and inets uses ssl for HTTPS). Also we like to keep having a HTTP-client that will be good enough for many kind of testing purposes and that will resemble the one we got in functionality but has a reworked API and cleaned up design. I think that it needs to be done in small iterative steps, as we do not mind to give guidance but waterfall design seldom is a good idea. Also, the server will be really hard to not support in some format, but I think it should focus on embedded simple use cases, where we phase out legacy features and dependencies fairly untested poorly documented and mostly unused, and maybe if possible simplify some things. Also there would need to be an upgrade path and maybe old API-written with help of the new one for a smooth transition.

7 Likes