13. Protocols

HTTP Status codes.

response_codes, https://www.iana.org/assignments/http-status-codes/http-status-codes.txt

RFC 5842 section 7.1: 208 Already Reported

RFC 2518 section 10.5: 424 Failed Dependency

RFC 2518 section 10.4: 423 Locked

RFC 5842 section 7.2: 508 Loop Detected

RFC 2518 section 10.2: 207 Multi-Status

RFC 2518 section 10.1: 102 Processing

RFC 2518 section 10.6: 507 Insufficient Storage

RFC 2518 section 10.3: 422 Unprocessable Entry

RFC 3229 section 10.4.1: 226 IM Used

RFC 2324 section 2.3.2: 418 I'm a teapot

RFC 2616 section 10.2.3: 202 Accepted

RFC 2616 section 10.4.1: 400 Bad Request

RFC 2616 section 10.5.3: 502 Bad Gateway

RFC 2616 section 10.4.17: 416 Requested Range Not Satisfiable

RFC 2616 section 10.4.10: 409 Conflict

RFC 2616 section 10.1.1: 100 Continue

RFC 2616 section 10.2.2: 201 Created

RFC 8297 section 2: 103 Early Hints

RFC 2616 section 10.4.18: 417 Expectation Failed

RFC 2616 section 10.4.4: 403 Forbidden

RFC 2616 section 10.3.3: 302 Found

RFC 2616 section 10.4.11: 410 Gone

RFC 2616 section 10.5.5: 504 Gateway Timeout

RFC 6585 section 5: 431 Request Header Fields Too Large

RFC 2616 section 10.5.1: 500 Internal Server Error

RFC 7725 section 3: 451 Unavailable For Legal Reasons

RFC 2616 section 10.4.12: 411 Length Required

RFC 2616 section 10.4.6: 405 Method Not Allowed

RFC 7540 section 9.1.2: 421 Misdirected Request

RFC 2616 section 10.3.2: 301 Moved Permanently

RFC 2616 section 10.3.1: 300 Multiple Choices

RFC 6585 section 6: 511 Network Authentication Required

RFC 2616 section 10.2.4: 203 Non-Authorative Information

RFC 2616 section 10.4.7: 406 Not Acceptable

RFC 2774 section 7: 510 Not Extended (obsolete)

RFC 2616 section 10.4.5: 404 Not Found

RFC 2616 section 10.5.2: 501 Not Implemented

RFC 2616 section 10.3.5: 304 Not Modified

RFC 2616 section 10.2.5: 204 No Content

RFC 2616 section 10.2.1: 200 OK

RFC 2616 section 10.2.7: 206 Partial Content

RFC 2616 section 10.4.3: 402 Payment Required

RFC 7538 section 3: 308 Permanent Redirect

RFC 2616 section 10.4.13: 412 Precondition Failed

RFC 6585 section 3: 428 Precondition required

RFC 2616 section 10.4.8: 407 Proxy Authentication Required

RFC 2616 section 10.4.14: 413 Request Entity Too Large

RFC 2616 section 10.2.6: 205 Reset Content

RFC 2616 section 10.3.4: 303 See Other

RFC 2616 section 10.1.2: 101 Switching protocols

RFC 2616 section 10.3.8: 307 Temporary Redirect

RFC 2616 section 10.4.9: 408 Request Timeout

RFC 6585 section 4: 429 Too Many Requests

RFC 2616 section 10.4.2: 401 Unauthorized

RFC 2616 section 10.5.4: 503 Service Unavailable

RFC 2616 section 10.4.16: 415 Unsupported Media Type

RFC 2616 section 10.5.6: 505 HTTP Version Not Supported

RFC 2616 section 10.4.15: 414 Request-URI Too Long

RFC 2616 section 10.3.6: 305 Use Proxy

RFC 2295 section 8.1: 506 Variant Also Negotiates

RFC 8470 section 5.2: 425 Too Early

RFC 2817 section 4.2: 426 Upgrade Required

Open and execute an HTTP query.

Errno copied from the connection or simulated for async operations.

In Pike 7.8 and earlier hardcoded Linux values were used in async operations, 110 instead of System.ETIMEDOUT and 113 instead of System.EHOSTUNREACH.

Headers as a mapping. All header names are in lower case, for convinience.

Connected host and port.

Used to detect whether keep-alive can be used.

Set this to a global mapping if you want to use a cache, prior of calling *request().

timeout is the time to wait in seconds on connection and/or data. If data is fetched asynchronously the watchdog will be reset every time data is received. Defaults to 120 seconds. maxtime is the time the entire operation is allowed to take, no matter if the connection and data fetching is successful. This is by default indefinitely.

These values only have effect in asynchroneous calls

Tells if the connection is successfull.

Protocol string, ie "HTTP/1.0".

Status number and description (eg 200 and "ok").

Wait for connection to complete.

Returns 1 on successfull connection, 0 on failure.

Fetch all data in background.

timed_async_fetch(), async_request(), set_callbacks()

Setup and run an asynchronous request, otherwise similar to thread_request().

request_ok(Protocols.HTTP.Query httpquery,...extra args) will be called when connection is complete, and headers are parsed.

request_fail(Protocols.HTTP.Query httpquery,...extra args) is called if the connection fails.

Returns the called object

The header mapping ORed with the following mapping.

Gives back the answer as a string.

Close all associated file descriptors.

Gives back the data as a string.

Gives back a pseudo-file object, with the methods read() and close(). This could be used to copy the file to disc at a proper tempo.

datafile() doesn't give the complete request, just the data.

file()

Gives back the number of downloaded bytes.

Gives back a pseudo-file object, with the methods read() and close(). This could be used to copy the file to disc at a proper tempo.

newheaders, removeheaders is applied as: (oldheaders|newheaders))-removeheaders Make sure all new and remove-header indices are lower case.

datafile()

Create a new query object and begin the query.

The query is executed in a background thread; call `() in the object to wait for the request to complete.

query is the first line sent to the HTTP server; for instance "GET /index.html HTTP/1.1".

headers will be encoded and sent after the first line, and data will be sent after the headers.

Returns the called object.

Like async_fetch(), except with a timeout and a corresponding fail callback function.

async_fetch(), async_request(), set_callbacks()

Gives back the size of a file if a content-length header is present and parsed at the time of evaluation. Otherwise returns -1.

Gives back data, but decoded according to the content-type character set.

data

A URL which is either a string a Standards.URI or a SessionURL.

Default HTTP headers.

The number of redirects to follow, if any. This is the default to the created Request objects.

A redirect automatically turns into a GET request, and all header, query, post or put information is dropped.

Default is 20 redirects. A negative number will mean infinity.

Loops will currently not be detected, only the limit works to stop loops.

Request.follow_redirects

Cache of hostname to IP lookups. Given to and used by the Query objects.

Maximum times a connection is reused. Defaults to 1000000. <2 means no reuse at all.

Maximum number of connections to the same server. Used only by async requests. Defaults to 10 connections.

Maximum total number of connections. Limits only async requests, and the number of kept-alive connections (live connections + kept-alive connections <= this number) Defaults to 50 connections.

The time to keep unused connections in seconds. Set to zero to never save any kept-alive connections. (Might be good in a for instance totaly synchroneous script that keeps the backend thread busy and never will get call_outs.) Defaults to 10 seconds.

Sends a HTTP GET, POST, PUT or DELETE request to the server in the URL asynchroneously, and call the corresponding callbacks when result arrives (or not). The callbacks will receive the created Request object as first argument, then the given callback_arguments, if any.

callback_headers_ok is called when the HTTP request has received headers.

callback_data_ok is called when the HTTP request has been received completely, data and all.

callback_fail is called when the HTTP request has failed, on a TCP/IP or DNS level, or has received a forced timeout.

The created Request object is returned.

Dump all cookies to a string and read them back. This is useful to store cookies in between sessions (on disk, for instance). decode_cookies will throw an error upon parse failures. Also note, decode_cookies will clear out any previously learned cookies from the Session object, unless no_clear is given and true.

Sends a HTTP GET, POST, PUT or DELETE request to the server in the URL and returns the created and initialized Request object. 0 is returned upon failure.

Get the cookies that we should send to this server, for this url. They are presented in the form suitable for HTTP headers (as an array). This will also take in count expiration of cookies, and delete expired cookies from the Session unless no_delete is true.

Returns an array of ({content_type,data}) and just the data string respective, after calling the requested server for the information. 0 is returned upon failure.

post* is similar to the get_url() class of functions, except that the query variables is sent as a POST request instead of as a GET.

Request a Query object suitable to use for the given URL. This may be a reused object from a keep-alive connection.

Return a previously used Query object to the keep-alive storage. This function will determine if the given object is suitable to keep or not by checking status and headers.

Set a cookie. The cookie will be checked against current security levels et al, using the parameter who. If who is zero, no security checks will be performed.

Parse and set a cookie received in the HTTP protocol. The cookie will be checked against current security levels et al.

This module contains various HTTP Authentication implementations for both server and client use. A Client implementation would typically call the make_authenticator method with the incoming WWW-Authenticate header to get a Client object. For each HTTP request the auth() method of the object can be called to get an appropriate Authorization header.

Server code should create an authentication class and inherit the concrete authentication scheme implementation. To add an actual user lookup, overload get_password or get_hashed_password. Hashed passwords must be hashed with the scheme appropriate digest.

Create an authenticator for a server responding with the given HTTP authentication header. Currently only works for one realm.

The WWW-Authenticate HTTP header or headers.

The username to use.

The plaintext password.

Optionally the realm the user and password is valid in. If omitted, the authentication will happen in whatever realm the server is presenting.

Split client generated Authorization header into its parts.

This HTTP client module utilises the Concurrent.Promise and Concurrent.Future classes and only does asynchronous calls.

Sends a GET, POST, PUT or DELETE request to url asynchronously. A Concurrent.Future object is returned on which you can register callbacks via Concurrent.Future->on_success() and Concurrent.Future.on_failure() which will get a Result object as argument.

For an example of usage see Protocols.HTTP.Promise

Fetch an URL with the http_method method.

set_timeout() sets the default timeout for connecting and data fetching. The watchdog will be reset each time data is fetched.

set_maxtime() sets the timeout for the entire operation. If this is set to 30 seconds for instance, the request will be aborted after 30 seconds event if data is still being received. By default this is indefinitely.

t is the timeout in seconds.

Arguments

Fast HTTP header parser.

Looks up the file extension in a table to return a suitable MIME type.

Determine the extension for a given filename.

Looks up the file extension in a table to return a suitable MIME type.

Makes a time notification suitable for the HTTP protocol.

The time in seconds since the 00:00:00 UTC, January 1, 1970

The date in the HTTP standard date format. Example : Thu, 03 Aug 2000 05:40:39 GMT

Decode a HTTP date to seconds since 1970 (UTC)

zero (UNDEFINED) if the given string isn't a HTTP date

Decodes an URL-encoded query into a mapping.

Alert packet.

Based on the base Packet.

String buffer with the possibility to read and write data as they would be formatted in structs.

Appends an array of unsigned integers of width item_size to the buffer, preceded with an unsigned integer len declaring the size of the array in bytes.

Appends an array of variable length strings with item_size bytes hollerith coding, prefixed by a len bytes large integer declaring the total size of the array in bytes.

Create a new buffer, optionally initialized with the value s.

Reads an array of integers as written by add_int_array from the buffer.

Reads an array of strings as written by add_string_array from the buffer.

Client-side connection state.

A few storage variables for client certificate handling on the client side.

Initialize a new ClientConnection.

Context to use.

Optional host name of the server.

Optional Session to resume.

Do handshake processing.

One of HANDSHAKE_*.

The contents of the packet.

The raw packet received (needed for supporting SSLv2 hello messages).

This function returns:

It uses the send_packet() function to transmit packets.

Renegotiate the connection (client initiated).

Sends a client_hello to force a new round of handshaking.

SSL.Connection keeps the state relevant for a single SSL connection. This includes the Context object (which doesn't change), various buffers, the Session object (reused or created as appropriate), and pending read and write states being negotiated.

Each connection will have two sets of read and write States: The current read and write states used for encryption, and pending read and write states to be taken into use when the current keyexchange handshake is finished.

This object is also responsible for managing incoming and outgoing packets. Outgoing packets are stored in queue objects and sent in priority order.

This class should never be created directly, instead one of the classes that inherits it should be used (ie either ClientConnection or ServerConnection) depending on whether this is to be a client-side or server-side connection. These in turn are typically created by File()->create().

ClientConnection, ServerConnection, Context, Session, File, State

Number of passed sequence numbers to keep track of. RFC 4347 section 4.1.2.5: A minimum window size of 32 MUST be supported, but a window size of 64 is preferred and SHOULD be employed as the default. Another window size (larger than the minimum) MAY be chosen by the receiver.

Selected ALPN (RFC 7301) protocol (if any).

Note that this is a connection property, and needs to be renegotiated on session resumption.

Random cookies, sent and received with the hello-messages.

The active Cipher.KeyExchange (if any).

Number of application data bytes sent by us.

Bitmask representing sequence numbers for accepted received packets in the interval [next_seq_num-window_size..next_seq_num-2].

The packet with seqence number next_seq_num-1 is implicitly known to have been received.

Bitfield with the current connection state.

Initialize the connection state.

The context for the connection.

Derive the master secret from the premaster_secret and the random seeds, and configure the keys.

Returns a string describing the current connection state.

Main receive handler.

String of data received from the peer.

Returns one of:

This function is intended to be called from an i/o read callback.

Handle an alert received from the peer.

Alert level; either ALERT_warning or ALERT_fatal.

Alert description code; one of indices(SSL.Constants.ALERT_descriptions).

Do handshake processing. Type is one of HANDSHAKE_*, data is the contents of the packet, and raw is the raw packet received (needed for supporting SSLv2 hello messages).

This function returns 0 if handshake is in progress, 1 if handshake is finished, and -1 if a fatal error occurred. It uses the send_packet() function to transmit packets.

Mark seqence number num as seen and accepted.

This will cause valid_seq_nump() to return 0 for it if it shows up again.

Generate new pending cipher states.

Returns the number of packets queued for writing.

Returns the number of times to_write() can be called before it stops returning non-empty strings.

Low-level receive handler. Returns a packet, an alert, or zero if more data is needed to get a complete packet.

Initiate close.

Queues a packet for write. Handshake and and change cipher must use the same priority, so must application data and close_notifies.

Renegotiate the connection.

Send an application data packet. If the data block is too large then as much as possible of the beginning of it is sent. The size of the sent data is returned.

Called with alert object, sequence number of bad packet, and raw data as arguments, if a bad packet is received.

Can be used to support a fallback redirect https->http.

Remove cyclic references as best we can.

Extracts data from the packet queues. Returns 2 if data has been written, 0 if there are no pending packets, 1 of the connection is being closed politely, and -1 if the connection died unexpectedly.

This function is intended to be called from an i/o write callback.

query_write_queue_size(), send_streaming_data().

Check whether num is a valid seqence number for a new packet.

Keeps the state that is shared by all SSL-connections on a client, or for one port on a server. It includes policy configuration, the server or client certificate(s), the corresponding private key(s), etc. It also includes the session cache.

The defaults are usually suitable for a client, but for a server some configuration is necessary.

Typical use is to:

• Call add_cert() with the certificates belonging to the server or client. Note that clients often don't have or need any certificates, and also that certificate-less server operation is possible, albeit discouraged and not enabled by default.

  Suitable self-signed certificates can be created with Standards.X509.make_selfsigned_certificate().

• Optionally call get_suites() to get a set of cipher_suites to assign to preferred_suites. This is only needed if the default set of suites from get_suites(128, 1) isn't satisfactory.

The initialized Context object is then passed to File()->create() or used as is embedded in Port.

File, Port, Standards.X509

List of advertised protocols using using TLS application level protocol negotiation.

Policy for client authentication. One of SSL.Constants.AUTHLEVEL_none, SSL.Constants.AUTHLEVEL_verify, SSL.Constants.AUTHLEVEL_ask and SSL.Constants.AUTHLEVEL_require.

Defaults to SSL.Constants.AUTHLEVEL_none.

The possible client authentication methods. Used only if auth_level is AUTH_ask or AUTH_require. Generated by set_authorities.

Supported elliptical curve cipher curves in order of preference. Defaults to all supported curves, ordered with the largest curves first.

If set enable SSL/TLS protocol renegotiation.

Defaults to 1 (enabled).

RFC 7540 section 9.2.1 requires this to be turned off after Protocols.HTTP2 communication has started.

Attempt to enable encrypt-then-mac mode. Defaults to 1.

Replaced by extensions.

A list of all extensions that will be considered in the handshake process. Extensions not listed will not be sent, and will be ignored if received.

The following values are included by default.

The following supported values are not included by default.

RFC 6066

Supported FFDHE groups for DHE key exchanges, in order of preference, most preferred first.

Defaults to the full set of supported FFDHE groups from the FFDHE draft, in order of size with the smallest group (2048 bits) first.

Server-side the first group in the list that satisfies the NIST guide lines for key strength (NIST SP800-57 5.6.1) (if any) for the selected cipher suite will be selected, and otherwise the largest group.

Client-side the list will be reversed (as a precaution if the server actually follows the clients preferences).

If set, the other peer will be probed for the heartbleed bug during handshake. If heartbleed is found the connection is closed with insufficient security fatal error. Requires Constants.EXTENSION_heartbeat to be set in extensions.

Maximum number of sessions to keep in the cache.

The accepted range of versions for the client/server. List specific versions in supported_versions instead.

Replaced by supported_versions.

The maximum amount of data that is sent in each SSL packet by File. A value between 1 and Constants.PACKET_MAX_SIZE.

Lists the supported compression algorithms in order of preference.

Defaults to ({ COMPRESSION_null }) due to SSL attacks that target compression.

Cipher suites we want to support, in order of preference, best first. By default set to all suites with at least 128 bits cipher key length, excluding RC4, and ephemeral and non-ephemeral certificate based key exchange.

DHE parameter lookup for the FFDHE private range.

Add any custom FFDHE-groups here.

Defaults to the empty mapping.

If you add any groups here, you will also need to update ffdhe_groups accordingly.

Used to generate random cookies for the hello-message. If we use the RSA keyexchange method, and this is a server, this random number generator is not used for generating the master_secret. By default set to random_string.

When set, require the chain to be known, even if the root is self signed.

Note that if set, and certificates are set to be verified, trusted issuers must be provided, or no connections will be accepted.

Replaced by auth_level.