NAME

openssl-quic-concurrency - OpenSSL QUIC Concurrency Model

DESCRIPTION

A QUIC domain is a group of QUIC resources such as listeners (see SSL_new_listener(3)) and connections which share common event processing resources, such as internal pollers, timers and locks. All usage of OpenSSL QUIC happens inside a QUIC domain.

These resources can be accessed and used concurrently depending on the circumstances. This man page discusses the available concurrency models and how they can be used.

EXPLICIT AND IMPLICIT QUIC DOMAINS

A QUIC domain is instantiated either explicitly (SSL_new_domain(3)) or implicitly by calling SSL_new(3) or SSL_new_listener(3):

An explicit QUIC domain is created by and visible to the application as a QUIC domain SSL object and has other QUIC SSL objects created underneath it, such as listeners or connections.
An implicit QUIC domain is one which is created internally due to the direct creation of a QUIC connection or listener SSL object; the application does not explicitly create a QUIC domain SSL object and never directly references the domain.

Explicit creation of a QUIC domain provides the greatest level of control for an application. Applications can use an implicit QUIC domain for ease of use and to avoid needing to create a separate QUIC domain SSL object.

Regardless of whether a QUIC domain is explicitly created, the internal processing model is the same and the application must choose an appropriate concurrency model as discussed below.

CONCURRENCY MODELS

The OpenSSL QUIC implementation supports multiple concurrency models to support a wide variety of usage scenarios.

The available concurrency models are as follows:

The Single-Threaded Concurrency Model (SCM), which supports only application-synchronised single-threaded usage.
The Contentive Concurrency Model (CCM), which supports multi-threaded usage.
The Thread-Assisted Concurrency Model (TACM), which also supports multi-threaded usage and provides assistance to an application for handling QUIC timer events.

The merits of these models are as follows:

The Single-Threaded Concurrency Model (SCM) performs no locking or synchronisation. It is entirely up to the application to synchronise access to the QUIC domain and its subsidiary SSL objects.
This concurrency model is also useful for an application which wants to use the OpenSSL QUIC implementation as a pure state machine.
The Contentive Concurrency Model (CCM) performs automatic locking when making API calls to SSL objects in a QUIC domain. This provides automatic synchronisation for multi-threaded usage of QUIC objects. For example, different QUIC stream SSL objects in the same QUIC connection can be safely accessed from different threads.
This concurrency model adds the overhead of locking over the Single-Threaded Concurrency Model in order to support multi-threaded usage, but provides limited performance in highly contended multi-threaded usage due to its simple approach. However, it may still prove a good solution for a broad class of applications which spend the majority of their time in application logic and not in QUIC I/O processing.

An advantage of this model relative to the more sophisticated concurrency models below is that it does not create any OS threads.
The Thread-Assisted Concurrency Model (TACM) is identical to the Contentive Concurrency Model except that a thread is spun up in the background to ensure that QUIC timer events are handled in a timely fashion. This ensures that QUIC timeout events are handled even if an application does not periodically call into the QUIC domain to ensure that any outstanding QUIC-related timer or network I/O events are handled. The assist thread contends for the same resources like any other thread. However, handshake layer events (TLS) are never processed by the assist thread.

The default concurrency model is CCM or TACM, depending on the SSL_METHOD used with a SSL_CTX. Using OSSL_QUIC_client_method(3) results in a default concurrency model of CCM, whereas using OSSL_QUIC_client_thread_method(3) results in a default concurrency model of TACM.

Additional concurrency models may be offered in future releases of OpenSSL.

BLOCKING I/O CAPABILITIES

All of the supported concurrency models are capable of supporting blocking I/O calls, where application-level I/O calls (for example, to SSL_read_ex(3) or SSL_write_ex(3) on a QUIC stream SSL object) block until the request can be serviced. This includes the use of SSL_poll(3) in a blocking fashion.

Supporting blocking API calls reliably with multi-threaded usage requires the creation of additional OS resources such as internal file descriptors to allow threads to be woken when necessary. This creation of internal OS resources is optional and may need to be explicitly requested by an application depending on the chosen concurrency model. If this functionality is disabled, depending on the chosen concurrency model, blocking API calls may not be available and calls to SSL_set_blocking_mode(3) attempting to enable blocking mode may fail, notwithstanding the following section.

Legacy Blocking Support Compatibility

OpenSSL 3.2 and 3.3 contained a buggy implementation of blocking QUIC I/O calls which is only reliable under single-threaded usage. This functionality is always available in the Single-Threaded Concurrency Model (SCM), where it works reliably.

For compatibility reasons, this functionality is also available under the default concurrency model if the application does not explicitly specify a concurrency model or disable it. This is known as Legacy Blocking Compatibility Mode, and its usage is not recommended for multi-threaded applications.

RECOMMENDED USAGE

New applications are advised to choose a concurrency model as follows:

A purely single-threaded application, or an application which wishes to use OpenSSL QUIC as a state machine and manage synchronisation itself, should explicitly select the SCM concurrency model.
An application which wants to engage in multi-threaded usage of different QUIC connections or streams in the same QUIC domain should a) select the CCM or TACM concurrency model and b) explicitly opt in or out of blocking I/O support (depending on whether the application wishes to make blocking I/O calls), disabling Legacy Blocking Compatibility Mode.
An application should select the CCM concurrency model if the application can guarantee that a QUIC domain will be serviced regularly (for example, because the application can guarantee that the timeout returned by SSL_get_event_timeout(3) will be handled). If an application is unable to do this, it should select the TACM concurrency model.
Applications should explicitly configure a concurrency model during initialisation.

CONFIGURING A CONCURRENCY MODEL

If using an explicit QUIC domain, a concurrency model is chosen when calling SSL_new_domain(3) by specifying zero or more of the following flags:

SSL_DOMAIN_FLAG_SINGLE_THREAD: Specifying this flag configures the Single-Threaded Concurrency Model (SCM).
SSL_DOMAIN_FLAG_MULTI_THREAD: Speciyfing this flag configures the Contentive Concurrency Model (CCM) (unless SSL_DOMAIN_FLAG_THREAD_ASSISTED is also specified).
SSL_DOMAIN_FLAG_THREAD_ASSISTED: Specifying this flag configures the Thread-Assisted Concurrency Model (TACM). It implies SSL_DOMAIN_FLAG_MULTI_THREAD.
SSL_DOMAIN_FLAG_BLOCKING: Enable reliable support for blocking I/O calls, allocating whatever OS resources are necessary to realise this. If this flag is specified, SSL_DOMAIN_FLAG_LEGACY_BLOCKING is ignored.
Details on the allocated OS resources can be found under "CONSUMPTION OF OS RESOURCES" below.
SSL_DOMAIN_FLAG_LEGACY_BLOCKING: Enables legacy blocking compatibility mode. See "Legacy Blocking Support Compatibility".

Mutually exclusive flag combinations result in an error (for example, combining SSL_DOMAIN_FLAG_SINGLE_THREAD and SSL_DOMAIN_FLAG_MULTI_THREADED).

The concurrency model for a domain cannot be changed after the domain is created.

Default Behaviour

If none of SSL_DOMAIN_FLAG_SINGLE_THREAD, SSL_DOMAIN_FLAG_MULTI_THREAD or SSL_DOMAIN_FLAG_THREAD_ASSISTED are provided to SSL_new_domain(3) or another constructor function which can accept the above flags, the default concurrency model set on the SSL_CTX is used. This default can be set and get using SSL_CTX_set_domain_flags(3) and SSL_CTX_get_domain_flags(3). Any additional flags provided (for example, SSL_DOMAIN_FLAG_BLOCCKING) are added to the set of inherited flags.

The default concurrency model set on a newly created SSL_CTX is determined as follows:

If an SSL_METHOD of OSSL_QUIC_client_thread_method(3) is used, the Thread-Assisted Concurrency Model (TACM) is used with the SSL_DOMAIN_FLAG_BLOCKING flag. This provides reliable blocking functionality.
Otherwise, if OpenSSL was built without threading support, the Single-Threaded Concurrency Model (SCM) is used, with the SSL_DOMAIN_FLAG_LEGACY_BLOCKING flag.
Otherwise, if an SSL_METHOD of OSSL_QUIC_client_method(3) is used, the Contentive Concurrency Model (CCM) is used with the SSL_DOMAIN_FLAG_LEGACY_BLOCKING flag.
Otherwise, the Contentive Concurrency Model (CCM) is used.

The default concurrency model may vary between releases of OpenSSL. An application may specify one or more of the domain flags above to ensure consistent usage of a specific concurrency model between releases.

Configuration of Concurrency Models with Implicit QUIC Domains

If an explicit QUIC domain is not explicitly created using SSL_new_domain(3), an implicit QUIC domain is created when calling SSL_new_listener(3) or SSL_new(3). Such a domain will use the default domain flags configured on the SSL_CTX as described above.

CONSUMPTION OF OS RESOURCES

If full blocking I/O support is selected using SSL_DOMAIN_FLAG_BLOCKING, at least one socket, socket-like OS handle or file descriptor must be allocated to allow one thread to wake other threads which may be blocking in calls to OS socket polling interfaces such as select(2) or poll(2). This is allocated automatically internally by OpenSSL.

If the Thread-Assisted Concurrency Model (TACM) is selected, a background thread is spawned. This also implies SSL_DOMAIN_FLAG_BLOCKING and the above.

The internal consumption by OpenSSL of mutexes, condition variables, spin locks or other similar thread synchronisation primitives is unspecified under all concurrency models.

The internal consumption by OpenSSL of threads is unspecified under the Thread-Assisted Concurrency Model.

The internal consumption by OpenSSL of sockets, socket-like OS handles or file descriptors, or other resources as needed to support inter-thread notification, is unspecified under the Thread-Assisted Concurrency Model or when using SSL_DOMAIN_FLAG_BLOCKING.

BEHAVIOUR OF SSL OBJECTS

A QUIC SSL object has blocking mode enabled by default where all of the following criteria are met:

SSL_DOMAIN_FLAG_BLOCKING or SSL_DOMAIN_FLAG_LEGACY_BLOCKING is enabled; and
The QUIC connection is being used with network read and write BIOs which expose supported poll descriptors. See openssl-quic(7) for details.

In all other cases, a QUIC SSL object has blocking mode disabled by default. The blocking mode can be changed explicitly using SSL_set_blocking_mode(3).

COPYRIGHT

Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliance with the License. You can obtain a copy in the file LICENSE in the source distribution or at <https://www.openssl.org/source/license.html>.