Frequently Asked Questions

1. Connection to the proxy cannot be established

Netty’s HTTP proxy support always uses CONNECT method in order to establish a tunnel to the specified proxy regardless of the scheme that is used http or https. (More information: Netty enforce HTTP proxy to support HTTP CONNECT method). Some proxies might not support CONNECT method when the scheme is http or might need to be configured in order to support this way of communication. Sometimes this might be the reason for not being able to connect to the proxy. Consider checking the proxy documentation whether it supports or needs an additional configuration in order to support CONNECT method.

2. What is the meaning of the information that is prepended to every log record?

Reactor Netty adds information for the connection at the beginning of every log record (when this is possible). There is a slight difference in the details for the connection when you use TCP, UDP, HTTP/1.1 or HTTP/2.

2.1. TCP and UDP

In case of TCP and UDP, the following is added at the beginning of every log record: the id of the underlying connection, local and remote addresses.

Examples
[a1566d55, L:/[0:0:0:0:0:0:0:1]:53446 - R:/[0:0:0:0:0:0:0:1]:53444]
[a1566d55, L:/[0:0:0:0:0:0:0:1]:53446 ! R:/[0:0:0:0:0:0:0:1]:53444]

Format
[<CONNECTION_ID>, L:<LOCAL_ADDRESS> <CONNECTION_OPENED_CLOSED> R:<REMOTE_ADDRESS>]
<CONNECTION_ID>: a1566d55
<LOCAL_ADDRESS>: [0:0:0:0:0:0:0:1]:53446
<CONNECTION_OPENED_CLOSED>: - (connection opened)
                            ! (connection closed)
<REMOTE_ADDRESS>: [0:0:0:0:0:0:0:1]:53444

2.2. HTTP/1.1

In case of HTTP/1.1, the following is added at the beginning of every log record: the id of the underlying connection, the serial number of the request received on that connection, local and remote addresses.

Examples
[a1566d55-5, L:/[0:0:0:0:0:0:0:1]:53446 - R:/[0:0:0:0:0:0:0:1]:53444]
[a1566d55-5, L:/[0:0:0:0:0:0:0:1]:53446 ! R:/[0:0:0:0:0:0:0:1]:53444]

Format
[<CONNECTION_ID>-<REQUEST_NUMBER>, L:<LOCAL_ADDRESS> <CONNECTION_OPENED_CLOSED> R:<REMOTE_ADDRESS>]
<CONNECTION_ID>: a1566d55
<REQUEST_NUMBER>: 5
<LOCAL_ADDRESS>: [0:0:0:0:0:0:0:1]:53446
<CONNECTION_OPENED_CLOSED>: - (connection opened)
                            ! (connection closed)
<REMOTE_ADDRESS>: [0:0:0:0:0:0:0:1]:53444

2.3. HTTP/2

In case of HTTP/2, the following is added at the beginning of every log record: the id of the underlying connection, local and remote addresses, the id of the stream received on that connection.

Examples
[a1566d55, L:/[0:0:0:0:0:0:0:1]:53446 - R:/[0:0:0:0:0:0:0:1]:53444](H2 - 5)
[a1566d55, L:/[0:0:0:0:0:0:0:1]:53446 ! R:/[0:0:0:0:0:0:0:1]:53444](H2 - 5)

Format
[<CONNECTION_ID>, L:<LOCAL_ADDRESS> <CONNECTION_OPENED_CLOSED> R:<REMOTE_ADDRESS>]<STREAM_ID>
<CONNECTION_ID>: a1566d55
<LOCAL_ADDRESS>: [0:0:0:0:0:0:0:1]:53446
<CONNECTION_OPENED_CLOSED>: - (connection opened)
                            ! (connection closed)
<REMOTE_ADDRESS>: [0:0:0:0:0:0:0:1]:53444
<STREAM_ID>: (H2 - 5)

3. How can I extract all log records for a particular HTTP request?

Reactor Netty adds information for the connection at the beginning of every log record (when this is possible). Use the id of the connection in order to extract all log records for a particular HTTP request. For more information see What is the meaning of the information that is prepended to every log record?

4. How can I debug a memory leak?

By default, Reactor Netty uses direct memory as this is more performant when there are many native I/O operations (working with sockets), as this can remove the copying operations. As allocation and deallocation are expensive operations, Reactor Netty also uses pooled buffers by default. For more information, see Reference Counted Objects.

To be able to debug memory issues with the direct memory and the pooled buffers, Netty provides a special memory leak detection mechanism. Follow the instructions for Troubleshooting Buffer Leaks to enable this mechanism. In addition to the instructions provided by Netty, Reactor Netty provides a special logger (_reactor.netty.channel.LeakDetection) that helps to identify where the memory leak might be located inside Reactor Netty or whether Reactor Netty already forwarded the ownership of the buffers to the application/framework. By default, this logger is disabled. To enable it, increase the log level to DEBUG.

Another way to detect memory leaks is to monitor reactor.netty.bytebuf.allocator.active.heap.memory and reactor.netty.bytebuf.allocator.active.direct.memory meters:

  • The reactor.netty.bytebuf.allocator.active.heap.memory provides the actual bytes consumed by in-use buffers allocated from heap buffer pools

  • The reactor.netty.bytebuf.allocator.active.direct.memory provides the actual bytes consumed by in-use buffers allocated from direct buffer pools

If the above meters are constantly growing, then it’s likely that there is a buffer memory leak.

Consider reducing the used memory when debugging memory leak issues (e.g -XX:MaxDirectMemorySize, -Xms, -Xmx). The less memory the application has, the sooner the memory leak will happen.

5. How can I debug "Connection prematurely closed BEFORE response"?

By default, Reactor Netty clients use connection pooling. When a connection is acquired from the connection pool, it is checked to see whether it is still open. However, the connection can be closed at any time after the acquisition. There are many reasons that can cause a connection to be closed. In most cases, the client might not send directly to the server. Instead, there might be other network components (proxies, load balancers, and so on) between them.

If, on the client side, you observe Connection prematurely closed BEFORE response, perform the following checks to identify the reason for the connection being closed:

  • Obtain a TCP dump and check which peer sends a FIN/RST signal.

  • Check your network connection.

  • Check your Firewall and VPN.

  • Check for any proxies and load balancers.

  • Check the target server.

    • Are there configurations related to any of the following?

      • idle timeout (the connection is closed when there is no incoming data for a certain period of time)

      • limit for buffering data in memory

      • multipart exceeds the max file size limit

      • bad request

      • max keep alive requests (the connection is closed when the requests reach the configured maximum number)

      • rate limit configuration

    • Is the target server in a shutting down state?

Consider checking Timeout Configuration. The section describes various timeout configuration options that are available for Reactor Netty clients. Configuring a proper timeout may improve or solve issues in the communication process.