aboutgitcodebugslistschat
path: root/tcp.c
Commit message (Collapse)AuthorAgeFilesLines
* pasta, tcp: Mask EPOLLIN and EPOLLRDHUP after sending FINStefano Brivio2021-09-161-2/+14
| | | | | | | Now that we dropped EPOLLET, we'll keep getting EPOLLRDHUP, and possibly EPOLLIN, even if there's nothing to read anymore. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Break splice() loop once we've written everything that was readStefano Brivio2021-09-161-2/+7
| | | | | | That's a guarantee that we don't need to retry writing. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Don't set SPLICE_FIN_BOTH state on EPOLLHUPStefano Brivio2021-09-161-3/+1
| | | | | | | EPOLLHUP just means we shut down one side of the connection on *one* socket: remember, we have two sockets here. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Don't reset 'never_read' flag on write retriesStefano Brivio2021-09-161-2/+3
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Don't set TCP_CORK on spliced socketsStefano Brivio2021-09-161-6/+1
| | | | | | | | ...throughput isn't everything: this leads (of course) to horrible latency with small, sparse messages. As a consequence, there's no need to set TCP_NODELAY either. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fix setting window from maximum ACK sequence in batchStefano Brivio2021-09-161-6/+2
| | | | | | | | | If we're at the first message in a batch, it's safe to get the window value from it, and there's no need to subtract anything for a comparison on that's not even done -- we'll override it later in any case if we find messages with a higher ACK sequence number. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Set pipe descriptor numbers to -1 after closingStefano Brivio2021-09-161-0/+4
| | | | | | ...so that we don't try to close them again, even if harmless. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta, tcp: Drop EPOLLET for spliced, established connectionsStefano Brivio2021-09-161-5/+5
| | | | | | | ...tcp_handler_splice() doesn't guarantee we read all the available data, the sending buffer might be full. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Read SO_SNDBUF unconditionallyStefano Brivio2021-09-161-17/+11
| | | | | | | | Checking it only when the cached value is smaller than the current window of the receiver is not enough: it might shrink further while the receiver window is growing. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta: Clean up FIN connection flags once a connection is deletedStefano Brivio2021-09-151-0/+1
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* pasta: Set spliced connection flag in epoll reference on compactionStefano Brivio2021-09-151-2/+2
| | | | | | ...otherwise, we'll mix indices with non-spliced connections. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Request retransmission with updated sequence also on partial write to ↵Stefano Brivio2021-09-141-11/+8
| | | | | | | | | | | socket If we couldn't write the whole batch of received packets to the socket, and we have missing segments, we still need to request their retransmission right away, otherwise it will take ages for the guest to figure out we're missing them. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: In ESTABLISHED state, acknowledge segments as they're sent to the socketStefano Brivio2021-09-141-6/+15
| | | | | | | | | ...instead of waiting for the remote peer to do that -- it's especially important in case we request retransmissions from the guest, but it also helps speeding up slow start. This should probably be a configurable behaviour in the future. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Properly time out ACK wait from tapStefano Brivio2021-09-141-3/+3
| | | | | | | | | Seen with iperf3: a control connection is established, no data flows for a while, all segments are acknowledged. The socket starts closing it, and we immediately time out because the last ACK from tap was one minute before that. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Don't mistake a FIN segment with no data for a Fast Retransmit requestStefano Brivio2021-09-141-1/+2
| | | | | | | It carries no data and usually duplicates the previous ACK sequence, but it's just a FIN. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Check errno on sendmmsg() failure, not just the return valueStefano Brivio2021-09-141-1/+1
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Make sure sending window is initialised before sending to tapStefano Brivio2021-09-141-1/+1
| | | | | | | Seen with iperf3: the first packet from socket (data connection) is 65520 bytes and doesn't fit in the window. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fixes for closing states, spliced connections, out-of-order packets, etc.Stefano Brivio2021-09-091-207/+400
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes a number of issues found with some heavier testing with uperf and neper: - in most closing states, we can still accept data, check for EPOLLIN when appropriate - introduce a new state, ESTABLISHED_SOCK_FIN_SENT, to track the fact we already sent a FIN segment to the tap device, for proper sequence number bookkeeping - for pasta mode only: spliced connections also need tracking of (inferred) FIN segments and clean half-pipe shutdowns - streamline resetting epoll_wait bitmaps with a new function, tcp_tap_epoll_mask(), instead of repeating the logic all over the place - set EPOLLET for tap connections too, whenever we are waiting for EPOLLRDHUP or an event from the tap to proceed with data transfer, to avoid useless loops with EPOLLIN set - impose an additional limit on the sending window advertised to the guest, given by SO_SNDBUF: it makes no sense to completely fill the sending buffer and send a zero window: stop a bit before we hit that - handle *all* interrupted system calls as needed - simplify the logic for reordering of out-of-order segments received from tap: it's not a corner case, and the previous logic allowed for deadloops - fix comparison of seen IPv4 address when we get a new connection from a socket directed to the configured guest address Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp, udp: Restore usage of gateway for guest to connect to local hostStefano Brivio2021-09-011-0/+5
| | | | | | | This went lost in a recent rework: if the guest wants to connect directly to the host, it can use the address of the default gateway. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* passt, pasta: Introduce command-line options and port re-mappingStefano Brivio2021-09-011-62/+75
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fixes for early data in SOCK_SYN_SENT, closing states, clamping windowStefano Brivio2021-09-011-23/+30
| | | | | | More details here after rebase. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Proper error handling for sendmmsg() to UNIX domain socketStefano Brivio2021-08-261-37/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As data from socket is forwarded to the guest, sendmmsg() might send fewer bytes than requested in three different ways: - failing altogether with a negative error code -- ignore that, we'll get an error on the UNIX domain socket later if there's really an issue with it and reset the connection to the guest - sending less than 'vlen' messages -- instead of assuming success in that case and waiting for the guest to send a duplicate ACK indicating missing data, update the sequence number according to what was actually sent and spare some retransmissions - somewhat unexpectedly to me, sending 'vlen' or less than 'vlen' messages, returning up to 'vlen', with the last message being partially sent, and no further indication of errors other than the returned msg_len for the last partially sent message being less than iov_len. In this case, we would assume success and proceed as nothing happened. However, qemu would fail to parse any further message, having received a partial descriptor, and eventually close the connection, logging: serious error: oversized packet received,connection terminated. as the length descriptor for the next message would be sourced from the middle of the next successfully sent message, not from its header. Handle this by checking the msg_len returned for the last (even partially) sent message, and force re-sending the missing bytes, if any, with a blocking sendmsg() -- qemu must not receive anything else than that anyway. While at it, allow to send up to 64KiB for each message, the previous 32KiB limit isn't actually required, and just switch to a new message at each iteration on sending buffers, they are already MSS-sized anyway, so the check in the loop isn't really needed. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Never send ACK because of pending unacknowleged data when sending SYNStefano Brivio2021-08-241-1/+3
| | | | | | | | | | | | | | With a kernel older than 5.3 (no_snd_wnd set), ack_pending in tcp_send_to_tap() might be true at the beginning of a new connection initiated by a socket. This means we send the first SYN segment to the tap together with ACK set, which is clearly invalid and triggers the receiver to reply with an RST segment right away. Set ack_pending to 0 whenever we're sending a SYN segment. In case of a SYN, ACK segment sent by the caller, the caller passes the ACK flag explicitly. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Drop EPOLLET for non-spliced connectionsStefano Brivio2021-08-241-4/+4
| | | | | | | Socket-facing functions don't guarantee that all data is handled before they return: stick to level-triggered mode for TCP sockets. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fast re-transmit, more fixes for closing states and no_snd_wndStefano Brivio2021-08-041-45/+73
| | | | | | | | | | | ...and while at it, fix an issue in the calculation of the last IOV buffer size: if we can't receive enough data to fill up the window, the last buffer can be filled completely. Also streamline the code setting iovec lengths if cached values are not matching. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Always allow ACKs when pending, fixes for no_snd_wnd and closing statesStefano Brivio2021-08-041-10/+23
| | | | | | | | | | | | | | | | We won't necessarily have another choice to ACK in a timely fashion if we skip ACKs from a number of states (including ESTABLISHED) when there's enough window left. Check for ACKed bytes as soon as it makes sense. If the sending window is not reported by the kernel, ACK as soon as we queue onto the socket, given that we're forced to use a rather small window. In FIN_WAIT_1_SOCK_FIN, we also have to account for the FIN flag sent by the peer in the sequence. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Lower TCP_TAP_FRAMES to 32Stefano Brivio2021-08-041-1/+1
| | | | | | | | Sending 64 frames in a batch looks quite bad when a duplicate ACK comes right at the beginning of it. Lowering this to 32 doesn't affect performance noticeably, with 16 the impact is more apparent. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Full batched processing for tap messagesStefano Brivio2021-07-271-121/+156
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Similar to UDP, but using a simple sendmsg() on iovec-style buffers from tap instead, as we don't need to preserve message boundaries. A quick test in PASTA mode, from namespace to init via tap: # ip link set dev pasta0 mtu 16384 # iperf3 -c 192.168.1.222 -t 60 [...] [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 80.4 GBytes 11.5 Gbits/sec receiver # iperf3 -c 2a02:6d40:3cfc:3a01:2b20:4a6a:c25a:3056 -t 60 [...] [ ID] Interval Transfer Bitrate [ 5] 0.00-60.01 sec 39.9 GBytes 5.71 Gbits/sec receiver # ip link set dev pasta0 mtu 65520 # iperf3 -c 192.168.1.222 -t 60 [...] [ ID] Interval Transfer Bitrate [ 5] 0.00-60.01 sec 88.7 GBytes 12.7 Gbits/sec receiver # iperf3 -c 2a02:6d40:3cfc:3a01:2b20:4a6a:c25a:3056 -t 60 [...] [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 79.5 GBytes 11.4 Gbits/sec receiver Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Limit TCP_INFO getsockopt() syscallsStefano Brivio2021-07-271-1/+2
| | | | | | | | There's no need to constantly query the socket for number of acknowledged bytes if we're far from exhausting the sending window, just do it if we're at least down to 90% of it. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp, udp: Map source address to gateway for any traffic from 127.0.0.0/8Stefano Brivio2021-07-261-3/+3
| | | | | | ...instead of just 127.0.0.1. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fix re-send mechanism to tap on ACK timeoutStefano Brivio2021-07-261-2/+3
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Simplify ACK accounting, skip some useless operations on tap handlingStefano Brivio2021-07-261-19/+10
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Introduce scatter-gather IO path from socket to tapStefano Brivio2021-07-261-45/+509
| | | | | | | | | | | | | | | | | ...similarly to what was done for UDP. Quick performance test with 32KiB buffers, host to VM: $ iperf3 -c 192.0.2.2 -N [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.47 GBytes 7.27 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec receiver $ iperf3 -c 2a01:598:88ba:a056:271f:473a:c0d9:abc1 [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.43 GBytes 7.24 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.41 GBytes 7.22 Gbits/sec receiver Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp, udp: Allow binding ports in init namespace to both tap and loopbackStefano Brivio2021-07-261-18/+37
| | | | | | | | Traffic with loopback source address will be forwarded to the direct loopback connection in the namespace, and the tap interface is used for the rest. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tap: Fill the IPv6 flow label field to represent flow associationStefano Brivio2021-07-261-1/+6
| | | | | | | | | This isn't optional: TCP streams must carry a unique, hard-to-guess, non-zero label for each direction. Linux, probably among others, will otherwise refuse to associate packets in a given stream to the same connection. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Don't open a new connection from tap if both SYN and ACK are setStefano Brivio2021-07-211-1/+1
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp, udp: Split IPv4 and IPv6 bound port setsStefano Brivio2021-07-211-23/+38
| | | | | | | | | | | Allow to bind IPv4 and IPv6 ports to tap, namespace or init separately. Port numbers of TCP ports that are bound in a namespace are also bound for UDP for convenience (e.g. iperf3), and IPv4 ports are always bound if the corresponding IPv6 port is bound (socket might not have the IPV6_V6ONLY option set). This will also be configurable later. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Replace source address also if it's the same as the guest addressStefano Brivio2021-07-211-2/+4
| | | | | | | | ...not just for loopback addresses, with the address of the default gateway. Otherwise, the guest might receive packets with source and destination set to the same address. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Increase maximum window scaling factor from 8 to 9Stefano Brivio2021-07-211-1/+1
| | | | | | | This is actually reasonable in terms of memory consumption and allows for better performance with local services. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fix partial (ACK) message coalescing, ACK timeout, MSG_MORE flag settingStefano Brivio2021-07-171-2/+5
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* passt: Add PASTA mode, major reworkStefano Brivio2021-07-171-458/+1085
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | PASTA (Pack A Subtle Tap Abstraction) provides quasi-native host connectivity to an otherwise disconnected, unprivileged network and user namespace, similarly to slirp4netns. Given that the implementation is largely overlapping with PASST, no separate binary is built: 'pasta' (and 'passt4netns' for clarity) both link to 'passt', and the mode of operation is selected depending on how the binary is invoked. Usage example: $ unshare -rUn # echo $$ 1871759 $ ./pasta 1871759 # From another terminal # udhcpc -i pasta0 2>/dev/null # ping -c1 pasta.pizza PING pasta.pizza (64.190.62.111) 56(84) bytes of data. 64 bytes from 64.190.62.111 (64.190.62.111): icmp_seq=1 ttl=255 time=34.6 ms --- pasta.pizza ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 34.575/34.575/34.575/0.000 ms # ping -c1 spaghetti.pizza PING spaghetti.pizza(2606:4700:3034::6815:147a (2606:4700:3034::6815:147a)) 56 data bytes 64 bytes from 2606:4700:3034::6815:147a (2606:4700:3034::6815:147a): icmp_seq=1 ttl=255 time=29.0 ms --- spaghetti.pizza ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 28.967/28.967/28.967/0.000 ms This entails a major rework, especially with regard to the storage of tracked connections and to the semantics of epoll(7) references. Indexing TCP and UDP bindings merely by socket proved to be inflexible and unsuitable to handle different connection flows: pasta also provides Layer-2 to Layer-2 socket mapping between init and a separate namespace for local connections, using a pair of splice() system calls for TCP, and a recvmmsg()/sendmmsg() pair for UDP local bindings. For instance, building on the previous example: # ip link set dev lo up # iperf3 -s $ iperf3 -c ::1 -Z -w 32M -l 1024k -P2 | tail -n4 [SUM] 0.00-10.00 sec 52.3 GBytes 44.9 Gbits/sec 283 sender [SUM] 0.00-10.43 sec 52.3 GBytes 43.1 Gbits/sec receiver iperf Done. epoll(7) references now include a generic part in order to demultiplex data to the relevant protocol handler, using 24 bits for the socket number, and an opaque portion reserved for usage by the single protocol handlers, in order to track sockets back to corresponding connections and bindings. A number of fixes pertaining to TCP state machine and congestion window handling are also included here. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Add support for kernels not exporting tcpi_snd_wnd via TCP_INFOStefano Brivio2021-06-081-6/+15
| | | | | | | | | | | | | | | | | | | | | Before commit 8f7baad7f035 ("tcp: Add snd_wnd to TCP_INFO"), the kernel didn't export tcpi_snd_wnd via TCP_INFO, which means we don't know what's the window size of the receiver, socket-side. To get TCP connections working in that case, ignore this value if it's zero during handshake, and use the initial window value as suggested by RFC 6928 (14 600 bytes, instead of 4 380 bytes), to keep network performance usable. To make the TCP dynamic responsive enough in this case, also check the socket for available data whenever we get an ACK segment from tap, instead of waiting until all the data from the tap is dequeued. While at it, fix the window scaling value sent for SYN and SYN, ACK segments: we want to increase the data pointer after writing the option, not the value itself. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Fix window size in initial SYN, ACK segment to guestStefano Brivio2021-06-051-4/+7
| | | | | | | | | | | | During handshake, the initial SYN, ACK segment to the guest, send as a response to the SYN segment, needs to report the unscaled value for the window, given that the handshake hasn't completed yet. While at it, fix the endianness for the window value in case TCP parameters can't be queried via TCP_INFO and we need to use the default value. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Properly initialise parameters for SO_ACCEPTCONN getsockopt()Stefano Brivio2021-05-211-2/+3
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Actually enforce MAX_CONNS limitStefano Brivio2021-05-211-1/+11
| | | | | | | | | | | | | | and, given that the connection table is indexed by socket number, we also need to increase MAX_CONNS now as the ICMP implementation needs 2^17 sockets, that will be opened before TCP connections are accepted. This needs to be changed later: the connection table should be indexed by a translated number -- we're wasting 2^17 table entries otherwise. Move initialisation of TCP listening sockets as last per-protocol initialisation, this will make it easier. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Close socket on EPOLLHUP or EPOLLRDHUP in non-data stateStefano Brivio2021-05-211-0/+2
| | | | | | | If the peer doesn't shut down orderly, this might happen: just close the socket then. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* udp, passt: Introduce socket packet buffer, avoid getsockname() for UDPStefano Brivio2021-04-301-1/+4
| | | | | | | | | | This is in preparation for scatter-gather IO on the UDP receive path: save a getsockname() syscall by setting a flag if we get the numbering of all bound sockets in a strict sequence (expected, in practice) and repurpose the tap buffer to be also a socket receive buffer, passing it down to protocol handlers. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* udp: Connection tracking for ephemeral, local ports, and related fixesStefano Brivio2021-04-291-34/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As we support UDP forwarding for packets that are sent to local ports, we actually need some kind of connection tracking for UDP. While at it, this commit introduces a number of vaguely related fixes for issues observed while trying this out. In detail: - implement an explicit, albeit minimalistic, connection tracking for UDP, to allow usage of ephemeral ports by the guest and by the host at the same time, by binding them dynamically as needed, and to allow mapping address changes for packets with a loopback address as destination - set the guest MAC address whenever we receive a packet from tap instead of waiting for an ARP request, and set it to broadcast on start, otherwise DHCPv6 might not work if all DHCPv6 requests time out before the guest starts talking IPv4 - split context IPv6 address into address we assign, global or site address seen on tap, and link-local address seen on tap, and make sure we use the addresses we've seen as destination (link-local choice depends on source address). Similarly, for IPv4, split into address we assign and address we observe, and use the address we observe as destination - introduce a clock_gettime() syscall right after epoll_wait() wakes up, so that we can remove all the other ones and pass the current timestamp to tap and socket handlers -- this is additionally needed by UDP to time out bindings to ephemeral ports and mappings between loopback address and a local address - rename sock_l4_add() to sock_l4(), no semantic changes intended - include <arpa/inet.h> in passt.c before kernel headers so that we can use <netinet/in.h> macros to check IPv6 address types, and remove a duplicate <linux/ip.h> inclusion Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Avoid SO_ACCEPTCONN getsockopt() by noting listening/data sockets numbersStefano Brivio2021-04-291-16/+30
| | | | | | | ...the rest is reshuffling existing macros to use the bits we need in TCP code. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tcp: Preserve data sent during SOCK_SYN_SENT stateStefano Brivio2021-04-291-3/+20
| | | | | | | | | | Seen with iperf3 server on tap side: connection state is SOCK_SYN_SENT, we haven't got an ACK from the tap yet (that's why we're not in ESTABLISHED), but a data packet comes. Don't read this data until we reach the ESTABLISHED state, by keeping EPOLLIN disabled until that point. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-11 07:01:49 +0000