FreeBSD Network Performance Optimization Guide

FreeBSD has one of the best network stacks in any operating system. The default configuration is sensible for general use, but if you are pushing 10 Gbps or higher, serving thousands of connections, or building a router/firewall, the defaults leave performance on the table.

This guide covers the real knobs that matter. Every sysctl, every driver parameter, every architectural decision -- tested on production systems.

Baseline: Measure Before You Tune

Never tune blind. Establish baselines first.

Install Benchmarking Tools

sh
pkg install iperf3 netperf nuttcp

Run a Baseline Test

On the server side:

sh
iperf3 -s

On the client side:

sh
iperf3 -c server_ip -t 30 -P 4

Record throughput, CPU usage (via top -SH), and retransmissions. You need numbers to compare against after each change.

Check Current Network Configuration

sh
ifconfig igb0
sysctl net.inet.tcp.sendspace
sysctl net.inet.tcp.recvspace
sysctl kern.ipc.maxsockbuf
netstat -m

Sysctl Tunables: The Core Knobs

TCP Buffer Sizes

The single most impactful change for throughput on high-bandwidth links. The default send/receive buffers are sized for 1 Gbps. For 10 Gbps and above, increase them:

sh
sysctl net.inet.tcp.sendspace=262144
sysctl net.inet.tcp.recvspace=262144
sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.recvbuf_max=16777216

Make persistent in /etc/sysctl.conf:

sh
cat >> /etc/sysctl.conf << 'EOF'
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
EOF

For high-latency links (WAN, cross-continent), increase sendspace and recvspace further. The bandwidth-delay product formula: buffer = bandwidth_bytes_per_sec * RTT_seconds. A 10 Gbps link with 50 ms RTT needs ~62 MB buffers.

TCP Congestion Control

FreeBSD supports multiple congestion control algorithms. For high-bandwidth WAN links, CUBIC or BBR often outperform the default NewReno:

sh
kldload cc_cubic
sysctl net.inet.tcp.cc.algorithm=cubic

To load at boot:

sh
echo 'cc_cubic_load="YES"' >> /boot/loader.conf
echo 'net.inet.tcp.cc.algorithm=cubic' >> /etc/sysctl.conf

Connection Tracking and Hash Tables

For servers handling many concurrent connections:

sh
sysctl net.inet.tcp.tcbhashsize=524288
sysctl net.inet.tcp.hostcache.hashsize=16384
sysctl kern.ipc.somaxconn=4096
sysctl kern.ipc.soacceptqueue=4096

The tcbhashsize must be set in /boot/loader.conf as it is read-only at runtime:

sh
echo 'net.inet.tcp.tcbhashsize=524288' >> /boot/loader.conf

TIME_WAIT Tuning

Servers with high connection churn accumulate TIME_WAIT sockets. Reduce the timeout:

sh
sysctl net.inet.tcp.msl=5000

This sets the Maximum Segment Lifetime to 5 seconds (default is 30 seconds), halving the TIME_WAIT duration to 10 seconds.

Enable TIME_WAIT socket recycling:

sh
sysctl net.inet.tcp.maxtcptw=200000
sysctl net.inet.tcp.nolocaltimewait=1

UDP Tuning

For applications using UDP (DNS, gaming, media streaming):

sh
sysctl net.inet.udp.maxdgram=65536
sysctl net.inet.udp.recvspace=262144
sysctl net.local.dgram.maxdgram=65536

Ring Buffers and Descriptor Counts

Network adapters have hardware ring buffers for transmit (TX) and receive (RX). Larger rings reduce packet drops under load but use more memory.

Check current ring size:

sh
sysctl dev.igb.0.rx_ring_size
sysctl dev.igb.0.tx_ring_size

Increase for Intel NICs:

sh
sysctl dev.igb.0.rx_ring_size=4096
sysctl dev.igb.0.tx_ring_size=4096

For Mellanox (mlx5):

sh
sysctl dev.mlx5en.0.rx_ring_size=8192
sysctl dev.mlx5en.0.tx_ring_size=8192

Add to /etc/sysctl.conf for persistence. The optimal size depends on your traffic pattern -- bursty traffic benefits more from large rings.

Interrupt Coalescing

Every packet that arrives generates a hardware interrupt. At high packet rates (millions of packets per second), interrupts consume entire CPU cores. Interrupt coalescing batches multiple packets into a single interrupt.

Intel igb/ixgbe

sh
sysctl dev.igb.0.rx_itr=450
sysctl dev.igb.0.tx_itr=450

Lower values = lower latency but more CPU. Higher values = higher throughput but more latency. For a firewall or router, use 100-200. For a file server, use 400-800.

Mellanox mlx5

sh
sysctl dev.mlx5en.0.conf.rx_coal_usecs=64
sysctl dev.mlx5en.0.conf.tx_coal_usecs=64

Polling

Polling flips the interrupt model. Instead of the NIC interrupting the CPU for each batch of packets, the CPU polls the NIC at regular intervals. This eliminates interrupt overhead at high packet rates.

Enable polling for a specific interface:

sh
ifconfig igb0 polling

Or system-wide:

sh
sysctl kern.polling.enable=1
sysctl kern.polling.each_burst=150
sysctl kern.polling.burst_max=1000

Polling is most beneficial when you are CPU-bound from interrupt processing. If your system is not saturating CPUs on interrupts, polling may actually add latency.

TSO, LRO, and Offloading

Modern NICs can offload work from the CPU.

TCP Segmentation Offload (TSO)

TSO lets the NIC split large TCP segments instead of the CPU doing it. Verify it is enabled:

sh
ifconfig igb0 | grep options

Look for TXCSUM and TSO4/TSO6. Enable if missing:

sh
ifconfig igb0 tso

Large Receive Offload (LRO)

LRO coalesces multiple incoming TCP segments into larger ones before passing them to the stack:

sh
ifconfig igb0 lro

Warning: Disable LRO on routers and firewalls. LRO modifies packets before they hit PF/IPFW, which breaks forwarding logic:

sh
ifconfig igb0 -lro

Checksum Offload

sh
ifconfig igb0 txcsum rxcsum

These are enabled by default on most modern NICs. Verify with ifconfig.

Receive Side Scaling (RSS)

RSS distributes incoming packets across multiple CPU cores using a hash of the packet header. Without RSS, a single core processes all packets from one NIC.

Check if your NIC supports RSS:

sh
sysctl dev.igb.0.queues

Enable RSS with multiple queues:

sh
echo 'hw.igb.num_queues=8' >> /boot/loader.conf

Match the queue count to your CPU core count, or half your core count on hyper-threaded systems.

Verify RSS distribution after reboot:

sh
vmstat -i | grep igb

You should see interrupts distributed across multiple IRQs, each bound to a different CPU.

CPU Affinity for NIC Interrupts

Manually bind NIC interrupts to specific CPUs for more predictable performance:

sh
cpuset -l 0 -x $(vmstat -ai | grep igb0:rxq0 | awk '{print $1}')
cpuset -l 1 -x $(vmstat -ai | grep igb0:rxq1 | awk '{print $1}')
cpuset -l 2 -x $(vmstat -ai | grep igb0:rxq2 | awk '{print $1}')
cpuset -l 3 -x $(vmstat -ai | grep igb0:rxq3 | awk '{print $1}')

LAGG: Link Aggregation

Combine multiple physical links for increased bandwidth or redundancy.

LACP (802.3ad)

Requires switch support:

sh
ifconfig lagg0 create
ifconfig lagg0 laggproto lacp
ifconfig lagg0 laggport igb0
ifconfig lagg0 laggport igb1
ifconfig lagg0 inet 10.0.0.1/24

Make persistent in /etc/rc.conf:

sh
cat >> /etc/rc.conf << 'EOF'
cloned_interfaces="lagg0"
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 inet 10.0.0.1/24"
EOF

Failover

For redundancy without switch configuration:

sh
ifconfig lagg0 create
ifconfig lagg0 laggproto failover
ifconfig lagg0 laggport igb0
ifconfig lagg0 laggport igb1

LAGG Hashing

LACP distributes traffic based on a hash. The default hash uses L2+L3+L4 headers. Verify:

sh
sysctl net.link.lagg.default_flowid_hash

For environments with many connections to different IPs, the default is fine. For few connections to few IPs (like a storage link), L4 hashing is critical to distribute across links.

Network Stack Architecture

NETISR Threading

FreeBSD processes network packets through NETISR (network interrupt service routine) threads. Increase the thread count for multi-core systems:

sh
sysctl net.isr.maxthreads=8
sysctl net.isr.dispatch=deferred

Check current NETISR statistics:

sh
netstat -Q

Zero-Copy Sockets

For applications that move large amounts of data (file servers, proxies), zero-copy sockets eliminate a memory copy:

sh
sysctl kern.ipc.zero_copy.send=1
sysctl kern.ipc.zero_copy.receive=1

Practical Benchmark Results

Here are typical throughput numbers on a dual Xeon system with Intel X710 10GbE NICs, before and after tuning:

|------|---------|-------|-------------|

| TCP connections/sec | 28K/s | 85K/s | +204% |

The biggest gains come from buffer sizing (throughput), RSS (multi-core distribution), and interrupt coalescing (packet rate).

Complete Tuning Template

Here is a consolidated /etc/sysctl.conf for a high-performance server:

sh
cat > /etc/sysctl.conf << 'EOF'
# TCP buffers
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216

# Congestion control
net.inet.tcp.cc.algorithm=cubic

# Connection handling
kern.ipc.somaxconn=4096
kern.ipc.soacceptqueue=4096
net.inet.tcp.msl=5000
net.inet.tcp.maxtcptw=200000
net.inet.tcp.nolocaltimewait=1

# UDP
net.inet.udp.maxdgram=65536
net.inet.udp.recvspace=262144

# NETISR
net.isr.maxthreads=8
net.isr.dispatch=deferred

# Zero-copy
kern.ipc.zero_copy.send=1
kern.ipc.zero_copy.receive=1
EOF

And /boot/loader.conf:

sh
cat >> /boot/loader.conf << 'EOF'
cc_cubic_load="YES"
net.inet.tcp.tcbhashsize=524288
hw.igb.num_queues=8
EOF

FAQ

Q: Should I tune network performance on every FreeBSD server?

A: Not necessarily. The defaults are good for moderate workloads. Tune when you measure a bottleneck -- not before.

Q: Is polling better than interrupt coalescing?

A: Polling works best at extreme packet rates (millions of packets per second). For most servers, interrupt coalescing with RSS is the better approach. Polling adds baseline latency even at low traffic.

Q: Should I enable TSO and LRO everywhere?

A: Enable TSO on all endpoints. Enable LRO only on endpoints, never on routers or firewalls. LRO modifies packet structure, which breaks packet forwarding.

Q: How many RSS queues should I configure?

A: Match your physical core count. More queues than cores wastes memory. Fewer queues than cores leaves CPU capacity unused.

Q: Does LACP double my bandwidth?

A: Not for a single TCP connection. LACP distributes flows across links based on a hash. A single flow uses one link. Many concurrent flows get distributed across all links.

Q: What congestion control algorithm should I use?

A: NewReno (the default) is fine for LAN and low-latency WAN. CUBIC is better for high-bandwidth, high-latency WAN links. BBR is experimental on FreeBSD but promising for lossy links.

Q: How do I verify my tuning is working?

A: Benchmark before and after every change with iperf3. Monitor CPU usage with top -SH and interrupt distribution with vmstat -i. One change at a time.

Q: Do these tunables work on FreeBSD 13?

A: Most of them. Some sysctl names changed between 13 and 14. Always check sysctl -d name to verify a tunable exists on your version.

Q: What about jumbo frames?

A: Jumbo frames (MTU 9000) help for bulk transfers on dedicated storage networks. Do not enable them on shared networks -- every device in the path must support the same MTU. Test with ping -c 5 -D -s 8972 target_ip.