TCP Monitoring

The TCP Event Interceptor captures TCP connection lifecycle events using eBPF kernel probes. It tracks connections from establishment to closure, recording network statistics including bytes transferred, segments sent/received, and connection metadata.

Overview

The TCP tracer attaches to the tcp_set_state kernel function to monitor TCP state transitions. When a connection closes (TCP_CLOSE state), it generates an event containing complete connection statistics.

Event Structure

Each TCP event contains the following fields:

struct tcp_event_t {
    uint64_t EventTime;        // Event timestamp (nanoseconds)
    uint32_t pid;              // Process ID
    uint32_t UserId;           // User ID
    uint64_t rx_b;             // Bytes received
    uint64_t tx_b;             // Bytes transmitted (acked)
    uint32_t tcpi_segs_out;    // TCP segments sent
    uint32_t tcpi_segs_in;     // TCP segments received
    uint16_t family;           // Address family (AF_INET or AF_INET6)
    uint16_t SPT;              // Source port
    uint16_t DPT;              // Destination port
    char task[128];            // Process/command name
    char SADDR[128];           // Source IP address (string)
    char DADDR[128];           // Destination IP address (string)
};

Initializing the TCP Tracer

Load the shared library

Load the TCP event interceptor library using dlopen:

#include <dlfcn.h>

#define SOFILE "/opt/RealTimeKql/lib/libtcpEvent.so"

void *handle = dlopen(SOFILE, RTLD_LAZY);
if (!handle) {
    fprintf(stderr, "Failed to load library: %s\n", dlerror());
    exit(EXIT_FAILURE);
}

Resolve the AddProbe function

Get the AddProbe function to attach the BPF program:

dlerror(); // Clear errors
void (*AddProbe)(const char *) = dlsym(handle, "AddProbe");

char *error = dlerror();
if (error) {
    fprintf(stderr, "Failed to resolve AddProbe: %s\n", error);
    exit(EXIT_FAILURE);
}

Resolve the DequeuePerfEvent function

Get the event dequeue function:

dlerror();
struct tcp_event_t (*DequeuePerfEvent)() = dlsym(handle, "DequeuePerfEvent");

error = dlerror();
if (error) {
    fprintf(stderr, "Failed to resolve DequeuePerfEvent: %s\n", error);
    exit(EXIT_FAILURE);
}

Resolve additional functions

Get status checking and cleanup functions:

// Get status function
unsigned (*getStatus)() = dlsym(handle, "getStatus");
if (dlerror()) {
    fprintf(stderr, "Failed to resolve getStatus\n");
    exit(EXIT_FAILURE);
}

// Get cleanup function
void (*cleanup)() = dlsym(handle, "cleanup");
if (dlerror()) {
    fprintf(stderr, "Failed to resolve cleanup\n");
    exit(EXIT_FAILURE);
}

Attach the BPF probe

Call AddProbe with your BPF program:

AddProbe(BPF_PROGRAM);

The BPF program should attach to tcp_set_state and define the event structure matching the kernel’s TCP socket fields.

Wait for initialization

Wait for the tracer to be ready:

while (!getStatus()) {
    puts("Waiting for tracer initialization...");
    sleep(1);
}

Complete Monitoring Example

Here’s a complete example based on the test implementation:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <signal.h>
#include <unistd.h>
#include "common.h"

#define SOFILE "/opt/RealTimeKql/lib/libtcpEvent.so"

void (*cleanup)();

void signalHandler(int signum) {
    printf("Interrupted by signal %u\n", signum);
    cleanup();
    exit(signum);
}

void printEvent(struct tcp_event_t *event) {
    if (!event) return;
    
    printf("---\n");
    printf("PID: %d\n", event->pid);
    printf("UID: %d\n", event->UserId);
    printf("Bytes received: %ld\n", event->rx_b);
    printf("Bytes sent: %ld\n", event->tx_b);
    printf("Segments out: %d\n", event->tcpi_segs_out);
    printf("Segments in: %d\n", event->tcpi_segs_in);
    printf("Command: %s\n", event->task);
    printf("Source: %s:%d\n", event->SADDR, event->SPT);
    printf("Destination: %s:%d\n", event->DADDR, event->DPT);
    printf("Event time: %ld\n", event->EventTime);
    printf("---\n");
}

int main() {
    // Load library
    void *handle = dlopen(SOFILE, RTLD_LAZY);
    if (!handle) {
        fprintf(stderr, "dlopen failed: %s\n", dlerror());
        exit(EXIT_FAILURE);
    }
    
    // Resolve symbols
    void (*AddProbe)(const char *) = dlsym(handle, "AddProbe");
    struct tcp_event_t (*DequeuePerfEvent)() = dlsym(handle, "DequeuePerfEvent");
    cleanup = dlsym(handle, "cleanup");
    unsigned (*getStatus)() = dlsym(handle, "getStatus");
    
    // Check for errors (simplified)
    if (!AddProbe || !DequeuePerfEvent || !cleanup || !getStatus) {
        fprintf(stderr, "Failed to resolve symbols\n");
        exit(EXIT_FAILURE);
    }
    
    // Setup signal handler
    signal(SIGINT, signalHandler);
    
    // Attach probe with your BPF program
    AddProbe(BPF_PROGRAM);
    
    // Wait for initialization
    while (!getStatus()) {
        sleep(1);
    }
    
    // Main event loop
    while (1) {
        struct tcp_event_t event = DequeuePerfEvent();
        printEvent(&event);
    }
    
    dlclose(handle);
    return 0;
}

Interpreting Event Data

Network Statistics

rx_b

Total bytes received on the connection (from tcp_sock->bytes_received)

tx_b

Total bytes acknowledged/transmitted (from tcp_sock->bytes_acked)

tcpi_segs_out

Number of TCP segments sent (from tcp_sock->data_segs_out)

tcpi_segs_in

Number of TCP segments received (from tcp_sock->data_segs_in)

Process Information

pid: Process ID that owns the socket
UserId: User ID of the process
task: Process name (up to 128 characters)

The process information is captured at connection establishment or close. For long-lived connections, the process may have changed ownership.

Connection Endpoints

family: AF_INET (2) for IPv4, AF_INET6 (10) for IPv6
SADDR/SPT: Source IP address and port
DADDR/DPT: Destination IP address and port

Ports are in host byte order. The source port is from sk->__sk_common.skc_num, and the destination port is converted from network byte order using ntohs().

Timestamps

EventTime: Nanosecond timestamp when the connection closed

The timestamp is adjusted to account for system boot time, providing an absolute wall-clock time rather than a monotonic kernel time.

IPv4 and IPv6 Support

The tracer automatically handles both IPv4 and IPv6 connections:

// From the BPF program
if (family == AF_INET) {
    event.family = AF_INET; 
    event.saddr = sk->__sk_common.skc_rcv_saddr;
    event.daddr = sk->__sk_common.skc_daddr;
} else if (family == AF_INET6) {
    event.family = AF_INET6;
    bpf_probe_read(&event.saddr, sizeof(event.saddr), 
                   sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
    bpf_probe_read(&event.daddr, sizeof(event.daddr), 
                   sk->__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}

Addresses are automatically converted to string format in the event structure.

Netlink Diagnostics Integration

The TCP tracer includes netlink socket diagnostics support for enriching connection data. This provides additional TCP metrics beyond what’s available from the BPF hooks.

See common.h for the complete anu_tcp_info structure which mirrors the kernel’s TCP info structure with fields like RTT, retransmissions, congestion window, and more.

Cleanup and Shutdown

Setup signal handler

void signalHandler(int signum) {
    printf("Caught signal %d, cleaning up...\n", signum);
    cleanup();
    exit(signum);
}

signal(SIGINT, signalHandler);
signal(SIGTERM, signalHandler);

Call cleanup function

The cleanup() function detaches all kprobes and releases BPF resources:

cleanup();

Close library handle

Close the dynamic library:

dlclose(handle);

Example Output

When running the TCP tracer, you’ll see output like this:

---
PID: 1177932
UID: 1000
Bytes received: 2988
Bytes sent: 3301
Segments out: 20
Segments in: 18
Command: ssh
Source: 2001:aaa:fff:eee:ccc:a627:f45f:9c0c:58532
Destination: 2601:xxx:yyy:zzz:aaa:db60:46cd:971c:22
Event time: 1628184562000000000
---

Best Practices

Error Handling

Always check return values from dlsym() and handle errors appropriately.

Signal Handling

Implement proper signal handling to ensure cleanup is called before exit.

Event Processing

Events are queued internally. Process them promptly to avoid queue overflow.

Root Privileges

eBPF programs require root or CAP_BPF capabilities to load and attach.

Troubleshooting

Library not found

Ensure the library is installed at /opt/RealTimeKql/lib/libtcpEvent.so. If installed elsewhere, update the SOFILE path.

Permission denied

eBPF requires elevated privileges. Run with sudo or grant CAP_BPF capability:

sudo ./tcpEventTest

No events appearing

Verify the probe attached successfully by checking kernel logs:

sudo dmesg | tail

Events are only generated when TCP connections close.

Incomplete event data

For short-lived connections, process information may be captured at different times. The tracer attempts to preserve the original process that established the connection.

Next Steps

UDP Monitoring

Learn how to monitor UDP traffic

Building from Source

Build and customize the interceptor

Testing

Run tests and verify functionality

API Reference

Detailed TCP API documentation

Get Started

Core Concepts

Guides

Overview

Event Structure

Initializing the TCP Tracer

Complete Monitoring Example

Interpreting Event Data

Network Statistics

rx_b

tx_b

tcpi_segs_out

tcpi_segs_in

Process Information

Connection Endpoints

Timestamps

IPv4 and IPv6 Support

Netlink Diagnostics Integration

Cleanup and Shutdown

Example Output

Best Practices

Error Handling

Signal Handling

Event Processing

Root Privileges

Troubleshooting

Next Steps

UDP Monitoring

Building from Source

Testing

API Reference

Get Started

Core Concepts

Guides

​Overview

​Event Structure

​Initializing the TCP Tracer

​Complete Monitoring Example

​Interpreting Event Data

​Network Statistics

rx_b

tx_b

tcpi_segs_out

tcpi_segs_in

​Process Information

​Connection Endpoints

​Timestamps

​IPv4 and IPv6 Support

​Netlink Diagnostics Integration

​Cleanup and Shutdown

​Example Output

​Best Practices

Error Handling

Signal Handling

Event Processing

Root Privileges

​Troubleshooting

​Next Steps

UDP Monitoring

Building from Source

Testing

API Reference

Overview

Event Structure

Initializing the TCP Tracer

Complete Monitoring Example

Interpreting Event Data

Network Statistics

Process Information

Connection Endpoints

Timestamps

IPv4 and IPv6 Support

Netlink Diagnostics Integration

Cleanup and Shutdown

Example Output

Best Practices

Troubleshooting

Next Steps