TCP Monitoring

Monitor TCP connections using eBPF with the TCP Event Interceptor

The TCP Event Interceptor captures TCP connection lifecycle events using eBPF kernel probes. It tracks connections from establishment to closure, recording network statistics including bytes transferred, segments sent/received, and connection metadata.

Overview

The TCP tracer attaches to the tcp_set_state kernel function to monitor TCP state transitions. When a connection closes (TCP_CLOSE state), it generates an event containing complete connection statistics.

Event Structure

Each TCP event contains the following fields:

struct tcp_event_t {
    uint64_t EventTime;        // Event timestamp (nanoseconds)
    uint32_t pid;              // Process ID
    uint32_t UserId;           // User ID
    uint64_t rx_b;             // Bytes received
    uint64_t tx_b;             // Bytes transmitted (acked)
    uint32_t tcpi_segs_out;    // TCP segments sent
    uint32_t tcpi_segs_in;     // TCP segments received
    uint16_t family;           // Address family (AF_INET or AF_INET6)
    uint16_t SPT;              // Source port
    uint16_t DPT;              // Destination port
    char task[128];            // Process/command name
    char SADDR[128];           // Source IP address (string)
    char DADDR[128];           // Destination IP address (string)
};

Initializing the TCP Tracer

1. Load the shared library

   Load the TCP event interceptor library using dlopen:

   #include <dlfcn.h>
   
   #define SOFILE "/opt/RealTimeKql/lib/libtcpEvent.so"
   
   void *handle = dlopen(SOFILE, RTLD_LAZY);
   if (!handle) {
       fprintf(stderr, "Failed to load library: %s\n", dlerror());
       exit(EXIT_FAILURE);
   }
   
2. Resolve the AddProbe function

   Get the AddProbe function to attach the BPF program:

   dlerror(); // Clear errors
   void (*AddProbe)(const char *) = dlsym(handle, "AddProbe");
   
   char *error = dlerror();
   if (error) {
       fprintf(stderr, "Failed to resolve AddProbe: %s\n", error);
       exit(EXIT_FAILURE);
   }
   
3. Resolve the DequeuePerfEvent function

   Get the event dequeue function:

   dlerror();
   struct tcp_event_t (*DequeuePerfEvent)() = dlsym(handle, "DequeuePerfEvent");
   
   error = dlerror();
   if (error) {
       fprintf(stderr, "Failed to resolve DequeuePerfEvent: %s\n", error);
       exit(EXIT_FAILURE);
   }
   
4. Resolve additional functions

   Get status checking and cleanup functions:

   // Get status function
   unsigned (*getStatus)() = dlsym(handle, "getStatus");
   if (dlerror()) {
       fprintf(stderr, "Failed to resolve getStatus\n");
       exit(EXIT_FAILURE);
   }
   
   // Get cleanup function
   void (*cleanup)() = dlsym(handle, "cleanup");
   if (dlerror()) {
       fprintf(stderr, "Failed to resolve cleanup\n");
       exit(EXIT_FAILURE);
   }
   
5. Attach the BPF probe

   Call AddProbe with your BPF program:

   AddProbe(BPF_PROGRAM);
   

   The BPF program should attach to tcp_set_state and define the event structure matching the kernel’s TCP socket fields.

6. Wait for initialization

   Wait for the tracer to be ready:

   while (!getStatus()) {
       puts("Waiting for tracer initialization...");
       sleep(1);
   }
   

Complete Monitoring Example

Here’s a complete example based on the test implementation:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <signal.h>
#include <unistd.h>
#include "common.h"

#define SOFILE "/opt/RealTimeKql/lib/libtcpEvent.so"

void (*cleanup)();

void signalHandler(int signum) {
    printf("Interrupted by signal %u\n", signum);
    cleanup();
    exit(signum);
}

void printEvent(struct tcp_event_t *event) {
    if (!event) return;
    
    printf("---\n");
    printf("PID: %d\n", event->pid);
    printf("UID: %d\n", event->UserId);
    printf("Bytes received: %ld\n", event->rx_b);
    printf("Bytes sent: %ld\n", event->tx_b);
    printf("Segments out: %d\n", event->tcpi_segs_out);
    printf("Segments in: %d\n", event->tcpi_segs_in);
    printf("Command: %s\n", event->task);
    printf("Source: %s:%d\n", event->SADDR, event->SPT);
    printf("Destination: %s:%d\n", event->DADDR, event->DPT);
    printf("Event time: %ld\n", event->EventTime);
    printf("---\n");
}

int main() {
    // Load library
    void *handle = dlopen(SOFILE, RTLD_LAZY);
    if (!handle) {
        fprintf(stderr, "dlopen failed: %s\n", dlerror());
        exit(EXIT_FAILURE);
    }
    
    // Resolve symbols
    void (*AddProbe)(const char *) = dlsym(handle, "AddProbe");
    struct tcp_event_t (*DequeuePerfEvent)() = dlsym(handle, "DequeuePerfEvent");
    cleanup = dlsym(handle, "cleanup");
    unsigned (*getStatus)() = dlsym(handle, "getStatus");
    
    // Check for errors (simplified)
    if (!AddProbe || !DequeuePerfEvent || !cleanup || !getStatus) {
        fprintf(stderr, "Failed to resolve symbols\n");
        exit(EXIT_FAILURE);
    }
    
    // Setup signal handler
    signal(SIGINT, signalHandler);
    
    // Attach probe with your BPF program
    AddProbe(BPF_PROGRAM);
    
    // Wait for initialization
    while (!getStatus()) {
        sleep(1);
    }
    
    // Main event loop
    while (1) {
        struct tcp_event_t event = DequeuePerfEvent();
        printEvent(&event);
    }
    
    dlclose(handle);
    return 0;
}

Interpreting Event Data

Network Statistics

rx_b

Total bytes received on the connection (from tcp_sock->bytes_received)

tx_b

Total bytes acknowledged/transmitted (from tcp_sock->bytes_acked)

tcpi_segs_out

Number of TCP segments sent (from tcp_sock->data_segs_out)

tcpi_segs_in

Number of TCP segments received (from tcp_sock->data_segs_in)

Process Information

• pid: Process ID that owns the socket
• UserId: User ID of the process
• task: Process name (up to 128 characters)

The process information is captured at connection establishment or close. For long-lived connections, the process may have changed ownership.

Connection Endpoints

• family: AF_INET (2) for IPv4, AF_INET6 (10) for IPv6
• SADDR/SPT: Source IP address and port
• DADDR/DPT: Destination IP address and port

Ports are in host byte order. The source port is from sk->__sk_common.skc_num, and the destination port is converted from network byte order using ntohs().

Timestamps

• EventTime: Nanosecond timestamp when the connection closed

The timestamp is adjusted to account for system boot time, providing an absolute wall-clock time rather than a monotonic kernel time.

IPv4 and IPv6 Support

The tracer automatically handles both IPv4 and IPv6 connections:

// From the BPF program
if (family == AF_INET) {
    event.family = AF_INET; 
    event.saddr = sk->__sk_common.skc_rcv_saddr;
    event.daddr = sk->__sk_common.skc_daddr;
} else if (family == AF_INET6) {
    event.family = AF_INET6;
    bpf_probe_read(&event.saddr, sizeof(event.saddr), 
                   sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
    bpf_probe_read(&event.daddr, sizeof(event.daddr), 
                   sk->__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}

Addresses are automatically converted to string format in the event structure.

Netlink Diagnostics Integration

The TCP tracer includes netlink socket diagnostics support for enriching connection data. This provides additional TCP metrics beyond what’s available from the BPF hooks.

See common.h for the complete anu_tcp_info structure which mirrors the kernel’s TCP info structure with fields like RTT, retransmissions, congestion window, and more.

Cleanup and Shutdown

1. Setup signal handler

   Register a signal handler for graceful shutdown:

   void signalHandler(int signum) {
       printf("Caught signal %d, cleaning up...\n", signum);
       cleanup();
       exit(signum);
   }
   
   signal(SIGINT, signalHandler);
   signal(SIGTERM, signalHandler);
   
2. Call cleanup function

   The cleanup() function detaches all kprobes and releases BPF resources:

   cleanup();
   
3. Close library handle

   Close the dynamic library:

   dlclose(handle);
   

Example Output

When running the TCP tracer, you’ll see output like this:

---
PID: 1177932
UID: 1000
Bytes received: 2988
Bytes sent: 3301
Segments out: 20
Segments in: 18
Command: ssh
Source: 2001:aaa:fff:eee:ccc:a627:f45f:9c0c:58532
Destination: 2601:xxx:yyy:zzz:aaa:db60:46cd:971c:22
Event time: 1628184562000000000
---

Best Practices

Error Handling

Always check return values from dlsym() and handle errors appropriately.

Signal Handling

Implement proper signal handling to ensure cleanup is called before exit.

Event Processing

Events are queued internally. Process them promptly to avoid queue overflow.

Root Privileges

eBPF programs require root or CAP_BPF capabilities to load and attach.

Troubleshooting

Library not found

Ensure the library is installed at /opt/RealTimeKql/lib/libtcpEvent.so. If installed elsewhere, update the SOFILE path.

Permission denied

eBPF requires elevated privileges. Run with sudo or grant CAP_BPF capability:

sudo ./tcpEventTest

No events appearing

Verify the probe attached successfully by checking kernel logs:

sudo dmesg | tail

Events are only generated when TCP connections close.

Incomplete event data

For short-lived connections, process information may be captured at different times. The tracer attempts to preserve the original process that established the connection.

Next Steps

UDP Monitoring

Learn how to monitor UDP traffic

Building from Source

Build and customize the interceptor

Testing

Run tests and verify functionality

API Reference

Detailed TCP API documentation