System Architecture

Overview

The eBPF Event Interceptor is built as a modular system with two primary components: tcpEvent and udpEvent. Each component operates as an independent library that attaches eBPF programs to kernel functions, streams events through perf buffers, and provides a C/C++ API for event consumption.

Both tcpEvent and udpEvent are compiled as shared objects (.so files) that can be loaded dynamically into applications for real-time network monitoring.

High-Level Architecture

The system follows a kernel-to-userspace event pipeline:

┌─────────────────────────────────────────────────────────────┐
│                     Kernel Space                            │
│  ┌──────────────┐    ┌──────────────┐   ┌──────────────┐  │
│  │ tcp_set_state│    │ udp_sendmsg  │   │ udp_recvmsg  │  │
│  └──────┬───────┘    └──────┬───────┘   └──────┬───────┘  │
│         │                   │                   │          │
│    ┌────▼────────────────────▼───────────────────▼─────┐   │
│    │         eBPF Programs (Kprobes)                   │   │
│    │  - kprobe__tcp_set_state                          │   │
│    │  - kprobe__udp_sendmsg / kprobe__udpv6_sendmsg   │   │
│    │  - kprobe_udp_recvmsg / kretprobe__udp_recvmsg   │   │
│    └────────────────────┬──────────────────────────────┘   │
│                         │                                  │
│                    ┌────▼─────┐                            │
│                    │ BPF Maps │                            │
│                    │ & Tables │                            │
│                    └────┬─────┘                            │
│                         │                                  │
│                  ┌──────▼───────┐                          │
│                  │ Perf Buffers │                          │
│                  │ - tcpEvents  │                          │
│                  │ - bpfPerfBuf │                          │
│                  └──────┬───────┘                          │
└─────────────────────────┼──────────────────────────────────┘
                          │
┌─────────────────────────▼──────────────────────────────────┐
│                    User Space                              │
│  ┌────────────────────────────────────────────────────┐   │
│  │              BCC Framework (libbcc)                │   │
│  │  - bpf.init()                                      │   │
│  │  - bpf.attach_kprobe()                             │   │
│  │  - bpf.open_perf_buffer()                          │   │
│  │  - bpf.poll_perf_buffer()                          │   │
│  └────────────┬───────────────────────────────────────┘   │
│               │                                            │
│       ┌───────▼────────┐                                   │
│       │ handle_output()│                                   │
│       │   callback     │                                   │
│       └───────┬────────┘                                   │
│               │                                            │
│      ┌────────▼─────────┐                                  │
│      │  std::deque      │                                  │
│      │  eventDeque      │                                  │
│      │  (MAXQSIZE=1024) │                                  │
│      └────────┬─────────┘                                  │
│               │                                            │
│    ┌──────────▼───────────┐                                │
│    │ DequeuePerfEvent()   │                                │
│    │  Public API          │                                │
│    └──────────────────────┘                                │
└────────────────────────────────────────────────────────────┘

Core Components

tcpEvent Library

The TCP event interceptor monitors TCP state changes and enriches events with socket diagnostics. Key Files:

tcpEvent/event.cc - Main implementation (710 lines)
tcpEvent/event.h - Public API declarations
tcpEvent/common.h - Data structure definitions

Functionality:

Attaches to tcp_set_state kernel function
Uses netlink socket diagnostics for detailed TCP statistics
Dual event collection: kprobes + netlink polling
Attributes events to processes via /proc filesystem

TCP Event Flow

Kprobe on tcp_set_state captures state changes
Events pushed to tcpEvents perf buffer
Parallel netlink thread polls socket diagnostics every 6.2s
Events enriched with PID, UID, command name, and TCP stats
Stored in deque for consumer retrieval

udpEvent Library

The UDP event interceptor tracks UDP send/receive operations across IPv4 and IPv6. Key Files:

udpEvent/udpTracer.cc - Main implementation (626 lines)
udpEvent/common.h - Data structure definitions

Kprobes Attached:

// Connection tracking
ip4_datagram_connect  → kprobe_ip4_datagram_connect
ip6_datagram_connect  → kprobe_ip6_datagram_connect

// Send operations
udp_sendmsg           → kprobe__udp_sendmsg
udpv6_sendmsg         → kprobe__udpv6_sendmsg

// Receive operations
udp_recvmsg           → kprobe_udp_recvmsg + kretprobe__udp_recvmsg
udpv6_recvmsg         → kprobe__udpv6_recvmsg + kretprobe__udpv6_recvmsg

// Cleanup
udp_destruct_sock     → kprobe_udp_destruct_sock

UDP tracking uses both entry kprobes and return kretprobes to capture actual bytes sent/received from function return values.

Perf Buffer Architecture

Buffer Configuration

Both libraries use BCC’s BPF_PERF_OUTPUT mechanism: TCP:

#define TABLE "tcpEvents"
bpf.open_perf_buffer(TABLE, &handle_output)

UDP:

BPF_PERF_OUTPUT(bpfPerfBuffer);
bpf.open_perf_buffer("bpfPerfBuffer", &handle_output)

Event Flow

Kernel → Perf Buffer: eBPF programs call perf_submit() to push events
Polling: bpf.poll_perf_buffer() runs in infinite loop
Callback: handle_output() invoked for each event
Queuing: Events pushed to std::deque<event_t*>
Consumption: DequeuePerfEvent() retrieves events for applications

Queue Management

// From event.cc:25
#define MAXQSIZE 1024

void handle_output(void *cb_cookie, void *data, int data_size) {
    auto event = static_cast<event_t*>(data);
    pthread_mutex_lock(&mtx);
    if (eventDeque.size() > MAXQSIZE) {
        // Drop oldest event to prevent memory overflow
        almostGone = eventDeque.front();
        eventDeque.pop_front();
        destroyEventPtr(almostGone);
        puts("Shedding TCP events..");
    }
    eventDeque.push_back(event);
    pthread_mutex_unlock(&mtx);
    pthread_cond_signal(&cond);
}

When event production exceeds consumption rate, the system automatically sheds the oldest events. Monitor for “Shedding” messages in logs.

Threading Model

Thread Architecture

Both libraries use POSIX threads for concurrent operation: tcpEvent threads:

pthread_t tid = 0;          // BPF polling thread
pthread_t netLinkTid = 0;   // Netlink diagnostics thread

Synchronization primitives:

pthread_cond_t cond;        // Condition variable for event signaling
pthread_mutex_t mtx;        // Protects eventDeque
pthread_mutex_t mapMu;      // Protects PtrMap for memory tracking
pthread_rwlock_t rwlock;    // Read-write lock for status

Thread Lifecycle

BPF Polling Thread

Created in: setupBPF() function (event.cc:176)Initialization:

int setupBPF(const char *BPF_PROGRAM) {
    tid = pthread_self();
    bpf.init(BPF_PROGRAM);
    bpf.attach_kprobe(FN_NAME, "kprobe__tcp_set_state");
    bpf.open_perf_buffer(TABLE, &handle_output);
    
    while (1) {
        bpf.poll_perf_buffer(TABLE);  // Blocking poll
    }
}

Termination: Cancelled in cleanup() via pthread_cancel(tid)

Netlink Polling Thread (TCP only)

Created in: DequeuePerfEvent() on first call (event.cc:117-121)Purpose: Continuously queries kernel socket diagnostics via netlinkOperation:

void netLinkProbe() {
    netLinkTid = pthread_self();
    while (1) {
        findSocketInodes();          // Scan /proc/[pid]/fd/*
        getEvents();                 // Query netlink for TCP stats
        sleep(NETLINKNAP);          // 6.2 seconds between polls
    }
}

Benefits: Captures connection statistics even without state changes

Consumer Thread

Applications call DequeuePerfEvent() which blocks until events are available:

struct tcp_event_t DequeuePerfEvent() {
    while (1) {
        pthread_mutex_lock(&mtx);
        if (!eventDeque.empty()) {
            auto event = eventDeque.front();
            eventDeque.pop_front();
            pthread_mutex_unlock(&mtx);
            // ... process and return event
        } else {
            pthread_cond_wait(&cond, &mtx);  // Wait for signal
            pthread_mutex_unlock(&mtx);
        }
    }
}

Data Flow: Kernel to User Space

Event Capture Path

Kernel Function Call
- Application calls socket operation (e.g., connect(), send())
- Kernel executes tcp_set_state, udp_sendmsg, etc.
Kprobe Activation
- eBPF program executes before/after kernel function
- Reads kernel memory: struct sock, addresses, ports
- Calls bpf_get_current_pid_tgid(), bpf_get_current_uid_gid()

Perf Buffer Submit

bpfPerfBuffer.perf_submit(ctx, eventPtr, sizeof(*eventPtr));

User Space Polling
- bpf.poll_perf_buffer() retrieves events
- Invokes handle_output() callback
Event Enrichment
- Timestamp adjustment: event->EventTime + notSoLongAgo
- Address conversion: inet_ntop() for IP addresses
- Process lookup: readCmdLine() from /proc
Consumer Delivery
- DequeuePerfEvent() returns structured event
- Application processes network telemetry

TCP Netlink Enrichment

TCP events get additional statistics via netlink socket diagnostics:

// Build socket inode map from /proc
findSocketInodes() → SockStrMap["socket:[12345]"] = pid

// Query kernel for TCP stats
sendDiagMsg() → sends inet_diag_req_v2
harvestEvents() → receives inet_diag_msg replies

// Extract statistics
struct anu_tcp_info {
    uint64_t tcpi_bytes_received;
    uint64_t tcpi_bytes_sent;
    uint32_t tcpi_segs_in;
    uint32_t tcpi_segs_out;
    // ... 40+ more fields
};

The custom anu_tcp_info struct (common.h:75-148) extends standard tcp_info to include tcpi_bytes_sent which is critical for bandwidth tracking.

Process Attribution Mechanism

Both libraries identify which process generated network events:

Inode-to-PID Mapping (TCP)

int findSocketInodes() {
    glob("/proc/[0-9]*/fd/*", &globbuf);  // Find all FDs
    
    for (path in globbuf.gl_pathv) {
        readlink(path, symlinkName);      // "socket:[12345]"
        
        if (starts_with(symlinkName, "socket:[")) {
            extract_pid_from_path(path);   // "/proc/1234/fd/5" → 1234
            SockStrMap[symlinkName] = pid;
        }
    }
}

Reading Command Names

int readCmdLine(uint32_t pid, char *writeTo, int maxLen) {
    sprintf(cmdLine, "/proc/%u/cmdline", pid);
    FILE *fp = fopen(cmdLine, "r");
    fread(writeTo, maxLen, 1, fp);
    return 0;
}

Process information is captured at event time. Short-lived processes may show PID but missing command names if they exit before enrichment.

Memory Management

Event Lifecycle

TCP: Uses pointer tracking map to prevent double-free

std::map<event_t*, int> PtrMap;  // event.cc:30

void destroyEventPtr(event_t* eventPtr) {
    pthread_mutex_lock(&mapMu);
    if (PtrMap.count(eventPtr) && PtrMap.erase(eventPtr)) {
        delete(eventPtr);
    }
    pthread_mutex_unlock(&mapMu);
}

UDP: Simpler model, events deleted after consumption

event = eventDeque.front();
eventDeque.pop_front();
// Event copied to toConsumer struct, original discarded

Time Synchronization

static uint64_t whenDidWeBootUp() {
    FILE *fp = fopen("/proc/uptime", "r");
    double firstWord;  // Seconds since boot
    fscanf(fp, "%lf", &firstWord);
    return (time(0) - firstWord);  // Epoch time of boot
}

// Event timestamp adjustment (event.cc:133)
toConsumer.EventTime = event->EventTime + notSoLongAgo;
// where: notSoLongAgo = whenDidWeBootUp() * 1000000000LLU

Component Communication

Public API

Both libraries expose a minimal C-compatible API:

extern "C" {
    void AddProbe(const char *BPF_PROGRAM);  // Start monitoring
    struct tcp_event_t DequeuePerfEvent();   // Get next event (blocking)
    void cleanup();                          // Stop and detach
    unsigned getStatus();                    // Check if initialized
}

Initialization Sequence

// Application code
AddProbe(BPF_PROGRAM_SOURCE);
    ↓
// Spawns detached thread
std::thread t(setupBPF, BPF_PROGRAM);
t.detach();
    ↓
// In setupBPF thread
bpf.init(BPF_PROGRAM);              // Compile eBPF
bpf.attach_kprobe(...);             // Attach to kernel
bpf.open_perf_buffer(...);          // Setup event stream
while(1) bpf.poll_perf_buffer(...); // Infinite polling

AddProbe() returns immediately. Call getStatus() to verify initialization completed successfully before consuming events.

Cleanup and Shutdown

Proper teardown sequence:

void cleanup() {
    // Detach kprobes
    bpf.detach_kprobe(FN_NAME);
    
    // Cancel threads
    if (tid) pthread_cancel(tid);
    if (netLinkTid) pthread_cancel(netLinkTid);
    
    // Queued events are NOT automatically freed
    // Consumer should drain queue before cleanup
}

Get Started

Core Concepts

Guides

Overview

High-Level Architecture

Core Components

tcpEvent Library

TCP Event Flow

udpEvent Library

Perf Buffer Architecture

Buffer Configuration

Event Flow

Queue Management

Threading Model

Thread Architecture

Thread Lifecycle

Consumer Thread

Data Flow: Kernel to User Space

Event Capture Path

TCP Netlink Enrichment

Process Attribution Mechanism

Inode-to-PID Mapping (TCP)

Reading Command Names

Memory Management

Event Lifecycle

Time Synchronization

Component Communication

Public API

Initialization Sequence

Cleanup and Shutdown

Performance Characteristics

Event Latency

Throughput

Next Steps

eBPF Overview

Event Collection

Get Started

Core Concepts

Guides

​Overview

​High-Level Architecture

​Core Components

​tcpEvent Library

TCP Event Flow

​udpEvent Library

​Perf Buffer Architecture

​Buffer Configuration

​Event Flow

​Queue Management

​Threading Model

​Thread Architecture

​Thread Lifecycle

​Consumer Thread

​Data Flow: Kernel to User Space

​Event Capture Path

​TCP Netlink Enrichment

​Process Attribution Mechanism

​Inode-to-PID Mapping (TCP)

​Reading Command Names

​Memory Management

​Event Lifecycle

​Time Synchronization

​Component Communication

​Public API

​Initialization Sequence

​Cleanup and Shutdown

​Performance Characteristics

Event Latency

Throughput

​Next Steps

eBPF Overview

Event Collection

Overview

High-Level Architecture

Core Components

tcpEvent Library

udpEvent Library

Perf Buffer Architecture

Buffer Configuration

Event Flow

Queue Management

Threading Model

Thread Architecture

Thread Lifecycle

Consumer Thread

Data Flow: Kernel to User Space

Event Capture Path

TCP Netlink Enrichment

Process Attribution Mechanism

Inode-to-PID Mapping (TCP)

Reading Command Names

Memory Management

Event Lifecycle

Time Synchronization

Component Communication

Public API

Initialization Sequence

Cleanup and Shutdown

Performance Characteristics

Next Steps