Kernel Level Data Extraction
Project description
The goal of this project is to create a service that monitors various resources, such as virtual machines (VMs), and collects performance data. This data will be stored in a database and displayed on a dashboard for real-time monitoring and analysis.
Context
Sue is a leading company in Cloud Native technologies and provides solutions to various customers across the Netherlands, including the Dutch Police, Booking.com and more. The company’s Cloud Native specialists are continuously expanding their knowledge in the newest technologies.
With modern operating systems running multiple applications simultaneously, effectively monitoring and managing system resources can become a challenge. As the core intermediary between hardware and applications, the kernel contains essential low-level data on running processes, resource usage and system behavior. Extracting and interpreting the data efficiently requires specialized approaches and tools, such as eBPF.
Results
Our project delivered a configurable Linux-based system monitoring solution leveraging YAML-based configuration files and performance monitoring agents. The key outcome is a modular and adaptable package that empowers users to monitor system metrics (CPU, memory, disk I/O) through simple file modifications, without altering the codebase. Validation involved testing across multiple Linux environments, confirming stability and ease of configuration.
We developed a lightweight agent capable of collecting and reporting metrics at user-defined intervals, with thresholds configurable for alerting. This flexibility allows the tool to scale across varied deployment scenarios, from single machines to large clusters. The solution’s design aligns with TRL 5–6: it demonstrates validated functionality in a relevant environment with a clear pathway toward operational integration and further prototyping.
An additional insight was the advantage of using YAML for configuration: it offers readability, ease of use, and supports complex structures without additional tooling. These characteristics enhance user adoption and reduce configuration errors.
Our findings emphasize the value of customizable monitoring infrastructure, setting the foundation for real-time optimization and predictive performance management.