Kernel and system traces

Both recorders are in the Targets view's board menu. Traces are copied off the board with a SHA-256 check and read on the build host or this computer, in the Kernel Trace editor.

ftrace and trace-cmd

State: modelled on QEMU arm64 boards with an Ubuntu 6.8.0 kernel and trace-cmd 3.2, and a 6.8.0 kernel without function tracers (tracefs alone); trace-cmd 3.2.0 and 2.9.6 on this machine and on a build host behind OpenSSH 9.6p1; trace.dat versions 6 and 7; wakeup, interrupt and softirq figures equal to trace-cmd report --profile; overruns and drops reported; the running app. Not run on a real board or a BitBake-built image, with real latencies (QEMU timing inflates them), on Windows, on Wind River Linux, or with a non-root board account.

Add trace-cmd to the image (and CONFIG_FUNCTION_GRAPH_TRACER for function graphs). On the build host install trace-cmd 3.x or set neuralInverse.embeddedLinux.traceCmd.path.

1. Run Record Kernel Trace on Target.... Pick the tool (the image's trace-cmd record, or tracefs alone when the image has no trace-cmd) and what to trace: Scheduling and Interrupts, Choose Events... from the board's available_events, Function Tracer... with function filters, or Function Graph... with a graph depth; every CPU, a process or a program; a duration and ring buffer size.
2. Events and filters are checked on the board before anything is traced (trace-cmd 3.2 silently traces every function when a filter matches nothing).
3. The confirmation names the board and the exact command. Stop Kernel Trace Recording stops early; cancel is in the notification.
4. The file is copied to the build host or this computer (neuralInverse.embeddedLinux.trace.localFolder) and read with trace-cmd report and dump (neuralInverse.embeddedLinux.traceCmd.localPath on Linux), or by the module itself for tracefs on any computer.
5. The editor shows a per-CPU Timeline of tasks, interrupts and softirqs, Events with filters, Latency Hot Spots (wakeup to switch, handler and function graph durations) and Trace Information: kernel, clock, CPUs, trace.dat version and every reported loss. Export Timing Record writes Markdown or CSV.

Open Kernel Trace... opens an earlier trace. Agent tools: four read-only; recording asks first.

LTTng

State: modelled on a QEMU arm64 board with kernel 6.8.0, lttng-tools and lttng-ust 2.13 and lttng-modules 2.14, read with babeltrace2: kernel tracepoints and system calls, lttng-ust events, snapshot mode, discarded events, PID tracking, an unknown event refused, stop and cancel, the copied trace's SHA-256, the running app. Not run on a physical board. On Windows, copying a trace to this computer is refused with the reason; reading on the build host works.

Install lttng-tools, lttng-ust and lttng-modules on the board and set its clock (without it lttng-ust writes a negative clock offset and babeltrace2 refuses the trace). On the build host install babeltrace2 2.0 or later, or set neuralInverse.embeddedLinux.babeltrace2.path (babeltrace2.localPath on this computer).

1. Run Record LTTng Trace on Target... and pick Recording Session (every event while the session runs, for a number of seconds) or Snapshot (overwrite mode, written once at the end).
2. Pick kernel tracepoints, system calls and lttng-ust user-space events, every process or one PID, and buffer sizes. The vpid, vtid and procname contexts are added. Unknown events are refused before anything is traced.
3. The confirmation names the board and the exact commands. Stop LTTng Recording stops; cancel leaves no session behind.
4. The CTF folder is copied with a SHA-256 check of every file and read with babeltrace2. The trace editor shows Resources (per CPU), Control Flow (per thread), CPU Usage per process, System Calls with latencies, user-space events beside kernel events, and a header stating lost and discarded events. The timing record exports as above.

Agent tools: probing and reading are read-only; recording asks first.

Next: AMD PetaLinux.