Tracing AI SDK C++ with Neural Inverse

This guide shows how to use the built-in Neural Inverse tracing module in AI SDK C++ to capture traces for ai::Client::generate_text calls, including the LLM round-trip and every tool invocation.

What is AI SDK C++? AI SDK C++ is a modern C++20 toolkit, maintained by ClickHouse, for building AI-powered applications with providers like OpenAI and Anthropic. It exposes a unified ai::Client API for text generation, streaming, tools, and multi-step agent loops.

What is Neural Inverse? Neural Inverse is an open-source LLM engineering platform that provides tracing, evaluation, prompt management, and metrics to debug and improve LLM applications.

Trace shape

The ai::langfuse component emits one trace per logical operation with the following structure:

1 trace – caller sets the input, output, metadata, and tags.
1 generation observation – model, model parameters, input messages, output text, aggregated usage, and finish reason.
1 span per tool call – parented to the generation, with tool arguments as input and the result (or error) as output.

Install AI SDK C++ with the Neural Inverse component

Add AI SDK C++ to your CMake project and link against the ai::langfuse target. The Neural Inverse component is built alongside the core, OpenAI, and Anthropic targets:

find_package(ai-sdk-cpp CONFIG REQUIRED)

target_link_libraries(my_app
  PRIVATE
    ai::sdk        # umbrella target (core + providers + langfuse)
    ai::langfuse   # or link this directly if you prefer fine-grained deps
)

The build defines AI_SDK_HAS_LANGFUSE=1 so you can guard tracing code behind a compile-time check if needed. See the AI SDK C++ README for full installation instructions.

Configure Neural Inverse credentials

Set the Neural Inverse credentials for the example. You can obtain these keys from your Neural Inverse Cloud project settings or from a self-hosted Neural Inverse instance.

# LLM provider (required by the example)
export OPENAI_API_KEY="sk-..."

# Neural Inverse credentials
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."

# Neural Inverse host (optional, defaults to https://cloud.langfuse.com)
# 🇪🇺 EU cloud:  https://cloud.langfuse.com
# 🇺🇸 US cloud:  https://us.cloud.langfuse.com
# 🇯🇵 Japan:     https://jp.cloud.langfuse.com
# ⚕️ HIPAA:     https://hipaa.cloud.langfuse.com
# Self-host:    http://localhost:3000
export LANGFUSE_HOST="https://cloud.langfuse.com"

Instrument a generate_text call

Construct a Tracer once per process and call tracer.start_trace(...) for each logical operation. Pass the trace to ai::langfuse::generate_text, which wraps ai::Client::generate_text and records the LLM call and tool invocations automatically.

#include <ai/langfuse.h>
#include <ai/openai.h>
#include <ai/tools.h>

#include <cstdlib>
#include <iostream>

int main() {
  // 1. Create a Tracer once and reuse it for all traces.
  ai::langfuse::Tracer tracer({
      .host = std::getenv("LANGFUSE_HOST")
                ? std::getenv("LANGFUSE_HOST")
                : "https://cloud.langfuse.com",
      .public_key = std::getenv("LANGFUSE_PUBLIC_KEY"),
      .secret_key = std::getenv("LANGFUSE_SECRET_KEY"),
      .environment = "ai-sdk-cpp-example",
  });
  if (!tracer.is_valid()) {
    std::cerr << "Neural Inverse tracer not configured.\n";
    return 1;
  }

  // 2. Configure the LLM client and tools as usual.
  auto client = ai::openai::create_client();

  ai::GenerateOptions options;
  options.model = ai::openai::models::kGpt4oMini;
  options.system =
      "You are a concise assistant. Use the available tools when helpful.";
  options.prompt = "Look up alice and tell me the weather where she lives.";
  options.max_steps = 4;
  options.temperature = 0.0;
  // options.tools = ... ;  // register your ai::ToolSet here

  // 3. Start a trace and attach input/metadata.
  auto trace = tracer.start_trace("langfuse_tracing_example");
  trace->set_input(options.prompt);
  trace->set_metadata({{"example", "langfuse_tracing"}, {"sdk", "ai-sdk-cpp"}});

  // 4. Run generate_text via the Neural Inverse wrapper.
  auto result = ai::langfuse::generate_text(client, std::move(options), *trace);

  if (result) {
    std::cout << "Output: " << result.text << "\n";
    trace->set_output(result.text);
  } else {
    trace->set_output(ai::JsonValue{{"error", result.error_message()}});
  }

  // 5. Flush the batch synchronously to Neural Inverse.
  trace->end();
  return result ? 0 : 2;
}

The wrapper hooks into the existing on_tool_call_start / on_tool_call_finish callbacks on GenerateOptions and chains any callbacks you have already installed, so tracing composes cleanly with your own instrumentation.

Tracer configuration

ai::langfuse::Config accepts the following options:

Field	Description
`host`	Base URL of your Neural Inverse instance. Defaults to `https://cloud.langfuse.com`.
`public_key`	Public API key (`pk-lf-...`).
`secret_key`	Secret API key (`sk-lf-...`).
`release`	Optional release identifier attached to all traces (e.g. a commit SHA).
`environment`	Environment tag attached to all traces. Defaults to `default`.
`connection_timeout_sec`	HTTP connection timeout for the ingestion request. Defaults to 10 seconds.
`read_timeout_sec`	HTTP read timeout for the ingestion request. Defaults to 30 seconds.
`error_policy`	`kStrict` surfaces HTTP/JSON failures via `Trace::end()`; `kBestEffort` swallows them.

Trace API

A Trace accumulates events in memory and POSTs them to /api/public/ingestion when end() is called.

auto trace = tracer.start_trace("sql-generation");

trace->set_input(user_prompt);
trace->set_user_id("user-123");
trace->set_session_id("session-abc");
trace->set_metadata({{"feature", "sql-assistant"}});
trace->add_tag("beta");

auto result = ai::langfuse::generate_text(client, std::move(options), *trace);

if (result) {
  trace->set_output(result.text);
} else {
  trace->set_output(ai::JsonValue{{"error", result.error_message()}});
}
trace->end();  // idempotent, synchronous flush

Trace methods are thread-safe, so callbacks fired from generate_text's worker threads can record into the same trace concurrently.

Notes and limitations

Trace::end() flushes synchronously; there is no background worker thread (yet).
Intermediate on_step_finish events are not emitted as separate generations — step text and usage roll up into the single parent generation observation.
The integration ships with AI SDK C++ as an optional CMake target. If you do not link ai::langfuse, your binary stays free of the additional HTTP and JSON code paths.

Learn more

Was this page helpful?

On this page