Who called that MCP tool? Building a toolchain for agent usage tracking
Submitted to NIC 2026 on . Rejected .
- Level
- 300
- Length
- 60 minutes
- Tags
- Operations Data/AI/ML DevOps
Elevator pitch
AI agents are calling your internal tools right now and nobody's tracking any of it, probably. I'll demo the telemetry to fill in the gap: an MCP gateway, OIDC auth, and the audit records that tell you who called what and what came back. Then I'll go over where the tooling isn't there yet and you have to build it yourself.
Abstract, as submitted
People are running AI agents against your internal systems right now. Those agents call tools - your APIs, your ticketing system, your source control - and in most organizations nobody can answer the obvious question afterwards: who called what, with which arguments, and what came back?
A gateway in front of your MCP servers is a good place to instrument. Every tool call through it can spit out a record with the end user's identity, the calling app, the tool name, the arguments, the result and the error, named along the lines of the OpenTelemetry and MCP semantic conventions.
I'll build that live: push an off-the-shelf MCP server, put a gateway in front of it with OIDC auth, point an agent at it, then get two people to ask the agent for things and watch the telemetry tell them apart.
What comes out isn't a trace, though. No trace IDs, no span IDs, no duration, just start and stop events. Getting to a real span waterfall means joining the pairs and minting the IDs yourself. I'll try to fill in the gaps with OpenTelemetry and OTLP where they fit, structured audit logs, Prometheus, and the other OTel backends where they don't. Then we'll look at the stuff that doesn't exist yet anywhere, like retrieval quality, user feedback as telemetry, cost attribution, and trace context that survives the hop from agent to gateway to tool.
Technology shown, live: Model Context Protocol, OpenTelemetry Collector and OTLP, Prometheus, OIDC, Grafana, and a few more.
This is the text that went into the CFP form, kept as submitted - not a later rewrite of it. Talks get retitled and reworked between submission and stage, so what was actually delivered may differ.