Instrumenting custom plugins

[avinashupadhya99]

Hello,

I am using the OpenTelemetry plugin to capture traces from Kong. I have a custom plugin that makes an HTTP request to an external service (Keycloak) for authz. The external service (Keycloak) is also instrumented with OTel and sends the traces to the same OTel collector. However, the trace is not propagated to the external service from Kong through the resty.http call.

As a workaround, I am manually propagating the trace from Kong by setting the traceparent header as follows -

local active_span = kong.tracing.active_span()
local trace_flags = "01"
if not active_span.should_sample then
    trace_flags = "00"
end
local traceparent = "00-" .. to_hex(active_span.trace_id) .. "-" .. to_hex(active_span.span_id) .. "-" .. trace_flags
    kong.log.debug("traceparent " .. traceparent)
    local res, err = client:request_uri(url, {
        method = "GET",
        headers = {
            ["traceparent"] = traceparent,
        },
    })

However, active_span.span_id returns the span ID of the root span and not the span of the plugin. This results in the HTTP call's span being a child of the root span and not the plugin span.
A snippet of how it looks in Honeycomb -

How to get the span of the plugin in the plugin?

[Oyami-Srk]

In which phase did you place that code?
IIUC, only access, rewrite, and header filter get span callback.
@samugi Could you have a look at this if possible?

[avinashupadhya99]

I am doing it in the access phase. As an alternative, I tried creating a new span in the access phase but that was also created as a child span of the root span and not the plugin span.

local active_span = kong.tracing.active_span()
local new_span = kong.tracing.start_span("resty.http.request")

local trace_flags = "01"
if not active_span.should_sample then
    trace_flags = "00"
end

local traceparent = "00-" .. to_hex(new_span.trace_id) .. "-" .. to_hex(new_span.span_id) .. "-" .. trace_flags

kong.log.debug("traceparent " .. traceparent)
local res, err = client:request_uri(url, {
    method = "GET",
    headers = {
        ["traceparent"] = traceparent,
    },
})

if not res then
    new_span:record_error(err)
    new_span:finish()
end

new_span:finish()

[samugi]

Hi @avinashupadhya99, what version of Kong did you test this on? I believe the issue was fixed here and after that change, active_span should be the plugin span, let me know if that isn't the case. Thank you.

[avinashupadhya99]

I'm using 3.7.1. Just to confirm, this was fixed in 3.8.0 right?

[samugi]

correct

[avinashupadhya99]

Thank you, I will try it out and let you know.

[avinashupadhya99]

It works in 3.8.0. Thank you @samugi and @Oyami-Srk for your help. Closing this discussion.