Semantic Entity Model

The semantic model turns flat OpenTelemetry attributes into entities and typed relationships.

Registry layers

The effective registry is a merge of:

1. the pinned OpenTelemetry semantic-conventions model under tools/semconv_codegen/upstream/otel-semconv/v1.43.0/model;
2. project extensions under tools/semconv_codegen/model/extensions.

Extensions may reference upstream attributes and entities, but may not redefine them. The upstream snapshot is build input and is never downloaded at runtime.

Entities

An entity definition has:

• a stable entity name such as service, k8s.pod, or app.endpoint;
• one or more attribute references;
• at least one identifying attribute if a runtime class should be generated;
• optional descriptive metadata such as stability and a brief.

- id: entity.app.endpoint
  type: entity
  name: app.endpoint
  stability: development
  attributes:
    - ref: service.name
      role: identifying
    - ref: service.namespace
      role: identifying
    - ref: http.request.method
      role: identifying
    - ref: http.route
      role: identifying

Java SemanticRegistry constructs a deterministic ID from identifying values:

app.endpoint:checkout-api:shop:POST:%2Fcheckout%2F%7Bcart_id%7D

The generated Python class is immutable, receives that stored ID as element_id, and validates exact registry field types without rebuilding the identity. Missing any identifying key means that Java does not observe the entity. Present but empty, incorrectly typed, or invalid enum values are rejected. Optional attributes enrich an entity but do not change its identity. Array and template attributes retain their registry types; template fields such as k8s.pod.label.<key> are exposed as canonical dotted attributes. Identity fields are limited to strings, integers, booleans, and enums without floating-point members. Floating-point values remain valid descriptive attributes but cannot participate in an entity ID.

ETL hierarchy

The local ETL extension models observed execution topology with hierarchical identity:

Entity	Identifying fields	Descriptive fields
etl.pipeline	etl.pipeline.id	etl.pipeline.name
etl.run	etl.pipeline.id, etl.run.id	etl.run.name
etl.part.run	etl.pipeline.id, etl.run.id, etl.part.run.id	etl.part.name

The registry defines Pipeline -> contains -> Run -> contains -> Part Run. Retries preserve the same logical Part Run identity. A later Pipeline Run uses a new Run ID and therefore creates a new Run and Part Run.

Extraction is additive: Pipeline identity alone creates only a Pipeline; adding Run identity creates the Run and first relationship; complete Part Run identity creates the full hierarchy. ETL executions without a matched interaction remain undiscovered.

Relationships

A relationship names a directed edge between entity types:

- id: relationship.service_exposes_app_endpoint
  type: relationship
  name: exposes
  source_entity: service
  target_entity: app.endpoint
  source_signals: [service_graph, span_metrics]
  evidence_scope: observation

An observation relationship is expanded when both endpoint entities are complete in one client, server, or root observation. An interaction relationship connects entities from the client observation to entities from the server observation and declares the Collector selector that chooses it:

- id: relationship.service_calls_service
  type: relationship
  name: calls
  source_entity: service
  target_entity: service
  source_signals: [service_graph]
  evidence_scope: interaction
  connection_type: unset

Missing, empty, and Collector-provided unset selectors normalize to unset. Unknown selectors are rejected without discarding valid client or server observation topology. Only a relationship explicitly allowed by the registry is emitted, and every expansion suppresses edges whose endpoint IDs are equal.

source_signals declares actual evidence eligibility. Servicegraph datapoints can expand definitions containing service_graph; root discovery datapoints can expand definitions containing span_metrics. For example, executes declares only span_metrics, so servicegraph observations cannot create transactions.

Code generation creates a frozen concrete Pydantic edge class for every relationship definition. For example, relationship.service_calls_service becomes ServiceCallsServiceEdge. Each class declares its relationship and endpoint semantic types, validates endpoint IDs, and computes the same deterministic edge ID used by Flink. Structural edge attributes are preserved without embedding endpoint entities.

From attributes to the live graph

The generation pipeline connects the model to runtime behavior:

1. python -m tools.semconv_codegen validates and merges both registry layers.
2. It builds a strict semantic intermediate representation and commits semantic-entities.schema.json.
3. Pinned datamodel-code-generator turns that JSON Schema into static Pydantic field classes, while the project generator adds semantic IDs, registries, and concrete edge models.
4. The generated Python remains static, importable, and IDE-friendly; no model generation or registry parsing occurs at runtime.
5. The same operation selects scalar attributes used by service-graph relationships.
6. Collector backends include those dimensions in their metrics.
7. Flink interprets client and server dimensions through the generated package.
8. Interaction events carry normalized graph nodes and edges.
9. Generic consumers can display custom types without embedding registry logic.

Template attributes ending in .label, .annotation, or .selector and non-scalar attributes are excluded from Collector dimensions.

Cardinality

Entity identity often requires high-cardinality values such as pod UIDs, routes, or instance IDs. Including them as metric dimensions increases service-graph time-series cardinality.

Treat every identifying attribute as a storage and throughput decision:

• use stable bounded names where possible;
• do not use request IDs, trace IDs, user IDs, or session IDs;
• estimate combinations across client, server, route, namespace, and workload;
• monitor Collector memory and Kafka throughput after adding dimensions.

The generator deliberately follows registry policy. It does not silently remove a dimension because its cardinality appears risky.