Traffic & Routing

This page documents how traffic flows through the platform — from external requests entering the cluster to internal service-to-service communication and async event processing.

Request Event Flow

External Traffic Path

External requests enter the cluster through Istio's ingress gateway, are matched by Gateway API HTTPRoute rules, and are forwarded to backend services with TLS termination at the gateway.

DNS Resolution

Each exposed service gets a hostname derived from the ingress prefix:

Endpoint	Hostname Pattern
OSDU API	`{prefix}.{dns_zone_name}`
Kibana	`{prefix}-kibana.{dns_zone_name}`
Keycloak	`{prefix}-keycloak.{dns_zone_name}`
Airflow	`{prefix}-airflow.{dns_zone_name}`

The ingress prefix (CIMPL_INGRESS_PREFIX) is auto-generated during pre-provision if not set. DNS records are created either manually or via ExternalDNS once its identity and DNS zone access have both been bootstrapped.

TLS Certificates

cert-manager provisions Let's Encrypt certificates for each HTTPRoute hostname. Certificates are stored as Kubernetes Secrets and referenced by the Gateway resource. The HTTP-01 challenge solver uses the Istio Gateway for validation.

See Platform Services — cert-manager for full configuration details.

Gateway API

Modern ingress routing using Kubernetes Gateway API with Istio as the implementation.

Architecture

Component	Resource	Purpose
Gateway	`Gateway` in `aks-istio-ingress`	External Istio ingress with Azure Load Balancer
Routes	`HTTPRoute` per endpoint	Path-based and host-based routing rules
TLS	`Certificate` per hostname	cert-manager provisions Let's Encrypt certificates
Access	`ReferenceGrant` per namespace	Allow cross-namespace secret references

HTTPRoute Pattern

Each external endpoint is defined as an HTTPRoute that matches on hostname and forwards to the backend service:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: osdu-api
  namespace: osdu
spec:
  parentRefs:
    - name: gateway
      namespace: aks-istio-ingress
  hostnames:
    - "myprefix.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/partition/
      backendRefs:
        - name: partition
          port: 80

Public vs Internal Ingress

The enable_public_ingress flag controls whether the gateway gets an external or internal Azure Load Balancer IP:

Public (default): Internet-reachable endpoints with Let's Encrypt TLS
Internal: Accessible only within the VNet; useful for private deployments

Cross-Namespace ReferenceGrants

Gateway API requires explicit ReferenceGrant resources to allow HTTPRoutes in the osdu or platform namespaces to reference the Gateway in aks-istio-ingress. The gateway module creates these grants automatically for each enabled endpoint.

Internal Service Communication

OSDU Service-to-Service (mTLS)

All OSDU services in the osdu namespace communicate over Istio's mTLS mesh:

mTLS mode: STRICT (enforced by PeerAuthentication in the osdu namespace)
Service discovery: Kubernetes DNS (e.g., partition.osdu.svc.cluster.local:80)
K8s Service port: 80 (targetPort 8080 on the pod)

See Security — Istio mTLS for policy details and sidecar opt-outs.

Middleware Access (Cross-Namespace)

OSDU services in the osdu namespace reach middleware in the platform namespace via standard Kubernetes DNS:

Middleware	Endpoint	Port
PostgreSQL (read-write)	`postgresql-rw.platform.svc.cluster.local`	5432
PostgreSQL (read-only)	`postgresql-ro.platform.svc.cluster.local`	5432
Elasticsearch	`elasticsearch-es-http.platform.svc.cluster.local`	9200
Redis	`redis-master.platform.svc.cluster.local`	6379
RabbitMQ	`rabbitmq.platform.svc.cluster.local`	5672
MinIO	`minio.platform.svc.cluster.local`	9000
Keycloak	`keycloak.platform.svc.cluster.local`	8080

Note

Both the platform and osdu namespaces have Istio sidecar injection enabled with STRICT mTLS. Cross-namespace calls between them stay within the mesh. A few pods in the platform namespace (e.g., RabbitMQ) opt out of sidecar injection at the pod level; calls to these pods use application-layer security (password auth).

Async Event Flow

RabbitMQ Messaging

OSDU services use RabbitMQ for asynchronous event processing:

Producer	Exchange	Consumer	Purpose
Storage	`storage`	Indexer	Record create/update triggers index refresh
Various	`notification`	Notification service	Downstream event delivery
Workflow	`workflow`	Airflow	DAG run triggers via message queue

Connection Details

Setting	Value
Host	`rabbitmq.platform.svc.cluster.local`
Port	5672
Credentials	From `rabbitmq-secret`

Data Storage Patterns

OSDU services use three storage backends, each serving a different access pattern:

Backend	Used For	Access Pattern	Endpoint
PostgreSQL	Structured data (OSM)	JDBC per-service database	`postgresql-rw.platform:5432/{db}`
Elasticsearch	Full-text search + analytics	REST API via Indexer/Search	`elasticsearch-es-http.platform:9200`
MinIO	Binary objects (files, datasets)	S3 API	`minio.platform:9000`

PostgreSQL (OSM)

Each OSDU service stores structured data in its own database within the shared CNPG cluster. The OSM pattern uses: id text (record ID), pk bigint IDENTITY (auto-increment), data jsonb NOT NULL (document), plus a GIN index on data for JSON path queries.

Elasticsearch (Search & Analytics)

Indexer maintains Elasticsearch indices for full-text search. When Storage creates or updates records, it publishes events to RabbitMQ. Indexer consumes these events and updates the corresponding Elasticsearch indices. The Search service queries these indices directly.

MinIO (Object Storage)

File and Dataset services store binary objects in MinIO via the S3 API. Objects are organized into buckets per data partition.