Managing settings and secrets for external integrations

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. Context

1.1. Information exchange with external systems

Interaction in the context of business scenarios was implemented with the following external systems:

Information exchange was implemented with the following registries (for details, see registry-develop:registry-admin/external-integration/api-call/trembita/overview.adoc):

  • Unified State Register (USR)

  • State Register of Civil Status Acts (SRCSA)

  • Unified Information Database of Internally Displaced Persons (UIDIDP)

1.2. Integration scenarios

Component Level External system/registry Configuration Integration method Scenario

dso-citizen-authenticator

Platform

USR

Integrations and secrets are configured in the registry regulation

Built-in

Authenticating business representatives in the electronic cabinet of a citizen (checking for an activated entry in the registry)

notification-service

Registry

Diia

Integration and secret are configured through the control-plane interface

Built-in

Sending push-notifications to citizens

bpms

Registry

USR

SRCSA

UIDIDP

Integrations and secrets are configured in the registry regulation

Standard SOAP connectors for integration

Information exchange with the registries through Trembita as part of business processes

Diia

Integrations are configured in the registry regulation, and secret is created manually

Universal REST connector for integration

Information exchange as part of business processes

<External system>

Integrations are configured in the registry regulation, and secrets are created manually

Universal REST connector for integration

Information exchange as part of business processes

2. Current technical design

registry-ext-systems-secrets-baseline

2.1. Configuring the citizen authenticator

To authenticate citizens, the system fetches user data from USR. Integration configuration and secret are stored at the regulation level and are used by the Regulation publishing pipeline to configure the registry’s dso-citizen-authenticator:

dso-citizen-authenticator

2.2. Configuring external integrations at the regulation level

Currently, integrations with registries through Trembita are implemented via standard SOAP connectors for integration.

For details, see SOAP integration connectors with other registries

REST-based integrations with external systems are implemented using the Universal REST connector, which supports the following authorization methods:

  • BASIC (username + password)

  • PARTNER_TOKEN (partner_token + Bearer token)

For details, see Integration with external services using the REST connector
registry-gerrit:<registry-regulation>.git/bp-trembita/configuration.yml
trembita-exchange-gateway:
  registries:
    edr-registry:
      user-id: 'DDM'
      protocol-version: '4.0'
      trembita-url: 'trembita.url/mockEDRService'
      authorization-token: 'token'
      client:
        x-road-instance: 'SEVDEIR-TEST'
        member-class: 'GOV'
        member-code: '43395033'
        subsystem-code: 'IDGOV_TEST_01'
      service:
        x-road-instance: 'SEVDEIR-TEST'
        member-class: 'GOV'
        member-code: '00015622'
        subsystem-code: '2_MJU_EDR_prod'
external-systems:
  diia:
    url: http://api2.diia.gov.ua
    methods:
      get-damaged-property:
        path: /api/v1/public-service/damaged-property/filtered
        method: GET
    auth:
      type: PARTNER_TOKEN
      secret-name: diia-partner-token
      partner-token-auth-url: https://api2t.diia.gov.ua/api/v1/auth/partner
      token-json-path: $.token
  httpbin:
    url: http://httpbin.org/
    methods:
      get:
        path: /get
        method: GET
    auth:
      type: BASIC
      secret-name: httpbin-basic-authentication

2.3. Disadvantages of the current implementation

  • Environment-specific integration settings are defined at the regulation level, which prohibits promoting the regulation between registry instances (such as locations and secrets of external systems).

  • The secrets for accessing external systems are defined at the regulation level.

  • Regulation administrator needs to handle secrets rotation.

  • Registry administrator needs to manually create OpenShift secrets for external systems.

  • Network policies must be configured manually (this includes creating Istio Service Entry for external systems).

  • Trembita client settings for the registry are duplicated at the regulation level.

3. Target technical design

3.1. General provisions

  • Registry regulation should not contain environment- or instance-specific settings.

  • Registry regulation should not contain confidential data in any form.

  • External integration parameters are stored centrally and not duplicated.

  • Integrating the registry with additional external systems does not require configuring network policies manually.

  • Secrets with access parameters to external systems are stored in a secure storage of the HashiCorp Vault secrets management service.

  • Registry administrator and security administrator determine the validity of registry’s interactions with external systems.

  • Registry administrator configures integrations with external systems (including integration protocol, locations, authentication protocol, and secrets) at the registry instance level.

  • Registry administrator handles rotation of secrets with access parameters to external systems.

  • Regulation administrator performs minimal initial configuration at the regulation level to use external integrations in business processes (BPs).

  • Integration between registries through Trembita is implemented as a catalogue of typical extensions-connectors to registries and does not require additional configuration at the regulation level.

  • Integration with third-party systems requires additional configuration at the regulation level in the form of a list of operations and their types used by the registry through a typical BP extension called Universal REST connector.

  • Control Plane Console and External Secrets Operator access the HashiCorp Vault secrets management service using a dedicated service account.

  • Every service account used to access HashiCorp Vault must have a configured policy with a minimum set of Capabilities to perform their tasks (using the principle of least privilege).

3.2. Solution technical design

Changes between HashiCorp Vault secrets and registry’s Secret resources are synchronized via External Secrets Operator.

Implementing this design requires making corresponding changes to the settings and usage of the notifications channels configuration with Diia within the notifications subsystem architecture/registry/operational/notifications/notifications-channels-configuration.adoc#_налаштування_каналу_звязку_для_відправки_push_повідомлень_у_мобільний_додаток_дія.
registry-ext-secrets-operator

  • The registry administrator creates and edits the registry configuration and enters the settings of the client registry into SEG Trembita through the control-plane-console. This results in:

    • Saving the trembita.consumer configuration entry to control-plane-gerrit:<registry>.git/deployment-templates/values.yaml

    • Initializing the platform-jenkins pipeline and applying the corresponding Helm chart to the registry namespace using the settings obtained from the Git repository.

  • The registry administrator creates and edits the registry configuration and defines the Diia integration properties through the control-plane-console. This results in:

    • Saving the secret and metadata to user-management:hashicorp-vault using the registry-kv/registry/<registry/>external-systems/diia path depending on the selected authentication method (AUTH_TOKEN+BEARER).

    • Saving the external-systems.diia configuration entry and the vault:-link for external Vault-secret to control-plane-gerrit:<registry>.git/deployment-templates/values.yaml

    • Initializing the platform-jenkins pipeline and applying the corresponding Helm chart to the registry namespace using the settings obtained from the Git repository.

    • Creating the ConfigMap resource diia-configuration in the registry namespace to be used by the bpms and ddm-notification-service services.

    • Creating the Istio ServiceEntry resource to give the bpms and ddm-notification-service services access to the external system.

    • External Secrets Operator creating the Secret resource diia-secret after processing the ExternalSecret resource diia-external-secret and receiving data from user-management:hashicorp-vault to be used by the bpms and ddm-notification-service services.

3.3. Configuring external integrations of the registry through the Platform Management Center

Connecting a registry to the information exchange requires specifying the location of SEG Trembita that serves as the only means of integration with other registries. We need to consider the possibility of defining it globally instead of duplicating it across each registry.

Currently, duplicating enables us to define separate mock services for the registries, but this approach needs to change going forward.

Currently, when making changes through the control-plane-console, the system automatically creates a merge request in Gerrit and integrates it into the target registry’s configuration repository: <registry>.git.
control-plane-gerrit:<registry>.git/deployment-templates/values.yaml
trembita:
# External registries used through Trembita / business processes specific integration connectors - can be updated & can't be removed by "control-plane" administrator
  registries:
    edr-registry:
      user-id: "DDM"
      protocol-version: "4.0"
      url: "https://trembita.mdtu-ddm.projects.epam.com"
      type: "platform" # non-removable record + secret metadata
      protocol: "SOAP"
      client:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "43395033"
        subsystem-code: "IDGOV_TEST_01"
      service:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "00015622"
        subsystem-code: "2_MJU_EDR_prod"
      auth:
        type: "AUTH_TOKEN"
        secret: "vault:registry-kv/registry/<registry>/trembita-registries/<trembita-registry-name>"
    dracs-registry:
      user-id: "DDM"
      protocol-version: "4.0"
      url: "https://trembita.mdtu-ddm.projects.epam.com"
      type: "platform" # non-removable record + secret metadata
      protocol: "SOAP"
      client:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "43395033"
        subsystem-code: "IDGOV_TEST_01"
      service:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "00015622"
        subsystem-code: "2_MJU_EDR_prod"
    idp-exchange-service-registry:
      user-id: "DDM"
      protocol-version: "4.0"
      url: "https://trembita.mdtu-ddm.projects.epam.com"
      type: "platform" # non-removable record + secret metadata
      protocol: "SOAP"
      client:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "43395033"
        subsystem-code: "IDGOV_TEST_01"
      service:
        x-road-instance: "SEVDEIR-TEST"
        member-class: "GOV"
        member-code: "00015622"
        subsystem-code: "2_MJU_EDR_prod"
external-systems:
# External system used both by registry services and business processes - can be updated & can't be removed by "control-plane" administrator
  diia:
    url: "https://api2t.diia.gov.ua"
    protocol: "REST"
    type: "platform" # non-removable record + secret metadata
    auth:
      type: "AUTH_TOKEN+BEARER"
      auth-url: "https://api2t-auth.diia.gov.ua/api/v1/auth/partner" # can be used both as an absolute url to external auth server or relative path to external system base url ('/api/v1/auth/partner')
      access-token-json-path: "$.token"
      secret: "vault:registry-kv/registry/<registry>/external-systems/<ext-system-name>"
# Example external systems added for particular registry and explicitly "used" on regulation level - can be added/updated/removed if necessary by "control-plane" administrator
  http-bin:
    url: "http://httpbin.org/"
    protocol: "REST"
    type: "registry" # removable record + secret metadata
    auth:
      type: "BASIC"
      secret: "vault:registry-kv/registry/<registry>/external-systems/<ext-system-name>"
  secured-service:
    url: "http://secured-service.org/"
    protocol: "REST"
    type: "registry" # removable record + secret metadata
    auth:
      type: "BEARER"
      secret: "vault:registry-kv/registry/<registry>/external-systems/<ext-system-name>"

For each record containing integration settings for external systems, an Istio Service Entry resource must be created automatically to allow the exchange according to the design.

3.4. Configuring external integrations at the regulation level

registry-gerrit:<registry-regulation>.git/bp-trembita/configuration.yml
# reusing external system names configured on registry level
external-systems:
  diia:
    operations:
      get-damaged-property:
        resource-path: "/api/v1/public-service/damaged-property/filtered"
        method: "GET"
      create-distribution:
        resource-path: "/api/v1/notification/distribution/push"
        method: "POST"
  http-bin:
    operations:
      get-operation:
        resource-path: "/get"
        method: "GET"

3.5. Creating ConfigMap resources when publishing regulation changes

ConfigMap: "external-systems-endpoint-configuration"
kind: ConfigMap
apiVersion: v1
metadata:
  name: external-systems-endpoint-configuration
  namespace: <registry-namespace>
data:
  external-systems-endpoint-configuration.yml: |
    external-systems:
      diia:
        operations:
          get-damaged-property:
            resource-path: "/api/v1/public-service/damaged-property/filtered"
            method: "GET"
          create-distribution:
            resource-path: "/api/v1/notification/distribution/push"
            method: "POST"
      http-bin:
        operations:
          get-operation:
            resource-path: "/get"
            method: "GET"

3.6. Creating ConfigMap resources when applying changes to registry settings

ConfigMap: "trembita-registries-configuration"
kind: ConfigMap
apiVersion: v1
metadata:
  name: trembita-registries-configuration
  namespace: <registry-namespace>
data:
  trembita-registries-configuration.yml: |
    trembita:
     registries:
        edr-registry:
          user-id: "DDM"
          protocol-version: "4.0"
          url: "https://trembita.mdtu-ddm.projects.epam.com"
          protocol: "SOAP"
          client:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "43395033"
            subsystem-code: "IDGOV_TEST_01"
          service:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "00015622"
            subsystem-code: "2_MJU_EDR_prod"
          auth:
            type: "AUTH_TOKEN"
        dracs-registry:
          user-id: "DDM"
          protocol-version: "4.0"
          url: "https://trembita.mdtu-ddm.projects.epam.com"
          protocol: "SOAP"
          client:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "43395033"
            subsystem-code: "IDGOV_TEST_01"
          service:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "00015622"
            subsystem-code: "2_MJU_EDR_prod"
        idp-exchange-service-registry:
          user-id: "DDM"
          protocol-version: "4.0"
          url: "https://trembita.mdtu-ddm.projects.epam.com"
          protocol: "SOAP"
          client:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "43395033"
            subsystem-code: "IDGOV_TEST_01"
          service:
            x-road-instance: "SEVDEIR-TEST"
            member-class: "GOV"
            member-code: "00015622"
            subsystem-code: "2_MJU_EDR_prod"
ConfigMap: "external-systems-configuration"
kind: ConfigMap
apiVersion: v1
metadata:
  name: external-systems-configuration
  namespace: <registry-namespace>
data:
  external-systems-configuration.yml: |
    external-systems:
      http-bin:
        url: "http://httpbin.org/"
        protocol: "REST"
        auth:
          type: "BASIC"
      secured-service:
        url: "http://secured-service.org/"
        protocol: "REST"
        auth:
          type: "BEARER"
ConfigMap: "diia-configuration"
kind: ConfigMap
apiVersion: v1
metadata:
  name: diia-configuration
  namespace: <registry-namespace>
data:
  diia-configuration.yml: |
    external-systems:
      diia:
        url: "https://api2t.diia.gov.ua"
        protocol: "REST"
        auth:
          type: "AUTH_TOKEN+BEARER"
          auth-url: "https://api2t-auth.diia.gov.ua/api/v1/auth/partner"
          access-token-json-path: "$.token"

3.7. Creating ExternalSecret resources when applying changes to registry settings

ExternalSecret: "trembita-registries-external-secrets"
kind: ExternalSecret
apiVersion: external-secrets.io/v1beta1
metadata:
  name: trembita-registries-external-secrets
  namespace: <registry-namespace>
spec:
  refreshInterval: "10s"
  secretStoreRef:
    name: user-management:hashicorp-vault
    kind: SecretStore
  target:
    name: trembita-registries-secrets
  dataFrom:
  - extract:
      key: "registry/<registry>/trembita-registries"
ExternalSecret: "external-systems-external-secrets"
kind: ExternalSecret
apiVersion: external-secrets.io/v1beta1
metadata:
  name: external-systems-external-secrets
  namespace: <registry-namespace>
spec:
  refreshInterval: "10s"
  secretStoreRef:
    name: user-management:hashicorp-vault
    kind: SecretStore
  target:
    name: external-systems-secrets
  dataFrom:
  - extract:
      key: "registry/<registry>/external-systems"
ExternalSecret: "diia-external-secret"
kind: ExternalSecret
apiVersion: external-secrets.io/v1beta1
metadata:
  name: diia-external-secret
  namespace: <registry-namespace>
spec:
  refreshInterval: "10s"
  secretStoreRef:
    name: user-management:hashicorp-vault
    kind: SecretStore
  target:
    name: diia-secret
data:
  - secretKey: "external-systems.diia.auth.secret.token"
    remoteRef:
      key: "registry/<registry>/external-systems"
      property: "external-systems.diia.auth.secret.token"

3.8. Applying changes to Kubernetes Secret resources via External Secrets Operator

External Secrets Operator supports creating a unified Secret resource based on N secret records from HashiCorp Vault with transformation options.
Secret: "trembita-registries-secrets"
kind: Secret
apiVersion: v1
metadata:
  name: trembita-registries-secrets
  namespace: <registry-namespace>
data:
  trembita.registries.<registry-name-1>.auth.secret.token: "<token>"
  trembita.registries.<registry-name-2>.auth.secret.token: "<token>"
  trembita.registries.<registry-name-3>.auth.secret.token: "<token>"
Secret: "external-systems-secrets"
kind: Secret
apiVersion: v1
metadata:
  name: external-systems-secrets
  namespace: <registry-namespace>
data:
  external-systems.<external-system-name-1>.auth.secret.username: "<username>"
  external-systems.<external-system-name-1>.auth.secret.password: "<password>"
  external-systems.<external-system-name-2>.auth.secret.token: "<token>"
  external-systems.diia.auth.secret.token: "<token>"
Secret: "diia-secret"
kind: Secret
apiVersion: v1
metadata:
  name: diia-secret
  namespace: <registry-namespace>
data:
  external-systems.diia.auth.secret.token: "<token>"

3.9. Mounting Secret resources onto the file system

Deployment: "bpms"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bpms
spec:
  template:
      containers:
        - name: bpms
          volumeMounts:
            - name: bpms-trembita-registries-secrets
              mountPath: /app/secrets/trembita-registries
            - name: bpms-external-systems-secrets
              mountPath: /app/secrets/external-systems
            - name: bpms-diia-secret
              mountPath: /app/secrets/diia
      volumes:
        - name: bpms-trembita-registries-secrets
          secret:
            secretName: trembita-registries-secrets
        - name: bpms-external-systems-secrets
          secret:
            secretName: external-systems-secrets
        - name: bpms-diia-secret
          secret:
            secretName: diia-secret
File system
Figure 1. File system

3.10. Types of supported authentication protocols used for integrations and secret storage in HashiCorp Vault

When saving secrets to user-management:hashicorp-vault, adding metadata related to the integration record type is required to enable further secrets filtering:

  • type: platform

  • type: registry

3.10.1. Integrations with other registries through Trembita

  • NO_AUTH — Interaction with the registry through SEG Trembita does not require additional authorization.

  • AUTH_TOKEN — Interaction with the registry through SEG Trembita requires additional authorization using an authorization token.

Secrets used to interact with the registries are stored in HashiCorp Vault (user-management:hashicorp-vault) using the path generated according to the following convention:

registry-kv/registry/<registry>/trembita-registries/<trembita-registry-name>

Where:

  • <registry> is the service name of the registry

  • <trembita-registry-name> is the service name of the registry for which the integration is configured through SEG Trembita

An example of storing the "AUTH_TOKEN" secret in HashiCorp Vault: "registry-kv/registry/<registry>/trembita-registries/<trembita-registry-name>"
{
  "trembita.registries.<registry-name>.auth.secret.token": "<authorization-token>"
}

3.10.2. Integrations with other systems

  • NO_AUTH — Interaction with an external system does not require authorization.

  • BASIC — Interaction with an external system requires standard authentication with a username and password.

  • AUTH_TOKEN — Interaction with an external system requires authorization with an authorization token.

  • AUTH_TOKEN+BEARER — Interaction with an external system requires two-step authorization, using the authorization token to obtain an access token.

  • BEARER — Interaction with an external system requires authorization with an authorization token.

Secrets used to interact with external systems are stored in HashiCorp Vault (user-management:hashicorp-vault) using the path generated according to the following convention:

registry-kv/registry/<registry/>external-systems/<ext-system-name>

Where:

  • <registry> is the service name of the registry

  • <ext-system-name> is the service name of the system for which the integration is configured

An example of storing a "BASIC" secret in HashiCorp Vault: registry-kv/registry/<registry/>external-systems/<ext-system-name>
{
  "external-systems.<external-system-name>.auth.secret.username": "<username>",
  "external-systems.<external-system-name>.auth.secret.password": "<password>"
}
An example of storing the "BEARER" | "AUTH_TOKEN" | "AUTH_TOKEN+BEARER" secrets in HashiCorp Vault: registry-kv/registry/<registry>/external-systems/<ext-system-name>
{
  "external-systems.<external-system-name>.auth.secret.token": "<authorization-token>"
}

3.10.3. Service accounts for accessing HashiCorp Vault

Every component accessing Vault must use a dedicated OpenShift service account. Service accounts created in HashiCorp Vault must have the Kubernetes Auth Method type and must be created during the initial HashiCorp Vault setup by running script-init ConfigMap.

Component

Service account name

Bound namespaces

Capabilities

External Secrets Operator

external-secrets-operator

Registry namespace

["read"]

Admin console

control-plane-console

control-plane

["create", "update"]
An example of HashiCorp Vault Capability Policy
{
      "policy": "path \"registry-kv/registry/<registry/>external-systems/\" \"{ capabilities = [ \"read\" ]}\""}
}
An example of binding an OpenShift service account in HashiCorp Vault
{
      "bound_service_account_names": ["control-plane-console"],
      "bound_service_account_namespaces": "ns",
      "policies": ["policy-name"],
      "ttl": "1h"
}

4. Regulation modeling

4.1. Changes to the USR connectors for integration

We need to switch to using the environment variable trembita.registries.edr-registry.auth.secret.token, which is based on the "trembita-registries-secrets" secret, to receive the authorization token in these typical extensions:

  • com.epam.digital.data.platform.bpms.extension.delegate.connector.registry.edr.SearchSubjectsEdrRegistryConnectorDelegate

  • com.epam.digital.data.platform.bpms.extension.delegate.connector.registry.edr.SubjectDetailEdrRegistryConnectorDelegate

4.2. Changes to the Universal REST connector

For an external system defined at the REST connector level, authorization type is identified by the environment variable external-systems.<ext-system-name>.auth-type set by the registry administrator ("NO_AUTH" | "BASIC" | "BEARER" | "AUTH_TOKEN+BEARER"). Depending on the type, the credentials required to authorize the request are obtained from the "external-systems-secrets" secret:

  • com.epam.digital.data.platform.bpms.extension.delegate.connector.rest.ExternalSystemConnectorDelegate

5. Managing registry settings

5.1. Interfaces for registry external integrations management

registry-integrations-management
Figure 2. Registry’s external integrations management
trembita-registry-integration-configuration
Figure 3. Managing interactions with a registry through Trembita
external-system-integration-configuration
Figure 4. Managing interactions with an external system

6. Security

6.1. Business data

Data category Description Confidentiality Integrity Availability

Technical data that contains information with restricted access

System settings, configs, and parameters that contain information with restricted access, modifying which can negatively affect system attributes

Medium

High

High

Technical data that contains service information

System settings, configs, and parameters that constitute service information

High

High

High

6.2. Simplified threat model

ext sec TM

6.3. Security risk mitigation and compliance

Risk Security controls Implementation Priority

Vault data compromise through a root token. Currently, a root token that has access to everything, including vault unseal, is used as the primary token by all services.

  • Create service accounts and differentiate access to Vault.

  • Set up RBAC for accessing secrets containing the root token.

Risk eliminated

Critical

Compromise of external integrations account data due to incorrect configuration of the error-handling system. Mounting OpenShift secrets onto the target service as an environment variable may lead to their reveal if software returns all information about operating environment when an error occurs.

  • Mount secrets for target services as files.

  • Set up a general error-handling mechanism.

Considered in the initial design

Critical

Vault data compromise through the secret operator’s access token. External Secrets Operator creates its own custom resources which may contain account information with access to Vault.

  • Create a dedicated service account to integrate with External Secrets Operator using the principle of least privilege.

  • Set up RBAC for accessing External Secrets Operator custom resource definition (CRD).

Not considered in the initial design

High

Lack of accountability. Absence of an audit log and information about accessing the Vault secrets.

Set up the logging and audit system for Vault.

Not considered in the initial design

High

Backdoor risk in the External Secrets Operator component.

Prohibit any communications between External Secrets Operator and external resources at the network policies level and allow communication with services involved according to business logic.

Not considered in the initial design

High

  • Unauthorized data access in the data center.

  • Incorrect decommissioning of data center components.

  • Unauthorized access to backups.

Set up encryption for partitions that use Vault.

Not considered in the initial design

Medium

Risk of detection evasion and anchoring in the system due to the absence of secrets rotation.

Set up the system/process of secrets rotation.

Not considered in the initial design

Medium