Complimentary Gartner® Report! 'A CTO's Guide to Open-Source Software: Answering the Top 10 FAQs.'Read more

Kubernetes Postgres Migration

Migrate from Bolt store to Postgres with Helm

As your BindPlane OP environment grows in size and importance, it can be desirable to migrate BindPlane's backend to Postgres. Using Postgres allows BindPlane to operate in High Availability.

Prerequisites

The following prerequisites must be met before performing the migration.

CLI

warning

Your CLI version should match your control plane version.

The migration requires the BindPlane CLI. Make sure you have a profile configured.

Example profile configuration:

bash
1bindplane profile set "bindplane-single" \
2  --remote-url "http://bindplane.corp.net" \
3  --username "username" \
4  --password "password"
5
6bindplane profile use "bindplane-single"

Credentials

The migration process will involve exporting and importing resources using the BindPlane CLI. The export process will not return sensitive data, such as credentials. You must have access to all credentials used by your resources, such as usernames and passwords, API keys, etc.

Deprecated Resources

When resources are deprecated by BindPlane, they are removed unless they are in use. When migrating from Bolt store to Postgres, any deprecated resources will need to be replaced. Generally, a resource is deprecated and replaced by a newer version if a breaking change is made.

Using the BindPlane CLI and jq, check to see if any of your resource types are deprecated.

bash
1bindplane get source-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'
2bindplane get processor-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'
3bindplane get destination-types -o json | jq 'map({ name: .metadata.name, deprecated: .metadata.deprecated })'

Example output. The parse_severity processor is deprecated. The delete_empty_values processor is not deprecated.

json
1[
2  {
3    "name": "parse_severity",
4    "deprecated": true
5  },
6  {
7    "name": "delete_empty_values",
8    "deprecated": null
9  }
10]

If there are deprecated resource types, it indicates that you have a configuration or resource library component that is deprecated. Remove the component from your configurations and delete it from the resource library, if it is present there.

Secret Key

Note the secret key. You can find your secret key with the bindplane get account command.

Example:

bash
1ID                        	NAME        SECRETKEY
201GR1PHH0K42NW2M0M868XTXDP	my-account  21GR1PHH0KB534010JGB9CXYNX

You will need your secret key during the migration process.

Postgres

Postgres should be pre-provisioned and include a database named bindplane or something similar. There should be a user with full access to the database named bindplane. The BindPlane Helm chart does not deploy or manage the Postgres instance.

Max Connections

Make a note of the maximum connections allowed to Postgres. When configuring BindPlane, you can set the maximum number of connections per BindPlane pod.

Prometheus

During the migration, a new Prometheus instance will be deployed by the Helm chart. This means you will lose existing Prometheus data. Prometheus is used by BindPlane to track agent throughput and data reduction metrics. It is not considered critical data when performing the migration.

If you manage your own Prometheus instance, outside of the Helm chart, you will not lose data.

Migration Steps

During the migration, users mustn't modify BindPlane resources. Doing so will cause the exported resources to become out of sync.

1. Export Resources

warning

Make sure there are no pending configuration changes. When exporting a configuration, the pending version will be exported if changes are present.

Export resources to a directory called resources. Avoid modifying resources after exporting.

bash
1mkdir resources
2bindplane get destination -o yaml --export > resources/destination.yaml
3bindplane get processor -o yaml --export > resources/processor.yaml
4bindplane get source -o yaml --export > resources/source.yaml
5bindplane get configuration -o yaml --export > resources/configuration.yaml

1.1 Sensitive Values

Using a text editor, find all instances of (sensitive) in your resource files. Make a note of each resource that contains a sensitive value. When the resources are imported, you will need to update them with the correct value.

2. Deploy High Availability BindPlane

warning

Make sure the new BindPlane deployment uses the same image tag as your current deployment.

2.1 Helm Values

Copy your Helm values file to a new file, values-ha.yaml or backup the existing values file.

Make the following changes to the HA values file.

  1. Add the replica top level key, and set it to 3
yaml
1replicas: 3
  1. Set backend.type to postgres
yaml
1backend:
2  type: postgres
  1. Configure backend.postgres with the values that are correct for your Postgres instance.
yaml
1backend:
2  postgres:
3    host: postgres.corp.net
4    port: 5432
5    database: bindplane
6    sslmode: disable # TLS is supported with additional configuration.
7    username: bindplane
8    password: bindplane-password
9    maxConnections: 20 # (replicas * maxConnections) should not exceed your Postgres instance's max connection
  1. Update the top-level resources section to reflect multiple pods. Follow the High Availability Sizing documentation for details.
yaml
1resources:
2  requests:
3    memory: 4192Mi
4    cpu: 2000m
5  limits:
6    memory: 4192Mi
  1. If using the Helm-managed Prometheus, ensure its resources and capacity are increased if you expect a high agent count.
yaml
1prometheus:
2  resources:
3    requests:
4      memory: 2048Mi
5      cpu: 1000m
6    limits:
7      memory: 2048Mi
8      cpu: 1000m
9  storage:
10    volumeSize: 120Gi
  1. Enable the NATS Event Bus.
yaml
1eventbus:
2  type: 'nats'
  1. Add the top-level nats configuration. This configuration is not nested under eventbus.
yaml
1nats:
2  resources:
3    requests:
4      memory: 500Mi
5      cpu: 500m
6    limits:
7      memory: 500Mi
  1. Update config.server_url and config.remote_url to use a "staging" hostname. During the migration, BindPlane will need to be accessed over a temporary hostname to allow the existing deployment to continue functioning.
yaml
1config:
2  server_url: http://bindplane-stage.corp.net:3001
3  remote_url: ws://bindplane-stage.corp.net:3001

2.2 Helm Deployment

Deploy the new BindPlane HA environment using Helm. Make sure to use a new name.

warning

Chart version 1.12.0 or newer is required for NATS event bus.

bash
1helm repo add bindplane \
2    https://observiq.github.io/bindplane-op-helm
3helm repo update
4helm search repo
bash
1helm install --values values-ha.yaml "bindplane-ha" bindplane/bindplane

Once deployed, the old and new HA deployments will be present side by side, if deployed to the same namespace. It is expected that some of the pods will have restarted during the initial deployment.

bash
1NAME                                            READY   STATUS    RESTARTS      AGE
2bindplane-0                                     1/1     Running   0             25m
3bindplane-ha-79c77f7f54-4qgb5                   1/1     Running   1 (14s ago)   16s
4bindplane-ha-79c77f7f54-ts62v                   1/1     Running   1 (14s ago)   17s
5bindplane-ha-79c77f7f54-z6p98                   1/1     Running   1 (14s ago)   16s
6bindplane-ha-jobs-5cfcb4dd67-h7k2m              1/1     Running   0             17s
7bindplane-ha-nats-0                             1/1     Running   0             17s
8bindplane-ha-nats-1                             1/1     Running   0             16s
9bindplane-ha-nats-2                             1/1     Running   0             16s
10bindplane-ha-prometheus-0                       1/1     Running   0             17s
11bindplane-ha-transform-agent-849c69fd96-djw4c   1/1     Running   0             17s
12bindplane-prometheus-0                          1/1     Running   0             25m
13bindplane-transform-agent-6546748c6f-jbwbv      1/1     Running   0             25m

2.3 Ensure Access

Make sure you can access the new environment using your existing solution (NGINX, Istio Gateway, etc). This may require you to create an additional ingress configuration.

It is important to keep the current ingress solution routing to your old BindPlane deployment, to prevent downtime. Access to the new HA environment should be handled using a temporary solution, allowing you to import resources.

3. Configure Project

3.1 Initial Sign In

Sign into BindPlane for the first time, and follow the onboarding prompts.

3.2 Configure CLI

Create a new CLI profile for accessing the new environment.

bash
1bindplane profile set "bindplane-ha" \
2  --remote-url "http://bindplane-stage.corp.net" \
3  --username "username" \
4  --password "password"
5
6bindplane profile use "bindplane-ha"

Ensure the CLI profile is working by issuing a get agent command. The command should not return any agents. If it does, you might be using the profile for the old BindPlane instance.

3.3 Update Project Secret Key

Update the new project's secret key with the secret key used by the old BindPlane deployment. This will allow agents to connect to the new BindPlane instance and project.

bash
1bindplane get account -o yaml --export > resources/ha-account.yaml

It will look similar to this.

yaml
1---
2apiVersion: bindplane.observiq.com/v1
3kind: Account
4metadata:
5  id: 01J2HTRNREDRKFXNTYDJD7S4MP
6  displayName: bindplane-ha
7spec:
8  secretKey: 01J2HTRNREX2H93C9TZV0TEF1J
9  alternateSecretKeys: []

Update spec.secretKey in ha-account.yaml to match the secret key from the original environment.

yaml
1apiVersion: bindplane.observiq.com/v1
2kind: Account
3metadata:
4  id: 01J2HTRNREDRKFXNTYDJD7S4MP
5  displayName: bindplane-ha
6spec:
7  # Secret key from old env
8  secretKey: 21GR1PHH0KB534010JGB9CXYNX # Orig: 01J2HTRNREX2H93C9TZV0TEF1J
9  alternateSecretKeys: []

Update the account using the apply command.

bash
1bindplane apply -f resources/ha-account.yaml

Check if it worked: bindplane get account. The output should show the original secret key. You can also check the UI.

4 Import Resources

4.1 Import

Import your resources using the apply command.

bash
1bindplane apply -f resources/destination.yaml
2bindplane apply -f resources/processor.yaml
3bindplane apply -f resources/source.yaml
4bindplane apply -f resources/configuration.yaml

4.2 Update Sensitive Values

Find all components that use sensitive values, and update them. The resources exported from the previous BindPlane instance will not contain sensitive values such as usernames and passwords or API keys.

4.3 Rollout Configurations

Trigger a rollout for all configurations to ensure they will be pushed to agents during the cut-over to the HA instance.

bash
1bindplane get config | grep -v NAME | awk '{print $1}' | xargs -I{} bindplane rollout start {}

5 Cut Over Access

Now that the new BindPlane instance has had its secret key configured and resources imported, it is ready to accept connections from agents.

5.1 Helm Configuration

Update config.server_url and config.remote_url to your primary hostname, the hostname used by the old BindPlane instance. Use Helm to deploy the change.

5.2

Update your ingress solution to point to the new BindPlane HA deployment.

Note that agents will likely remain connected to the old BindPlane instance until one of two things happens:

  1. The agent(s) are restarted
  2. The old BindPlane instance is restarted

Restart the old BindPlane StatefulSet pod to force agents to reconnect, allowing them to connect to new HA system.

6 Verify Functionality

  1. Ensure resource utilization is acceptable
  2. Ensure agents are connecting
  3. Ensure Recent Telemetry is working
  4. Ensure Live Preview is working
  5. Ensure configuration rollout can be triggered and finished

6.1 Abort

If the new system is not working correctly or appears unstable, revert your ingress change to point to the old BindPlane instance. Restart all BindPlane HA pods to force agents to reconnect to the old BindPlane system.

7 Finish and Cleanup

At this point, you have successfully migrated to BindPlane High Availability with NATS and Postgres.

You can safely and optionally remove the old BindPlane instance using helm uninstall. The New BindPlane deployment does not rely on any resources from the old BindPlane instance.