Upgrading

The mpc chart manages upgrades through standard Helm lifecycle hooks. DB schema migrations run automatically as a pre-upgrade job before any workload pods are replaced.

Before you upgrade

Step 1: Review the release notes

Read the release notes for every chart and application version between your current version and the target version. Pay particular attention to:

• Breaking value changes - fields that have been renamed, removed, or changed type. Helm validates values against values.schema.json; upgrading with stale values will fail with a schema validation error.
• DB migration notes - migrations are additive and run automatically, but irreversible migrations (column drops, type changes) require a maintenance window and cannot be rolled back without restoring a database snapshot.
• Image version requirements - the mpc chart version and the versions.wallet / versions.mpa image versions are coupled. Do not mix chart versions with incompatible image versions.

Step 2: Dry-run the upgrade

Render the chart with your current values against the new version to catch schema errors before touching the live cluster:

helm upgrade mpc oci://iv.sepior.net/charts/mpc \
  --version <new-chart-version> \
  --namespace <tenant-namespace> \
  -f my-instance-values.yaml \
  --dry-run

Compare the rendered output against the running workloads to confirm the expected changes:

helm diff upgrade mpc oci://iv.sepior.net/charts/mpc \
  --version <new-chart-version> \
  --namespace <tenant-namespace> \
  -f my-instance-values.yaml

helm diff is a plugin - install it with helm plugin install https://github.com/databus23/helm-diff.

Step 3: Snapshot stateful data

Before any upgrade that includes a DB migration, take a snapshot of the managed database (RDS, Azure Database for PostgreSQL, or Cloud SQL). This is the only way to roll back a migration that has already run.

# Azure
az postgres server backup show ...

# AWS
aws rds create-db-snapshot \
  --db-instance-identifier <instance-id> \
  --db-snapshot-identifier iv-pre-upgrade-$(date +%Y%m%d)

# GCP
gcloud sql backups create --instance=<instance-name> --project=<project>

Step 4: Mirror new images

If you are upgrading to new application versions, mirror the new images to your container registry before running helm upgrade (see the cloud-specific installation guide for your platform). Use the delivery tags for each image group (IV_VERSION for mothership / wallet-frontend / evm-tracker / nats, POLICY_NODE_VERSION for policy-node, CONFIGMAP_INIT_VERSION for configmap-init). The upgrade will fail if pods cannot pull the new image.

Running the upgrade

helm upgrade mpc oci://iv.sepior.net/charts/mpc \
  --version <new-chart-version> \
  --namespace <tenant-namespace> \
  -f my-instance-values.yaml

What happens during upgrade

1. Pre-upgrade hooks - the db-setup job runs DB migrations against the configured database. It completes before any workload pods are replaced.
2. Rolling update - workload Deployments (wallet, Policy Nodes, wallet-frontend, etc.) roll out with maxUnavailable: 0, maxSurge: 1. A new pod becomes Ready before the old pod is terminated, so the service remains available throughout.
3. StatefulSet update - NATS rolls out one pod at a time. During the rollout, the NATS cluster continues to operate with reduced redundancy.

Expected downtime

Destructive migrations are noted in the release notes. For these, scale the wallet to zero replicas before upgrading, run the migration, then scale back up:

kubectl scale deployment mpc-wallet -n <tenant-namespace> --replicas=0
helm upgrade mpc ... --set wallet.replicaCount=0
# after migration completes:
helm upgrade mpc ... -f my-instance-values.yaml   # restores normal replica count

Verifying the upgrade

After helm upgrade completes, confirm all pods are running the new version:

kubectl get pods -n <tenant-namespace> -o wide

Check the wallet health endpoint:

kubectl port-forward -n <tenant-namespace> svc/mpc-wallet 8080:80 &
curl http://localhost:8080/ready

Check the DB migration job completed successfully:

kubectl get job -n <tenant-namespace>
kubectl logs job/mpc-db-setup -n <tenant-namespace>

Rolling back

If the upgrade introduces a regression, roll back to the previous release:

helm rollback mpc <revision> --namespace <tenant-namespace>

Find the revision number with:

helm history mpc --namespace <tenant-namespace>

🚧

Caution:

helm rollback reverses the Helm release but does not reverse DB migrations. If the migration was additive (new columns), rollback is safe - the old application version will ignore the new columns. If the migration was destructive, you must restore the database snapshot taken before the upgrade.

Updating secrets without a chart upgrade

When secrets in the cloud secret store change (for example, rotating NATS credentials), the running pods do not automatically pick up new values - configmap-init runs only at pod startup. Trigger a rolling restart to reload secrets:

kubectl rollout restart deployment/mpc-wallet -n <tenant-namespace>
kubectl rollout restart deployment/mpc-policy-node-0 -n <tenant-namespace>
kubectl rollout restart deployment/mpc-policy-node-1 -n <tenant-namespace>
kubectl rollout restart deployment/mpc-policy-node-2 -n <tenant-namespace>

🚧

Caution:

EncryptorMasterPassword must never be rotated. It is the root key for database encryption. Changing it corrupts the encrypted data and requires a full key ceremony to recover.