`
In this section:
Modified: for 12.1.1
Caution
Use caution before upgrading any fields or parameters in deployed statefulset pods. The following parameters may be updated:
replicasordinalstemplateupdateStrategypersistentVolumeClaimRetentionPolicyminReadySeconds
If new parameters are updated other than those listed, a Helm upgrade failure occurs with the following message:
Error: UPGRADE FAILED: cannot patch "dssbc-cache" with kind StatefulSet: StatefulSet.apps "dssbc-cache" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Kubernetes Rolling Update and Rollback
Pre-Upgrade Checks
- Run commands on the OAM that checks the overall health of the SBC CNe.
Check if all the pods of the SBC CNe deployment are running correctly.
Check the overall health of the SBC CNe deployment.
SBC CNe - Healthadmin@vsbc1> show table cnfGlobal cnfHealth Pod CONTAINER CONTAINER NAME NAME STATUS ---------------------------- ALL ALL Healthy Reference page: https://wiki.rbbn.com/display/NPI/CNF+pods+life+cycle+CLI+commands
- A new SBC CNe helm chart is available with the required changes.
For example:- an image tag of a specific container or multiple containers set to the correct version such as the upgrade version
- new parameters are added to the deployment/configmap
- In the new SBC CNe helm chart confirm the rolling update strategy of individual deployments/pods is set properly.
Rolling Update Strategy in helm chart
The rolling update strategy of the different pods are captured in the deployment/statefulset helm charts. Few sample helm charts with the rolling update strategy
SC Deployment:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
Redis Cache Statefulset:
There is no explicit mention about the Rolling Update strategy for Statefulset as that's the default setting. Details about Rolling Update in statefulset is captured under https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets.
Upgrade
Run the "helm upgrade" command.
#helm upgrade <deployment_name> rbbn-core-cnf --values rbbn-core-cnf/values.yaml)
Check if the SBC CNe helm upgrade is executed successfully.
[prrao@cli-blr-1 ~]$ helm history vgsbc REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Tue Mar 5 12:03:13 2024 superseded rbbn-core-cnf-12.1.1-135 12.1.1-135 Install complete 2 Tue Mar 5 16:31:14 2024 superseded rbbn-core-cnf-12.1.1-148 12.1.1-148 Upgrade complete
Post-Upgrade Checks
- Check if the pods are successfully upgraded.
Check the health of the CNe by running the cnfHealth CLI command from OAM.
admin@vsbc1> show table cnfGlobal cnfHealth POD CONTAINER CONTAINER NAME NAME STATUS ---------------------------- ALL ALL Healthy
- Verify the changes are applied.
If the pod upgrade fails or if a call failure occurs post-upgrade, you can roll back to the last stable version by running "helm rollback".
[prrao@cli-blr-1 ~]$ helm history vgsbc REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Tue Mar 5 12:03:13 2024 superseded rbbn-core-cnf-12.1.1-135 12.1.1-135 Install complete. 2 Tue Mar 5 16:31:14 2024 failed rbbn-core-cnf-12.1.1-148 12.1.1-148 Failed to deploy [mshanmugam@cli-server PD]$ helm rollback vgsbc 1 (Here 1 is the revision # that you want to rollback to) Rollback was a success! Happy Helming! [prrao@cli-blr-1 ~]$ helm history vgsbc REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Tue Mar 5 12:03:13 2024 superseded rbbn-core-cnf-12.1.1-135 12.1.1-135 Install complete 2 Tue Mar 5 16:31:14 2024 superseded rbbn-core-cnf-12.1.1-148 12.1.1-148 Upgrade complete 3 Tue Mar 5 17:00:14 2024 deployed rbbn-core-cnf-12.1.1-135 12.1.1-135 Rollback to 1
Once the rollback completes, make sure the pods are stable and no call failures exist.
admin@vsbc1> show table cnfGlobal cnfHealth POD CONTAINER CONTAINER NAME NAME STATUS ---------------------------- ALL ALL Healthy
Note: The SBC should process calls continuously throughout all stages.
Helm History
By default, the maximum number of revisions that displays as part of "helm history" command is 10. To increase the revision limit to more than 10 while performing the helm upgrade (Provide "--history-max=xxx", where xxx is 1-255). This parameter should always pass when running the helm upgrade command.
helm upgrade prargo . --values values.yaml --history-max=100
Git-based Upgrade and Rollback
Pre-Upgrade Checks
Check that all SBC CNe deployment pods are operational.
Check the overall health of the SBC CNe deployment.
SBC CNe - Healthadmin@vsbc1> show table cnfGlobal cnfHealth POD CONTAINER CONTAINER NAME NAME STATUS ---------------------------- ALL ALL Healthy Reference page: https://wiki.rbbn.com/display/NPI/CNF+pods+life+cycle+CLI+commands
- A new SBC CNe helm chart should be available either in the git repository or artifactory.
- In the new SBC CNe helm chart, ensure that the rolling update strategy of individual deployments/pods is set properly.
Upgrade
Avoid simultaneous upgrade of redis-cache pods and pods using rbbn-cache-proxy container
For SBC CNF pods (SC, CS, RAC and NS) that interact with redis-cache pods using rbbn-cache-proxy, do not trigger the upgrade simultaneously with the redis-cache pods to avoid the following issue.
If the rbbn-cache-proxy's connection is not completely established with the redis-cache pods during the redis-cache pod upgrade, a connection failure will occur causing the isbc-container (SC pod)/cs-container (CS pod)/rac-container (RAC pod)/network-service (NS pod) to restart. Once the connection is restored between the rbbn-cache-proxy and redis-cache pod, no further container restarts occur.
Edit the SBC CNe deployment YAML file (created during the initial SBC CNe deployment) based on the requirements.
Git - SBC CNe Helm Release file# Values for RBBN-CORE-CNF chart # This is a YAML-formatted file. # Declare variables to be passed into your templates. apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: lpl-cnf-1 namespace: cnf-demolab spec: releaseName: lpl-cnf-1 targetNamespace: cnf-demolab chart: spec: chart: rbbn-core-cnf version: 12.0.0-14515 --> Point to the new version of the helm chart available in Helm Repository sourceRef: kind: HelmRepository name: sbx-helm-prod-plano namespace: cnf-demolab install: disableWait: true upgrade: disableWait: true interval: 2m values: # Global chart values global: serviceAccount: name: default # namespace where the core-cnf solution has to be deployed. namespace: cnf-demolab # Platform on which the cluster is deployed. kubernetesPlatform: ocp # Storage Class for the PVC creation. # ----------------------------------- # Available options - netapp-nfs-san(default), managed-nfs-storage storageClass: managed-nfs-storage : :Commit and Push the changes using git commands.
git add/push/commit example[llokesh@cli-blr-2 jenkinsbuild-dev]$ git add -A; git commit -a -m "upgrading sbc"; git push origin master [master ae8c499] upgrading sbc 1 file changed, 10 insertions(+), 10 deletions(-) Enumerating objects: 11, done. Counting objects: 100% (11/11), done. Delta compression using up to 16 threads Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 654 bytes | 654.00 KiB/s, done. Total 6 (delta 3), reused 0 (delta 0), pack-reused 0 To https://bitbucket.rbbn.com/scm/sbc/rbbn-gitops-cnf-demolab.git dbfb561..ae8c499 master -> master
Monitor the deployment in the Kubernetes cluster.
oc describe helmrelease lpl-blr1-sbc-llokesh Status: Conditions: Last Transition Time: 2023-05-30T12:04:49Z Message: Release reconciliation succeeded Reason: ReconciliationSucceeded Status: True Type: Ready Last Transition Time: 2023-05-30T12:04:49Z Message: Helm upgrade succeeded Reason: UpgradeSucceeded Status: True Type: Released Helm Chart: cnf-demolab/cnf-demolab-lpl-blr1-sbc-llokesh Last Applied Revision: 12.0.0-latest Last Attempted Revision: 12.0.0-latest Last Attempted Values Checksum: 7b8f71a0794f3d2abb39afdf157a3d998a1a3dba Last Release Revision: 2 Observed Generation: 2 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal info 9h helm-controller Helm install has started Normal info 9h helm-controller Helm install succeeded Normal info 8h (x2 over 9h) helm-controller HelmChart 'cnf-demolab/cnf-demolab-lpl-blr1-sbc-llokesh' is not ready Normal info 8h helm-controller Helm upgrade has started Normal info 8h helm-controller Helm upgrade succeededCheck if all the pods in the SBC CNe are deployed successfully. After the successful upgrade, all the pods should be in "Running State". This completes the deployment of SBC CNe solution using helm.
During Upgrade: lpl-blr1-sbc-cac-6b7bcc9559-8b944 3/3 ContainerCreating 0 15s lpl-blr1-sbc-cac-65db468df9-pa25f 3/3 Running 0 122m lpl-blr1-sbc-cache-0 2/2 Running 0 122m lpl-blr1-sbc-cache-1 2/2 Running 0 122m lpl-blr1-sbc-cache-2 2/2 Running 0 122m lpl-blr1-sbc-cache-3 2/2 Running 0 122m lpl-blr1-sbc-cache-4 2/2 Running 0 122m lpl-blr1-sbc-cache-5 2/2 Running 0 122m lpl-blr1-sbc-cs-785b6b497d-6qpsf 5/5 ContainerCreating 0 15s lpl-blr1-sbc-cs-856886ff46-r2rgb 5/5 Running 0 122m lpl-blr1-sbc-epu-56c6796c88-7bf7g 3/3 ContainerCreating 0 15s lpl-blr1-sbc-epu-768558964d-spedg 3/3 Running 0 122m lpl-blr1-sbc-hpa-548c5898bc-qxd4s 3/3 ContainerCreating 0 15s lpl-blr1-sbc-hpa-86cb7bcbbc-fp3n4 3/3 Running 0 122m lpl-blr1-sbc-ns-5bcb59954b-4gh5j 4/4 Running 0 122m lpl-blr1-sbc-ns-646cb8c5f4-bmn7j 4/4 ContainerCreating 0 15s lpl-blr1-sbc-oam-75fc957dc-kgvh6 2/2 Running 0 122m lpl-blr1-sbc-oam-8f696f9dc-fdcvb 2/2 ContainerCreating 0 122m lpl-blr1-sbc-rac-5b565c4cdf-p9257 4/4 ContainerCreating 0 15s lpl-blr1-sbc-rac-5f9dcb9c97-adft3 4/4 Running 0 122m lpl-blr1-sbc-sc-8c644b4b4-2jglm 5/5 Running 0 122m lpl-blr1-sbc-sc-8c644b4b4-e5jd 5/5 Running 0 9m14s lpl-blr1-sbc-sc-7d4c4d485b-pxvf4 5/5 ContainerCreating 0 15s lpl-blr1-sbc-slb-7f4d8d966b-22obj 4/4 Running 0 122m lpl-blr1-sbc-slb-d76655578-x2v9b 4/4 ContainerCreating 0 15s Once the upgrade is successful, all the pods will be in Up & Running state. Check the ovrall health of the SBC CNe after the upgrade is complete by executing the following command admin@vsbc1> show table cnfGlobal cnfHealth POD CONTAINER CONTAINER NAME NAME STATUS ---------------------------- ALL ALL Healthy
Overview
Content Tools