> ## Documentation Index > Fetch the complete documentation index at: https://hanabiaiinc-fish-772-self-host-docs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes deployment > Install, validate, upgrade, and roll back the Fish Audio Enterprise Helm release Both Helm delivery forms use the same chart and the same procedure. They differ only in how usage is accounted, which is expressed in your values file. ## Prerequisites * A cluster that meets the [requirements](/developer-guide/self-hosting/requirements), preferably a clean cluster dedicated to this deployment. * Shared storage mounted at the same path on every node. * [Registry access](/developer-guide/self-hosting/registry-access) configured, and a chart version chosen on the Self Host page — see [Releases](/developer-guide/self-hosting/enterprise-releases). * Prometheus installed, or approved for installation. * Reference voice archives prepared, if your traffic uses `reference_id`. Set the values you will reuse: ```bash theme={null} CHART_REF='' CHART_VERSION='' NAMESPACE=fish-audio RELEASE=fish-audio ``` `fish-audio` is the supported default namespace. If you must use a different one, change **both** the Helm `--namespace` argument **and** `MODEL_DISCOVERY_NAMESPACE` in your values file. Otherwise the edge API looks for the model services in the wrong namespace. ## Choose a delivery form | | Offline | Online | | -------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | `global.offlineMode` | `true` | `false` | | Model assets | Served in-cluster by the bundled model warehouse | The same — in-cluster, from the same warehouse | | Usage accounting | Local signed ledger on shared storage | Validated and billed against the Fish Audio service | | Object storage credentials | Yours to choose — a password shared inside the cluster | The same — the value is yours, but both sides read one secret | | Runtime egress | None | The billing endpoint on 443. Enabling `qwen3-aligner` adds huggingface.co in either mode | "Offline" here means runtime-offline. Installation still pulls images and the chart from a registry. For an install with no network at all, see [Air-gapped deployments](/developer-guide/self-hosting/air-gapped). ## Preflight ```bash theme={null} kubectl version helm version kubectl get nodes -o wide kubectl get storageclass kubectl describe nodes | grep -E "nvidia.com/gpu|Taints|Labels" || true ``` Verify on **every** node that the shared storage path is a real network mount, not a local directory, **and that it is writable**. Both matter and they fail differently: a node missing the mount quietly gets a local directory and looks healthy while data splits across nodes, and a read-only export stops every Pod before it starts, because Kubernetes creates directories on the volume for each workload. With node access, `findmnt -T /mnt/share` should report an `nfs` or `nfs4` filesystem. Without node access: ```bash theme={null} for node in $(kubectl get nodes -o name); do echo "== ${node}" kubectl debug "${node}" -it --image=busybox:1.36 -- \ sh -c 'grep " /mnt/share " /proc/1/mounts || echo "MISSING: /mnt/share is not a mount"; probe=/host/mnt/share/.fish-audio-write-probe.$$; if touch "$probe" 2>/dev/null; then echo "writable"; else echo "NOT WRITABLE: Kubernetes cannot create directories here"; fi; rm -f "$probe" 2>/dev/null' done ``` ## Create the namespace and secrets ```bash theme={null} kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f - ``` The chart expects three existing secrets. The registry pull secret is created in [Registry access](/developer-guide/self-hosting/registry-access#create-the-kubernetes-pull-secret). Create the other two: ```bash theme={null} # Runtime secret for the edge API. kubectl -n "$NAMESPACE" create secret generic fish-audio-edge-api \ --from-literal=jwt-secret="$(openssl rand -hex 32)" \ --dry-run=client -o yaml | kubectl apply -f - # Object storage credentials. Models are served from inside the cluster in # either mode, so these authenticate nothing outside it and a placeholder is # fine — but the in-cluster store serves with them and the workers present # them back, so change them and restart both sides together. kubectl -n "$NAMESPACE" create secret generic fish-audio-model-assets \ --from-literal=aws-access-key-id='' \ --from-literal=aws-secret-access-key='' \ --dry-run=client -o yaml | kubectl apply -f - ``` | Secret | Keys | Used by | | ------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fish-audio-registry` | Docker registry credentials | Every workload that pulls a private image. | | `fish-audio-edge-api` | `jwt-secret` | The edge API. Generate once and keep it stable; changing it invalidates issued tokens. | | `fish-audio-model-assets` | `aws-access-key-id`, `aws-secret-access-key` | Inference worker and vocoder model asset loading. Required in both forms. Models are served from inside the cluster, so these authenticate nothing outside it — the in-cluster store serves with them and the workers present them back. A placeholder is fine; change it and restart both sides together. | For production, prefer External Secrets, Sealed Secrets, or your cloud secret manager over plain Kubernetes Secrets. ## Prepare a values file The chart carries a values file for each delivery form under `profiles/`, so pulling the chart gets you a starting point: ```bash theme={null} helm pull "$CHART_REF" --version "$CHART_VERSION" --untar --untardir ./chart cp ./chart/*/profiles/offline.yaml values.yaml # or online.yaml ``` Edit the copy. A later `helm pull --untar` fails rather than overwriting the directory it created, so upgrading means deleting it first. A profile sets the deployment mode and the shared-storage path. What a cluster usually has to change: | Setting | When | | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `global.sharedStorageHostPath` | Shared storage is not mounted at `/mnt/share`. | | The `MODEL_DISCOVERY_NAMESPACE` entry under `edge-api-production.env` | You install into a namespace other than `fish-audio`. | | `sglang.worker.tolerations`, `vocoder.tolerations`, and the matching `nodeSelector` | Your GPU nodes are tainted or you run more than one GPU model. | | `offlineS3.enabled: false`, then `sglang.worker.env.S3_ENDPOINT_URL` **and** `vocoder.env.S3_ENDPOINT_URL` | You serve model assets from a bucket of your own instead of the in-cluster warehouse. Also the one case where `fish-audio-model-assets` must hold real credentials — they become what the workers authenticate to your store with — and you will need the weights themselves from your account team, since they ship inside the warehouse image. All three: leaving the warehouse enabled while pointing the workers elsewhere is refused, and the vocoder is easy to miss. | Image references are pinned in the chart. Change them only when mirroring into your own registry, as [Air-gapped deployments](/developer-guide/self-hosting/air-gapped) describes — a version is otherwise selected with `--version`, not by editing tags. Talk to your account team before changing replica counts. The profiles ship the single-worker shape the delivery was sized against, and the API, GPU, aligner, normalizer and router deployments read a replica change in a values file on install but not on `helm upgrade`, where the running count wins. `oxidengine`, the model warehouse and Redis are the other way round: they take the value every time, so a `kubectl scale` on those is undone by the next upgrade. Ingress and TLS are disabled by default. Enable them only if you want Kubernetes to manage your access layer; otherwise put your own ingress controller or load balancer in front of the `edge-api-production` service. ## Install Prometheus The chart exposes metrics through `prometheus.io/*` pod annotations and creates no ServiceMonitors. An annotation-scraping Prometheus works out of the box: ```bash theme={null} helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm upgrade --install prometheus prometheus-community/prometheus \ --namespace monitoring --create-namespace \ --set alertmanager.enabled=false ``` If you already run kube-prometheus-stack, add an annotation-based scrape config instead. ## Stage reference voices A self-hosted deployment never downloads reference audio from the network. Requests that pass a `reference_id` resolve from Redis, then from a local archive on shared storage. Stage one zip per voice before enabling that traffic: ```text theme={null} ///reference-archives/.zip ``` Each archive holds one or more audio files at the zip root, each paired with a UTF-8 `.txt` transcript of the same basename: ```text theme={null} voice-alice-001.zip |-- sample-01.wav |-- sample-01.txt |-- sample-02.mp3 `-- sample-02.txt ``` Supported audio extensions are `.wav`, `.mp3`, and `.opus`. Do not nest directories inside the archive, and do not add prefixes to the filename — the file name must match the reference id exactly. If you have no node shell, stage archives through a helper pod that mounts the share, then confirm the edge API can see the file: ```bash theme={null} kubectl -n "$NAMESPACE" exec deploy/edge-api-production -- \ ls -lh /mnt/share/reference-archives/voice-alice-001.zip ``` These archives are durable data, not cache. Back them up and preserve the path across upgrades. ## Validate the release Render the manifests before touching the cluster: ```bash theme={null} helm show values "$CHART_REF" --version "$CHART_VERSION" > defaults.yaml helm template "$RELEASE" "$CHART_REF" \ --version "$CHART_VERSION" \ --namespace "$NAMESPACE" \ --values values.yaml ``` ## Install ```bash theme={null} helm upgrade --install "$RELEASE" "$CHART_REF" \ --version "$CHART_VERSION" \ --namespace "$NAMESPACE" \ --create-namespace \ --values values.yaml \ --timeout 30m ``` ## Wait for readiness On the first install the inference worker compiles its GPU graphs and the vocoder builds its inference engine, which takes far longer than later rollouts. Wait on every workload in the release, Deployments and StatefulSets alike: ```bash theme={null} ( set -e resources="$(kubectl -n "$NAMESPACE" get deployment,statefulset \ -l app.kubernetes.io/instance="$RELEASE" -o name)" test -n "$resources" || { echo "No Fish Audio workloads found" >&2; exit 1; } printf '%s\n' "$resources" | while read -r r; do kubectl -n "$NAMESPACE" rollout status "$r" --timeout=90m done ) ``` The timeout applies per workload, not to the loop. Watch progress with: ```bash theme={null} kubectl -n "$NAMESPACE" get pods,svc kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp ``` ## Smoke test The acceptance signal is a successful generation, not a health probe. With no ingress configured, port-forward the edge API: ```bash theme={null} kubectl -n "$NAMESPACE" port-forward svc/edge-api-production 8080:80 ``` ```bash theme={null} curl -fsS --noproxy '*' -X POST http://127.0.0.1:8080/v1/tts \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'model: ' \ -d '{"text": "Hello from Fish Audio Enterprise.", "format": "mp3"}' \ -o smoke-test.mp3 file smoke-test.mp3 ``` The test passes when the request returns HTTP 200 and the file contains playable audio. | Detail | Notes | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` header | Must match the model name registered by your values file. | | `Authorization`, online form | Your Fish Audio API key. Every request is authorized and billed against the Fish Audio service. | | `Authorization`, offline form | Any non-empty bearer token is accepted and recorded verbatim as the billing identity. An empty or missing token still returns 401. Use a stable, distinct token per tenant. | | Proxies | A local HTTP or SOCKS proxy can break a port-forwarded request with an empty reply. `--noproxy '*'` and `127.0.0.1` avoid it. | | Latency | Latency measured through `kubectl port-forward` includes the round trip to the API server and your own network. Measure from inside the cluster for representative numbers. | Also confirm that the GPU workloads see the shared storage mount, that Redis is ready, that reference archives are visible if you use them, and that no pod is in a crash or image-pull loop. ## Upgrade ```bash theme={null} helm diff upgrade "$RELEASE" "$CHART_REF" \ --version "$CHART_VERSION" \ --namespace "$NAMESPACE" \ --values values.yaml helm upgrade "$RELEASE" "$CHART_REF" \ --version "$CHART_VERSION" \ --namespace "$NAMESPACE" \ --values values.yaml \ --timeout 30m ``` If the `helm-diff` plugin is not installed, review `helm template` output before applying. Re-run the smoke test after every upgrade and keep the previous revision available until the deployment is accepted. ## Roll back ```bash theme={null} helm -n "$NAMESPACE" history "$RELEASE" helm -n "$NAMESPACE" rollback "$RELEASE" '' --timeout 30m kubectl -n "$NAMESPACE" get pods ``` ## Uninstall ```bash theme={null} helm -n "$NAMESPACE" uninstall "$RELEASE" ``` Uninstall does not remove data on shared storage. Reference archives, the usage ledger, and compile caches survive and should be deleted deliberately. ## Next steps * [Air-gapped deployments](/developer-guide/self-hosting/air-gapped) to install with no network access. * [Operations](/developer-guide/self-hosting/operations) for monitoring signals, scaling, and troubleshooting.