Use this Kubernetes configuration validation checklist to catch invalid manifests, scheduling problems, probe failures, unsafe defaults, and rollout issues before they affect a shared or production cluster. It covers a practical workflow from rendered YAML and schema checks through dry runs, policy tests, rollout observation, and post-failure triage.
Overview
Kubernetes validation is more reliable when treated as a sequence of checks rather than a single command. YAML can be syntactically valid while still containing the wrong API version, an invalid field, an unusable image reference, a missing Secret, or resource requests that prevent scheduling. A manifest can also pass client-side checks and later be rejected by the API server or an admission policy.
A useful validation path has four layers:
- Shape: Confirm that YAML parses and that rendered templates produce the objects you expect.
- Schema: Check resource kinds, API versions, field names, and field types against the target Kubernetes version.
- Behavior: Test scheduling assumptions, Services, configuration references, health probes, resources, and security settings.
- Delivery: Apply safely, observe the rollout, and retain enough information to diagnose a failure.
Run these checks against the exact output that will be deployed. If you use Helm, Kustomize, Jsonnet, or another templating system, validating only the source templates can miss errors introduced during rendering. Keep the target cluster version and enabled admission policies in mind; a manifest that works in one environment may be rejected or behave differently in another.
Manifest-field reference
| Area | Check before deployment | Typical failure signal |
|---|---|---|
| Deployment | Use the intended API version; verify selector and pod labels match; review replicas, strategy, and update limits. | Rejected selector, orphaned ReplicaSets, or a rollout that cannot make progress. |
| Service | Confirm the selector matches ready pod labels and that service ports map to the container port or application listener. | No endpoints, connection refusal, or traffic sent to the wrong port. |
| ConfigMap | Check key names, mounted paths, and whether values are consumed as files or environment variables. | Application starts with missing or stale configuration. |
| Secret | Verify the Secret name, keys, namespace, and reference style. Treat values as sensitive during testing. | Pod creation errors, authentication failures, or credentials exposed in logs. |
| Health probes | Check path, port, scheme, initial delay, timeout, period, and failure thresholds against actual startup behavior. | Restart loops, failed rollouts, or traffic reaching an unready process. |
| Resources | Set realistic CPU and memory requests and limits where appropriate; check init containers separately. | Pending pods, eviction, throttling, or out-of-memory termination. |
| Security | Review service accounts, privilege escalation, capabilities, filesystem writability, host access, and security contexts. | Admission rejection or a workload with broader access than intended. |
Checklist by scenario
1. Validating a new or changed manifest locally
- Run a YAML parser or validator first. Confirm indentation, quoting, list structure, and scalar types. A YAML formatter and validator can help isolate indentation and type errors before Kubernetes is involved.
- Inspect the complete object metadata: namespace, name, labels, annotations, and owner references where applicable.
- Check that every referenced ConfigMap, Secret, ServiceAccount, volume, and image name is intentional.
- Use schema-aware validation for the Kubernetes version you plan to target. Look for deprecated or removed API versions and fields rather than assuming a client will translate them.
- Render templates before validating. For example, inspect the output of
helm templateorkubectl kustomizeand save it as an auditable build artifact.
2. Performing a dry run
A dry run is a useful boundary check, but it is not a complete deployment test. A client-side dry run can catch local interpretation problems. A server-side dry run sends the request to the API server for validation and admission processing without persisting the object.
kubectl apply --dry-run=client -f rendered.yaml
kubectl apply --dry-run=server -f rendered.yaml
Use the server-side form when you need to test against the cluster's API resources and admission configuration. It still does not prove that an image can be pulled, a pod can be scheduled, a probe will succeed, or a dependency is reachable. Those require runtime checks.
3. Checking a Deployment before rollout
- Confirm
spec.selector.matchLabelsmatches the labels underspec.template.metadata.labels. Do not change an existing selector casually; it is commonly immutable. - Review the update strategy. A rolling update needs enough capacity to create replacement pods while old pods remain available, unless the chosen strategy explicitly accepts downtime.
- Check
maxUnavailableandmaxSurgeas percentages or counts and consider how they interact with replicas and cluster capacity. - Make sure the container command, arguments, working directory, and exposed ports match the image.
- Confirm that environment variables are sourced from the intended ConfigMaps and Secrets, with optional references used only when absence is acceptable.
- Review the pod-level and container-level security context, especially if the image expects a writable directory or a particular user.
4. Testing Services and connectivity
Start with labels. A Service selector must match the labels on the pods it is meant to expose, and those pods must become ready before they are normally included as ready endpoints. Compare the Service port and target port with the process's actual listening port; a declared container port alone does not make an application listen there.
When a Service appears broken, inspect both sides:
kubectl get svc <service-name> -n <namespace> -o yaml
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>
kubectl get pods -n <namespace> --show-labels
For deeper HTTP behavior, compare status codes, headers, and response bodies using the techniques in the HTTP request debugging guide. DNS or external routing issues may require a separate check of records, TTLs, and propagation assumptions; see the DNS records troubleshooting guide.
5. Diagnosing a failed rollout
First establish whether the failure is validation, scheduling, startup, readiness, or application-level. Then collect evidence before changing several fields at once:
kubectl rollout status deployment/<name> -n <namespace>
kubectl describe deployment/<name> -n <namespace>
kubectl get pods -n <namespace> -l app=<label>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers
Events often distinguish an unschedulable pod from an image-pull error or a failing probe. If the process starts and then exits, inspect the container command and logs. If it stays running but never becomes ready, test the probe path and port from inside the relevant network context.
What to double-check
Health probes
A liveness probe answers whether Kubernetes should restart a container; a readiness probe answers whether it should receive traffic. A startup probe can give a slow-starting application time to initialize before liveness and readiness checks become decisive. Select endpoints that represent the intended condition, not merely any endpoint that returns a response.
Check that the probe protocol, port, path, and host are correct. Set timing values according to observed startup and dependency behavior. An overly aggressive liveness probe can create a restart loop that hides the original problem. A readiness probe that depends on an optional downstream service may remove all capacity when that dependency has a temporary issue.
Resources and scheduling
Requests influence scheduling, while limits constrain consumption. Review both regular and init containers; a large init-container request can affect scheduling even if the main application is small. Compare requests with node capacity, namespace quotas, and any limits imposed by the cluster. A pod stuck in Pending should be investigated through its events rather than by immediately reducing requests.
Configuration and secrets
Check namespace boundaries carefully. A reference to a ConfigMap or Secret in another namespace will not behave like a cross-namespace lookup. Verify key names exactly, distinguish absent values from empty values, and determine whether a configuration change requires a pod restart. Never paste real credentials into online formatters, issue trackers, or shell history while troubleshooting. The environment variables guide provides a broader workflow for organizing and debugging configuration across environments.
Security and policy
Use the least privilege that the application can support. Review the service account, Linux capabilities, privilege escalation, host networking, host paths, writable filesystems, and user identity. Test these settings with the same admission policies used by the destination cluster. Policy examples and security defaults can change with platform upgrades, so treat them as versioned configuration rather than permanent assumptions.
CI validation workflow
A repeatable pipeline can use this order:
- Parse YAML and fail on syntax errors.
- Render Helm or Kustomize output and archive the rendered result.
- Run Kubernetes schema validation against the target API version.
- Run policy tests for namespaces, images, resources, security, and required labels.
- Run a server-side dry run against a suitable validation cluster or target context.
- Deploy to a disposable or staging environment and wait for rollout status.
- Exercise a small health or smoke test before promotion.
Keep validation tools pinned or centrally managed where practical, and document which checks are advisory versus deployment-blocking. This makes failures easier to interpret when a client, chart, cluster, or admission policy changes.
Common mistakes
- Validating only raw YAML: Templates and overlays can alter names, labels, values, and resource counts. Validate the rendered output.
- Confusing client and server dry runs: Client-side checks do not exercise the target API server's schema or admission path.
- Using a liveness probe as a readiness probe: A dependency outage can cause unnecessary restarts instead of temporarily removing a pod from traffic.
- Forgetting selectors: A Deployment, Service, NetworkPolicy, or monitoring rule can target different labels than the pod actually has.
- Assuming an image tag is immutable: Record the resolved image reference or digest when reproducibility matters, and verify pull access from the cluster.
- Ignoring events: Pod status gives a summary; events and descriptions often contain the scheduling, mount, admission, or image error that explains it.
- Changing several variables at once: Capture the failing manifest, events, logs, and rollout state before applying a broad fix.
- Testing only the happy path: Include missing configuration, slow startup, unavailable dependencies, insufficient capacity, and rollback behavior in staging checks.
For YAML-specific problems, compare the rendered file with a known-good version and use a formatter or validator to make indentation and scalar-type differences visible. For adjacent API contracts, an OpenAPI validation workflow can help ensure that the application endpoint behavior matches its documented interface.
When to revisit
Use this checklist before every meaningful manifest change, chart upgrade, image change, namespace migration, cluster migration, or rollout strategy adjustment. Revisit it after a failed deployment, a new admission rule, a Kubernetes API upgrade, or a change to node capacity and namespace quotas. It is also useful during periodic configuration audits and before seasonal planning cycles when workloads or traffic patterns may change.
Because API versions, deprecated fields, security guidance, validation tools, and policy examples evolve, review this workflow every four to six months and before a cluster upgrade. Do not assume that a previously accepted manifest remains valid in a new environment.
Reusable pre-deployment record
Change: ______________________________
Commit or image: _____________________
Target cluster and namespace: ________
Rendered manifest reviewed: yes / no
Schema validation: pass / fail
Policy checks: pass / fail / not run
Client dry run: pass / fail
Server dry run: pass / fail / not run
Dependencies and referenced objects checked: yes / no
Probe paths and ports tested: yes / no
Requests, limits, and capacity reviewed: yes / no
Security context reviewed: yes / no
Rollout owner: _______________________
Rollback or recovery plan: ___________
When a deployment fails, record the first observed symptom, the affected object, the relevant events, pod status, logs, recent manifest difference, and the next reversible action. Apply one focused change, rerun the appropriate validation layer, and observe the rollout again. This turns Kubernetes deployment troubleshooting from guesswork into a repeatable DevOps workflow.