What 499 Means
NGINX logs 499 when the client closes the connection before NGINX can send a response. It is a symptom, not a root cause. The failure may be upstream latency, PHP-FPM port mismatch, database saturation, deploy regression or client timeout.
Service and Target-Port Validation
kubectl get svc,pod -n app
kubectl describe svc php-app -n app
kubectl get endpoints php-app -n app
kubectl exec deploy/php-app -n app -- ss -lntpA common failure is an NGINX or Service targetPort pointing to the wrong PHP-FPM listen port. The pod can be Running while the upstream connection still fails.
Logs and Probes
- Compare ingress logs with PHP-FPM logs at the same timestamp.
- Check readiness and liveness probes for false positives.
- Look for max_children exhaustion or slow PHP workers.
- Review HPA decisions and CPU/memory throttling.
Database Saturation
If PHP workers wait on MySQL, NGINX can see client disconnects even when ports are correct. Inspect slow queries, connection counts and recent migration or traffic changes.
Rollback and Prevention
- Rollback the deployment if the failure aligns with a release.
- Add pre-deploy checks for Service targetPort and container listen ports.
- Alert on PHP-FPM saturation and upstream errors, not only pod restarts.
- Keep a validation checklist for ingress, service, endpoints, pods and database health.
Build a Timestamped Request Path
Pick representative failed request IDs and follow them through external load balancer, ingress, NGINX sidecar or web pod, PHP-FPM and database. Record the timeout configured at every boundary. The shortest timeout usually determines who closes first, but the slow component can be several layers deeper.
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=30m | grep ' 499 '
kubectl logs -n app deploy/web --since=30m
kubectl logs -n app deploy/php --since=30m
kubectl top pod -n app --containersProve Service Wiring From Both Directions
Compare the Service port and targetPort with the named containerPort and the real PHP-FPM listen directive. Then connect from the NGINX pod to each endpoint IP. EndpointSlice readiness, NetworkPolicy and sidecar interception can make a Service fail even while localhost works inside the PHP container.
kubectl get svc,endpoints,endpointslice -n app -o wide
kubectl exec -n app deploy/web -- getent hosts php-app
kubectl exec -n app deploy/web -- nc -vz php-app 9000
kubectl exec -n app deploy/php -- sh -c "grep -R '^listen' /usr/local/etc/php-fpm.d && ss -lnt"Measure PHP-FPM Queueing
A reachable port can still be saturated. Enable the FPM status path on an internal-only endpoint and inspect listen queue, max listen queue, active processes and max children reached. Correlate these with CPU throttling and memory limits. Increasing pm.max_children without accounting for per-worker memory can turn latency into OOM restarts.
Distinguish CPU throttling from low CPU demand
A container at its CPU limit may report modest averaged utilization while cgroup throttling delays requests. Inspect container_cpu_cfs_throttled_seconds_total, request versus limit, and node pressure. Compare latency before and after a controlled resource adjustment rather than removing limits during an incident.
Follow the Wait Into MySQL
When FPM workers are occupied but not consuming CPU, inspect database connections, lock waits and slow queries. A release can introduce an unindexed query or hold transactions longer; traffic growth can exhaust the connection pool. Raising ingress timeouts hides the symptom while queues and resource use continue to grow.
mysql -e "SHOW FULL PROCESSLIST"
mysql -e "SHOW ENGINE INNODB STATUSG"
mysql -e "SHOW GLOBAL STATUS LIKE 'Threads_connected'"
mysql -e "SHOW GLOBAL STATUS LIKE 'Slow_queries'"Containment Versus Permanent Repair
- Rollback when the latency change aligns with a known release and data compatibility permits it.
- Scale FPM replicas only if database and node capacity can absorb the concurrency.
- Temporarily raise a timeout only when measured work can finish and queue growth is bounded.
- Correct target ports and probes together so Kubernetes stops routing to unusable endpoints.
- Fix the query, lock, resource limit or worker policy that created the delay.
Validation Under Load
Replay representative requests at expected concurrency, not a single curl. Confirm p50, p95 and p99 latency, 499 and upstream error rates, FPM queue depth, max_children events, database waits, pod restarts and CPU throttling. Continue through a rolling restart to prove readiness prevents traffic before PHP-FPM can accept it.
Prevent Repeat Incidents
- Carry a request ID across ingress, application and database logs.
- Alert on FPM queue depth and max children reached.
- Test Service targetPort and process listen ports in deployment checks.
- Set readiness against real runtime dependencies without making it excessively fragile.
- Review timeout budgets as one request path, not isolated defaults.
A Safe Incident Runbook
Start by freezing unrelated deployments and recording the failure window, affected routes, client timeout and recent release identifiers. Select several failed and successful requests with timestamps or request IDs. Capture ingress configuration, Service and EndpointSlice state, pod revisions, resource limits, autoscaling events and database health before restarting pods. A restart can drain the queue temporarily while erasing the evidence needed to explain why it formed.
Build a timeout budget from the outside inward: client, CDN or load balancer, ingress proxy, NGINX FastCGI, application HTTP client and database statement limits. Record both configured values and observed durations. If the client closes at thirty seconds while PHP completes at thirty-five, raising the client timeout may hide the 499 but still deliver unacceptable latency. The repair must reduce or intentionally bound the work.
Deployment and Probe Analysis
Compare the failing ReplicaSet with the prior revision, including image digest, ConfigMap and Secret checksums, command, ports, probes and resources. A named targetPort can follow a renamed containerPort differently from a numeric target. Readiness should fail until NGINX can reach PHP-FPM and required local initialization is complete, but it should not depend on every remote service in a way that removes all pods during a downstream outage.
Review termination behavior as carefully as startup. Kubernetes removes a terminating endpoint, but existing connections need enough preStop and terminationGracePeriodSeconds time to drain. NGINX and PHP-FPM must receive signals they handle correctly. A fast termination during rolling deployment can create client disconnects that resemble runtime saturation, while an excessive grace period can slow rollback and consume cluster capacity.
Capacity and Queue Mathematics
Estimate per-worker memory from real processes, then set pm.max_children within the pod limit with headroom for the master, opcache and sidecars. Measure service time and arrival rate; when utilization approaches capacity, queueing rises non-linearly. Scaling replicas helps only if sessions, cache, storage and database connections support concurrency. Coordinate the FPM pool, horizontal autoscaler and database connection budget rather than tuning each independently.
Database Evidence
Use slow-query samples and lock-wait evidence from the incident window. Compare execution plans and row counts for representative parameters because a query can be fast for one tenant and pathological for another. Check connection creation, idle sessions and transaction duration. If rollback includes a schema migration, establish whether old application code remains compatible before changing the deployment; application recovery must not create data corruption.
Closure Criteria
Close only after representative load runs through at least one rolling replacement with stable latency percentiles, no unexplained 499 growth, bounded FPM queues, no max_children events, acceptable database waits and no CPU throttling or OOM restart. Preserve dashboards and example request traces in the incident record. Add a deployment check that compares Service ports with listeners and a synthetic transaction that crosses the complete request path.
Questions to Answer Before the Incident Closes
Confirm whether every observed 499 belongs to the same failure mode. Client navigation, cancelled searches and health-check behavior can create harmless 499s alongside production latency. Segment by route, upstream time, user agent, ingress and release revision. Establish a normal baseline and alert on deviation coupled with latency or upstream errors. Verify the repaired target port and FPM listener from a deployment test, and ensure configuration generation cannot drift between chart values, Service definitions and container images. Keep one representative request trace showing the complete healthy path for future comparison.
Review the capacity boundary under failure as well as normal load. Test a slow database, terminating pod and unavailable dependency to confirm queues remain bounded and clients receive intentional errors before arbitrary disconnects. Check PodDisruptionBudget, rollout surge, node capacity and autoscaler delay so a deployment does not remove too much FPM capacity at once. Record safe rollback conditions for application and schema changes. The incident is closed when the system handles expected concurrency and controlled degradation with measured timeout budgets, useful telemetry and repeatable deployment validation, not merely when the current 499 graph returns to zero.
Include client and edge behavior in the final review. A CDN, load balancer or mobile client may retry after closing the first request, doubling work on an already saturated backend. Look for repeated request IDs, identical payloads and overlapping upstream execution. Protect non-idempotent operations with idempotency keys or transaction controls before enabling retries. Verify buffering settings for uploads and large responses because request bodies can consume ingress resources before PHP begins. For streaming responses, standard proxy buffering and timeout assumptions may be wrong. Classify routes by workload and apply timeout, body-size and concurrency controls intentionally instead of one global ingress annotation.