Symptoms
Users experienced intermittent failures while ingress logs showed client-closed 499 responses. Pod status alone did not explain the behavior.
Evidence
- Service target ports and container listen ports were compared.
- PHP-FPM and NGINX timestamps were correlated.
- Database saturation was checked because slow upstream responses can surface as 499s.
- Recent deployment changes were reviewed for rollback risk.
Fix
Runtime wiring was corrected, rollback remained available, and validation covered ingress, service endpoints, PHP-FPM availability and database responsiveness.
Lessons
499 investigations need request-flow evidence across ingress, service, pod and database layers; otherwise teams can chase client behavior while the upstream remains unhealthy.
Context and Architecture
The request path crossed a public load balancer, NGINX ingress, a Kubernetes Service, web and PHP-FPM containers, and a MySQL-backed application. Deployments and autoscaling could change endpoints while traffic remained active.
Business Risk
Intermittent failures affected user transactions while healthy pod counts suggested the platform was available. Increasing timeouts risked converting visible failures into a larger queue and exhausting workers or database connections.
Observable Symptoms
NGINX recorded 499 responses and upstream timing outliers. Some application requests completed, which ruled out a total network break. Failure frequency changed around deployment and load conditions.
Hypotheses Considered
- Clients had an unrealistically short timeout.
- Service targetPort did not match the FPM listener.
- Readiness admitted pods before FPM was usable.
- FPM max_children created queueing.
- MySQL lock or query latency occupied workers.
- CPU throttling delayed the request path.
Root Cause
Cross-layer evidence identified incorrect runtime wiring as an immediate failure and worker/database pressure as the condition that amplified client disconnects. The 499 code described who closed the connection, not why the request was still unfinished.
Containment and Permanent Fix
The risky release path remained rollbackable while Service and FPM ports were aligned. Probes were adjusted to represent runtime readiness, and PHP-FPM capacity and database behavior were tuned from measured queue and query evidence rather than by extending every timeout.
Validation
Representative traffic was tested through a rolling deployment. The team tracked 499s, upstream errors, latency percentiles, FPM queue depth, database waits, endpoint readiness, restarts and CPU throttling until the request path remained stable.
Operational Lessons
A Kubernetes status page cannot validate an application request path. Port contracts, request IDs, worker queues and database waits need to be visible together, and timeout changes must follow a measured latency budget.