Challenge
The workload needed to stay recoverable while suspicious outbound behavior was contained and investigated. The priority was to stop unnecessary egress without destroying evidence.
Evidence
- Outbound SSH-style behavior was treated as a compromise signal.
- Temporary directories, web logs, cron and SSH keys were reviewed for wget, tar and persistence patterns.
- Security groups and NACLs were checked to separate required DNS/NTP paths from unnecessary outbound traffic.
Fix
Egress was restricted, vulnerable web entry points were reviewed, suspicious runtime artifacts were documented, and restoration was staged so each reopened path could be observed.
Outcome
The case produced a reusable incident checklist for EC2 abuse notices, containment decisions, evidence collection and AWS communication without exposing customer or attacker details.
Context and Architecture
The affected Linux web workload ran on EC2 with public application traffic, an instance role, attached EBS storage and normal outbound dependencies for DNS, time synchronization, package access and monitoring. Customer, account and attacker identifiers are omitted; the investigation pattern is preserved.
Business Risk
The abuse report created two simultaneous risks: continued outbound activity could harm third parties and trigger account restrictions, while an aggressive shutdown could destroy volatile evidence and extend customer downtime. The recovery plan therefore separated containment, evidence preservation and replacement.
Observable Symptoms
The initial signal was suspicious outbound connection behavior rather than an application outage. That meant normal health checks were not evidence of safety. External flow records, the AWS notice window and host process data became the primary timeline.
Hypotheses Considered
- A vulnerable web component spawned an outbound process.
- An exposed SSH credential allowed interactive access.
- A scheduled task or service provided persistence.
- A legitimate process was misclassified.
- A neighboring resource, NAT address or stale observation was involved.
Root Cause and Containment
Evidence concentrated the investigation on the web execution path and writable runtime locations rather than normal administrative SSH. Outbound access was narrowed, volume evidence was preserved, and public exposure was controlled while file timestamps, web requests, process ancestry and persistence locations were correlated.
Permanent Fix
The recoverable path was a trusted rebuild with the exposed application path patched, reviewed data restored, instance-readable credentials rotated and application code made less writable. Required egress was reopened by dependency instead of restoring unrestricted outbound access.
Validation
Validation covered application functions, clean process and socket baselines, expected DNS and monitoring traffic, new Flow Log observations, IAM activity and staged public traffic. The prior volume remained isolated for follow-up analysis.
Operational Lessons
Abuse response is not simply malware removal. A defensible recovery needs external logs, a trustworthy rebuild path, constrained egress, credential scope analysis and a timeline that can be communicated to AWS and internal owners.