How to recover AWS resources affected by the CrowdStrike Falcon agent?

Recovering AWS Resources Affected by the CrowdStrike Falcon Agent

On July 18, 2024, an update to the CrowdStrike Falcon agent (csagent.sys) caused unplanned stop errors or blue screens on Windows-based devices, including Amazon EC2 instances and Amazon WorkSpaces. Here’s how to resolve the issue if a reboot doesn’t work:

Steps to Restore Amazon EC2 Instances:

• Identify Impaired Instances:

  aws ec2 describe-instance-status --filters Name=instance-status.status,Values=impaired --query "InstanceStatuses[*].InstanceId" --region your-region

• Option 1: AWS Systems Manager Automation Runbook:
  • Use AWSSupport-StartEC2RescueWorkflow to automate recovery.
  • Ensure your IAM user/role has the required permissions, including kms:CreateGrant.
  • The runbook mounts the root volume of the selected instances and deletes the problematic file using:

    get-childitem -path "$env:EC2RESCUE_OFFLINE_DRIVE\Windows\System32\drivers\CrowdStrike\" -Include C-00000291*.sys -Recurse | foreach { $_.Delete()}


• Option 2: Manual Recovery Method:
  • Create a snapshot of the instance’s EBS root volume.
  • Create a new EBS volume from the snapshot in the same Availability Zone.
  • Launch a new Windows instance in the same Availability Zone.
  • Attach the new EBS volume to the new instance as a data volume.
  • Download and run the EC2Rescue tool as an administrator.
  • Select "Offline Instance" mode and navigate to the attached volume to delete the problematic file C-00000291*.sys.
  • Detach the EBS volume from the new instance and create an AMI from it.
  • Replace the root volume on the original EC2 instance using this AMI.

Steps to Restore Amazon WorkSpaces:

• Restore from Snapshot:
  • Restore the WorkSpace to a previous snapshot taken up to 12 hours before the issue occurred.
  • If beyond 12 hours, rebuild the WorkSpace to restore the root volume.

Notes:

• If instance store volumes are used, data will not persist when stopped, hibernated, or terminated.
• If the root EBS volume is encrypted, ensure the encryption key exists and you have the necessary permissions.
• If issues persist, contact AWS Support and provide a screenshot of the unreachable instance.

For detailed instructions and additional support, visit the CrowdStrike Blog and AWS documentation. This will ensure that you have the most accurate and up-to-date information for resolving these issues.