Losing credentials or log-in details to any platform or Infrastructure can be frustrating and can have a lot of consequences which is why we must protect our security credentials with all intentions.
Imagine losing credentials to your vault and you can't access your money
Never lost my ssh keys tho. So this article is a request from someone and I'm writing just to show a walk-through of how to recover missing Pem file to an AWS instance. let's go
RECOVERING ACCESS TO MY EC2 INSTANCE AFTER LOSING PEM FILE
For Pem file recovery to be possible, the lunched instance with the missing Pem file must be an EBS-Backed Instance because it is not possible on an instance backed by an instant store.
Assume that your instance with lost Pem file is “Lost pem server”
I. To confirm the backed-storage type
-
Click/Check the instance.
-
Select the “storage” tab.
II. Launch a Recovery Instance
-
Launch the “Recovery Instance” in the same AZ as the “Lost pem server”
-
To Launch a New Instance in the same Availability Zone as the instance(Lost pem server) with the missing Pem file….Make sure to check which AZ the instance is running(The above is running in us-east-2a).
-
Select the same network(VPC) and subnet as the old instance(Lost pem server).
III. Create key pair for the new instance(Recovery server) and connect to it using ssh. Be sure you can connect to the instance.
IV. Now stop the old instance(Lost pem server).
V. Detach the EBS volume connected to the “Lost pem server”.
-
Click on the instance with lost Pem and select the “Storage” Tab.
-
Scroll down to blocked devices and click on the ID for the Root Device. In my case, the root device is dev/sda1.
-
Make sure the Root Storage is still selected, then click on Action
-
Finally, select “Detach volume”
- Go to “volumes” under EBS(Elastic Block Store) and select the volume you previously detached from the “Lost pem server”
- Select the volume and click “Action”
- Select “Attach Volume”.
- Select the recovery server(Recovery Server).
VI. Go back to the shh terminal to the “Recovery server” and check the EBS volume
NOTE: The attached EBS volume won’t mount automatically, so you have to mount it.
- Go to your terminal and ssh into the Recovery server. In my case, I’m using WSL in visual studio.
Enter the following commands
I. Optionally: you might want to copy the pem file from where it is to the .ssh folder
cp /mnt/c/Users/YourName/desktop/pem/recovery.pem ~/.ssh/recovery.pem II. Connect to the instance using SSH
III. To check the list of storage on the device to see if the attached EBS was mounted or not. Though it doesn’t mount automatically.
lsblk 
NOTE: The attached EBS(XVDF) has not been mounted to any path as you can see in the screenshot above.
IV. Also, confirm that the attached EBS is not empty because u can’t mount an empty EBS.
sudo file -s /dev/xvdf Output: /dev/xvdf: DOS/MBR boot sector, extended partition table (last)
Output like this means the drive is not empty, so you can go ahead and mount.
V. Create a temporary directory to mount the EBS volume(for the Lost pem server) in the Recovery server.
sudo mkdir /mnt/tempvolume The path = /mnt/tempvolume
VI. Mount the storage on path /mnt/tempvolume. Mount the drive with a number at the end “/dev/xvdf1”
/dev/xvdf1 is the disk to be mounted.
sudo mount /dev/xvdf1 /mnt/tempvolume VII. Check if the EBS has mounted
lsblk 
To check the list of storage on the device again.
VIII. Copy the SSH key of your “Recovery server” into the attached drive belonging to the “Lost pem server”
cp .ssh/authorized_keys /mnt/tempvolume/home/ubuntu/.ssh/ NOTE: Ubuntu in path /mnt/tempvolume/home/ubuntu/.ssh/ is the server name, it can be ec2-user if u launched linux server.
IX. Check the list of contents in the temporary location on the storage you copied the key to.
ls -lah /mnt/tempvolume/home/ubuntu/.ssh/ 
The key “Authozied_keys” is now in the folder you copied it to.
X. Unmount the attached storage from the “Recovery server so you can attach it back to the “Lost pem server”.
sudo umount /mnt/tempvolume/ XI. Check if the disk has unmounted.
lsblk 
The mount point of /dev/xvdf1 is no longer /mnt/tempvolume
XII. Now detach the attached “Lost pem EBS” from the “Recovery server” to attach it back to its original server “Lost pem server”.
XIII. Attached the EBS to the “Lost pem server” and edit the name to /dev/sda1 and save.
To confirm the time the EBS was attached, check “Attachment time”
XIV. Connect to the “Lost pem server” with the key of the “Recovery Server”.
ssh -i "ggfgvfv.pem" ubuntu@ec2-3-145-xxx-92.us-east-2.compute.amazonaws.com XV. Connect to the “Lost pem server” with the key of the “Recovery Server”.
NOTE: Don’t forget to terminate the instance if it’s not a free tier instance to avoid extra cost.
Thank you and next time don't misplace your key 😄