Tuesday, December 13, 2011

Snapshot removal failed: unable to access file since it is locked

Current Running environment is ESX 4.1 and Veeam 5 (running on a VM) for backups.

The current backup job never completed successfully, it simply stopped mid backup. I couldn't kill the job or do anything so I rebooted the Veeam backup server.

To finish the clean up I selected the VM that was being processed in the stuck backup and told the system to remove the snapshot.

An error message was received and the snapshot was never removed. The error message was "unable to access file since it is locked"

SOLVED: unable to access file unspecified filename since it is locked
SOLVED: unable to access file since it is locked

This error message is so generic, and appeared to me at least in two different situations.  First time, trying to remove a snapshot of a VM, and second time, trying to power on a VM.  This is the second time now where I cannot power on the VDR virtual machine.  The message "Unable to access file since it is locked appears in the Recent tasks panel.

Since the filename is unspecified, it makes it hard to figure out what the issue is. Why did it happen? Well, first I have to tell you that something went wrong in a backup process of one of my VMs, specifically the Virtual Center Server (in my case, this is a VM). It could be ANY VM, but in this case is this one. The last time, the same issue happened with a different VM, but anyway, the problem is the same.

I figured out that the job failed, and It couldn't remove the snapshot. (VDR appliance creates a snapshot, and then copies the contents to the destination; after that, the snapshot is removed). But something went wrong, and the snapshot was not removed. It caused all future jobs also failed. I tried manually creating a snapshot for the failed VM, and then remove it, to force it to "Delete All" the unused snapshots, but the procedure failed, giving me the same error: "unable to access file since it is locked" ... mmmm, who is locking and which file ?????

I tried also moving the VM to another ESX in the cluster; restarted vmware management services; restarted the VM; restarted the ESX host itself, but no luck.

Ok, you got it, VDR is locking it .... but which file ???? I searched hours in google and Vmware KB, but nothing ... I opened a ticked with VMware. It took one day for them to call me back, just to acknowledge the ticket. It took another 3 days to have an email from them asking me for uploading the log files (I already did it at the time I opened the ticket !).It took another 3 days for them to call me, but I was busy and I couldn't work with them, after three more days, I called them again, but they told me they will call me back ... GRRR. I hate VMware support procedures and times.

Don't worry, I solved it by myself, and let me tell you how:

1) Shutdown the VDR appliance. It will free up the locked files in your VMs that were not sucessfuly backed up.

2) Create manually a snapshot in every VM with the problem, then "Delete all" snapshots will work !, you won't get that error message again.

3) "Try" to power on the VDR (VMware Data Recovery) appliance... Oh, no, the same message again! And now, I cannot power up the Virtual Machine !

4) I found the VDR "mounts" the hard disks of the VMs it is creating the backups, so, go to the VDR, and in commands "Edit settings".

By default, the VDR has only one hard disk, but mine shows three: those two extra hard disks corresponds to the Virtual Machine the backup failed!

Look at the hard disk description path for the first hard disk drive, and disk mode independent checkbox is not checked.

Look at the extra hard disks added to the VDR, see the hard disk description path (it corresponds to the VM that was in progress of backup). It also has the independent checkbox checked.

Select one by one the extra hard disks and click "Remove". Be very careful here, selecting just remove, and DO NOT delete files from disk.

Also, confirm that you selected the right hard disks ! If you make a mistake, just hit Cancel and do it again.

5) Verify that you have the single and right hard disk in place.

6) Power on the (VMware Data Recovery) appliance. Problem solved !

No comments:

Post a Comment