Backup job failed. Cannot create a shadow copy of the volumes containing Exchange writer’s data.

Been facing the Exchange VSS Writer issue  when I backup my Exchange Server 2007 with Veeam Backup & Replication for quite sometime, and the only way to clear this out is by rebooting the Exchange Server or sometime it will pass through from the second or third job retry.

Last Friday 16th Nov, 2012 my Veeam Backup jobs suppose to run as Full Active Backup as per my configuration, Backup went fine for all the Backup Jobs, except the Backup Job which contains the Exchange Mailbox Server, it gives the usual error;

11/16/2012 8:43:20 PM :: Unable to release guest. Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{8ea7190d-337c-448f-b264-3401303b586b}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]

I have reboot it, and retry the job but no joy it didn’t help. I rebooted second and third time, but no joy The error is persistent.

I searched and searched for a solution but the usual result showed up, either reboot the server to clear the VSS writer Timed Out or restart the Microsoft Information Store which will clear the VSS Timed Out. I have done, both but without any luck.

I fed up from troubleshooting during my holiday and I left home for fishing. While I’m at the sea, my mind triggered out that why don’t you Exclude the C:\Drive of the Exchange VM and select only those drives which contains the Exchange Database?

Hummm, it seems it’s a brilliant idea. As soon as I reached home, I immideltly, logged in remotely again and I Excluded the C:\Drive VMDK SCSI (0:0) from the job and selected only those which contains the Exchange Database, “the disks are vRDM”;


SCSI (0:1)
SCSI (0:2)
SCSI (0:3)
SCSI (0:4)
SCSI (0:5)
SCSI (0:6)
SCSI (0:8)
SCSI (0:9)
SCSI (0:10)
SCSI (0:11)
SCSI (0:12)
SCSI (0:13)
SCSI (0:14)
SCSI (0:15)
SCSI (1:0)
SCSI (1:1)
SCSI (1:2)
SCSI (1:3)
SCSI (1:4)
SCSI (1:5)
SCSI (1:6)
SCSI (1:8)


Imagine, the Job passed the Snapshot Process so fast and I’m surprised that the job is started reading :) and the Exchange Database has been put in Backup Mode and the backup Speed is a bit faster 1.6 TB finished in 4 Hours :)

Wondering:
Why Excluding the SCSI (0:0) which is the C:\SystemDrive of the Exchange 2007 Virtual Machine and Including only the vRDM SCSI Drives the process will pass successfully without VSS Error or Timed Out?

What is the restoration impact of Backing up only the those drives where the Exchange Database resides without the the C:\Drive SCSI (0:0).

Will it be possible configuring another job against the Exchange VM to backup only the C:\Drive SCSI (0:0) without the rest of other drives? And when I want to restore it, I have to restore the Job which contains the C:\Drives and followed by the other Job which backed up the Database Drives?

Update:
I can confirm that, I have Selected All Disks to process under the Disk Exclusions and the job Failed. But I have amended the Selection and I included SCSI (0:0) which contains the Virtual Machine System Drive and the backup successful.
I think the reason why it didn’t work when the All Disks radio button is selected but it does when the SCSI Disks are selected including the SCSI (0:0) which includes the System Drive.

The Virtual Machine is limited to 60 Virtual SCSI Controllers/Targets and when Veeam Backup is initiating job to process the Backup, it creates a Loop within the SCSI Controllers selected in the VM. If any drives / SCSI Controller is presented, it will be added for the Backup. While Exchange is a bit sensitive to wait for the entire process of Veeam Backup Job to go from SCSI (0:0) till SCSI (3:15) which is the last SCSI Controller, it failed because VSS Snapshot it didn’t pass within the time frame that specified by Veeam.
But by selecting the correct SCSI Controller/Disks which are presented within the Virtual Machine, I think Veeam Intelligent enough to pickup only those SCSI Disks which are presented/Selected.

Scheduling Veeam Backup Jobs for Daily Incremental and Weekly Full using PowerShell

As every Veeam Backup & Replication User/Administrator can see that only one schedule can be configured for each job, which will includes First Full backup, Active Full backup and Daily Increments.

Daily incremental takes less time than the full backup and if you have configured the jobs to run each after another with minimum 20 – 30 minutes cap. In the same job there you are restricted to select either Synthetic Backup with Transforming the Backups into one Full Backup *.VBK file chain of all previous incremental or you have to select Active Full.

Synthetic takes less time to finish incremental, but if you don’t have good CPU/Memory resources and Disk I/O, the transformation process could take time or you might end up with full VM freezing.

But if you are in Active Backup and selected Weekly Active Full Backup on Friday let assume, which again if Friday trigger it will run as Full in the same time that the daily incremental runs which will take longer to finish and in the same time the second job will start before the first job finish and will end up having whole jobs running and will slow down the read and write performance of the disks and Veeam server.

Since Friday is the day off and the servers/systems are being utilized very less, I have the whole day to run my backup, but can I can configure the Active Full Backup to run on different day other than the one which is configured in the Job Scheduler of the Backup Job?

Currently there is no way to configure it from the Veeam GUI and it would be very fantastic feature if Veeam.com would consider to add a separate schedule for the Active Full Backup.

The only way to make sure you run your jobs without overlapping with other jobs and will run into performance issue; with very simple Veeam PowerShell script and a Windows batch script to call the PS1 script will make sure all jobs runs as per your needs.
In my environment, I’m backing up 44 VMs those VMs divided into 5 Jobs; I have created 5 scripts for each job and I disabled the Backup Job Schedule;

Critical VMs-1

Critical VMs-2

Critical VMs-3

Critical VMs-4

Critical VMs-5

I have created two scripts for each job as follows;

PowerShell Script “Critical VMs-1.PS1”

Add-PSSnapIn -Name VeeamPSSnapIn -ErrorAction SilentlyContinue
Get-VBRJob -Name "Critical VMs-1" | Start-VBRJob

This Power Shell Script, will find the job name “Critical VMs-1” via the Get-VBRJob command and will start the Backup job via Start-VBRJob.

And another Windows Batch Script to call the Critical VMs-1.PS1 PowerShell script and invoke it into the PowerShell;

Windows Batch Script “RunCritical-1.bat”

PowerShell.exe G:\Scripts\Critical VMs-1.PS1

I have repeated the same scripts for each job. Then in the task Scheduler in windows server 2008 R2, I configured two schedules for each backup job, so one schedule will run the same script RunCritical-1.bat from Sunday to Thursday as daily incremental and the same script will run on Friday on different time than the daily schedule, this will make sure that I have enough time window to run my backup jobs from early morning on Friday.  And since the Veeam Backup Job already configured as Active Full on Friday, it will automatically triggered and run as Full.

Hope it helps;

Integration Veeam Backup with Symantec Backup Exec 2010

Integration Veeam Backup with Symantec Backup Exec 2010

Introduction:

This document highlights some necessary steps required to setup a successful Veeam Backup and Symantec Backup Exec 2010 for tape integration and off-site storage.

Steps to a successful integration:

Veeam B& R v6.0.0.181 is installed on a Virtual Machine running guest operating system Windows 2008 R2 64bit with 8 GB of Ram and 2 Virtual vCPU Sockets and 4 Cores for each socket. The Veeam resources it depends on the environments and workload that put on the backup during backup operation.

All backup repositories’ are mapped through iSCSI Initiator with MPIO. The sizing of the repositories also it depends on each environment and how much data needs to be restored from disk and for how long. Again, this retention period plays a big role in the repository storage and how big your storage is.

In my environments, I have multiple targets;

No	Repository Target	Size
1.	IBM iSCSI – DS3500	2.0 TB
2.	IBM iSCSI – DS3500	745 GB
3.	EMC iSCSI – AX4-5i	800 GB
4.	OpenFiler iSCSI 2.3	1.4 TB
5.	IBM iSCSI – DS3500	2.0 TB
	Total	6.9 TB

Veeam Backup Configuration:

All the above repositories are added to the Veeam Backup & Replication v.6.0.0.181. Backup jobs configured for each VM to one of the above repositories.

Due to lack of disk space in the repositories and which limits the number of days  ofbacked up VMs should remain on disk, all the backup jobs are configured as follows;

Categorizing the backup jobs based on roles of the Virtual Machines helps a lot from backing up static Virtual Machines which doesn’t change very often.

No	Backup Category	Descriptions
1.	Critical Virtual Machines	Virtual machines are changing dynamically, such as Mail Server, file Server, Archive Server, Sharepoint, etc..
2.	Non-Critical Virtual Machines	Virtual machines are not changing on a daily basis, such as AntiVirus, Proxy Servers, WSUS, Deployment Serves

Critical Virtual Machines Job Settings:

Since those Virtual Machines are changing dynamically and daily, weekly and monthly backup requires for them backup job settings are configured based on the disk space and how long we can restore from disk.

Restore points to keep on disk: = 5

Deleted VMs data retention period: = 5

Backup Mode set as Incremental and Weekly Friday Active Full Backup. And schedule backup Sunday through Friday.

This settings result 1 Full Backup on Friday .vbk file and Sunday through Thursday is incremental which will gives 5 .vibs files.

Note: Since the Deleted VMs data retention period set to 5, CheckDataValue() when it reaches to 5 days, the full last backup chain will be deleted from disk and will have a new .vib file changed with last .vbk file.

Example:

Full Backup runs on 27^th April, 2012 it will have a chain of Sunday through Thursday.

Week#1

No	Backup Date	Day	Backup Mode
1.	04/27/2012  02:43 VM001012012-04-27T143033.vbk	Friday	Full
2.	04/29/2012  02:37 VM001012012-04-29T143033.vib	Sunday	Incremental
3.	04/30/2012  09:42 VM001012012-04-30T093346.vib	Monday	Incremental
4.	05/01/2012  02:37 VM001012012-05-01T143039.vib	Tuesday	Incremental
5.	05/02/2012  02:37 VM001012012-05-02T143047.vib	Wednesday	Incremental
6.	05/03/2012  02:38 VM001012012-05-03T143029.vib	Thursday	Incremental

Week #2

No	Backup Date	Day	Backup Mode
1.	05/05/2012  11:18 VM001012012-05-04T143026.vbk	Friday	Full
2.	05/06/2012  02:36 VM001012012-05-06T143052.vib	Sunday	Incremental
3.	05/07/2012  02:36 VM001012012-05-07T143038.vib	Monday	Incremental
4.	05/08/2012  02:36 VM001012012-05-06T143052.vib	Tuesday	Incremental
5.	CheckDateValue()
6.

When CheckDateValue() reach next week Wednesday, the previous backup chain will be deleted entirely.

Non-Critical Virtual Machines:

The same applies to the non-critical virtual machines backup jobs, except those backup jobs are set to 1 for both values, Restore points to keep on disk:= 1 && Deleted VMs data retention period:= 1.

This will allow only one restore point on disk.

Symantec Backup Exec 2010 Configuration:

All traditional backup software, the well-known backup methods are Full, Incremental or Differential. With Symantec Backup Exec 2010 and Veeam Integration there is a caution in here to select which backup methods to use when you Backup the Veeam Images *.VBK or *.VIB files.

In Symantec when Full Backup is configured to backup Veeam Images it will Backup everything on the drive and will reset the file Archive Bits.

When Veeam Backup runs again and created a VIB file or VBK file and again Symantec Runs as Incremental against veeam files, only the last changed files will be backed up.

But, when Symantec Full backup runs against Veeam, it will backup everything on the drive, means all Full *.VBK Files along with all *.VIB files. This increase the possibility of taking long time to finish the backup as well as doubling the backup twice.

The best option in here to backup Veeam Image using Symantec is to set all the backup methods as Incremental with Reset-Archive-Bit including Daily, weekly and Monthly Backup.

Also, Since Veeam Backup & Replication doesn’t have the option to set a monthly Backup, the last Wednesday of Last Week in the month can be considered as monthly backup and again the Backup Method has to be Copy otherwise backup will not backup anything to tapes because the Archive-Bit has been reset already.

In my example, Wednesday is my CheckDateValue() that configured on the Veeam backup jobs. So, all previous chain will be deleted.