Archive for September, 2018

Validation of Virtual Machines Backup using VMWare vSphere and Veeam Backup

PValidation of Virtual Machines Backup using Veeam & Replication 5.0

The objective of this document is to validate the backup of the Virtual machines at ORGANIZATION NAME production environment that to be restored at Dublin test environment or ORGANIZATION NAME DR environment or test environment.
This validation report focus of Veeam technology that makes the restoration possible and successful of all the application running on Virtual Machines that backed up using Veeam software.
With Veeam backup the successful of any virtual machine backup can be restored to any virtual environment or they can be run directly from the backup disk-image or using Virtual Lab for instance exchange servers or domain controller.
No# Software Version Remarks
1. Symantec Backup Exec 2010 13.0 Rev 2896 (32bit)
2. Veeam Backup & Replication (64bit)

Environment Details:
No# Software Version Remarks
1. VMware vSphere ESX 4.1.0, 260247 (64bit) With Intel-VT / AMD-V in order to un (64bit) Nested VMs
2. VMware vCenter Server 4.1.0 Build 345043 Not necessary as it can restore to ESX directly
3. Dell Powervault TL2000 LTO-3

Backup Validation:
The Lab performed backup evaluation/validation testing at ORGANIZATION NAME DR Site, Bahrain. The testing began by performing backup of Virtual Machines as image backup to disk and transfers them to tapes using Dell PowerVault TL2000 on LTO-3 media. Virtual Machines full backup and incremental backup files *.vbk and *.vib was restored to a LUN at the DR Site where this LUN is presented to Virtual Machine running Veeam software and formatted as NTFS.

Virtual Machines Validated at DR Site:
• Infrastructure
Name Status Host Provisioned Space Used Space
DC001_replica Normal devesx03.esx.local 21.00 GB 16.97 GB
v-Terminal01_replica Normal devesx03.esx.local 22.00 GB 12.39 KB
PRODSQL-01_replica Normal devesx03.esx.local 83.12 GB 12.39 KB
v-OpManager01_replica Normal devesx03.esx.local 59.00 GB 12.43 KB
sFTP01_replica Normal devesx02.esx.local 26.00 GB 11.70 KB
Catalog01_replica Normal devesx02.esx.local 28.00 GB 11.80 KB
ACC01_replica Normal devesx02.esx.local 44.08 GB 12.39 KB
v-Helpdesk01_replica Normal devesx02.esx.local 26.08 GB 11.72 KB
INT01_replica Normal devesx02.esx.local 72.06 GB 11.73 KB
PRODEV-01_replica Normal devesx02.esx.local 194.11 GB 12.41 KB
PRINT01_replica Normal devesx01.esx.local 27.02 GB 11.80 KB
AV001_replica Normal devesx01.esx.local 27.02 GB 11.85 KB
RAD01_replica Normal devesx01.esx.local 26.06 GB 11.75 KB
PRODFILE01_replica Normal devesx01.esx.local 407.03 GB 12.65 KB

• Mail Servers
Name Status Host Provisioned Space Used Space
v-EXCHHUB001_replica Normal devesx03.esx.local 59.00 GB 11.94 KB
IMSS01_replica Normal devesx03.esx.local 27.00 GB 11.88 KB
NMS01_replica Normal devesx01.esx.local 46.01 GB 11.78 KB

• Proxy Servers
Name Status Host Provisioned Space Used Space
PR02_replica Normal devesx02.esx.local 28.00 GB 11.77 KB
ISA02_replica Normal devesx02.esx.local 26.00 GB 12.31 KB
ISA01_replica Normal devesx02.esx.local 26.00 GB 12.31 KB
CSS01_replica Normal devesx02.esx.local 26.00 GB 11.80 KB
PR001_replica Normal devesx01.esx.local 27.02 GB 11.95 KB

• Quercus
Name Status Host Provisioned Space Used Space
v-papp01_replica Normal devesx03.esx.local 67.02 GB 13.24 KB
v-qpweb_replica Normal devesx01.esx.local 38.03 GB 12.49 KB
v-qpdb01_replica Normal devesx01.esx.local 207.03 GB 14.47 KB

Name Status Host Provisioned Space Used Space
v-VLE-DB01_replica Normal devesx01.esx.local 204.00 GB 14.83 KB

Figure 1. Backup / Restoration Process

Restoration Process:
1. Load the LTO-3 Tape into the Tape Library to be Catalogued
2. Restore Veeam Backup Images using Symantec Backup Exec 2010 to a LUN that Shared between Backup Server and Veeam Server
3. Once restoration is completed, Import Backup files in Veeam Software into the Database and catalogued.
4. Once the Veeam imported succeeded, restore process can be initiated to the target ESX.
5. VM will be restored using source configuration such as network portgroup, this can be change manually to facilitate the DR environment / Test environment.
Instant Recovery:
Instant Recovery is a feature with Veeam Backup & Replication which will allow you to run the Backup Image *.vbk directly from backup target. This will mount the VM Directly to the vSphere Infrastructure using NFS storage technology and helps to test the validity of the VM Backup without restoring it to test environment or production environment.
When you run VM directly from the Backup, put in mind that the size of the local C:\ drive where the OS installed of the Veeam Machine will filled up with Veeam Logs. Make sure enough disk space available if you are planning to test the entire backup set.

Replication to DR Site:
OgranizationName/Site Name Bahrain has tested the Virtual Machine integrity and validation by two methods, Backup and restore and Replication to DR Site. Replication jobs configured on all the production VMs to be replicated from production site to DR site. The integrity checked of the replication by pointing the Replica machine to a private portgroup that does not contact with production site when VMs are Powered ON. The private portgroups are created with pNIC to be able to contact other virtual machines located on another ESX host on the same portgroup this portrgroup doesn’t have vLAN ID to avoid address conflicts.
Note: For replication, the target ESX server must be of the same or later version than the source ESX server.
Replicated VMs to DR Site:
Job Name Type State Last Result Target Host / LUN
Rep-CSS01 Replica Stopped Success [] ESX02-LocalStorage
Rep-INT01 Replica Stopped Success []ESX02-VMFS
Rep-PRODFILE01 Replica Stopped Success [] BootLUN02
Rep-ISA02 Replica Stopped Success [] ESX02-LocalStorage
Rep-ISA01 Replica Stopped Success [] ESX02-LocalStorage
Rep-v-Terminal01 Replica Stopped Success []iSCSI-01-SAN02
Rep-PRODSQL-01 Replica Stopped Success []iSCSI-01-SAN02
Rep-sFTP01 Replica Stopped Success []ESX02-VMFS
Rep-v-OpManager01 Replica Stopped Success []iSCSI-01-SAN02
Rep-v-HelpDesk01 Replica Stopped Success [] ESX02-VMFS
Rep-v-rcsiqpdb01 Replica Stopped Success [] BootLUN02
Rep-Catalog01 Replica Stopped Success []ESX02-VMFS
Rep-DC001 Replica Stopped Success []iSCSI-01-SAN02
Rep-PRINT01 Replica Stopped Success [] ESX01
Rep-PR02 Replica Stopped Success [] ESX02-LocalStorage
Rep-IMSS01 Replica Stopped Success []iSCSI-01-SAN02
Rep-v-rcsiqpweb Replica Stopped Success [] ESX01
Rep-HUB001 Replica Stopped Success [ iSCSI-01-SAN02
Rep-v-rcsiqpapp01 Replica Stopped Success [ iSCSI-01-SAN02
Rep-ACC01 Replica Stopped Success [] ESX02-VMFS
Rep-vCenter01 Replica Stopped Success [] BootLUN02
Rep-v-VLE-01 Replica Stopped Warning []VLE-VMs
Rep-PRODEV-01 Replica Stopped Success [] BootLUN02
Rep-v-VLE-DB01 Replica Stopped Success []VLE-VMs
Rep-PR001 Replica Stopped Success [] ESX01
Rep-RAD01 Replica Stopped Success [] ESX01
Rep-AV001 Replica Stopped Success [] ESX01
Rep-NMS01 Replica Stopped Success [] ESX01

DR Site Networking Overview:

The design has been implemented the same as production site, all the vLANs has been stretched over to DR Site. On the ESX Servers, portgroups has been created to match the DR Site. So, once the VM replicated from Production Site to DR Site, the VM will be replicated and will be placed in the same portgroup as it were in production site.

The DMZ network will be used using 2Connect ADSL Line that connected to the DR Site to access the internet. No services are published for inbound / outbound emails nor for published web servers.

This script is used to find all the VMs that a name ending with _replica as this will indicates the VM is replica from production and will change the network name from Production to Private-Prod.
Get-VM -Name *_replica |Get-NetworkAdapter |Set-NetworkAdapter -NetworkName Private-Prod -Confirm:$false
To run this script, connect to the vCenter using VMware PowerCLI to initiate it.
Figure 2. Replication Process

Figure 3. A VMware vCenter view of the DR Site

Virtual Machine Application Configuration and Requirements:
The table below shows the configuration and the requirement of the virtual machines that tested at the ORGANIZATION NAME DR Site. During the testing all the services for each server came up without any issue.

Server / Role OS Version / Application Services
HelpDesk System Windows XP Professional /Helpdesk • ManageEngine ServiceDesk Plus
• ManageEngine ADSelfService Plus
Account System Windows Server 2003, Enterprise Edition / Microsoft Dynamic • MSSQLSERVER
• Microsoft Dynamc
Antivirus Server Windows Server 2008 / Trend Micro • Trend Micro Local Web Classification Server
• Trend Micro Smart Scan Server
• Trend Micro Unauthorized Change Prevention Service
Library System Windows Server 2003, Standard Edition / Heritage • Heritage Data Server v3.4
Domain Controller Windows Server 2008 R2 / Microsoft • Active Directory Domain Services
• DNS Server
• DHCP Server
• Active Directory Web Services
• DFS Replication
Intranet Server Windows Server 2008 / IIS • IIS
Print Server Windows Server 2003, Enterprise Edition • Print Spooler
Archive Server Windows Server 2008 R2 /Symantec • EnterpriseVaultTaskControllerService
• Enterprise Vault Storage Service
• Enterprise Vault Shopping Service
• Enterprise Vault Indexing Service
• Enterprise Vault Admin Service
• Enterprise Vault Directory Service
File Server Windows Server 2008 R2
SQL Server Windows Server 2008 R2 /Microsoft • SQL Server (MSSQLSERVER)
• SQL Server Analysis Services (MSSQLSERVER)
• SQL Server Reporting Services (MSSQLSERVER)
Radius Server Windows Server 2003, Standard Edition • Remote Access Connection Manager
• Internet Authentication Service•
FTP Server Windows Server 2003, Standard Edition /WinSSHD • WinSSHD
Terminal Server Windows Server 2008 R2 • Remote Desktop Services
vCenter Server Windows Server 2008 R2 / VMware • VMWare vCenter Server
Mail Server Windows Server 2008 / Microsoft • Microsoft Exchange Transport Log Search
• Microsoft Exchange Service Host
• Microsoft Exchange Search Indexer
• Microsoft Exchange System Attendant
• Microsoft Exchange Replication Service
• Microsoft Exchange Monitoring
• Microsoft Exchange Mail Submission
• Microsoft Exchange Mailbox Assistants
• Microsoft Exchange Information Store
• Microsoft Exchange Active Directory Topology Service
HUB Server Windows Server 2008 / Microsoft • Microsoft Exchange Transport Log Search
• Microsoft Exchange Transport
• Microsoft Exchange Service Host
• Microsoft Exchange POP3
• Microsoft Exchange Monitoring
• Microsoft Exchange File Distribution
• Microsoft Exchange EdgeSync
SMTP Gateway Windows Server 2003, Enterprise Edition / Trend Micro IMSS • TrendMicro IMSS SMTP
• TrendMicro IMSS Web Console
• TrendMicro IMSS Manager
ISA CSS Windows Server 2003, Standard Edition / Microsoft • Microsoft ISA Server Storage
ISA Server01 Windows Server 2003, Standard Edition / Microsoft • Microsoft ISA Server Storage
• Microsoft ISA Server Job Scheduler
• Microsoft ISA Server Control
• Microsoft Firewall
ISA Server02 Windows Server 2003, Standard Edition / Microsoft • Microsoft ISA Server Storage
• Microsoft ISA Server Job Scheduler
• Microsoft ISA Server Control
• Microsoft Firewall
Staff ISA Windows Server 2003, Enterprise Edition / Microsoft • Microsoft ISA Server Storage
• Microsoft ISA Server Job Scheduler
• Microsoft ISA Server Control
• Microsoft Firewall
Students ISA Windows Server 2003, Standard Edition / Microsoft • Microsoft ISA Server Storage
• Microsoft ISA Server Job Scheduler
• Microsoft ISA Server Control
• Microsoft Firewall
Quercus App Windows Server 2003, Enterprise Edition / Quercus • OracleServiceASDB
• OracleJobSchedulerASDB
• OracleDBConsoleasdb
• OracleCSService
• Oracle10gInfraTNSListener
• Oracle10gInfraProcessManager
• Oracle10gInfraASControl
• Oracle10ASToolsProcessManager
• Oracle10ASToolsASControl
Quercus DB Windows Server 2003, Enterprise Edition / Quercus • OracleServiceTEST
• OracleServiceLIVE
• OracleOraDb11g_home1TNSListener
• OracleDBConsoleTEST
• OracleDBConsoleLIVE
Quercus Web Windows Server 2003, Enterprise Edition / Quercus

VLE Database Linux Debian

VLE Application / Web Linux Debian

Virtual Machine Power on Procedure:

To ensure a proper operation on the Virtual Machines that replicated to the DR Site or imported from Backup their start-up order has to be correct and based on the services dependencies from other servers. Usually in any Microsoft Directory Services (Active Directory), the domain controller and the DNS server has to power on first, if the DNS service is not installed on the Domain Controller, this has to be taken into consideration and DNS has to be power up first followed by the Domain Controller. Domain Controllers in another environment that not backed up by Veeam, also consideration must be taken for Authoritative Restore.
With Veeam Backup & Replication, the VSS integration of Veeam Backup can commit the state of the Virtual Machine and backing / replicate them up without any issue. Thus, will allow us to restore / re-operate the VM with regards to its role whether it’s Domain Controller, Exchange or SQL Server.

Restoring Active Directory / Domain Controller using Veeam Backup & Replication:
Below is the procedure to restore Domain Controller after successfully backed up using veeam software.
1. Restore the Virtual Machine to the datastore that accessible by ESX host where the VM will run.
2. Once restoration is successful, make sure before you Power On the VM to put it in a private PortGroup. This will avoid conflicting the machine name / IP address with the production VM.
3. First boot it goes into safe mode Non-Authoritative Restore it’s by default
4. Second boot it goes into Directory Restore Mode. This will ask you to provide the local username and password of the domain controller / during DS setup:
a. Username: Administrator
b. PWD: password
5. Third boot it goes again into Directory Restore Mode. If this is the case here’s the trick. It goes into DRM because it doesn’t uncheck the Safe Mode and Active Directory Repair in the Boot Option under the MSConfig System Configuration.
6. Run MSConfig and unchecked the Safe Mode boot and reboot the machine
7. At this step, the DC starts normally and it gives the below indications as the restoration successfully done.
Below event IDs indicates the restoration of the Domain Controller succeeded.
Event ID Source Description
1004 DFSR The DFS Replication Service Started
6102 DFSR The DFS Replication Service successfully registered with WMI
1206 DFSR The DFS Replication Service successfully contacted Domain Controller “Name”
1000 ActiveDirectory_DomainService Microsoft Active Directory Domain Service Startup Completes
1394 ActiveDirectory_DomainService All problem preventing updates to the Active Directory Domain Services database has been cleared.

Note: This procedure tested on an Additional Domain Controller that backed up from production site ADC. If this procedure will be tested on a private environment where the Primary Domain Controller doesn’t exist or the Primary Domain Controller is Physical and doesn’t want to be interrupted it, then Sezing the FSMO Roles is required to have full function of Active Directory Service.

The following table shows the servers and services dependencies:

No# Servers / Role Dependencies
1. Domain Controller • DNS Service / Server
2. Great Plains • Domain Controller for login
3. HelpDesk • Domain Controller for Services to start & Login
4. Heritage • Domain Controller for Service to start
5. Symantec Archive • Domain Controller
• SQL Server
6. SQL Server • Domain Controller
7. Radius Server • Domain Controller
8. sFTP Server N/A
9. Exchange HUB • Domain Controller
11. Exchange Mailbox • Domain Controller – GC
• Exchange HUB Server
12. SMTP Gateway • Exchange HUB Server – For Inbound / Outbound
13. Quercus App • Domain Controller
• Quercus DB Server
14. Quercus DB N/A
15. Quercus Web • Quercus App
• Quercus DB
16. VLE App • Domain Controller – LDAP Authentication
• VLE DB Server
17. VLE DB N/A

Virtual Machine Passwords
The below table lists all the passwords of the virtual machines that were tested at ORGANIZATION NAME DR site. These passwords are local administrator as for some application will be used such as for Domain Controller
Virtual Machine Password
DC001 DSRM: password
Admin: password

Leave a comment