Maintenance Task Fails On VDP When Connected To Data Domain

There are many instances where the maintenance task fails on VDP. This article is in specific to VDP when integrated with data domain and moreover when the DDoS version is 6.1 and above.

The checkpoint and HFS tasks were completing fine without issues:

# dumpmaintlogs --types=cp | grep "<4"

2018/03/19-12:01:04.44235 {0.0} <4301> completed checkpoint maintenance
2018/03/19-12:04:17.71935 {0.0} <4300> starting scheduled checkpoint maintenance
2018/03/19-12:04:40.40012 {0.0} <4301> completed checkpoint maintenance

# dumpmaintlogs --types=hfscheck | grep "<4"

2018/03/18-12:00:59.49574 {0.0} <4002> starting scheduled hfscheck
2018/03/18-12:04:11.83316 {0.0} <4003> completed hfscheck of cp.20180318120037
2018/03/19-12:01:04.49357 {0.0} <4002> starting scheduled hfscheck
2018/03/19-12:04:16.59187 {0.0} <4003> completed hfscheck of cp.20180319120042

Garbage collection task was the one that was failing:

# dumpmaintlogs --types=gc --days=30 | grep "<4"

2018/03/18-12:00:22.29852 {0.0} <4200> starting scheduled garbage collection
2018/03/18-12:00:36.77421 {0.0} <4202> failed garbage collection with error MSG_ERR_DDR_ERROR
2018/03/19-12:00:23.91138 {0.0} <4200> starting scheduled garbage collection
2018/03/19-12:00:41.77701 {0.0} <4202> failed garbage collection with error MSG_ERR_DDR_ERROR

From ddrmaint.log located under /usr/local/avamar/var/ddrmaintlogs had the following entry:

Mar 18 12:00:31 VDP01 ddrmaint.bin[14667]: Error: gc-finish::remove_unwanted_checkpoints: Failed to retrieve snapshot checkpoints: LSU: avamar-1488469814 ddr: data-domain.home.local(1), DDR result code: 5009, desc: I/O error

Mar 18 12:00:34 VDP01 ddrmaint.bin[14667]: Info: gc-finish:[phase 4] Completed garbage collection for data-domain.home.local(1), DDR result code: 0, desc: Error not set

Mar 19 12:00:35 VDP01 ddrmaint.bin[13409]: Error: gc-finish::remove_unwanted_checkpoints: Failed to retrieve snapshot checkpoints: LSU: avamar-1488469814 ddr: data-domain.home.local(1), DDR result code: 5009, desc: I/O error

Mar 19 12:00:39 VDP01 ddrmaint.bin[13409]: Info: gc-finish:[phase 4] Completed garbage collection for data-domain.home.local(1), DDR result code: 0, desc: Error not set

It was basically failing to retrieve checkpoint list from the data domain.
Also, the get checkpoint list was failing:

Mar 20 11:16:50 VDP01 ddrmaint.bin[27852]: Error: cplist::body - auto checkpoint list failed result code: 0

Mar 20 11:16:50 VDP01 ddrmaint.bin[27852]: Error: <4750>Datadomain get checkpoint list operation failed.

Mar 20 11:17:50 VDP01 ddrmaint.bin[28021]: Error: cplist::execute_cplist: Failed to retrieve snapshot checkpoints from LSU: avamar-1488469814, ddr: data-domain.home.local(1), DDR result code: 5009, desc: I/O error

Mar 20 11:17:50 VDP01 ddrmaint.bin[28021]: Error: cplist::body - auto checkpoint list failed result code: 0

Mar 20 11:17:50 VDP01 ddrmaint.bin[28021]: Error: <4750>Datadomain get checkpoint list operation failed.

From the mTree LSU of this VDP Server, we noticed that the checkpoints were not expired:

# snapshot list mtree /data/col1/avamar-1488469814

Snapshot Information for MTree: /data/col1/avamar-1488469814
----------------------------------------------
Name                Pre-Comp (GiB)   Create Date         Retain Until   Status
-----------------   --------------   -----------------   ------------   ------
cp.20171220090039         128533.9   Dec 20 2017 09:00
cp.20171220090418         128543.0   Dec 20 2017 09:04
cp.20171221090040         131703.8   Dec 21 2017 09:00
cp.20171221090415         131712.9   Dec 21 2017 09:04
.
cp.20180318120414         161983.7   Mar 18 2018 12:04
cp.20180319120042         162263.9   Mar 19 2018 12:01
cp.20180319120418         162273.7   Mar 19 2018 12:04
cur.1515764908            125477.9   Jan 12 2018 13:49
-----------------   --------------   -----------------   ------------   ------
Snapshot Summary
-------------------
Total:          177
Not expired:    177
Expired:          0

Due to this, all the recent checkpoints on VDP were invalid:

# cplist

cp.20180228120038 Wed Feb 28 12:00:38 2018 invalid --- ---  nodes   1/1 stripes     76
.
cp.20180318120414 Sun Mar 18 12:04:14 2018 invalid --- ---  nodes   1/1 stripes     76
cp.20180319120042 Mon Mar 19 12:00:42 2018 invalid --- ---  nodes   1/1 stripes     76
cp.20180319120418 Mon Mar 19 12:04:18 2018 invalid --- ---  nodes   1/1 stripes     76

The case here is the VDP version was 6.1.x and the data domain OS version was 6.1

# ddrmaint read-ddr-info --format=full

====================== Read-DDR-Info ======================
System name        : xxx.xxxx.xxxx
System ID          : Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx4
DDBoost user       : ddboost
System index       : 1
Replication        : True
CP Backup          : True
Model number       : DDxxx
Serialno           : Cxxxxxxxx
DDOS version       : 6.1.0.21-579789
System attached    : 1970-01-01 00:00:00 (0)
System max streams : 16

6.1 DD OS version is not supported for VDP 6.1.x. 6.0.x is the last DD OS version supported for VDP.

So if your DD OS is on 6.1.x then the choice would be to:
> Migrate the VDP to Avamar Virtual Edition (Recommended)
> Rollback DD OS to 6.0.x

Hope this helps!

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

In this part, we will replicate VDP backup data to Avamar and then migrate the jobs. First, to replicate the data from VDP to Avamar, we create a regular replication job. Let's have a look at this.

Connect to the VDP appliance from the web client and click the Restore tab. To give you a brief overview, here is the restore points for my VM-A

Click the Replication tab and click Replication job actions. and click Create New Replication Job.

You will be presented with the below wizard. I want to replicate my current Image backups, so the first option and click Next.

I will replicate the individual client, and select VM-A and then click Next.

Select which backup schedule you would like to replicate. This is as per your need and then click Next.

Enter the AVE FQDN/IP as the hostname, the port for replication is 29000. The username to be used is repluser and the password for repluser is the one you setup during the Avamar configuration. Click Verify Authentication. Once the connection test is completed successfully, click Next.

Select how frequently you need to replicate. This does not really matter here, because it is going to be one time replication, because post the migration source VDP will be in a unusable state. Click Next.

Select a retention policy for the replicated backup as per your requirement and click Next.

Provide a name to this replication job and then click Next and then Finish the wizard.

Once the job is created, select the job and click Replicate Now to begin the task.

Once the replication is completed, you can see the replicated data in the Avamar end.

You will still notice there is a red X for Backup Jobs as the jobs are not transferred. It is only the VDP backup data that is replicated to Avamar.

Next we will begin migrating the jobs. To do this, open a browser and go to:

https://vdp-ip/fqdn:8543/vdp-configure

Once on this page, click Avamar Migration and click Migrate.

Check the Terms and conditions to proceed further and click Next.

Provide the AVE FQDN, the root credentials would be for Avamar root and not the OS root password. Click Verify Authentication.

Once the authentication succeeds, click OK and then click Next.

You can select the backup jobs you need to migrate here. 

You can switch the tab to Backup Verification and select the required verification jobs and then click Next.

A validation will be performed and if everything is successful, click Next.

You will be prompted for migration. Click Yes to begin the migration.

You can then monitor the migration progress.

Then you can review the migration status. Click Close.

Now the status is, Jobs have been migrated and the appliance is disabled.

The same is seen in the VDP page in web client as well.

On the Avamar end, you can see the jobs have now been migrated successfully.

So, in the end, the VDP would be used only for performing restore operations with the existing data. And once new valid backups are performed by Avamar, you can discard VDP completely.

Hope this series was helpful. Let me know your thoughts in the comments.

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

Previously we configured AVE services. Next we need to download the Avamar Client, install it and then configure a vCenter and deploy a proxy. Let's look at this step by step.

Let's start off by downloading the Avamar Client. To get this, go to the below URL:

http://avamar-fqdn

You should see the below screen:

Click Downloads and you should see the below screen. First download the jre.exe and have it installed on the machine where you plan to run the Avamar Client. Then download AvamarConsoleMultipleWindows.exe

Run the installer, it's a pretty straightforward wizard to follow.

Once the client installation is done, open the Avamar client and you will be presented with the below login screen. The Server would be FQDN or IP of your AVE. The Username would be MCUser and the Password for the same with / Domain.

Once the login is successful, we will see the below Avamar management UI.

Notice there are no backup jobs, hence a red X for Backup Jobs tab. Now, we will have to configure vCenter to the AVE. Click Administration from the management UI and you will see a new window:

The vCenter client, should always be added to the / domain. So select the ave and then click Actions > Account Management and then click New Client.

You will see the following section. Select the Client Type as VMware vCenter. Provide the vCenter FQDN or IP for Client name. For username provide an admin user defined on the vCenter level. Disable vCenter Certificate parameter before performing this. You can find this step in this link here.
Click OK and the vCenter should be registered successfully to the AVE machine.

You will notice the vcenter-prod (my vCenter) under the ave domain added as a client.

Next, you will have to deploy a proxy, without this your migration pre-check will fail. In the same window, click VMware and Proxy Deployment Manager.

Here you can input your required setting and create a recommendation.

Click Apply and then click New Proxy and you will be presented with the below window. Enter a name for the proxy. The domain would be your /vcenter-fqdn/VirtualMachines. Provide an IP for the proxy, datastore and a network. Please note the proxy should be resolvable before performing the deployment. Click Save.

The proxy deployment will begin and you can monitor the progress here as well as the vCenter Recent Tasks pane.

This should be it. Until now, you have deployed AVE ovf, configured it, downloaded the client and configured it to a vCenter and added a proxy.

Next we will look at the migration of data and jobs from VDP to AVE.

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

In part 1 we saw how to deploy and setup the basic Avamar 7.2 machine. In this article, we will have the appliance configured for usage.

Open a browser and type in:

https://avamar-server-ip/fqdn:7543/avi/avigui.html

This should bring up the below screen and you can login with root as the user and changeme as the password.

Post the login you will see the below page. Here on the top left corner you will see a lock icon. Click this and enter the password as Supp0rtHarV1.  Once unlocked, click Install to begin the package installation.

The initialization will take a few minutes to complete and once done you will be presented with the below window. Here, you will need to fill all the fields with the red exclamation marks.

Under Server Settings, the Avamar Server Address should be the hostname and select an appropriate time zone.

The number of storage node I will be using is 0, just like how my VDP is setup. A single node server. This is the node you see when you run status.dpn

Fill in the remaining fields and click Continue on the bottom right. This should begin the package installation like seen below:

Once the configuration is complete you will see the below task. At this point, your AVE setup is complete.

Now you can SSH into the AVE machine with admin credentials and run the below command to verify system and service status.

admin@ave:~/>: dpnctl status
Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: suspended.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.

admin@ave:~/>: status.dpn

Tue Apr 18 00:49:01 IST 2017  [AVE] Mon Apr 17 19:19:01 2017 UTC (Initialized Mon Apr 17 17:51:04 2017 UTC)

Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk

0.0   10.109.10.169   7.2.1-32  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.49 5209    33104   0.0%   0%(onl:8  )  0%(onl:8  )  0%(onl:8  )

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1492451464@00:50:56:9A:52:F6

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

Last checkpoint: cp.20170417180550 finished Mon Apr 17 23:36:09 2017 after 00m 19s (OK)

No GC yet

No hfscheck yet

Maintenance windows scheduler capacity profile is active.

  WARNING: Scheduler is WAITING TO START until Wed Apr 19 20:30:00 2017 IST.

  Next backup window start time: Thu Apr 20 20:00:00 2017 IST

  Next maintenance window start time: Thu Apr 20 08:00:00 2017 IST

That should be it with the configuration part. Next, we will look into how to Configure AVE to vCenter Server. 

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

Since VMware announced, the end of vSphere Data Protection, there is a choice to migrate existing deployment to EMC Avamar. More about EOA can be found in this link here.

In this article, we will be looking at deploying Avamar Virtual Edition 7.2. You can go ahead and download the required version of AVE from EMC download portal. The version I will be using is Avamar 7.2.1.

Login to Web Client or vSphere Client, Select the ESXi host where you want to deploy your AVE and select File > Deploy OVF. Browse the location for the AVE download and add the file. Click Next.

Review the details of the OVF template and click Next.

Accept the EULA and click Next.

Provide a name for this AVE virtual machine. Click Next.

If available and required select a resource pool in which you want to place this VM. Click Next.

Select the datastore where you want to deploy this. Remember the vmdk bundled with AVE is just the OVF, the data drives are configured later just like a VDP appliance. Click Next.

Select a disk provisioning type. Thick provision is recommended. Click Next.

Select a network where this AVE should be connected to. Click Next.

Review the changes and click Finish. Do not check Power On after deployment, because there are couple of steps to be done once the OVF deployment is completed.

Just like VDP, the AVE comes with 4 supported available backup storage. You can refer the below table to size your AVE accordingly.

Once you choose the deployment type, you will need to refer the below table to plan the drive sizes. Just like in VDP a 512GB deployment will have 3 drives of 256 GB each. The additional space is for the checkpoint maintenance overhead.

So the rule goes like:
Total size = GSAN capacity + 1/2 of GSAN capacity.

GSAN capacity would be the actual space for storing backup data.

So go ahead and add three disks manually (Depending on your AVE configuration) to this VM. Only Thick Provisioning is supported for AVE. I will be using Thin because of space constraints.

Once the drives are added, power on the AVE virtual machine.

The default login is root and changeme

First, we will have to configure network settings for the AVE machine. Post the login to AVE from VM console, run yast2 to begin the networking configuration. You will see a similar interface:

Select Network Devices and then Network Services to begin the network configuration wizard and you should see something similar:

You will need to Set IP Configuration in Overview, Hostname / DNS settings and Gateway under Routing.

Once the appliance is configured with network, restart the guest and then verify the network by ping and nslookup. If this works good proceed to Part 2 in the below link.

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2