How to deploy 2 node direct connect vSAN cluster.

Begining with v6.1 VMware supports direct connect for ESXi hosts when using vSAN. You just need to simply cross-connect your ESXi hosts using either cross UTP cable or 10 Gb SFP+ cable. This way you can save that expensive 10 Gb switch you would normally use for you vSAN and vMotion network. Please, do not proceed with the article before you have tested your direct connection! Ensure to use IP addresses that reflect your environment.

You will need to download the VMware vSAN Witness Appliance. Please note that I am using VMware ESXi 6.5 U1 and vSAN 6.6. In this example, I already have a fully functional vSAN cluster with my vCenter running. Here is my network configuration:

ESXi host 1 on the remote site:

  • vmk0: (10.1.1.8/24) VLAN1 – Management network 1 Gbps through MPLS
  • vmk1: (10.1.10.8/24) VLAN10 – vSAN network 10 Gbps SFP+ directly connected to host 2
  • vmk2: (10.1.11.8/24) VLAN11 – vMotion network 10 Gbps SFP+ directly connected to host 2
  • vmk3: (10.1.11.108/24) VLAN11 – vMotion network 10 Gbps SFP+ directly connected to host 2
  • vmk4: (10.1.12.8/24) VLAN12 – Witness traffic 1 Gbps through MPLS

ESXi host 2 on the remote site:

  • vmk0: (10.1.1.9/24) VLAN1 – Management network 1 Gbps through MPLS
  • vmk1: (10.1.10.9/24) VLAN10 – vSAN network 10 Gbps SFP+ directly connected to host 1
  • vmk2: (10.1.11.9/24) VLAN11 – vMotion network 10 Gbps SFP+ directly connected to host 1
  • vmk3: (10.1.11.109/24) VLAN11 – vMotion network 10 Gbps SFP+ directly connected to host 1
  • vmk4: (10.1.12.9/24) VLAN12 – Witness traffic 1 Gbps through MPLS

Witness ESXi host appliance on my main site:

  • vmk0: (10.0.1.10/24) VLAN1 – Management network
  • vmk1: (10.0.12.10) VLAN12 – Witness traffic (witnessPg)

First, we need to configure ESXI hosts to use vmk4 as witness traffic. Open SSH connection to each of your hosts and type the following command:

esxcli vsan network ip add -i vmk4 -T=witness

To confirm that vmk4 is used for witness traffic, use the following command:

esxcli vsan network list

You should get output like this:

Interface
   VmkNic Name: vmk4
   IP Protocol: IP
   Interface UUID: 8cf3ec57-c9ea-148b-56e1-a0369f56dcc0
   Agent Group Multicast Address: 224.2.3.4
   Agent Group IPv6 Multicast Address: ff19::2:3:4
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: 224.1.2.3
   Master Group IPv6 Multicast Address: ff19::1:2:3
   Master Group Multicast Port: 12345
   Host Unicast Channel Bound Port: 12321
   Multicast TTL: 5
   Traffic Type: witness

Interface
   VmkNic Name: vmk1
   IP Protocol: IP
   Interface UUID: 6df3ec57-4fb6-5722-da3d-a0369f56dcc0
   Agent Group Multicast Address: 224.2.3.4
   Agent Group IPv6 Multicast Address: ff19::2:3:4
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: 224.1.2.3
   Master Group IPv6 Multicast Address: ff19::1:2:3
   Master Group Multicast Port: 12345
   Host Unicast Channel Bound Port: 12321
   Multicast TTL: 5
   Traffic Type: vsan

Now we need to add static route. On each of your ESXi hosts on the remote site add the following:

esxcli network ip route ipv4 add --gateway 10.1.12.254 --network 10.0.12.0/24

And on your witness host add the following:

esxcli network ip route ipv4 add -gateway 10.0.12.254 network 10.1.12.0/24

Now we are ready to enable vSAN on our remote site.

1. Select Configure two host vSAN clister and click Next

2. On the Network validation dialog click Next.

3. On the Claim disks dialog, select disks you want to use for your vSAN and click Next.

4. Now we will select our ESXi vSAN Witness Appliance. Click Next

5. Select already predefined disks for vSAN from your witness appliance and click Next.

6. On the Ready to complet dialog, check that all the information is correct and click Finish.

Once vSAN creation tasks are complete, confirm that your vSAN is functioning properly using Configuration Assist that is located under Monitor tab of your cluster.

One Reply to “How to deploy 2 node direct connect vSAN cluster.”

Leave a Reply

Your email address will not be published. Required fields are marked *