**** UPDATE **** A disclaimer is important here before doing the below. This should ONLY be used for migrations on a temporary basis! Long term there is no scalability or failover capability. *****
NOTE: For the below to work, the non-nutanix hosts and nutanix hosts must be on the same layer 2 network.
With ESXi, NFS datastores are recognized discretely by the IP address of the server they connect to in order to mount the volume. Nutanix hosts use a private network for the NFS mount in a normal basis, to keep read traffic off of the external network — only reaching out over the network when data happens not to be local, or on write for redundancy to one or two hosts based on the replication factor setting. This private network is 192.168.5.0/24, and the NFS server is always 192.168.5.2. The Nutanix platform allows you to whitelist other external IP addresses to connect to the same container (or NFS based datastore in vSphere) by non-nutanix hosts. Primarily for migration purposes to our platform.
However, when these external hosts mount the volume, it would be on the external network of our Controller Virtual Machines (CVMs). This means that the datastore will show up like below — even trying to name it the same wont work. Screenshot below shows this in my lab environment. In the screenshots esxi00 is a non-nutanix ESXi host, and the nesxi## hosts are nutanix hosts in a single nutanix cluster.
The primary way this has been addressed in the past, is to move data twice. First you would mount the container from the Nutanix hosts both internally per usual, and externally using a different datastore name. The external mount is also mounted by the non-nutanix hosts. It is then used to complete the first storage vMotion, followed by the compute vMotion, and then ultimately a final storage vMotion to complete the migration. Not difficult, but a bit time consuming. However, I had a requirement to try and reduce these steps due to another platform accessing the vCenter server upstream, where the virtual machines could not be easily moved at the vCenter/vSphere layer directly. The upstream platform had the ability to trigger storage migrations, but it is extraordinarily tedious — I had an idea occur to me that could hopefully remedy this situation, and make migrations easier in general.
During Acropolis upgrade processes, or when a CVM fails, a static route is injected into the Nuatnix host that lost its connection to the CVM that is offline. This functionality is called autopathing (brief detail here, but a google search will find you blogs explaining it in depth). The IP address of 192.168.5.2/32 is set to route to the external IP of another working CVM as the next hop temporarily. When the CVM comes back up on the host, the static route is removed. In this way, the host with the CVM that is offline never loses connectivity to the datastore. I wondered if this could be used another way, where the non-nutanix hosts could have a static route of 192.168.5.2/32 configured as a next hop to one of the CVM IP addresses. If this were to work, the IP of the datastore would be the same for both non-nutanix and nutanix hosts, and the result would be that you could migrate data just one time.
In order to do this, log into the subject non-nutanix host. The command to set the static route would be esxcfg-route -a 192.168.5.2/32 <IP of a CVM or the Cluster IP>. You’ll get an error that says the IP isn’t valid (it believes it’s not on the same network), but it will still add the route.
At this point, we can mount the volume as before on the non-nutanix host, but using 192.168.5.2 as the server rather than the external IP address of the CVM.
After hitting the OK button, and the container successfully mounts as a datastore, the IP addresses match and the datastore is identified as the same by vSphere.
Now you can do a storage vMotion just once, followed by a compute migration. Hopefully this saves you some time!