HCX Bulk and RAV Migration Stuck at 0% Base Sync: Causes, Diagnosis and Fixes

A Consult Circle technical guide to the replication handshake failure that stalls HCX migrations silently, and the sequence that identifies which of the five causes you have.

This is the HCX failure that costs the most change windows, and it does so precisely because it does not look like a failure. A Bulk Migration or Replication Assisted vMotion (RAV) enters the base sync phase, reports zero percent, and then stays there. The workflow remains Active. No error appears in the HCX Manager interface. The migration is still actionable, so it can be halted or cancelled, which reinforces the impression that it is simply slow.

Meanwhile vMotion and Cold Migration on the same Service Mesh work perfectly, which sends people to investigate the workload rather than the path.

The cause is nearly always the same in shape: the vSphere Replication handshake between the source ESXi host and the HCX Interconnect appliance never completes. Bulk and RAV both depend on that handshake and the other migration types do not, which is exactly why the failure is selective in a way that looks confusing at first.

What follows is how to confirm you have this problem, the five underlying causes, and a diagnostic sequence that isolates which one applies. It draws on Broadcom KB 319757 together with how this presents on live migrations.

What This Guide Covers

1. Why the failure is silent, and why only Bulk and RAV are affected
2. Confirming the diagnosis from the logs
3. The five causes, ranked by how often they turn out to be the answer
4. A diagnostic sequence with the commands to run at each step
5. The dedicated replication interface problem and its workaround
6. Static routes on a separate vMotion network
7. Prevention checklist and frequently asked questions

Why Only Bulk and RAV Fail

Bulk Migration and Replication Assisted vMotion are both built on vSphere Replication. Before any data moves, the source ESXi host has to establish a replication session directly with the Interconnect appliance, over a specific port, from the VMkernel interface tagged for vSphere Replication. That handshake is the thing failing here.

HCX vMotion and Cold Migration use a different path entirely and do not require that handshake, which is why they continue working while the replication types stall. Teams frequently read that pattern as evidence that the network is fine, since something is clearly getting through. What it actually tells you is the opposite: it isolates the problem to the replication path specifically, which is useful information rather than a contradiction.

The silence is a design consequence. The migration has been accepted and the session is in progress from the orchestration layer’s point of view, so there is nothing to raise an error about. It will sit in that state for as long as you let it.

Confirming the Diagnosis

Two logs confirm this quickly. On the source ESXi host, the kernel log records the replication service failing to establish its connection to the Interconnect appliance, timing out against the replication port. The entry names the target address and port, which immediately tells you whether the host is even attempting to reach the appliance you expect.

On the HCX Manager, the application log carries two useful signatures. The first is an error reporting that communication to the replication server failed. The second is the transfer monitor observing that the current bytes transferred value is not greater than the already transferred value, both being zero. That second line is the machine-readable version of nothing is happening, and it is the one that distinguishes a stalled migration from a slow one.

If you see the timeout on the host side and zero-against-zero on the manager side, you have this problem and the rest of this guide applies. If the host log shows a successful connection, the issue lies elsewhere and the replication path is not where to spend your time.

The Five Causes

Every instance of this we have worked has come down to one of the following. They are ordered by how frequently they turn out to be the answer.

Cause	What is happening	Tell
Port blocked	TCP 31031 closed between the ESXi VMkernel interface and the Interconnect appliance, or 44046 for RAV. Physical firewalls and NSX distributed firewall rules both apply	Port test fails outright
MTU fragmentation	Mismatch between the ESXi VMkernel, the distributed switch and the HCX network profile, causing large replication packets to be dropped silently	Port test succeeds, sized ping fails
Interface tagging conflict	More than one VMkernel adapter tagged for vSphere Replication, so traffic egresses through a path with no route to the appliance	Two or more tagged adapters on the host
Subnet mask mismatch	The mask in the HCX network profile differs from the mask on the host, for example a /25 against a /24, producing ARP failures across the underlay	Addresses look correct, masks do not match
Routing failure on a dedicated replication network	A separate replication segment exists but Layer 3 routing or physical connectivity on it is broken	Ping fails on the dedicated interface even with the port open

The last of these is worth flagging because it defeats the usual reasoning. The firewall team can demonstrate that the port is permitted, and they are correct, and the migration still stalls, because permission is not the same as reachability. If a dedicated replication segment has no working route, the packets never arrive to be permitted.

Diagnostic Sequence

Work through these in order. Each step eliminates a cause, and the order is chosen so that the cheapest tests come first.

Step one: validate port connectivity

From the source ESXi shell, test the replication port against the Interconnect appliance address.

nc -zv <IX-appliance-IP> 31031

A successful connection confirms the port is reachable. A refusal or a timeout means the path is blocked, and both physical firewalls and NSX distributed firewall rules are candidates. The distributed firewall is the one more often overlooked, because rules written at the perimeter do not describe what is happening between workloads inside the environment. For Replication Assisted vMotion, confirm TCP 44046 as well.

Step two: verify MTU consistency

If the port test passes and replication still stalls, MTU is the next candidate. Large replication packets are dropped silently by any hop configured below the size in use, which produces a timeout that looks identical to a firewall block. Test with a do-not-fragment ping sized to the payload rather than the frame, from the specific VMkernel interface carrying replication.

# for a 1500 byte MTU path
vmkping -I vmk# -d -s 1472 <IX-appliance-IP>

# for a 9000 byte MTU path
vmkping -I vmk# -d -s 8972 <IX-appliance-IP>

The sizes account for the 28 bytes of IP and ICMP header, so a successful test at 1472 confirms a genuine 1500 byte path. Also confirm that the distributed switch MTU is equal to or greater than the value set in the HCX network profile. A profile configured above the switch it rides on will fail in exactly this way, and the configuration will look correct in both places when inspected separately.

Step three: review VMkernel tagging

Check the host networking configuration and confirm that exactly one VMkernel adapter is tagged for vSphere Replication. Where more than one carries the tag, the host may select an interface that has no route to the Interconnect appliance, and it will do so consistently rather than intermittently.

Remove the tag from the unused adapter, then cancel the stalled migration and start it again. The re-initiation matters: an existing session will not pick up the corrected tagging on its own.

Step four: check subnet mask alignment

Confirm that the subnet mask in the HCX network profile matches the mask on the ESXi management or replication network exactly. A host on a /25 with an Interconnect appliance configured for a /24 produces ARP failures across the underlay, and this is a genuinely easy one to miss because both addresses are valid and appear to be on the same network at a glance.

Step five: check the virtual machine itself

If the path is clean, two workload-level conditions can produce the same stall.

• Disk count. A virtual machine exceeding 64 disks is supported only on ESXi 8.0 U1 and later. Below that build, the migration will not proceed.
• Stale files in the virtual machine home directory. Zero-byte .vmxf files and leftover .lck files from previous failed attempts both block a new replication session. Remove them before retrying, which also means that repeated retries without cleanup will keep failing for a reason that has nothing to do with the original cause.

That second point explains a pattern we see regularly: the underlying network problem gets fixed, the migration is retried, and it fails again, leading the team to conclude the fix did not work. The fix did work. The debris from the first failure is now the problem.

The Dedicated Replication Interface Problem

This deserves separate treatment because it presents differently from the others and has a workaround worth knowing under time pressure.

Where the ESXi host has a dedicated interface for vSphere Replication rather than using management, that interface is what talks to the Interconnect appliance. If Layer 3 routing or physical connectivity on that specific segment is broken, standard ping tests fail and the replication timeouts continue regardless of firewall configuration, because the traffic has no route to be filtered.

The workaround is to fail replication back onto the management path temporarily. Untag vSphere Replication on the dedicated VMkernel adapter, tag it on the management adapter, and update the Interconnect appliance network configuration to match. Migrations then proceed over the management network, which is usually enough to save a change window that would otherwise be lost.

Static Routes for a Separate vMotion Network

A related configuration issue arises when the vMotion network differs from the management network. The compute profile needs to know how to reach it, and without that the same class of connectivity failure appears.

Edit the compute profile, continue through to the vMotion network profile selection, and open the advanced configuration. Add a static route using the vMotion network as the destination and the vMotion gateway as the next hop. Complete the profile, then resynchronise the Service Mesh so the change is applied to the deployed appliances.

The resynchronisation step is the one that gets forgotten. Editing the profile alone changes the intended configuration without changing what is running, and the migration continues to fail against appliances that have not received the update.

Preventing This Before the First Wave

Every cause above is detectable during the pre-check, at a cost of perhaps two hours. Discovered during a change window instead, each one costs the window.

• TCP 31031 confirmed open from every source ESXi VMkernel interface to the Interconnect appliance, with 44046 confirmed as well where RAV is in the plan
• NSX distributed firewall rules checked in addition to physical firewalls, since perimeter rules do not describe internal traffic
• MTU validated end to end with a do-not-fragment ping at the sized payload, from the interface that actually carries replication
• Distributed switch MTU confirmed equal to or greater than the value in the HCX network profile
• Exactly one VMkernel adapter tagged for vSphere Replication on every source host, verified rather than assumed
• Subnet masks compared between the HCX network profile and the host networks, digit by digit
• Dedicated replication segments proven end to end with a routing test, not just a firewall rule review
• Static routes configured in the compute profile where vMotion or replication networks are separate, and the Service Mesh resynchronised afterwards
• Virtual machines with more than 64 disks identified against the destination ESXi build
• Virtual machine home directories checked for stale lock files before any retry

Frequently Asked Questions

Why do vMotion and Cold Migration work when Bulk and RAV do not?

Because Bulk and RAV are built on vSphere Replication and require a handshake between the source ESXi host and the Interconnect appliance over the replication port. vMotion and Cold Migration use a different path and do not need it. The selective failure is diagnostic rather than contradictory: it isolates the problem to the replication path.

Which port does HCX replication actually use?

TCP 31031 for the replication path from the source ESXi host to the Interconnect appliance, with 44046 also relevant for Replication Assisted vMotion. Both need to be open from the VMkernel interface tagged for vSphere Replication, and both need checking against the distributed firewall as well as any physical firewall.

How do I tell a firewall problem from an MTU problem?

The port test distinguishes them. If the connection to the replication port fails, the path is blocked. If it succeeds but a do-not-fragment ping at the sized payload fails, the port is open and packets above a certain size are being dropped somewhere along the path, which is fragmentation.

I fixed the network problem and the retry still failed. Why?

Most likely stale files left behind by the first attempt. Zero-byte .vmxf files and leftover .lck files in the virtual machine home directory will block a new replication session on their own. Clear them before retrying, otherwise a correct fix looks like a failed one.

Is it safe to move replication onto the management network?

As a temporary measure to save a change window, yes, and it is the documented workaround when a dedicated replication segment has broken routing. It puts replication traffic onto a network that was not sized for it, so treat it as an exception with a date attached rather than a permanent configuration.

How long should I let a migration sit at 0% before investigating?

Not long. Because the workflow stays Active and raises no error, the natural instinct is to give it more time, and that instinct is what turns a twenty minute diagnosis into a lost window. If no bytes have transferred after a few minutes of base sync, check the logs rather than waiting.

Can Consult Circle help if our migrations keep stalling?

Yes. This particular failure is one we work through regularly, and the diagnostic sequence above usually identifies the cause within an hour. We review builds that are already in place, take over stalled migrations, and run full programmes from eligibility audit through to decommissioning.

Where Consult Circle Fits

The reason this failure costs so much time is not that it is difficult. It is that it presents as a performance problem when it is a connectivity problem, and the natural response of waiting longer is the one thing that guarantees the window is lost.

We run HCX migrations often enough that this pattern is recognised rather than researched. If your migrations are stalling and the cause is not clear, a short review of the build is usually the fastest route to an answer, and it is a small piece of work.

Planning a wider move? See our step-by-step guide to migrating to VCF using HCX, or get a timeline in minutes with the VCF Migration Estimator.