VMware HCX Migration Problems and How to Fix Them: Service Mesh, Network Extension and Failed Transfers

HCX TROUBLESHOOTING

A Consult Circle troubleshooting guide to the HCX failures that actually happen, in the order you are likely to meet them, with the checks that resolve each one.

HCX fails in a fairly small number of ways. That is the useful thing about it. The error messages are frequently generic, the logs are verbose, and the symptom often appears several layers away from the cause, but the underlying set of problems is finite and most of them come down to name resolution, time, certificates, ports, MTU or eligibility.

This guide is organised by symptom in roughly the order you meet them during a migration: activation and deployment, site pairing, Service Mesh and tunnels, throughput, migrations that stall or fail, Network Extension, switchover, and problems that only appear after the workload has landed. Each section gives the symptom, the causes worth checking in order of likelihood, and the fix.

If you are reading this mid-incident, the diagnostic sequence at the end of the introduction resolves a surprising proportion of HCX problems before you open a single log file.

What This Guide Covers

• Activation and appliance deployment failures
• Site pairing failures
• Service Mesh deployment and tunnel problems
• Throughput far below expectation
• Migrations that stall, fail or never start
• Network Extension problems
• Switchover failures and overrunning windows
• Problems that appear after the workload has landed

Activation and Deployment Failures

HCX Manager will not activate

The symptom is a licence key that will not validate, often with an unhelpfully generic error. The cause is almost always that the appliance cannot reach the activation service, and the reason is usually a proxy or an outbound restriction that the appliance has not been told about.

• Confirm the appliance has outbound HTTPS access to the activation endpoint, testing from the appliance rather than from a workstation on the same subnet.
• Configure the proxy settings in the appliance management interface if outbound access is proxied, since HCX will not inherit them from anywhere else.
• Confirm DNS is working from the appliance, because activation failures caused by name resolution present identically to those caused by blocked access.
• Check the system clock. A significant time offset will cause certificate validation against the activation service to fail.
• Confirm the licence key is the correct type and has not already been consumed by a previous deployment that was never properly removed.

Appliance deployment fails or hangs

The Service Mesh commits but appliances do not deploy, or they deploy and never come up. This is usually a resource or placement problem rather than an HCX problem.

• Check capacity on the target cluster or resource pool. Appliances need real CPU and memory, and full reservations on the Interconnect appliance will fail deployment outright if the headroom is not there.
• Confirm the datastore selected in the compute profile has sufficient free space and is accessible from every host in the deployment cluster.
• Check the IP pool in the network profile has free addresses. Pools sized exactly to the initial deployment fail the moment you scale out.
• Confirm the service account still has the required permissions, particularly if a permissions review has happened since the profiles were built.
• Look for DRS or admission control policies preventing placement, especially on clusters running close to capacity.

Site Pairing Failures

Pairing is a single outbound HTTPS connection authenticated with a service account. When it fails, the error rarely identifies which of the three usual causes applies, so check all three.

Symptom	Most likely cause	Check
Connection times out	Port 443 not open on the path between sites	Test from the appliance itself, not from a jump host on a different segment
Certificate error during pairing	Destination certificate not trusted at the source	Import the destination certificate into the source HCX Manager trust store
Authentication fails	Service account lacks required roles	Confirm the account can enumerate inventory in the destination vCenter, not just log in
Pairs then shows unhealthy	Name resolution asymmetry between sites	Confirm forward and reverse DNS resolve correctly in both directions
Version incompatibility error	Connector and Cloud Manager builds mismatched	Check both against the interoperability matrix and upgrade the older side

The asymmetric DNS case is worth dwelling on because it produces the most confusing behaviour. Pairing succeeds, the interface shows a healthy relationship, and then operations fail intermittently in ways that look like network problems. HCX pairs on fully qualified domain names and both sides need to resolve each other correctly, forward and reverse, from the appliances themselves. Test it from the appliance shell rather than assuming.

Service Mesh and Tunnel Problems

Tunnel status will not come up

The Interconnect appliances deploy at both sites and the tunnel stays down. This is a uplink connectivity problem in the overwhelming majority of cases, and it is nearly always UDP.

• Confirm UDP 500 and UDP 4500 are open between the uplink interfaces at both sites. TCP 443 being open proves nothing about UDP, and firewall rules written for one frequently omit the other.
• Confirm the uplink network profile addresses are routable between sites and that return traffic follows a symmetric path. Asymmetric routing through a stateful firewall will break tunnel establishment while leaving ping working.
• Check for NAT between the sites. It is supported but needs to be declared in the network profile configuration rather than discovered by HCX.
• Confirm the uplink profile is using the interface you expect. On appliances with several network profiles attached, selecting the management profile as the uplink is an easy and confusing mistake.
• Check MTU on the uplink path. A tunnel that establishes and then drops under load frequently indicates fragmentation rather than a firewall issue.

Tunnel comes up then drops repeatedly

Intermittent tunnel behaviour points at the path rather than the configuration, since a wrong configuration fails consistently. Check for a stateful device timing out idle sessions, an MTU mismatch that only manifests under load, or contention on the host running the Interconnect appliance. If the appliance shares a host with other heavy dataplane workloads, move it and apply anti-affinity.

Service Mesh will not delete

Deletion hangs, usually because Network Extension segments are still extended or migrations are still registered against the mesh. Un-extend every segment first, confirm no migrations remain in a non-terminal state, then retry. Forcing removal without un-extending first will drop connectivity for workloads that have already migrated onto those segments.

Throughput Far Below Expectation

The tunnel is up, migrations run, and the rate is a fraction of what the circuit should deliver. This is the most commonly reported HCX problem and it is usually not a network problem at all.

• The 2 Gbps appliance ceiling. A single Interconnect appliance has an architectural transfer ceiling of roughly 2 Gbps regardless of underlay capacity. If your circuit is 10 Gbps and you are seeing 2, this is the answer. The fix is additional Service Meshes each with their own Interconnect appliance, not a larger appliance.
• Appliances co-located on one host. Scaling out to four Interconnect appliances achieves nothing if placement puts three of them on the same ESXi host. Apply anti-affinity rules and verify actual placement rather than intended placement.
• No resource reservations. Without full CPU and memory reservations, the Interconnect and WAN Optimization appliances compete for scheduler time with production workloads, and throughput becomes erratic in a way that correlates with business hours.
• Source storage read performance. Replication reads from source datastores. On older storage carrying a production workload, read performance frequently binds before the network does. Check source storage latency during a migration rather than at rest.
• Source host resource pressure. Migration work happens on hosts that are also running production. A cluster at ninety percent CPU utilisation will not deliver the concurrency the documentation suggests.
• The circuit does not deliver its rating. A link rated at 1 Gbps rarely sustains 1 Gbps once TCP overhead, encryption and existing production traffic are accounted for. Baseline with Transport Analytics rather than arguing from the contract.

Work through those in order. The first three account for most cases and all three are configuration rather than capacity, which means they are fixable this afternoon rather than through procurement.

Migrations That Stall, Fail or Never Start

Workload does not appear in the migration wizard

If a virtual machine is missing from the selectable inventory, the usual causes are that it sits outside the compute profile scope, that the service account cannot see it, or that its configuration makes it ineligible for the selected mechanism. Check the compute profile boundaries first, then permissions, then eligibility.

Validation fails before migration starts

HCX validates before it transfers, which is a feature even when it does not feel like one at the time. The common blockers are consistent.

• Virtual hardware below version 9, which blocks Replication Assisted vMotion and requires a reboot to remediate.
• Attached ISO images, frequently from datastores that no longer exist, which are trivially fixed once identified.
• Independent Persistent or Non-Persistent disks, virtual NVMe controllers, or serial port devices, each of which blocks RAV and pushes the workload towards Bulk Migration.
• Fault Tolerance, multi-writer disks or shared SCSI configurations, which require the workload to move by a different mechanism entirely.
• Snapshots present on the source, which should be consolidated before migration rather than carried through it.
• Destination resource or datastore selections that are no longer valid, particularly where the destination has changed since the wave was planned.

Migration stalls at a percentage and does not progress

A transfer that sits at the same percentage for an extended period is usually blocked rather than slow. Check whether the tunnel is still up, since a dropped tunnel mid-transfer presents as a stall rather than an error. Check source storage latency, which is the most common cause of genuinely slow progress. Check whether concurrency limits have been reached and the migration is queued behind others rather than actually running. Finally check destination datastore capacity, because filling the destination mid-wave produces exactly this symptom.

Migration fails part way through

The reassuring property of Bulk and RAV is that the source workload keeps running throughout, so a failure before switchover leaves the original untouched and the migration can be retried. Before retrying, establish what changed: a tunnel drop, a capacity limit, a permissions change, or a source snapshot created by a backup job mid-transfer. Retrying without identifying the cause usually produces the same failure several hours later.

Network Extension Problems

Segment will not extend

Check the distributed switch version on the source, since older versions may not be supported for extension and need upgrading first. Confirm the port group is a type that can be extended, and confirm the destination NSX configuration is ready to receive it. Permissions on the destination NSX side are a frequent cause where the service account was scoped to vCenter only.

Extended segment works but performance is poor

MTU is the first thing to check, every time. Network Extension encapsulates traffic and needs at least 1600 bytes end to end, and a single hop that silently fragments will produce exactly this symptom while every configuration file you inspect looks correct. Test the actual path rather than reading switch configurations.

The second cause is tromboning. Without Mobility Optimized Networking, a migrated workload on an extended segment routes north-south back through the source site gateway, so traffic crosses the WAN twice. The workload has moved and its traffic has not. MON allows the destination gateway to handle routing for migrated workloads, which removes most of the penalty, at the cost of a routing design conversation with the network team.

The third is appliance placement. If the Network Extension appliance shares a host with NSX Edge nodes or other heavy dataplane workloads, they contend. Separate them and apply anti-affinity.

Connectivity breaks after un-extending

Almost always because a workload was still on the segment when it was un-extended, or because routing on the destination side was not normalised as part of the un-extension. The sequence matters: confirm every workload has moved and been validated, normalise routing on the destination, then un-extend. Doing it in a different order is how outages happen at the end of otherwise successful migrations.

Switchover Failures and Overrunning Windows

The window overruns even though replication finished early

This is the single most common scheduling failure in HCX projects and it is not a fault. Replication runs in parallel across many workloads. Switchover happens serially. A wave of two hundred workloads does not switch over in the time it takes to switch over one, and a window sized on the largest single workload rather than the cumulative serial time will overrun predictably. Size the window against the whole batch and add contingency.

Switchover fails for an individual workload

Check whether a snapshot was taken during the replication phase, commonly by a backup job that nobody excluded. Check destination capacity, since the final switchover needs headroom. Check whether the destination network mapping is still valid, particularly on waves planned weeks earlier. Check the workload has not been modified on the source since replication started, since configuration changes mid-flight cause switchover to fail rather than to silently reconcile.

Workload comes up on the wrong network

A network mapping error in the wizard. It is recoverable by correcting the port group assignment, and it is best prevented by documenting mappings per mobility group in the runbook rather than selecting them from memory during the window.

Problems That Appear After the Workload Lands

The migration reports success, the workload is running, and something is wrong anyway. These are the ones that generate escalations days later.

• Workload runs but slower than before. Check the storage policy applied on the destination, which may be a weaker default than the source protection level. Check whether the destination cluster is more heavily contended than the source. Check whether traffic is tromboning back across the WAN on an extended segment.
• Backups stop running. The backup job is still pointed at the source location or at an object that no longer exists there. Re-point it and confirm one successful run before considering the wave complete.
• Monitoring alerts on the old object. The source object was left in place and continues to generate alerts that nobody actions, which trains the team to ignore alerts from that system.
• Application partially works. Usually a dependency that was not mapped, where half an application moved and half did not. This is why mobility groups should follow applications rather than clusters.
• Licensing breaks in the guest. Some software licences bind to hardware identifiers including MAC address. Retaining the MAC during migration prevents this and needs deciding as policy before the wave rather than fixing afterwards.
• Time drift in the guest. Check NTP configuration inside the workload and the time source on the destination hosts, particularly where source and destination use different time infrastructure.

A Diagnostic Sequence That Works

When something is wrong and it is not obvious which layer, work outwards in this order rather than starting with the logs. It resolves most problems faster than reading them will.

• Check time synchronisation on both HCX managers. Clock drift causes certificate and pairing failures that present as almost anything.
• Check DNS resolution, forward and reverse, from the appliances themselves in both directions.
• Check tunnel status at both sites, since a dropped tunnel presents as a stalled migration rather than an error.
• Check the ports, actually, from the appliance addresses. Documented as open and confirmed as open are different states.
• Check capacity: destination datastore, IP pool addresses, cluster CPU and memory headroom.
• Check the service account still has the permissions it had when the profiles were built.
• Check source storage latency during a transfer rather than at rest, since this is the most common cause of genuinely slow throughput.
• Only then read the logs, with a specific hypothesis rather than looking for something that stands out.

Reading the Logs Without Drowning

HCX produces a great deal of log output and very little of it is relevant to the problem you have. Approaching it by reading until something looks wrong is slow and unreliable, because plenty of things always look wrong. Approach it with a hypothesis instead.

• Form the hypothesis first. Decide what you think is happening before opening anything: the tunnel dropped at this time, the transfer is blocked on storage, the service account lost a permission. Then look for evidence that confirms or eliminates it. A search is fast; a read is not.
• Work from the timestamp. Establish exactly when the symptom started and search around that window. Most HCX investigations are resolved by correlating an event in the appliance log with something entirely outside HCX, such as a firewall change, a backup job or a host that rebooted.
• Check the appliance before the manager. For tunnel and transfer problems, the Interconnect appliance is closer to the symptom than the manager. For pairing, permissions and inventory problems, the manager is the right place.
• Correlate across sites. A problem visible only at one end is frequently caused at the other. Pull the equivalent window from both sides before concluding anything about direction.
• Use the support bundle for handover, not for diagnosis. Generate it when you are escalating so somebody else has what they need. Trying to diagnose from a bundle you have just generated is usually slower than working from the live system.

The one pattern worth internalising is that HCX reports the symptom at the layer where it was noticed, not the layer where it occurred. A migration failure is frequently logged as a migration failure when the actual event was a tunnel drop forty seconds earlier, which was itself caused by a firewall session timeout. Work outwards from the symptom rather than accepting the first plausible explanation.

When to Stop and Escalate

Persistence is a virtue up to a point and past that point it costs change windows. These are reasonable triggers for stopping and bringing in either the vendor or an external party.

• The diagnostic sequence has been completed twice with no change in the symptom, which usually means the problem is not where you are looking.
• The same failure has recurred across three or more waves after a fix that appeared to work, indicating the actual cause has not been identified.
• The problem is affecting workloads that have already migrated, since that is a production issue rather than a project one and should be treated accordingly.
• You are considering forcing an operation that HCX is refusing to perform. The refusal is usually protecting something, and forcing past it converts a delay into an outage.
• The next change window is within forty eight hours and the cause is still unknown, at which point postponing is cheaper than proceeding hopefully.
• You have started making multiple changes at once in the hope that one of them works, which reliably makes the problem harder to identify.

When escalating, generate the support bundle from both sites, record the exact timestamps of the symptom, and describe what has already been eliminated rather than only what is failing. The elimination list saves more time than the description of the symptom does.

What Prevents Most of This

Nearly every problem in this guide is preventable at the pre-check stage, which is unhelpful advice mid-incident and worth acting on before the next wave.

• End to end MTU tested on the real path, not read from a configuration
• Throughput baselined with Transport Analytics before wave dates are published
• Full CPU and memory reservations on the Interconnect and WAN Optimization appliances
• Anti-affinity rules applied and actual placement verified after deployment
• Eligibility audit completed across the whole estate before the wave plan is built
• A single disposable workload migrated end to end before any production wave
• Maintenance windows sized against cumulative serial switchover time for the batch
• Backup jobs excluded from workloads that are mid-migration
• Written rollback trigger and procedure for every wave
• Un-extension sequence documented, with a named owner and a date for every extended segment

Frequently Asked Questions

Why does my site pairing keep failing?

Three causes account for nearly all of it: name resolution that works from your workstation but not from the appliance, a destination certificate that has not been trusted at the source, and a service account with enough rights to log in but not to enumerate inventory. Check all three before investigating further.

Why is my throughput so much lower than my circuit?

Most often the roughly 2 Gbps ceiling per Interconnect appliance, which is architectural and not tunable. Scaling out to additional Service Meshes with anti-affinity rules is the fix. After that, check resource reservations on the appliances and source storage read performance, both of which bind before the network on many estates.

My migration is stuck at a percentage. Is it going to fail?

Not necessarily, but a transfer sitting at the same percentage is usually blocked rather than slow. Check tunnel status first, then source storage latency, then destination datastore capacity, then whether it is queued behind other migrations against a concurrency limit.

Is it safe to retry a failed migration?

Yes for Bulk and RAV, since the source workload continues running throughout and is untouched by a failure before switchover. Establish the cause before retrying, because retrying blindly usually reproduces the same failure several hours later.

Why is performance poor on an extended network?

Check MTU first, every time, on the actual path. After that, consider tromboning: without Mobility Optimized Networking, migrated workloads on an extended segment route back through the source site gateway, so their traffic crosses the WAN twice.

Can I un-extend a segment while workloads are still on it?

You can, and it will drop their connectivity. Confirm every workload has moved and been validated, normalise routing on the destination, then un-extend. The order is not optional.

Our maintenance window keeps overrunning. What are we doing wrong?

Almost certainly sizing it against the largest workload in the batch rather than the cumulative serial switchover time for the whole batch. Replication is parallel, switchover is not, and a wave of two hundred workloads takes proportionally longer to switch over than one.

Can Consult Circle help with a migration that is already in trouble?

Yes, and it is a common way we get involved. The usual first step is a short review of the current build against the failure modes above, which frequently identifies the cause within a day. From there we can either hand back a fix or take over execution, depending on what the situation needs.

Where Consult Circle Fits

We run HCX migrations regularly rather than occasionally, which mostly means we have already encountered these failures on somebody else’s estate. That is the entire value on offer here: not that the problems are exotic, but that recognising them takes ten minutes when you have seen them before and two days when you have not.

We do design reviews on builds that are already in place, we take over migrations that have stalled, and we run the full lifecycle from audit through execution to decommissioning. If you are mid-project and something is not behaving, a short review is usually the fastest route to an answer.

Related Guides in This Series

• VMware Migration Services: How to Scope, Price and Run a VCF Move
• How to Migrate to VCF Using HCX: A Step by Step Walkthrough from Site Pairing to Cutover
• vSphere to VCF Migration and Upgrade Paths: In-Place, Side-by-Side or Greenfield
• VCF Migration Services for vSphere 7 and 8 Estates: What Changes After End of Support
• The Complete VMware HCX Migration Checklist: 70 Checks Before, During and After Cutover
• VMware Cloud Foundation 5.x vs VCF 9.x: Key Differences, Architecture Changes and Upgrade Considerations
• Migrating from Legacy vSphere to VMware Cloud Foundation (VCF) Using HCX