Troubleshooting HCX Migration Failures: Which Log to Read and What to Gather Before You Open a Case

HCX TROUBLESHOOTING

A Consult Circle technical guide to the HCX log estate, matching the failure to the component that owns it, and assembling an evidence pack that gets a support case moving on the first reply.

An HCX migration failure is rarely reported by the component that caused it. The manager tells you a migration failed. The failure itself happened somewhere further down: a replication session on the Interconnect appliance, a relocate task on a destination host, an authentication attempt against vCenter that expired forty seconds earlier. The message you are looking at is the last link in a chain, and the useful information sits in whichever log belongs to the link that actually broke.

HCX spreads its diagnostic information across at least five components, and the right one to open depends on which migration type you were running and which phase it failed in. Opening the wrong log is not neutral, it costs an hour and produces a lot of plausible-looking noise. The material here draws on Broadcom KB 377590, which sets out the log locations and the evidence required for a support case, alongside how we use it on live migrations.

What This Guide Covers

• Working out which component actually failed
• The HCX log estate, file by file, and what each one is good for
• Matching the migration type to the log that will tell you something
• Establishing the timeline before you start reading
• Three known failure patterns worth ruling out first
• Assembling the evidence pack for a Broadcom support case
• A diagnostic sequence, and a checklist you can work from

Working Out Which Component Actually Failed

Each component in the migration path keeps its own record. Deciding which layer the failure belongs to eliminates most of the log estate before you open anything.

• Orchestration layer. HCX Manager, at both the Connector and Cloud ends, coordinates the migration as a task: validation, scheduling, state transitions, and the interaction with vCenter. If the migration failed during validation, never started, or failed with an error about permissions or endpoints, this is the layer.
• Data movement layer. The Interconnect appliance carries the actual transfer. If replication started and then stalled or failed part way, or if a switchover failed after data had already moved, the Interconnect owns the detail.
• Platform layer. The ESXi hosts and vCenter at either end perform the underlying operations: the relocate, the power state change, the disk access. Failures that surface as a task error rather than an HCX error usually originate here.
• Guest layer. The virtual machine itself, for anything involving customisation, tools, or a workload that comes up but comes up wrong.

The phase is usually enough to identify the layer. Validation failures are orchestration, transfer failures are data movement, and switchover failures are platform with the Interconnect providing context.

The HCX Log Estate

These are the files worth knowing. Each sits on a different component, so getting to them means knowing which appliance or host to connect to as much as which path to open.

Log	Component	What it tells you
/common/logs/admin/app.log	HCX Manager, Connector and Cloud	The primary back-end record of HCX activity, including migration orchestration. The starting point for most investigations
/var/log/vmware/hbrsrv.log	HCX Interconnect appliance	Replication activity. The log that matters for any migration type built on vSphere Replication, and for disaster recovery
/var/log/vmware/mobility-agent.log	HCX Interconnect appliance	Non-replication migration activity, and the place to look for cutover problems on those types
/var/log/vpxa.log	ESXi host, source and destination	The host agent that communicates with vCenter
/var/log/hostd.log	ESXi host, source and destination	Host and virtual machine management operations performed on that host
/var/log/vmkernel.log	ESXi host, source and destination	Kernel level activity relating to virtual machines and the host itself
vmware.log in the VM folder	Source and destination datastore	Per virtual machine: power events, hardware changes, vMotion activity, tools status
/var/log/vmware/vpxd/vpxd.log	vCenter, source and destination	The main vCenter log, covering internal tasks, events and communication with host agents

The per virtual machine log lives alongside the workload files on the datastore rather than on the host, so the path depends on where the machine was running at the time. If the migration partially succeeded there may be a version at both ends, each telling a different half of the story.

Match the Migration Type to the Log

This is the part that saves the most time, and it is the part most commonly got wrong. The two Interconnect logs cover different mechanisms and reading the wrong one produces an hour of activity that looks relevant and is not.

Bulk Migration and Replication Assisted vMotion

Both are built on vSphere Replication, so the replication log on the Interconnect appliance is the authoritative record. It covers the base sync, ongoing delta replication, and the state of the replication session throughout. If a migration of either type stalls, fails during transfer, or behaves oddly around sync progress, this is where the detail lives. It is also the correct log for HCX disaster recovery activity, since that shares the same replication foundation.

HCX vMotion and Cold Migration

Neither uses vSphere Replication, so the replication log will have nothing useful to say about them. The mobility agent log on the Interconnect appliance is the one that tracks these, and it is particularly useful for cutover problems, which is where both types tend to fail if they fail at all.

Establish the Timeline Before You Start Reading

Log analysis without a timestamp is browsing. These files are verbose, they record concurrent activity across many workloads, and there is always something in them that looks like a problem. Decide what you are looking for and when it happened before opening anything.

• Capture the exact error string as it was presented, rather than a paraphrase of it. The precise wording is frequently the difference between a known issue and an unknown one.
• Record the migration ID. It is the identifier that lets you follow a single migration through logs that are recording dozens of concurrent operations.
• Record the virtual machine name as HCX knows it, which is not always the guest hostname.
• Record the timestamp and the time zone. Components in a migration frequently report in different zones, and reconciling them afterwards without knowing what you started from wastes real time.
• Note which Service Mesh the migration ran through, since that determines which Interconnect appliance holds the relevant logs on a scaled-out deployment.

Those five items are also, not coincidentally, the information Broadcom asks for when a support case is opened. Capturing them at the time of the failure means the case can be raised immediately rather than after a second failure has been engineered to reproduce the conditions.

Rule Out the Known Patterns First

Three failure patterns come up often enough to be worth checking against your symptom before committing to a full investigation. Each has a documented Broadcom knowledge base article, and matching one of them can shorten a two day investigation to twenty minutes.

Migration will not progress past zero percent

A Bulk or Replication Assisted vMotion migration reports the base sync as initiated and never advances. Because it is technically running rather than failed, no error is raised and it can sit there while the change window closes around it. Broadcom document this as KB 319757. The tell in the replication log is a session that established with no meaningful data movement behind it.

Switchover fails with a relocate task error

Data has moved successfully and the final relocate fails. This one hurts because it lands at the end of a long transfer, inside the window, with the least time available to react. The relocate is a platform operation, so the destination host and vCenter logs carry the cause while the Interconnect log supplies the context. Covered by KB 369919.

Operations fail with a vCenter endpoint login error

A reported failure to log into the vCenter endpoint points at credentials or the HCX to vCenter connection rather than at the migration. Usually a service account, a rotated password, an expired session or a certificate change. Documented as KB 372347, and worth checking early because it presents as a migration failure while being nothing of the sort.

Assembling the Evidence Pack

What determines how quickly a case moves is whether the first submission contains everything needed. A case opened with a screenshot and a description spends its first exchange requesting the material below, which costs a day at minimum.

The five facts

• Exact error message as displayed, not summarised
• Migration ID
• Virtual machine name as it appears in HCX
• Timestamps with the time zone stated explicitly
• Service Mesh through which the migration ran

The log bundles

• HCX Connector and HCX Cloud bundles, generated with the core HCX logs and the database dump included, and with the affected Interconnect appliance selected
• ESXi support bundle from the source host on which the virtual machine was running at the time of the failure
• ESXi support bundle from the destination host where relevant, particularly for switchover and relocate failures
• vCenter diagnostic bundle from both the source and destination environments

Two details cause most second-round requests. The database dump is a separate selection when generating the HCX bundle and is frequently omitted. The Interconnect appliance must be selected explicitly, and on a scaled-out deployment a bundle collected without the right one contains everything except the component that carried the failed migration.

Generate the bundles as close to the failure as practical. Logs roll, and on a busy migration platform carrying concurrent waves they roll faster than people expect. A bundle collected three days later may not contain the window you need.

A Diagnostic Sequence That Works

Applied in this order, most HCX migration failures resolve or become clearly escalatable within an hour.

1. Capture the five facts before touching anything, because some of them become harder to retrieve once the interface is refreshed.
2. Check the obvious environmental conditions: tunnel status on the Service Mesh, destination datastore capacity, and whether anything changed in the environment in the preceding hours.
3. Compare the symptom against the three known patterns above, since matching one shortens everything that follows.
4. Open the manager log and locate the migration by its ID, to establish which phase the failure occurred in and what the orchestration layer believed was happening.
5. Move to the Interconnect log that matches the migration type: the replication log for Bulk and Replication Assisted vMotion, the mobility agent log for vMotion and Cold.
6. If the failure is a platform operation such as a relocate, follow it into the host and vCenter logs at the relevant end, searching on the timestamp rather than reading forward.
7. Correlate across both sites before drawing a conclusion, since a problem visible at one end is frequently caused at the other.
8. If the cause is not established, generate the bundles and open the case with the full evidence pack rather than continuing to read.

Step eight is a decision rather than a defeat. There is a point at which continuing to investigate costs more than escalating, and with change windows booked it arrives sooner than engineers instinctively want it to.

If you are mid-project and still planning the wider move, our step by step HCX migration walkthrough and the complete HCX migration checklist cover the build and execution sides, while common HCX migration problems and fixes deals with the failures that happen before the logs get involved.

Frequently Asked Questions

Which HCX log should I open first?

The manager log at /common/logs/admin/app.log, on either the Connector or the Cloud side depending on which end reported the failure. It carries the orchestration record and tells you which phase failed, which determines where to look next. It is rarely where the root cause sits, and it is almost always the fastest route to identifying which component owns it.

What is the difference between hbrsrv.log and mobility-agent.log?

Both sit on the Interconnect appliance and they cover different mechanisms. The replication log records vSphere Replication activity, which means Bulk Migration, Replication Assisted vMotion and disaster recovery. The mobility agent log records the non-replication types, meaning HCX vMotion and Cold Migration, and is the better source for cutover problems on those.

My migration is stuck at zero percent. Is it failing?

Not in the sense that it will report an error, which is what makes this pattern awkward. A base sync that initiates and never progresses can sit indefinitely while the change window closes. It is a documented pattern in Broadcom KB 319757, and the replication log on the Interconnect will show whether the session established and whether data is genuinely moving.

What does Broadcom need when I open a support case?

Five facts and a set of bundles. The facts are the exact error message, migration ID, virtual machine name, timestamps with time zone, and the Service Mesh involved. The bundles are the HCX Connector and Cloud logs generated with the database dump and the affected Interconnect appliance selected, plus ESXi bundles from the relevant hosts and vCenter diagnostics from both sides.

Can Consult Circle help with a migration that keeps failing?

Yes, and it is a common way we are brought in. The usual first step is a short review of the build and the failure pattern, which frequently identifies the cause within a day because we have generally seen it before. From there we can hand back a fix or take over execution, depending on what the situation needs.

Where Consult Circle Fits

We run HCX migrations often enough that this diagnostic sequence is routine rather than researched. That is the whole of the advantage on offer: these failures are not exotic, but recognising one takes ten minutes with prior exposure and the better part of two days without it.

We review builds that are already in place, take over migrations that have stalled, and run full programmes from eligibility audit through execution to decommissioning. If you are mid-project and losing change windows to a failure you cannot pin down, a short review is usually the quickest route to an answer.