Why HMS Vanguard was unlikely to receive US 5-inch guns

Could HMS Vanguard have been refitted with the American 5-inch/54 dual-purpose gun?

Yes in engineering terms, but the extensive integration work would be difficult to justify unless the new weapon produced a clear operational advantage.

The proposed refit was technically possible but far from a simple mount substitution. Vanguard would need structural work, ammunition handling, power, fire control, and support arrangements compatible with the American weapon. Those changes would cost money and time on a ship whose existing battery already served the intended role. The deciding issue is justification rather than physical possibility.