What did an experienced crew mean by a "strange sail"?

Was identifying a "strange sail" mainly a command team's experienced gut feeling?

Yes; it meant that a vessel's course or behaviour looked wrong to experienced seamen, which was enough reason to investigate.

The phrase is compared to an experienced police officer sensing that the visible details do not fit. A captain, first lieutenant, and sailing master could bring close to a century of combined sea time to that judgment. On detached convoy duty they could not ask a distant headquarters for immediate guidance, so the captain had to decide whether to send boats or alter course. Investigating a harmless vessel cost little, while ignoring unusual behaviour could allow a privateer or other threat to approach the trade route unchecked.