This study proposes a modal online differential fault detection and localization (MODF) scheme for VSC‐MTDC cable transmission. Modal theory is used to remove the coupling between layers of two‐core extruded high‐voltage direct current cable. As a result, the decoupled modes are determined and the modal matrix resulted. In high frequencies, the modes have different velocities, which result in a time difference between modal initial peaks. This time delay is used to design modal fault detection and localization scheme, which can work in a variety of strategies like pilot or non‐communicated protection systems, one‐sided or two‐sided relaying, and single‐zone or double‐zone relay settings. Novel modal time–distance characteristics are introduced for adjusting the zone setting of relays in the MODF scheme. Multiple faults are applied to the CIGRE VSC‐MTDC model in the real‐time digital simulator with a Beckhoff CX5130 controller in the hardware‐in‐the‐loop setup for testing the proposed MODF scheme. The advantages and weak points of each strategy are analyzed and compared for the modelled faults. Hilbert–Huang transform is used for the accurate detection of modal initial peaks. It is shown that the method gives accurate and selective results for a wide range of VSC‐MTDC configurations.