Shadowmask mode is a lighting mode shared by all Mixed Lights in a Scene. To set Mixed lighting to Shadowmask, open the Lighting window (menu: Window > Lighting > Settings), click the Scene tab, navigate to Mixed Lighting and set the Lighting Mode to Shadowmask.
You also need to select the desired Shadowmask mode to use from the Quality Settings (menu: Edit > Project Settings > Quality):
See documentation on Mixed lighting and Light modes to learn more about the other modes available.
2017–09–18 Page amended
Shadowmask modes added to Quality Settings in 2017.1 NewIn20171