This is used to render parts of the Scene selectively.
For example if a GameObject's layer is set to 2, and the second bit of the camera's cullingMask is set to zero, the GameObject will be invisible to the camera. See Layers for more information.
Additional resources: camera component.