path | Path to tokenize. |
entrySeparators | Entry separators used to tokenize the path. |
IEnumerable<string> Returns list of tokens and variations in lowercase.
Split a file entry according to a list of separators and find all the variations on the entry name.
public static void IndexWordComponents(int documentIndex, string word) { foreach (var c in SearchUtils.SplitFileEntryComponents(word, SearchUtils.entrySeparators)) index.AddWord(c.ToLowerInvariant(), 0, documentIndex); }