Function tokenize

tokenize(text: string): string[]
Simple whitespace + punctuation tokenizer. Lowercases and strips punctuation.
Parameters
- text: string
Returns string[]
- Defined in src/score/nlpMetrics.ts:19