Logo Search packages:      
Sourcecode: ucto version File versions  Download package

ucto Documentation


Unicode Tokenizer
Ucto can tokenize UTF-8 encoded text files (i.e. separate words from
punctuation, split sentences, generate n-grams), and offers several other
basic preprocessing steps (change case, count words/characters and reverse
lines) that make your text suited for further processing such as indexing,
part-of-speech tagging, or machine translation.
Ucto is a product of the ILK Research Group, Tilburg University (The
If you are interested in machine parsing of UTF-8 encoded text files, e.g. to
do scientific research in natural language processing, ucto will likely be of
use to you.
Generated by  Doxygen 1.6.0   Back to index