jptranstokenizer.mainword.juman

class jptranstokenizer.mainword.juman.JumanTokenizer(do_lower_case: bool = False, normalize_text: bool = True, ignore_max_byte_error: bool = False)

Bases: MainTokenizerABC

Tokenizer to split into words using Juman. Juman++ and pyknp are required to use. You can import this module shortly:

>> from jptranstokenizer.mainword import JumanTokenizer

Parameters:

• do_lower_case (bool, optional, defaults to False) – Whether or not to lowercase the input when tokenizing.Defaults to None.

• normalize_text (bool, optional, defaults to True) – Whether to apply unicode normalization to text before tokenization.

• ignore_max_byte_error (bool, optional, defaults to False) – Whether or not to ignore error of max bytes (only valid with Juman and Sudachi). If valid, the tokenizer return empty list.

See also

• Juman++ https://github.com/ku-nlp/jumanpp

• pyknp https://github.com/ku-nlp/pyknp

tokenize(text: str, **kwargs: Dict[str, Any]) → List[str]

Converts a string in a sequence of words. Other kwargs (such as never_split) are ignored.

Parameters:

text (str) – A sequence to be encoded.

Returns:

A list of words.

Return type:

List[str]