Index

C

convert_tokens_to_string() (jptranstokenizer.tokenization_utils.JapaneseTransformerTokenizer method)

F

from_pretrained() (jptranstokenizer.tokenization_utils.JapaneseTransformerTokenizer class method)

G

get_word_tokenizer() (in module jptranstokenizer.tokenization_utils)

J

JapaneseTransformerTokenizer (class in jptranstokenizer.tokenization_utils) jptranstokenizer.mainword.base module jptranstokenizer.mainword.juman module jptranstokenizer.mainword.spacy_luw module jptranstokenizer.mainword.sudachi module

jptranstokenizer.subword.sentencepiece module jptranstokenizer.tokenization_utils module jptranstokenizer.version module JumanTokenizer (class in jptranstokenizer.mainword.juman)

M

MainTokenizerABC (class in jptranstokenizer.mainword.base) module jptranstokenizer.mainword.base jptranstokenizer.mainword.juman jptranstokenizer.mainword.spacy_luw jptranstokenizer.mainword.sudachi jptranstokenizer.subword.sentencepiece jptranstokenizer.tokenization_utils jptranstokenizer.version

N

Normalizer (class in jptranstokenizer.mainword.base)

S

SentencepieceTokenizer (class in jptranstokenizer.subword.sentencepiece)

SpacyluwTokenizer (class in jptranstokenizer.mainword.spacy_luw) SudachiTokenizer (class in jptranstokenizer.mainword.sudachi)

T

tokenize() (jptranstokenizer.mainword.base.MainTokenizerABC method) (jptranstokenizer.mainword.base.Normalizer method) (jptranstokenizer.mainword.juman.JumanTokenizer method) (jptranstokenizer.mainword.spacy_luw.SpacyluwTokenizer method) (jptranstokenizer.mainword.sudachi.SudachiTokenizer method) (jptranstokenizer.subword.sentencepiece.SentencepieceTokenizer method)