jptranstokenizer
latest
User Guide
Quickstart
Available Pretrained Models
Featured API
JapaneseTransformerTokenizer
JumanTokenizer
SpacyluwTokenizer
SudachiTokenizer
SentencepieceTokenizer
All API
List
jptranstokenizer
Index
Edit on GitHub
Index
C
|
F
|
G
|
J
|
M
|
N
|
S
|
T
C
convert_tokens_to_string() (jptranstokenizer.tokenization_utils.JapaneseTransformerTokenizer method)
F
from_pretrained() (jptranstokenizer.tokenization_utils.JapaneseTransformerTokenizer class method)
G
get_word_tokenizer() (in module jptranstokenizer.tokenization_utils)
J
JapaneseTransformerTokenizer (class in jptranstokenizer.tokenization_utils)
jptranstokenizer.mainword.base
module
jptranstokenizer.mainword.juman
module
jptranstokenizer.mainword.spacy_luw
module
jptranstokenizer.mainword.sudachi
module
jptranstokenizer.subword.sentencepiece
module
jptranstokenizer.tokenization_utils
module
jptranstokenizer.version
module
JumanTokenizer (class in jptranstokenizer.mainword.juman)
M
MainTokenizerABC (class in jptranstokenizer.mainword.base)
module
jptranstokenizer.mainword.base
jptranstokenizer.mainword.juman
jptranstokenizer.mainword.spacy_luw
jptranstokenizer.mainword.sudachi
jptranstokenizer.subword.sentencepiece
jptranstokenizer.tokenization_utils
jptranstokenizer.version
N
Normalizer (class in jptranstokenizer.mainword.base)
S
SentencepieceTokenizer (class in jptranstokenizer.subword.sentencepiece)
SpacyluwTokenizer (class in jptranstokenizer.mainword.spacy_luw)
SudachiTokenizer (class in jptranstokenizer.mainword.sudachi)
T
tokenize() (jptranstokenizer.mainword.base.MainTokenizerABC method)
(jptranstokenizer.mainword.base.Normalizer method)
(jptranstokenizer.mainword.juman.JumanTokenizer method)
(jptranstokenizer.mainword.spacy_luw.SpacyluwTokenizer method)
(jptranstokenizer.mainword.sudachi.SudachiTokenizer method)
(jptranstokenizer.subword.sentencepiece.SentencepieceTokenizer method)