jptranstokenizer documentation

jptranstokenizer provides various combinations of main-word and sub-word tokenizers. You can use JapaneseTransformerTokenizer like transformers.BertJapaneseTokenizer.

Install

pip install jptranstokenizer
python
>> from jptranstokenizer import JapaneseTransformerTokenizer

User Guide

• Quickstart
• Available Pretrained Models

Featured API

• JapaneseTransformerTokenizer
• JumanTokenizer
• SpacyluwTokenizer
• SudachiTokenizer
• SentencepieceTokenizer

All API

• List

• Index