jptranstokenizer
latest

User Guide

  • Quickstart
  • Available Pretrained Models

Featured API

  • JapaneseTransformerTokenizer
  • JumanTokenizer
  • SpacyluwTokenizer
  • SudachiTokenizer
  • SentencepieceTokenizer

All API

  • List
jptranstokenizer
  • jptranstokenizer
  • Edit on GitHub

jptranstokenizer

  • jptranstokenizer.mainword
    • jptranstokenizer.mainword.base
      • MainTokenizerABC
        • MainTokenizerABC.tokenize()
      • Normalizer
        • Normalizer.tokenize()
    • jptranstokenizer.mainword.juman
      • JumanTokenizer
        • JumanTokenizer.tokenize()
    • jptranstokenizer.mainword.spacy_luw
      • SpacyluwTokenizer
        • SpacyluwTokenizer.tokenize()
    • jptranstokenizer.mainword.sudachi
      • SudachiTokenizer
        • SudachiTokenizer.tokenize()
  • jptranstokenizer.subword
    • jptranstokenizer.subword.sentencepiece
      • SentencepieceTokenizer
        • SentencepieceTokenizer.tokenize()
  • jptranstokenizer.tokenization_utils
    • JapaneseTransformerTokenizer
      • JapaneseTransformerTokenizer.convert_tokens_to_string()
      • JapaneseTransformerTokenizer.from_pretrained()
    • get_word_tokenizer()
  • jptranstokenizer.version
Previous Next

© Copyright 2022, Masahiro Suzuki. Revision 1d7f7c2d.

Built with Sphinx using a theme provided by Read the Docs.