jptranstokenizer
latest

User Guide

  • Quickstart
  • Available Pretrained Models

Featured API

  • JapaneseTransformerTokenizer
  • JumanTokenizer
  • SpacyluwTokenizer
  • SudachiTokenizer
  • SentencepieceTokenizer

All API

  • List
jptranstokenizer
  • jptranstokenizer
  • jptranstokenizer.mainword
  • Edit on GitHub

jptranstokenizer.mainword

  • jptranstokenizer.mainword.base
    • MainTokenizerABC
      • MainTokenizerABC.tokenize()
    • Normalizer
      • Normalizer.tokenize()
  • jptranstokenizer.mainword.juman
    • JumanTokenizer
      • JumanTokenizer.tokenize()
  • jptranstokenizer.mainword.spacy_luw
    • SpacyluwTokenizer
      • SpacyluwTokenizer.tokenize()
  • jptranstokenizer.mainword.sudachi
    • SudachiTokenizer
      • SudachiTokenizer.tokenize()
Previous Next

© Copyright 2022, Masahiro Suzuki. Revision 1d7f7c2d.

Built with Sphinx using a theme provided by Read the Docs.