User Tools

Site Tools


hugging_face_transformers

Hugging Face - Transformers

Model

Training

Tokenizer

  • tokenizers.BertWordPieceTokenizer - for vocab generation - impl
    • tokenizers.normalizers.BertNormalizer - impl
  • transformers.tokenization_bert.BertTokenizer - tokenizer for normal usage - impl
  • transformers.tokenization_bert.BertTokenizerFast - fast tokenizer for normal usage - impl
  • tokenizers.AutoTokenizer - doc - impl
  • PreTrainedTokenizerBase.__call__ - doc

Pipelines

Data Handling

Important Torch Classes

hugging_face_transformers.txt · Last modified: 2020/10/12 10:27 by pmay