hugging_face_transformers

Table of Contents

Hugging Face - Transformers

Hugging Face - Transformers

Transformers Doc
- Examples
Transformers GitHub
Messageboard
HF @ Medium: https://medium.com/huggingface
Model sharing and uploading: https://huggingface.co/transformers/model_sharing.html

Model

transformers.AutoConfig - doc - impl
Save a pretrained model: https://huggingface.co/transformers/main_classes/model.html?highlight=save_pretrained#transformers.PreTrainedModel.save_pretrained

Training

transformers.TrainingArguments - doc
transformers.Trainer - doc - impl
Metrics (scroll down): https://huggingface.co/transformers/training.html#trainer
Learning Rate Schedules: https://huggingface.co/transformers/main_classes/optimizer_schedules.html?highlight=warm%20restart#learning-rate-schedules-pytorch
Freezing the encoder: https://huggingface.co/transformers/training.html?highlight=freezing#freezing-the-encoder
Early Stopping
- PR
- https://github.com/huggingface/transformers/pull/7431
Custom Loss Function to add class weights for unbalanced datasets: Subclass Trainer and override the compute_loss method (see example here).
Multi Class (multi head) classification: https://discuss.huggingface.co/t/how-do-i-do-multi-class-multi-head-classification/1140

Tokenizer

tokenizers.BertWordPieceTokenizer - for vocab generation - impl
- tokenizers.normalizers.BertNormalizer - impl
transformers.tokenization_bert.BertTokenizer - tokenizer for normal usage - impl
transformers.tokenization_bert.BertTokenizerFast - fast tokenizer for normal usage - impl
tokenizers.AutoTokenizer - doc - impl
PreTrainedTokenizerBase.__call__ - doc

Pipelines

GitHub: https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py
TextClassificationPipeline → Pipeline → _ScikitCompat

Data Handling

https://github.com/huggingface/nlp
Usage Example: https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM
Fine-tuning with custom datasets: https://huggingface.co/transformers/master/custom_datasets.html#fine-tuning-with-custom-datasets

Important Torch Classes

DataLoader

Links & Know-how

hugging_face_transformers.txt · Last modified: 2020/10/12 10:27 by pmay