| --- |
| license: apache-2.0 |
| datasets: |
| - TIGER-Lab/MATH-plus |
| language: |
| - en |
| tags: |
| - torchtune |
| - minerva-math |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # jrc/phi3-mini-math |
|
|
| <!-- Provide a quick summary of what the model is/does. --> |
|
|
| Math majors - who needs em? This model can answer any math questions you have. |
|
|
| ## How to Get Started with the Model |
|
|
| Use the code below to get started with the model. |
|
|
| ```python |
| # Load model directly |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True) |
| ``` |
|
|
| ## Training Details |
|
|
| Phi3 was trained using [torchtune](https://github.com/pytorch/torchtune) and the training script + config file are located in this repository. |
|
|
| ```bash |
| tune run lora_finetune_distributed.py --config mini_lora.yaml |
| ``` |
|
|
| You can see a full Weights & Biases run [here](https://api.wandb.ai/links/jcummings/hkey76vj). |
|
|
| ### Training Data |
|
|
| <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
| This model was finetuned on the following datasets: |
|
|
| * [TIGER-Lab/MATH-plus](https://huggingface.co/datasets/TIGER-Lab/MATH-plus): An advanced math-specific dataset with 894k samples. |
|
|
| #### Hardware |
|
|
| * Machines: 4 x NVIDIA A100 GPUs |
| * Max VRAM used per GPU: 29 GB |
| * Real time: 10 hours |
|
|
| ## Evaluation |
|
|
| The finetuned model is evaluated on [minerva-math](https://research.google/blog/minerva-solving-quantitative-reasoning-problems-with-language-models/) using [EleutherAI Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) through torchtune. |
|
|
| ```bash |
| tune run eleuther_eval --config eleuther_evaluation \ |
| checkpoint.checkpoint_dir=./lora-phi3-math \ |
| tasks=["minerva_math"] \ |
| batch_size=32 |
| ``` |
|
|
| | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |
| |------------------------------------|-------|------|-----:|-----------|-----:|---|-----:| |
| |minerva_math |N/A |none | 4|exact_match|0.1670|± |0.0051| |
| | - minerva_math_algebra | 1|none | 4|exact_match|0.2502|± |0.0126| |
| | - minerva_math_counting_and_prob | 1|none | 4|exact_match|0.1329|± |0.0156| |
| | - minerva_math_geometry | 1|none | 4|exact_match|0.1232|± |0.0150| |
| | - minerva_math_intermediate_algebra| 1|none | 4|exact_match|0.0576|± |0.0078| |
| | - minerva_math_num_theory | 1|none | 4|exact_match|0.1148|± |0.0137| |
| | - minerva_math_prealgebra | 1|none | 4|exact_match|0.3077|± |0.0156| |
| | - minerva_math_precalc | 1|none | 4|exact_match|0.0623|± |0.0104| |
| |
| This shows a large improvement over the base Phi3 Mini model. |
| |
| ## Model Card Contact |
| |
| Drop me a line at @official_j3rck |
