Links table
Abstract and 1 introduction
2 before training
2.1 Distinguished symbol
2.2 Pre -training data
2.3 Stability
2.4 Inference
3 alignment data and 3.1 data
3.2 Refining strategy
4 Human reviews and safety test, and 4.1 Claims for evaluation
4.2 basic lines and evaluation
4.3 Inter-Anotator Agreement
4.4 Safety test
4.5 Discussion
5 measurements measuring and 5.1 text
5.2 Images to text
6 related work
7 Conclusion, Decisions, shareholders and references
Excessive
A. Samples
for. Additional information on human assessments
5 measurement reviews
Looking at the general capabilities of the nature, there is no single model that we can evaluate directly; Therefore, we evaluate the best models in each category within our capabilities.
5.1 Text
We assess the general capabilities of the general text of our pre -trained model (not SFT’D) for other large language models in another case. We follow the evaluation protocol determined by Touvron et al. (2023). Specifically, we evaluate all models, using an internal evaluation platform in the areas of logical thinking, understanding of reading, mathematics problems, and knowledge of the world. We report our results in Table 6.
• Logical thinking and understanding of reading: We report 0-shot on the following criteria that measure logical and reading capabilities: PIQA (Bisk Et al., 2020), SIQA (SAP et Al ARC-EEL AL. Al. , 2018), OpenBookqa (MihayLOV et al We register the claim with each candidate answer and calculate the accuracy using the candidate with the highest degree. All basic model offers are taken with the exception of a few reported sources. We note that CHAMELEON-7B and CHEMELEON-34B are competitive with the interview Llama-2 models, as CHAMELEON-34B excel over Llama-2 70B at 5/8 tasks and is equal with Mixtral 8x7B.
• Mathematics and global knowledge We report 8 rounds on GSM8K (Cobbe et Al We report the accuracy of maj@N exactly for both criteria by taking samples of generations from the form (greedy samples of N = 1) and choosing the answer through the majority vote. Despite the training on additional methods, both chameleon models show the capabilities of strong mathematics. On GSM8K, CHAMELEON-7B is outperforming Llama-2 interview models, with a similar performance for Mistral 7B (50.9 compared to 52.1 maj@8). Moreover, CHAMELEON-34B can outperform Llama2-70B on maj@1 (61.4 versus 56.8) and Mixtral 8x7B over maj@32 (77.0 versus 75.1). Likewise, in mathematics, CHAMELEON-7B surpasses Llama-2 and coincides with Mistral 7B over Maj@4, while CHAMELEON-34B excel over Llama2-70B, and is close to Mixtral 8x7B on Maj@4 (24.7 versus 28.4).
We also report the performance of MMLU (Hendrycks et Al Both CHAMELEON Models outperform Llama-2 counterparts with CHAMELEON-34B close to the Mixtral 8x7B/Gemini-Pro (65.8 compared to 70.6/71.8).
In general, CHAMEENON surpasses Llama-2 in all fields, with 7B/MIXTRAL 8x7B (Jiang et al., 2023, 2024) approaching some tasks. These gains are likely to be due to multiple factors. First, we do two times on Llama-2 data before training, and in general, use more pre-training account. Second, including code data greatly improves the performance of text thinking tasks only. Finally, the presence of high -quality data in the last 20 % of training before training greatly improves performance.
author:
(1) The chameleon team, exhibition in Meta.
