Links table
Abstract and 1 introduction
2 related work
3 Method and 3.1 Stage 1: Classification generation
3.2 Stage 2: LLM-UAGTED text classification
4 Evaluation set and 4.1 rating strategies for the first stage
4.2 stage 2 evaluation strategies
5 experiments and 5.1 data
5.2 Classification generation
5.3 Llm washing text classification
5.4 Summary of results and suggestions
6 discussion, future work, references
Categories
Additional results
Implementation details
Quick templates
4 evaluation suite
Due to the nature that is not under the supervision of the problem we study and the lack of a standard standard, a quantitative evaluation of the generation of comprehensive classification and the classification of the text can be difficult. So we design a set of evaluation strategies TNT -lm. Our evaluation strategies may be classified into three buckets, depending on the type and source of evaluation criteria. The three categories are as follows:
• automatic automatic evaluation: This type of approach is developed and consistent, but it requires well -defined standard rules and explanations. It is less applicable to assess the abstract aspects that have been studied in this paper, such as the quality and benefit of classification of stickers.
Human evaluation: These methods are useful for evaluating abstract aspects that automatic evaluations can address. However, it takes a long and expensive time, and it may face the limitations of data privacy and compliance.
• LLM assessments: Here, LLMS is used to perform the same or similar tasks as human residents. This type of evaluation is more able to develop and costly effective than human evaluation, although it is likely to be subject to biases and errors if they are not applied properly. Therefore, we aim to combine the LLM assessment and verify the validity of the human evaluation measures on small companies so that we can devise conclusions with sufficient statistical power.
4.1 Phase 1 Evaluation Strategies
After previous studies [23, 30]We assess the classification of stickers on three criteria: coverage, accuracy and importance to teach the state of use. Note that we are asking the implementation of the original basic designation for the application of these measures. For assembly -based methods, this is created through the assembly algorithm. to TNTLM, This is done by the designation of the designation as shown in section 3.2. We also note that the accuracy of stickers and standards is the importance of the state of use discussed here. Preamble and Llm Residents.
Crade Cover. This scale measures comprehensively the classification of the stickers generated by the body. Traditional text assembly methods (for example, the inclusion -based means) often achieve 100 % coverage depending on the design. In the LLM classification pipeline, we add “another” or “unlimited” category in the designation of the designation by design and measuring the data points assigned to this category. The lower this percentage, the higher the classification coverage.
The accuracy of the name. This scale determines the extent to which the designated mark reflects the text data point, for other stickers in the same classification. Similar to the collection of the mixture model, the basic poster should be the most likely to be text. We assume that human residents and LLM can evaluate the poster that fits and described it. We deal with accuracy as the task of comparing the spouses: for each text, we get the basic poster and a random negative naming from the same classification, and we ask Rater to choose the most accurate poster based on their names and descriptions.[1] If the Rater properly determines the positive poster, we consider it a “blow” and report the average rate of a scalp of the designation accuracy. We do not explicitly evaluate the overlap through the category stickers and expect that they will implicitly reflect on the marital stickers accuracy scale.
The importance of case instructions. This scale measures the extent to which the naming classification is corrected with the state of use state. For example, “creation of content” is related to the instructions “Understanding the user’s intention in conversation”, while “history and culture” is not. We manage this as a dual classification task: for each case, we offer the basic name name and describe it to Rather or LLM Rater, and we ask them to decide whether the stickers are related to teaching the specified case or not. Note that we direct Rater to use the prescribed counterpart as the context, and evaluate the conditional importance of the poster’s ability to describe some aspects of the introduction of the text accurately. The aim of this scale is not to assess the accuracy of the poster, but rather to exclude the randomness that was presented through classifications that appear related to the education of the situation, but it is not related to the group sample-and thus benefit from the estuary applications.
[1] The “nothing” option is presented to the side of the husband, but they are directed to reduce its use.
Authors:
(1) Mingting Wan, Microsoft Corporation and Microsoft Corporation;
(2) Tara Safavi (opposite authors), Microsoft Corporation;
(3) Sujay Kumar Jauhar, Microsoft Corporation;
(4) Yujin Kim, Microsoft Corporation;
(5) Scott Counts, Microsoft Corporation;
(6) Jennifer Neville, Microsoft Company;
(7) Siddhath Suri, Microsoft Corporation;
(8) Chirag Shah, Washington University and work accomplished while working in Microsoft;
(9) Ryen W. White, Microsoft Corporation;
(10) Longqi Yang, Microsoft Corporation;
(11) Red Andersen, Microsoft;
(12) George Bosher, Microsoft;
(13) Dhruv Joshi, Microsoft Corporation;
(14) Nago Ranjan, Microsoft Company.
