AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21
This paper presents a new method called AT-BERT for identifying acronyms in scientific documents. It combines advanced language models with techniques to make the model more reliable.
Content & Liability Disclaimer
This article and its accompanying video are automated summaries derived from the original research paper by Unknown authors. The original research was conducted solely by the paper's authors; PDFdigest did not conduct any of the research and makes no claims of ownership over the underlying scientific work.
The video narration is generated by artificial intelligence and references the paper's authors for attribution. The video is not narrated by any of the paper's authors. This content may contain inaccuracies, omissions, or misinterpretations of the original research. First-person language (e.g., "we found", "our results") reflects the original authors' voice, not PDFdigest's. Always read the original paper for accurate, verified information before making any decisions based on this content.
This content is provided "as is" without any warranties, express or implied. Simulated systems OÜ, its officers, directors, employees, and agents shall not be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from your use of, reliance on, or access to this content, including but not limited to errors, omissions, or misinterpretations of the original research. This disclaimer applies to the fullest extent permitted by applicable law.
- 1 Acronyms are widespread used in many technical documents to reduce duplicate references to the same concept.
- 2 Adversarial training, in which a network is trained on adversarial examples, is an important way to enhance the robustness of neural networks.
- 3 Considering the dataset for acronym identification is relatively small that is easily to be overfit, we incorporate the adversarial training strategy into the BERT-based models to achieve a more robust and generalized performance.
- 4 In addition, due to the complexity of the acronyms in scientific documents and the relatively small training dataset, the model is prone to overfitting.
Introduction
As the growing amount of scientific papers published every year, the number of acronyms is also constantly climbing. Thus, automatic identification of acronyms and discovery of associated definitions are crucial for text understanding tasks, such as question answering , slot filling and definition extraction .
Several approaches have been proposed to solve the acronym identification problem in the last two decades.
The majority of the prior methods are rule-based or feature-based , which employs manually designed rules or features for the acronym and long form predictions.
Methodology
According to the reports , after an analysis of more than 24 million article titles and 18 million article abstracts published between 1950 and 2019, there was at least one acronym in 19% of the titles and 73% of the abstracts. Motivated by the above observations, the first publicly available and the largest manually annotated acronym identification the dataset in scientific domain is released , and the Scientific Document Understanding.
Study Design
The task aims to identify acronyms (i.e., short-forms) and their meanings (i.e.,long-forms) from the documents, a toy example is shown in Table 1 .
To the best of our knowledge, it is the first work to incorporate adversarial training strategy into BERT-based 1 https:\/\/sites.google.com\/view\/sdu-aaai21\/shared-task arXiv:2101.03700v2 [cs.CL] 12 Jan 2021.
Results & Findings
Acronyms are widespread used in many technical documents to reduce duplicate references to the same concept. However, not all acronyms are standard written (i.e., take the first letter of each word and put them together in all capital letters), there are many different ways of writing, e.g., XGBoost is an acronym of eXtreme Gradient Boosting .
- Acronyms are widespread used in many technical documents to reduce duplicate references to the same concept.
- However, not all acronyms are standard written (i.e., take the first letter of each word and put them together in all capital letters), there are many.
- On the contrast, taking advantage of pre-trained word embeddings and deep architecture, deep learning models like LSTM-CRF show promising results for acronym identification .
- Although these works have made great progress, there are still some limitations that hinder further improvement, such as the limited size of manually annotated acronyms and.
- In this paper, we formulate the problem as a sentence-level sequence labeling problem, and design a novel BERT-based ensemble model called Adversarial Training BERT (AT-BERT).
Although these works have made great progress, there are still some limitations that hinder further improvement, such as the limited size of manually annotated acronyms and the noises in the automatically created datasets.
Acronyms are widespread used in many technical documents to reduce duplicate references to the same concept.
How PDFdigest Helps You Understand Research
Instant Paper Analysis
Get structured summaries and key findings from dense PDFs in seconds.
Visual Explanations
Turn complex methods, figures, and results into clearer visual breakdowns.
AI-Powered Q&A
Ask focused questions and get answers grounded in the paper.
Practical Applications
In order to solve the problem that the models may be overfitted and have poor generalization due to less training data, we used the FGM algorithm for adversarial training on various BERT models.
Sequence Labeling and BERT-based Models
The paper formulates acronym identification as a sequence labeling problem, contrasting traditional rule-based methods with modern deep learning approaches, particularly BERT-based models that provide superior contextualized representations.
Adversarial Training
This section explains adversarial training techniques, particularly the Fast Gradient Sign Method (FGSM) and its variants, emphasizing their role in enhancing the robustness of neural networks, especially in the context of small datasets.
Frequently Asked Questions
Several approaches have been proposed to solve the acronym identification problem in the last two decades. Due to the rules\/features are specially designed for finding long forms, these methods have high precision.
The task aims to identify acronyms (i.e., short-forms) and their meanings (i.e.,long-forms) from the documents, a toy example is shown in Table 1 . To the best of our knowledge, it is the first work to incorporate adversarial training strategy into BERT-based.
Acronyms are widespread used in many technical documents to reduce duplicate references to the same concept. Adversarial training, in which a network is trained on adversarial examples, is an important way to enhance the robustness of neural networks.
The overall architecture of the proposed AT-BERT is shown in Figure 1 . In order to solve the problem that the models may be overfitted and have poor generalization due to less training data, we used the FGM algorithm for adversarial training.
Although these works have made great progress, there are still some limitations that hinder further improvement, such as the limited size of manually annotated acronyms and the noises in the automatically created datasets.
This paper presents a new method called AT-BERT for identifying acronyms in scientific documents. It combines advanced language models with techniques to make the model more reliable.