-
CodeFort: Robust Training for Code Generation Models, (EMNLP2024)
- Abstract: Code generation models are not robust to small perturbations, which often lead to inconsistent and incorrect generations and significantly degrade the performance of these models. Improving the robustness of code generation models is crucial to better user experience when these models are deployed in real-world applications. However, existing efforts have not addressed this issue for code generation models. To fill this gap, we propose CodeFort, a framework to improve the robustness of code gene...
- Labels: code generation, code model, code model training, code model, code model robustness
-
LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Models, (TOSEM2024)
- Abstract: Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-a...
- Labels: code model, code model robustness
-
RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code, (ASE2024)
- Abstract: Warning: Please note that this article contains potential harmful or offensive content. This content is only for the evaluating and analysis of LLMs and does not imply any intention to promote criminal activities.The emergence of Large Language Models (LLMs) has significantly influenced various aspects of software development activities. Despite their benefits, LLMs also pose notable risks, including the potential to generate harmful content and being abused by malicious developers to create mal...
- Labels: code generation, benchmark, code model, code model robustness
-
ReCode: Robustness Evaluation of Code Generation Models, (ACL2023)
- Abstract: Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in ...
- Labels: code model, code model training, source code model, code model, code model robustness