Awesome-Code-LLM

"DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production" [2024-12] [ ]

"NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System" [ ] [ ]

"Mapping Language to Code in Programmatic Context" [ ] [ ]

"JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation" [ ] [ ]

"Measuring Coding Challenge Competence With APPS" [ ] [ ]

"Evaluating Large Language Models Trained on Code" [ ] [ ]

"Program Synthesis with Large Language Models" [ ] [ ] [ ]

"PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context" [ ] [ ]

"Training and Evaluating a Jupyter Notebook Data Science Assistant" [ ] [ ]

"Competition-Level Code Generation with AlphaCode" [ ] [ ]

"MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages" [ ] [ ]

"AixBench: A Code Generation Benchmark Dataset" [ ] [ ]

"MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation", [ ] [ ]

"Multi-lingual Evaluation of Code Generation Models" [ ] [ ]

"Multi-lingual Evaluation of Code Generation Models" [ ] [ ]

"Multi-lingual Evaluation of Code Generation Models" [ ] [ ]

"Execution-based Evaluation for Data Science Code Generation Models" [ ] [ ]

"DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation" [ ] [ ]

"Execution-Based Evaluation for Open-Domain Code Generation" [ ] [ ]

"CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models" [ ] [ ]

"XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval" [ ] [ ]

"CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X" [ ] [ ]

"Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation" [ ] [ ]

"StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code" [ ] [ ]

"OctoPack: Instruction Tuning Code Large Language Models" [ ] [ ]

"Guiding Language Models of Code with Global Context using Monitors" [ ] [ ]

"CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models" [ ] [ ]

"VerilogEval: Evaluating Large Language Models for Verilog Code Generation" [ ] [ ]

"ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks" [ ] [ ]

"TACO: Topics in Algorithmic COde generation dataset" [ ] [ ]

"PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs" [ ] [ ]

"Can Large Language Models Write Parallel Code?" [ ] [ ]

"OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models" [ ] [ ]

"HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization" [ ] [ ]

"Can Language Models Solve Olympiad Programming?" [ ] [ ]

"PECC: Problem Extraction and Coding Challenges" [ ] [ ]

"Constrained Decoding for Secure Code Generation" [ ] [ ]

"NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts" [ ] [ ]

"MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation" [ ] [ ]

"VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation" [ ]

"AICoderEval: Improving AI Domain Code Generation of Large Language Models" [ ] [ ]

"VersiCode: Towards Version-controllable Code Generation" [ ] [ ]

"ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation" [ ]

"BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions" [ ] [ ]

"CodeUpdateArena: Benchmarking Knowledge Editing on API Updates" [ ] [ ]

"On Leakage of Code Generation Evaluation Datasets" [ ] [ ]

"NoviCode: Generating Programs from Natural Language Utterances by Novices" [ ] [ ]

"Case2Code: Learning Inductive Reasoning with Synthetic Data" [ ] [ ]

"SciCode: A Research Coding Benchmark Curated by Scientists" [ ] [ ]

"Generating Unseen Code Tests In Infinitum" [ ]

"WebApp1K: A Practical Code-Generation Benchmark for Web App Development" [ ] [ ]

"CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow" [ ] [ ]

"DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation" [ ] [ ]

"ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code" [ ] [ ]

"Contextualized Data-Wrangling Code Generation in Computational Notebooks" [ ] [ ]

"Evaluation of Code LLMs on Geospatial Code Generation" [ ] [ ]

"mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation" [ ] [ ]

"Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists" [ ] [ ]

"GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models" [ ] [ ]

"One-to-many testing for code generation from (just) natural language" [ ] [ ]

"LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation" [ ]

"Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code" [ ] [ ]

"Evaluating and Aligning CodeLLMs on Human Preference" [ ] [ ]

"Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar" [ ] [ ]

"MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems" [ ] [ ]

"Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots" [ ] [ ]

"ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation" [ ] [ ]

"HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks" [ ] [ ]

"TurtleBench: A Visual Programming Benchmark in Turtle Geometry" [ ] [ ]

"BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks" [ ] [ ]

"CodeQA: A Question Answering Dataset for Source Code Comprehension" [ ] [ ]

"CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course" [ ] [ ]

"CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models" [ ] [ ]

"CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution" [ ] [ ]

"Multiple-Choice Questions are Efficient and Robust LLM Evaluators" [ ] [ ]

"Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering" [ ] [ ]

"RepoQA: Evaluating Long Context Code Understanding" [ ] [ ]

"CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution" [ ] [ ]

"SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications" [ ] [ ]

"CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs" [ ] [ ]

"Leveraging Large Language Models in Code Question Answering: Baselines and Issues" [ ] [ ]

"ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges" [ ] [ ]

"Deep learning driven natural languages text to SQL query conversion: A survey", 2022-08, arXiv, [ ]

"Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect", 2022-08, COLING 2022, [ ]

"A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions", 2022-08, arXiv, [ ]

"A survey on deep learning approaches for text-to-SQL", 2023-01, VLDB J., [ ]

"Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning" [ ] [ ]

"Improving Text-to-SQL Evaluation Methodology" [ ] [ ]

"Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task" [ ] [ ]

"SParC: Cross-Domain Semantic Parsing in Context" [ ] [ ]

"Text-to-SQL Generation for Question Answering on Electronic Medical Records" [ ] [ ]

"CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases" [ ] [ ]

"Dataset and Enhanced Model for Eligibility Criteria-to-SQL Semantic Parsing" [ ] [ ]

"On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries" [ ] [ ]

"Structure-Grounded Pretraining for Text-to-SQL" [ ] [ ]

"Towards Robustness of Text-to-SQL Models against Synonym Substitution" [ ] [ ]

"Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data" [ ] [ ]

"KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers" [ ] [ ]

"Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization" [ ] [ ]

"Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignment" [ ] [ ]

"Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs" [ ] [ ]

"XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations" [ ] [ ]

"EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records" [ ]

"BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain" [ ] [ ]

"MultiSQL: A Schema-Integrated Context-Dependent Text2SQL Dataset with Diverse SQL Operations" [ ] [ ]

"BEAVER: An Enterprise Benchmark for Text-to-SQL" [ ]

"PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries" [ ]

"BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios" [ ] [ ]

"Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows" [ ] [ ]

"Unsupervised Translation of Programming Languages" [ ] [ ]

"CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation" [ ] [ ]

"AVATAR: A Parallel Corpus for Java-Python Program Translation" [ ] [ ]

"Multilingual Code Snippets Training for Program Translation" [ ] [ ]

"XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence" [ ] [ ]

"xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval" [ ] [ ]

"CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X" [ ] [ ]

"On the Evaluation of Neural Code Translation: Taxonomy and Benchmark" [ ] [ ]

"CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation" [ ] [ ]

"Escalating LLM-based Code Translation Benchmarking into the Class-level Era" [ ]

"Repository-level Code Translation Benchmark Targeting Rust" [ ] [ ]

"Neural Program Repair: Systems, Challenges and Solutions", 2022-02, Internetware 2022, [ ]

"A Survey of Learning-based Automated Program Repair", 2023-01, arXiv, [ ]

"A Survey on Automated Program Repair Techniques", 2023-03, arXiv, [ ]

"Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs" [ ] [ ]

"The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs" [ ] [ ]

"Discovering Bug Patterns in JavaScript" [ ] [ ]

"DeepFix: Fixing Common C Language Errors by Deep Learning" [ ] [ ]

"DeepFix: Fixing Common C Language Errors by Deep Learning" [ ] [ ]

"QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge" [ ] [ ]

"Bugs.jar: a large-scale, diverse dataset of real-world Java bugs" [ ] [ ]

"An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation" [ ] [ ]

"Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies" [ ] [ ]

"On Learning Meaningful Code Changes via Neural Machine Translation" [ ] [ ]

"BugsJS: a Benchmark of JavaScript Bugs" [ ] [ ]

"BugSwarm: mining and continuously growing a dataset of reproducible failures and fixes" [ ] [ ]

"Graph-based mining of in-the-wild, fine-grained, semantic code change patterns" [ ] [ ]

"How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset" [ ] [ ]

"Re-factoring based program repair applied to programming assignments" [ ] [ ]

"CoCoNuT: combining context-aware neural translation models using ensemble for program repair" [ ] [ ]

"Review4Repair: Code Review Aided Automatic Program Repairing" [ ] [ ]

"BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies" [ ] [ ]

"TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer" [ ] [ ]

"Megadiff: A Dataset of 600k Java Source Code Changes Categorized by Diff Size" [ ] [ ]

"TSSB-3M: Mining single statement bugs at massive scale" [ ] [ ]

"FixJS: a dataset of bug-fixing JavaScript commits" [ ] [ ]

"PyTER: Effective Program Repair for Python Type Errors" [ ] [ ]

"xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval" [ ] [ ]

"RunBugRun -- An Executable Dataset for Automated Program Repair" [ ] [ ]

"OctoPack: Instruction Tuning Code Large Language Models" [ ] [ ]

"DebugBench: Evaluating Debugging Capability of Large Language Models" [ ] [ ]

"MdEval: Massively Multilingual Code Debugging" [ ]

"A Survey of Automatic Source Code Summarization", 2022-02, Symmetry, [ ]

"Summarizing Source Code using a Neural Attention Model" [ ] [ ]

"A parallel corpus of Python functions and documentation strings for automated code documentation and code generation" [ ] [ ]

"Deep code comment generation" [ ] [ ]

"Summarizing Source Code with Transferred API Knowledge" [ ] [ ]

"Improving Automatic Source Code Summarization via Deep Reinforcement Learning" [ ] [ ]

"CodeSearchNet Challenge: Evaluating the State of Semantic Code Search" [ ] [ ]

"OctoPack: Instruction Tuning Code Large Language Models" [ ] [ ]

"Benchmarking Software Vulnerability Detection Techniques: A Survey", 2023-03, arXiv, [ ]

"VulDeePecker: A Deep Learning-Based System for Vulnerability Detection" [ ] [ ]

"Cross-Project Transfer Representation Learning for Vulnerable Function Discovery" [ ] [ ]

"Automated Vulnerability Detection in Source Code Using Deep Representation Learning" [ ] [ ]

"SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities" [ ] [ ]

"A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software" [ ] [ ]

"Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks" [ ] [ ]

"Software Vulnerability Discovery via Learning Multi-Domain Knowledge Bases" [ ] [ ]

"Global Relational Models of Source Code" [ ] [ ]

"μVulDeePecker: A Deep Learning-Based System for Multiclass Vulnerability Detection" [ ] [ ]

"Deep Learning-Based Vulnerable Function Detection: A Benchmark" [ ] [ ]

"Deep Learning based Vulnerability Detection: Are We There Yet?" [ ] [ ]

"A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries" [ ] [ ]

"D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis" [ ] [ ]

"Self-Supervised Bug Detection and Repair" [ ] [ ]

"CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software" [ ] [ ]

"CrossVul: a cross-language vulnerability dataset with commit data" [ ] [ ]

"DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection" [ ] [ ]

"Limits of Machine Learning for Automatic Vulnerability Detection" [ ] [ ]

"How Far Have We Gone in Vulnerability Detection Using Large Language Models" [ ] [ ]

"Vulnerability Detection with Code Language Models: How Far Are We?" [ ]

"VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models" [ ] [ ]

"CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?" [ ] [ ]

"CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics" [ ] [ ]

"A Survey of Deep Code Search", 2023-05, arXiv, [ ]

"StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow" [ ] [ ]

"Deep Code Search" [ ] [ ]

"Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow" [ ] [ ]

"Neural Code Search Evaluation Dataset" [ ] [ ]

"CodeSearchNet Challenge: Evaluating the State of Semantic Code Search" [ ] [ ]

"Are the Code Snippets What We Are Searching for? A Benchmark and an Empirical Study on Code Search with Natural-Language Queries" [ ] [ ]

"Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent" [ ] [ ]

"Deep Graph Matching and Searching for Semantic Code Retrieval" [ ] [ ]

"CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation" [ ] [[data]]

"CoSQA: 20,000+ Web Queries for Code Search and Question Answering" [ ] [ ]

"ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search" [ ] [ ]

"CoSQA+: Enhancing Code Search Dataset with Matching Code" [ ] [ ]

"CoIR: A Comprehensive Benchmark for Code Information Retrieval Models" [ ] [ ]

"What can Large Language Models Capture about Code Functional Equivalence?" [ ]

"TypeWriter: Neural Type Prediction with Search-based Validation" [ ] [ ]

"Typilus: Neural Type Hints" [ ] [ ]

"LambdaNet: Probabilistic Type Inference using Graph Neural Networks" [ ] [ ]

"ManyTypes4Py: A Benchmark Python Dataset for Machine Learning-based Type Inference" [ ] [ ]

"ManyTypes4TypeScript: a comprehensive TypeScript dataset for sequence-based type inference" [ ] [ ]

"Do Machine Learning Models Produce TypeScript Types That Type Check?" [ ] [ ]

"TypeT5: Seq2seq Type Inference using Static Analysis" [ ] [ ]

"Type Prediction With Program Decomposition and Fill-in-the-Type Training" [ ] [ ]

"On the Evaluation of Commit Message Generation Models: An Experimental Study", 2021-07, ICSME 2021, [ ]

"Towards Automatic Generation of Short Summaries of Commits" [ ] [ ]

"A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes" [ ] [ ]

"Automatically Generating Commit Messages from Diffs using Neural Machine Translation" [ ] [ ]

"Neural-machine-translation-based commit message generation: how far are we?" [ ] [ ]

"Generating commit messages from diffs using pointer-generator network" [ ] [[data(

"Commit message generation for source code changes" [ ] [ ]

"ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking" [ ] [ ]

"CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model" [ ] [ ]

"On the Evaluation of Commit Message Generation Models: An Experimental Study" [ ] [ ]

"Context-aware Retrieval-based Deep Commit Message Generation" [ ] [ ]

"Delving into Commit-Issue Correlation to Enhance Commit Message Generation Models" [ ] [ ]

"From Commit Message Generation to History-Aware Commit Message Completion" [ ] [ ]

"RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation" [ ] [ ]

"RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems" [ ] [ ]

"Guiding Language Models of Code with Global Context using Monitors" [ ] [ ]

"RepoFusion: Training Code Models to Understand Your Repository" [ ] [ ]

"BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models" [ ] [ ]

"CodePlan: Repository-level Coding using LLMs and Planning" [ ] [ ]

"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" [ ] [ ]

"CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion" [ ] [ ]

"EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories" [ ] [ ]

"DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories" [ ] [ ]

"Can AI Beat Undergraduates in Entry-level Java Assignments? Benchmarking Large Language Models on JavaBench" [ ] [ ]

"Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond" [ ] [ ]

"REPOEXEC: Evaluate Code Generation with a Repository-Level Executable Benchmark" [ ]

"RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale" [ ] [ ]

"SWE-bench-java: A GitHub Issue Resolving Benchmark for Java" [ ] [ ]

"Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?" [ ] [ ]

"SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?" [ ] [ ]

"SWE-Bench+: Enhanced Coding Benchmark for LLMs" [ ] [ ]

"DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models" [ ] [ ]

"Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'" [ ]

"M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation" [ ] [ ]

"A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models" [ ] [ ]

"Commit0: Library Generation from Scratch" [ ] [ ]