From Automation to Autonomy: A Survey of Agentic Workflows in CI/CD Orchestration

Ali Amar; Ibrahim Qaiser; Ayesha Kanwal

Authors

Ali Amar School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan
Ibrahim Qaiser School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan
Ayesha Kanwal School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan

Keywords:

CI/CD, Agentic AI, Large Language Models, Multi-Agent Systems, DevOps

Abstract

Continuous Integration and Continuous Deployment (CI/CD) pipelines are foundational to modern software delivery, yet remain reliant on rigid, pre-defined scripts that lack the flexibility to handle unforeseen anomalies. While prior surveys have examined Large Language Model (LLM) agents for general software engineering tasks, none have focused specifically on CI/CD orchestration and the transition from automation to autonomy. Through a structured review of 72 papers published between 2023 and 2025 and sourced from IEEE Xplore, ACM Digital Library, and arXiv, this survey addresses that gap. The 72 studies are distributed across three primary application domains: autonomous code generation and repair (28 papers), intelligent verification and environment setup (23 papers), and incident management with root cause analysis (21 papers). We propose the PARA (Perception, Action, Reasoning, Reflection) framework as an operational lens for analyzing agentic CI/CD systems. Comparative analysis of five representative systems yields concrete performance figures: SWE-agent resolves 12.5% of issues against a 3.8% scripted baseline; the DEI multi-agent committee reaches 34.3% versus 27.3% for single-agent baselines; CXXCrafter achieves 71.2% success on C/C++ builds compared with 45% for general-purpose agents; MACOG reaches 74.02% on Terraform synthesis, dropping to 61.45% when its Security Prover is ablated; and Flow reports a 67% reduction in Mean Time to Resolution for incident triage. Reported task success across the 72 studies ranges from 15% to 75% as a function of task complexity, and self-correction loops add a further 4–5% per iteration at exponential token cost. Challenges spanning economic viability, security risks, and reliability concerns are systematically analyzed. We conclude that the shift from scripted automation to autonomous agents represents a significant evolution in DevOps practices toward intent-driven orchestration, and outline future directions, including Knowledge Graph-augmented LLMs and standardized Agent-Tool Protocols.

References

Junda He, Christoph Treude, David Lo, “LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead,” arXiv:2404.04834, 2024, [Online]. Available: https://arxiv.org/abs/2404.04834

Lingzhe Zhang, Tong Jia, Mengxi Jia, Yifan Wu, Aiwei Liu, Yong Yang, Zhonghai Wu, Xuming Hu, Philip S. Yu, Ying Li, “A Survey of AIOps in the Era of Large Language Models,” arXiv:2507.12472, 2025, [Online]. Available: https://arxiv.org/abs/2507.12472

Chunqiu Steven Xia, Yinlin Deng, “Demystifying LLM-Based Software Engineering Agents,” Proc. ACM Softw. Eng., vol. 2, 2025, [Online]. Available: https://dl.acm.org/doi/10.1145/3715754

Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, Huaming Chen, “From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future,” arXiv:2408.02479, 2025, [Online]. Available: https://arxiv.org/abs/2408.02479

M. S. C. Mr. Balajee Asish Brahmandam, Mr. Vishal Narender Punjabi, “AI-Augmented DevOps: Autonomous Software Delivery with Large Language Models,” IJIRMP, vol. 13, no. 3, 2025, [Online]. Available: https://www.ijirmps.org/papers/2025/3/232448.pdf

Yihong Dong, Xue Jiang, “A Survey on Code Generation with LLM-based Agents,” arXiv:2508.00083v1, 2025, [Online]. Available: https://arxiv.org/html/2508.00083v1

Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, Wei Wang, “CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories,” arXiv:2502.06111, 2025, [Online]. Available: https://arxiv.org/abs/2502.06111

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer, “Survey on Evaluation of LLM-based Agents,” arXiv:2503.16416, 2025, [Online]. Available: https://arxiv.org/abs/2503.16416

Hatalis, K., Christou, D., Myers, J., Jones, S., Lambert, K., Amos-Binks, A., Dannenhauer, Z., & Dannenhauer, D, “Memory Matters: The Need to Improve Long-Term Memory in LLM-Agents,” Proc. AAAI Symp. Ser., vol. 2, no. 1, pp. 277–280, 2024, doi: https://doi.org/10.1609/aaaiss.v2i1.27688.

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang, “Large Language Model based Multi-Agents: A Survey of Progress and Challenges,” arXiv:2402.01680, 2024, [Online]. Available: https://arxiv.org/abs/2402.01680

Xinyi Hou, Yanjie Zhao, “Large Language Models for Software Engineering: A Systematic Literature Review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 1–79, 2024, [Online]. Available: https://dl.acm.org/doi/10.1145/3695988

Yixin Liu, Guibin Zhang, Kun Wang, Shiyuan Li, Shirui Pan, “Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects,” arXiv:2507.21407, 2025, [Online]. Available: https://arxiv.org/abs/2507.21407

Zhengmin Yu, Yuan Zhang, Ming Wen, Yinan Nie, Wenhui Zhang, Min Yang, “CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building,” arXiv:2505.21069, 2025, [Online]. Available: https://arxiv.org/abs/2505.21069

Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, Yufei Ding, “KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows,” arXiv:2507.07400, 2025, [Online]. Available: https://arxiv.org/abs/2507.07400

Dheer Toprani, Vijay K. Madisetti, “LLM Agentic Workflow for Automated Vulnerability Detection and Remediation in Infrastructure-as-Code,” IEEE Access, vol. 13, 2025, [Online]. Available: https://ieeexplore.ieee.org/document/10965635

Thorsten Händler, “Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures,” arXiv:2310.03659, 2023, [Online]. Available: https://arxiv.org/abs/2310.03659

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin, “CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges,” arXiv:2401.07339, 2024, [Online]. Available: https://arxiv.org/abs/2401.07339

Haolin Jin, Zechao Sun, Huaming Chen, “RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance,” arXiv:2410.01242, 2024, [Online]. Available: https://arxiv.org/abs/2410.01242

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji, “Executable Code Actions Elicit Better LLM Agents,” arXiv:2402.01030, 2024, [Online]. Available: https://arxiv.org/abs/2402.01030

V. Gogineni, “LLM-Powered Multi-Agent Systems: A Technical Framework for Collaborative Intelligence Through Optimized Knowledge Retrieval and Communication,” 2025 6th Int. Conf. Artif. Intell. Robot. Control. AIRC 2025, pp. 452–456, 2025, doi: 10.1109/AIRC64931.2025.11077480.

Yoichi Ishibashi, Yoshimasa Nishimura, “Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization,” arXiv:2404.02183, 2024, [Online]. Available: https://arxiv.org/abs/2404.02183

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang, “Agentless: Demystifying LLM-based Software Engineering Agents,” arXiv:2407.01489, 2024, [Online]. Available: https://arxiv.org/abs/2407.01489

Siru Liu, Allison B. McCoy, “Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines,” J. Am. Med. Inform. Assoc., vol. 32, no. 4, 2025, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/39812777/

Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, “MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework,” arXiv:2308.00352, 2023, [Online]. Available: https://arxiv.org/abs/2308.00352

Zhuoyun Du, Chen Qian, Wei Liu, Zihao Xie, YiFei Wang, Rennai Qiu, Yufan Dang, Weize Chen, Cheng Yang, Ye Tian, Xuantang Xiong, Lei Han, “Multi-Agent Collaboration via Cross-Team Orchestration,” arXiv:2406.08979, 2024, [Online]. Available: https://arxiv.org/abs/2406.08979

Zeeshan Rasheed, Malik Abdul Sami, Kai-Kristian Kemell, Muhammad Waseem, Mika Saari, Kari Systä, Pekka Abrahamsson, “CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology,” arXiv:2402.01411, 2024, [Online]. Available: https://arxiv.org/abs/2402.01411

Muhammad Haseeb, “Context Engineering for Multi-Agent LLM Code Assistants Using Elicit, NotebookLM, ChatGPT, and Claude Code,” arXiv:2508.08322, 2025, [Online]. Available: https://arxiv.org/abs/2508.08322

Yihong Dong, Xue Jiang, Zhi Jin, Ge Li, “Self-collaboration Code Generation via ChatGPT,” arXiv:2304.07590, 2023, [Online]. Available: https://arxiv.org/abs/2304.07590

Louis Milliken, Sungmin Kang, Shin Yoo, “Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects,” arXiv:2412.06294, 2024, [Online]. Available: https://arxiv.org/abs/2412.06294

Patara Trirat, Wonyong Jeong, Sung Ju Hwang, “AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML,” arXiv:2410.02958, 2024, [Online]. Available: https://arxiv.org/abs/2410.02958

Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, Tongliang Liu, “Flow: Modularized Agentic Workflow Automation,” arXiv:2501.07834, 2025, [Online]. Available: https://arxiv.org/abs/2501.07834

Anna Kalyuzhnaya, Sergey Mityagin, “LLM Agents for Smart City Management: Enhancing Decision Support Through Multi-Agent AI Systems,” Smart Cities, vol. 8, no. 1, p. 19, 2025, doi: https://doi.org/10.3390/smartcities8010019.

Masoud Shokrnezhad, Tarik Taleb, “An Autonomous Network Orchestration Framework Integrating Large Language Models with Continual Reinforcement Learning,” arXiv:2502.16198, 2025, [Online]. Available: https://arxiv.org/abs/2502.16198

Zelong Li, Wenyue Hua, Hao Wang, He Zhu, Yongfeng Zhang, “Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents,” arXiv:2402.00798, 2024, [Online]. Available: https://arxiv.org/abs/2402.00798

Joshua Owotogbe, “Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering,” arXiv:2505.03096, 2025, [Online]. Available: https://arxiv.org/abs/2505.03096

Krishna Ronanki, “Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering,” arXiv:2505.04251, 2025, [Online]. Available: https://arxiv.org/abs/2505.04251

C. Sun, S. Huang, and D. Pompili, “LLM-Based Multi-Agent Decision-Making: Challenges and Future Directions,” IEEE Robot. Autom. Lett., vol. 10, no. 6, pp. 5681–5688, 2025, doi: 10.1109/LRA.2025.3562371.

Amine Ben Hassouna, Hana Chaari, Ines Belhaj, “LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architectures,” arXiv:2409.11393, 2024, [Online]. Available: https://arxiv.org/abs/2409.11393

Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing, Dehai Zhao, Hao Zhang, “Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture,” arXiv:2411.13768, 2024, [Online]. Available: https://arxiv.org/abs/2411.13768

Xiang Fei, Xiawu Zheng, Hao Feng, “MCP-Zero: Active Tool Discovery for Autonomous LLM Agents,” arXiv:2506.01056, 2025, [Online]. Available: https://arxiv.org/abs/2506.01056

Xiaoyu Tan, Bin Li, “Meta-Agent-Workflow: Streamlining Tool Usage in LLMs through Workflow Construction, Retrieval, and Refinement,” WWW Companion 2025 - Companion Proc. ACM Web Conf. 2025, 2025, [Online]. Available: https://dl.acm.org/doi/10.1145/3701716.3715247

Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, Weinan Zhang, “AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems,” arXiv:2504.00587, 2025, [Online]. Available: https://arxiv.org/abs/2504.00587