AIDev: Studying AI Coding Agents on GitHub

README.md 2025-11-05 1 / 6 AID e v : S tu d y i n g AI C o di n g A ge nts on G i t H u b ( T he R i s e o f AI T ea mm a t e s i n S o f tw a r e E n gi n ee r i n g 3.0) 📢 W e ʼ r e h ost i n g t he MSR 2026 M i n i n g C ha ll e n ge ( c o - lo ca t ed w i t h ICSE 2026 i n R i o de J a n ei ro , B r a z i l ). D e t ai ls a n d su b m i ss i ons : ⚠ ⚠ ⚠ h ttps ://2026. msr c on f . or g / tr ack / msr -2026m i n i n g - cha ll e n ge ⚠ ⚠ ⚠ arXiv arXiv 2507.15003 2507.15003 GitHub GitHub Code Code DOI DOI 10.5281/zenodo.16919272 10.5281/zenodo.16919272 P a p e r : h ttps :// a rx i v . or g / ab s /2507.15003 G i t H u b : h ttps :// gi t h u b . c om / SAILR e s ea r ch / AI _ T ea mm a t e s _ i n _ SE 3 E x a mpl e N ot eb oo k s : D e s c r i pt i on N ot eb oo k L i n k O p e n i n C ol ab B a s ic us age lo ad _ AID e v . i pyn b Open in Colab Open in Colab D a t a s e t ov e rv ie w da t a s e t _ ov e rv ie w . i pyn b Open in Colab Open in Colab A n a lys i s o f pro g r a mm i n g us age l a n g u age _ us age . i pyn b Open in Colab Open in Colab PR m e r ge r a t e a n d turn a roun d t i m e pro d u c t i v i ty . i pyn b Open in Colab Open in Colab ⚠ U p da t e ( O c t 28, 2025): pr_commit_details ha s bee n up da t ed to i n c lu de a ll p a t che s fe t ched f rom G i t H u b API ( w hich d o e s not prov ide c ont e nt f or l a r ge p a t che s ). U s e rs must v e r if y a n d c omply w i t h t he sp ecific l ice ns e o f each sour ce r e pos i tory . ⚠ U p da t e ( O c t 16, 2025): pr_review_comments.parquet d o e s not c ont ai n f ull da t a po i nts , us e pr_review_comments_v2.parquet i nst ead . ⚠ U p da t e ( A u g 10, 2025): T he da t a s e t ha s bee n r ef r e s hed to i n c lu de da t a up to A u g ust 1, 2025 , e nsur i n g our da t a s e t r ef l ec ts t he most r ece nt tr e n d s i n c o di n g age nts . O v e rv ie w AID e v i s a l a r ge - s ca l e da t a s e t ca ptur i n g t he e m e r ge n ce o f a utonomous c o di n g age nts ( AI t ea mm a t e s ) w i t hi n r ea l - worl d op e n - sour ce so f tw a r e e n gi n ee r i n g . I t sp a ns n ea rly 1 m i ll i on pull r e qu e sts ac ross README.md 2025-11-05 2 / 6 116,000+ r e pos i tor ie s , a ut h or ed b y fi v e AI c o di n g age nts : O p e n AI C o de x , D e v i n , G i t H u b C op i lot , C ursor , a n d C l a u de C o de , a n d i nvolv i n g 72,000+ h um a n de v e lop e rs . Y ou ca n ea s i ly lo ad t he da t a s e t b y f our l i n e s o f c o de : import pandas as pd all_pr_df = pd.read_parquet("hf://datasets/haoli/AIDev/all_pull_request.parquet") all_repo_df = pd.read_parquet("hf://datasets/haoli/AIDev/all_repository.parquet") all_user_df = pd.read_parquet("hf://datasets/haoli/AIDev/all_user.parquet") I f you ' r e i nt e r e st ed i n t he r a w da t a o f AID e v - pop , you ca n fi n d t he m he r e : h ttps :// d r i v e . g oo g l e . c om / fi l e / d /1 l 0_ R j S 7 ZT 0 Y 27 V 3 mv 0 o JK - jfe R kh q 5 l 5/ v ie w ? usp = d r i v e _ l i n k I nt e n ded U s e s F i n e - tun i n g or post - tr ai n i n g : fi n e - tun i n g or post - tr ai n i n g your LLM s / age nts ba s ed on t he p a t che s E mp i r ica l SE r e s ea r ch : a n a lys e c oll ab or a t i on p a tt e rns , r e v ie w l a t e n c y , v e lo ci ty A ge nt e v a lu a t i on : m ea sur e b u g - fi x su cce ss , c o de qu a l i ty , PR acce pt a n ce r a t e H um a n – AI i nt e r ac t i on : stu d y c onv e rs a t i on a l r e v ie w d yn a m ic s a n d s e nt i m e nt Q u ick L oo k T he ov e rv ie w o f t he AID e v da t a s e t i s a s f ollows : # PR # D e v e lop e r # R e po OpenAI Codex 814,522 61,653 84,704 Devin 29,744 NA 4,747 GitHub Copilot 50,447 NA 14,492 Cursor 32,941 9,658 12,699 Claude Code 5,137 1,643 1,915 T ot a l 932,791 72,189 116,211 README.md 2025-11-05 3 / 6 D a t a s e t S tru c tur e README.md 2025-11-05 4 / 6 A de t ai l ed e xpl a n a t i on ab out t he fie l d s o f t he t ab l e s ca n be f oun d i n da t a _ t ab l e . m d . AID e v i s or ga n i z ed i nto norm a l i z ed t ab l e s ( a v ai l ab l e a s CS V s ) t ha t ca n be j o i n ed v ia c ons i st e nt ke ys . T he c or e c ompon e nts i n c lu de : all_pull_request : PR - l e v e l da t a ( ID , t i tl e , b o d y , age nt l abe l , us e r i n f o , st a t e , t i m e st a mps ) all_repository : M e t ada t a i n c lu di n g l ice ns e , l a n g u age , st a rs , f or k s , a n d pro jec t - l e v e l i n f o all_user : U s e r i n f orm a t i on su ch a s id , lo gi n , a n d c r ea t ed da t e ( p e rson a lly i n f orm a t i on ha s bee n r e mov ed to add r e ss pr i v ac y c on ce rns ) AID e v - pop : F i lt e r ed (>100 st a rs ) # PR # D e v e lop e r # R e po OpenAI Codex 21,799 1,284 1,248 Devin 4,827 NA 288 GitHub Copilot 4,970 NA 1,012 Cursor 1,541 363 327 README.md 2025-11-05 5 / 6 # PR # D e v e lop e r # R e po Claude Code 459 236 213 T ot a l 33,596 1,796 2,807 F or t he AID e v - pop su b s e t ( r e pos i tor ie s w i t h mor e t ha n 100 st a rs ) o f AID e v , w e prov ide e xtr a t ab l e s : pull_request : PR - l e v e l da t a ( ID , t i tl e , b o d y , age nt l abe l , us e r i n f o , st a t e , t i m e st a mps ) repository : M e t ada t a i n c lu di n g l ice ns e , l a n g u age , st a rs , f or k s , a n d pro jec t - l e v e l i n f o pr_timeline : C ompl e t e PR e v e nt hi story ( op e n / c los e / m e r ge , l abe l , a ss ig n , e t c .) pr_comments & pr_reviews & pr_review_comments_v2 : R e v ie w di s c uss i ons , a pprov a ls , t i m e st a mps , ac tors , pr_review_comments_v2 c ont ai ns i nl i n e r e v ie w c omm e nts pr_commits & pr_commit_details : C omm i t m e t ada t a , diff s , fi l e - l e v e l cha n ge s , p a t ch . N ot e t ha t t he patch da t a d o e s not i n c lu de l a r ge p a t che s s i n ce t he G i t H u b API d o e s not prov ide t he m . I f you w a nt t he l a r ge p a t che s , you n eed to d ownlo ad t he m yours e l f . pr_task_type : A uto - c l a ss ifica t i on o f PR purpos e us i n g C onv e nt i on a l C omm i t ca t eg or ie s v ia LLM s issue & related_issue : L i n ked G i t H u b i ssu e s a n d t hei r m a pp i n g to PR s user : U s e r i n f orm a t i on su ch a s id , lo gi n , a n d c r ea t ed da t e ( p e rson a lly i n f orm a t i on ha s bee n r e mov ed to add r e ss pr i v ac y c on ce rns ) H um a n - PR H um a n - PR s w e r e s a mpl ed f rom t he s a m e r e pos i tor ie s a s A ge nt ic - PR s , b ut only f rom t h os e t ha t ha v e mor e t ha n 500 st a rs : human_pull_request : PR - l e v e l da t a ( ID , t i tl e , b o d y , age nt l abe l , us e r i n f o , st a t e , t i m e st a mps ) human_pr_task_type : A uto - c l a ss ifica t i on o f PR purpos e us i n g C onv e nt i on a l C omm i t ca t eg or ie s v ia LLM s L ice ns e T hi s da t a s e t agg r ega t e s c ont e nt f rom G i t H u b r e pos i tor ie s . E ach sour ce r e pos i tory r e t ai ns i ts or igi n a l c opyr igh t a n d l ice ns e ( e . g ., MI T , A p ache -2.0, GPL fa m i ly , C r ea t i v e C ommons v a r ia nts , e t c .). F i l e s , p a t che s / diff s , a n d a ny ot he r a rt ifac ts or igi n a t i n g f rom t h os e r e pos i tor ie s r e m ai n g ov e rn ed b y t hei r or igi n a l l ice ns e s . U s e rs must v e r if y a n d c omply w i t h t he sp ecific l ice ns e o f a ny sour ce r e pos i tory or fi l e t he y e xtr ac t or us e f rom t hi s c oll ec t i on . D o not a ssum e a un i v e rs a l r e - l ice ns e . I f you be l ie v e c ont e nt a pp ea rs he r e i n a w a y t ha t c on f l ic ts w i t h i ts l ice ns e , pl ea s e c ont ac t t he m ai nt ai n e rs , a n d i t w i ll be r e mov ed . I mport a nt : R e pos i tory c ont e nts m ai nt ai n t hei r or igi n a l l ice ns e s . P l ea s e r e sp ec t i n di v id u a l pro jec t l ice ns e s w he n us i n g t hi s da t a . C i t a t i on I f you us e AID e v i n your wor k , pl ea s e ci t e : README.md 2025-11-05 6 / 6 @misc{li2025aiteammates_se3, title={The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering}, author={Hao Li and Haoxiang Zhang and Ahmed E. Hassan}, year={2025}, eprint={2507.15003}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2507.15003} }