Blog
tech design

Introducing the Coding LLM Leaderboard

Meng Zhang
November 23, 2023
•
2
-minute read

In our previous post on Cracking the Coding Evaluation, we shed light on the limitations of relying on HumanEval pass@1 as a code completion benchmark. In response, we've launched the Coding LLMs Leaderboard, embracing Next Line Accuracy as a metric inspired by academic works such as RepoCoder, RepoBench, and CCEval.

Leaderboard

But what exactly is line accuracy? In code completion, model predicts a block of code spanning multiple lines. A naive approach would involve comparing the predicted block with the actual code being committed directly. While this approach might seem ideal, it is often considered "too sparse" as a revealing metric. On the other hand, next-line accuracy serves as a dependable proxy for overall block match accuracy.

Next Line Accuracy

Only content inside red box are used to compared with ground truth to compute accuracy metric

CCEval utilizes the next statement, but based on our observations, it strongly correlates with next line exact match. Therefore, we've opted for next line accuracy due to its ease of implementation across languages, eliminating the need for language-specific Tree Sitter parsers.

For data preparation, our initial release exclusively leverages the dataset from CCEval. This dataset provides well-structured left context, right context, cross-file context with BM25, and oracle information.

At present, evaluation is limited to prefix text + cross-file context. Our future plans involve more in-depth analyses:

  1. Comparing accuracy in completing a function's argument list.
  2. Computing accuracy in completing a function's docstring.

We genuinely believe that this leaderboard can assist Tabby's users in navigating the tradeoff between service cost, quality, and other factors. We are committed to enhancing and refining this leaderboard in the future.

Share this post
tech design

Stay Updated with Tabby News

Subscribe to our newsletter for the latest updates and news about Tabby.

By joining, you agree to our Terms and Conditions.
Thank you! We've received your submission.
Oops! Something went wrong. Please try again.
Stream laziness in Tabby
September 30, 2023
  ▪︎  
4
min read
tech design
Repository context for LLM assisted code completion
October 16, 2023
  ▪︎  
5
min read
tech design
repository context
Decode the Decoding in Tabby
October 21, 2023
  ▪︎  
4
min read
tech design
[ View our Full blog ]
Discover Tabby Unlock Your Coding Potential
Explore the Power of Tabby, the Self-Hosted AI Coding Assistant

  
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
333                                                                            333333                        
444   7                                                                       66466                          
00   313333                                                                 0000                             
   55555                                                                                                  331
  666                                                                                                    444 
888       777777                                                                                        888  
0       3311                                                                                            0    
    222222                                                                                                   
  455555         77777777                                                                                    
 666664       1111117                                                                                        
999999     3333333                                                                                    7      
8888     2222222   77777777                                                                    777           
000    5555555   1111111                                                                     33333           
0   4444444    1111113                                                                     55555             
  6666666    2333333                                                                      66664              
 999999    2222222                           7                                           8888                
888888    555555      77777777     77777777                                              00                  
0000   44444444     77777777    177777777                                                                    
00  4444444446   111111111    11111111                                                                       
0  666666666   13313131    3331313        777777                                                             
 999999999    333333     3333333        777777                                                               
8888888      222223    2222222        1111117                                                                
0000       222222     2222225      11111111                                                                  
000     55555555    5555555     333333333    7                                                               
0      5444444   544444445    3333322                                                                        
      444444   4444444444  2222222                                                             7             
      6666  66666666666 5552522   7 777   777 77    7 7   7                                  7               
     6666 66666666666  55555  7777777777  7777     7777   7777       7 7777       7 77 77 7      7 7   777   
   999999999999999    4444  7777777777 777777    177777  777777     777777  7777777777        7777 7777777   
 88898889888898     4444 111111111   111111     1111111  77777    777777 1777777777         717777777777     
88888888888        666 11111111    111111     11111111  11111    711111111111111          1111111111111      
0000000          666 3131313     3131313     11313133  11113    111111111111           111111111111111       
00000         9999  333333      3333333    33333333   13333   3333333333         333333333333333311          
000        999999  33333      3333333     333333     3333    3333333          33333333333333                 
0       8888888  222222    222222222    222222     3232    32332            3323232323223                    
    88888880    22222    222222222   2222222     2222    2222             2222222222222                      
 00000000     555555    55555555  255555552    2222    22222            222222222222                         
000000       55555    55555555 5555555555    5555   255555            5555555555552                          
0000       555555    555555555555555554    5555   55555555          555555555555                             
00      4444444     44444444444444445    44444 4444444445         44444444445                                
      44444444     444444444444444      444444444444 4444       4444444444                                   
    46666664      66666666666664       44444444644   444      46444444                                       
 66666666        6666666666666        666666666    6666      666666                                          
6969666        66969696969696       96666666     66666      66666     777777777    777777                    
99999        99999999999999       9999999      999999      99999     111111113    11111                      
99         999999999999        9999999        999999      99999    333333333     33333                       
         888888888      9988888888898       8888899     888889    22222222      22222   77777777             
      888888888      88888888888888      8888888     88888888   55555555       5555    111111                
    088888888      0888888888888       088888      88888888   444444444      44444    22222                  
  000000000      00000000000        000000      0000000000  666666666      66666    55555      1111111       
0000000000     000000000     00000000000    000000000000   99999999      999999    66666     2222222         
0 000000    000000000   0000000000000    0000000000000    8888888      888888   9999999    4444444           
000000    00000000    0000000000000   00000000000000     00000      0000000   0000000    89999998         7  
000    00000000     0000000000000   000000000000000    00000       000000   00000000    000000   9999999     
  


Get Started with our Community Plan Today
Get Started
Simple self-onboarding
Free community plan
Local-first deployment




  
333                                                                            333333                        
444   7                                                                       66466                          
00   313333                                                                 0000                             
   55555                                                                                                  331
  666                                                                                                    444 
888       777777                                                                                        888  
0       3311                                                                                            0    
    222222                                                                                                   
  455555         77777777                                                                                    
 666664       1111117                                                                                        
999999     3333333                                                                                    7      
8888     2222222   77777777                                                                    777           
000    5555555   1111111                                                                     33333           
0   4444444    1111113                                                                     55555             
  6666666    2333333                                                                      66664              
 999999    2222222                           7                                           8888                
888888    555555      77777777     77777777                                              00                  
0000   44444444     77777777    177777777                                                                    
00  4444444446   111111111    11111111                                                                       
0  666666666   13313131    3331313        777777                                                             
 999999999    333333     3333333        777777                                                               
8888888      222223    2222222        1111117                                                                
0000       222222     2222225      11111111                                                                  
000     55555555    5555555     333333333    7                                                               
0      5444444   544444445    3333322                                                                        
      444444   4444444444  2222222                                                             7             
      6666  66666666666 5552522   7 777   777 77    7 7   7                                  7               
     6666 66666666666  55555  7777777777  7777     7777   7777       7 7777       7 77 77 7      7 7   777   
   999999999999999    4444  7777777777 777777    177777  777777     777777  7777777777        7777 7777777   
 88898889888898     4444 111111111   111111     1111111  77777    777777 1777777777         717777777777     
88888888888        666 11111111    111111     11111111  11111    711111111111111          1111111111111      
0000000          666 3131313     3131313     11313133  11113    111111111111           111111111111111       
00000         9999  333333      3333333    33333333   13333   3333333333         333333333333333311          
000        999999  33333      3333333     333333     3333    3333333          33333333333333                 
0       8888888  222222    222222222    222222     3232    32332            3323232323223                    
    88888880    22222    222222222   2222222     2222    2222             2222222222222                      
 00000000     555555    55555555  255555552    2222    22222            222222222222                         
000000       55555    55555555 5555555555    5555   255555            5555555555552                          
0000       555555    555555555555555554    5555   55555555          555555555555                             
00      4444444     44444444444444445    44444 4444444445         44444444445                                
      44444444     444444444444444      444444444444 4444       4444444444                                   
    46666664      66666666666664       44444444644   444      46444444                                       
 66666666        6666666666666        666666666    6666      666666                                          
6969666        66969696969696       96666666     66666      66666     777777777    777777                    
99999        99999999999999       9999999      999999      99999     111111113    11111                      
99         999999999999        9999999        999999      99999    333333333     33333                       
         888888888      9988888888898       8888899     888889    22222222      22222   77777777             
      888888888      88888888888888      8888888     88888888   55555555       5555    111111                
    088888888      0888888888888       088888      88888888   444444444      44444    22222                  
  000000000      00000000000        000000      0000000000  666666666      66666    55555      1111111       
0000000000     000000000     00000000000    000000000000   99999999      999999    66666     2222222         
0 000000    000000000   0000000000000    0000000000000    8888888      888888   9999999    4444444           
000000    00000000    0000000000000   00000000000000     00000      0000000   0000000    89999998         7  
000    00000000     0000000000000   000000000000000    00000       000000   00000000    000000   9999999     




Explore Full Features with Team or Enterprise Plans 
BOOK A DEMO 🚀
Enterprise-first experience
Flexible deployment options
Enhanced security support