Benchmarking Web API Integration Code Generation

Appendix to Benchmarking Web API Integration Code Generation Daniel Maninger∗†, Leon Chemnitz§∗, Amir Molzam Sharifloo∗, Jannis Brugger∗†, Mira Mezini∗†‡ ∗Technische Universit¨ at Darmstadt, Germany †Hessian Center for Artificial Intelligence (hessian.AI), Germany ‡National Research Center for Applied Cybersecurity ATHENE, Germany §Pariton AI, Germany [email protected], [email protected], amir[email protected], [email protected], [email protected] APPENDIX A. Data Availability WAPIIBench is available on GitHub 1 . In addition, we provide all model-generated codes in our artifact2. B. Background OpenAPI provides a structured, humanand machinereadable way to document an API’s endpoints, requests, response formats, authentication methods, and other details. For this paper, we define the term endpoint as a unique combination of path and HTTP method. Specification files are written in JSON or YAML. Listing 1a shows an excerpt of the OpenAPI specification of the Google Calendar API 3 . Besides some meta information, it contains a list of paths (e.g., /calendars ) and, for each path, a list of supported HTTP methods (e.g., post ). Each endpoint has properties for documentation purposes and a list of named parameters ( prettyPrint ). Parameters can be passed in different locations, indicated by the property in : path parameters are inserted directly into of the URL, query parameters are appended to the URL as key–value pairs, and header parameters become part of the HTTP request header. Additionally, methods like POST use the requestBody to transfer data to the server (e.g., summary , timeZone ). We consider this data as another kind of parameter. Parameters have a schema that specifies their data type and constraints on permissible values. The security property determines available authentication schemes, such as OAuth 2.0. Listing 1b shows how a request to the endpoint from the specification in Listing 1a could be implemented in JavaScript using the Axios library. The method called on the axios object determines the HTTP method, and the first argument determines the server URL and path. This is followed by one object containing the request body and another object containing header and query parameters ( headers , params ). Query parameters are automatically serialized and appended to the URL by Axios. If we execute the code in Listing 1b, 1https://github.com/stg-tud/WAPIIBench 2https://doi.org/10.5281/zenodo.13758414 3 Adapted from https://api . apis . guru/v2/specs/googleapis . com/calendar/v3/ openapi.yaml; the original comprises 3190 lines. a configuration object as shown in Listing 1c is created and a request, configured accordingly, is sent to the given URL. The configuration contains all arguments explicitly given in the code, as well as some implicit parameters, such as Accept (for the expected response media type) and Content-Type (for the request body media type). The representation of requests as configuration objects is key to our evaluation method described in the paper. C. Technologies We implemented WAPIIBench using the following technologies: •OpenAPI •OpenAPI 3 parser •Axios •axios-mock-adapter •Hugging Face Transformers D. Models These are the exact names of the models we evaluated: •bigcode/starcoderbase •bigcode/starcoder2-3b •bigcode/starcoder2-7b •bigcode/starcoder2-15b •deepseek-ai/deepseek-coder-1.3b-base •deepseek-ai/deepseek-coder-6.7b-base •deepseek-ai/deepseek-coder-7b-base-v1.5 •deepseek-ai/deepseek-coder-33b-base •deepseek-ai/DeepSeek-Coder-V2-Lite-Base •google/gemini-pro-1.5 •meta-llama/CodeLlama-7b-hf •meta-llama/CodeLlama-13b-hf •meta-llama/CodeLlama-70b-hf •meta-llama/Llama-3.1-8B •meta-llama/Llama-3.1-70B •openai/gpt-4o •openai/gpt-4o-mini •Qwen/Qwen2.5-Coder-0.5B •Qwen/Qwen2.5-Coder-1.5B •Qwen/Qwen2.5-Coder-3B •Qwen/Qwen2.5-Coder-7B openapi: 3.0.0 servers: -url: https://www.googleapis.com/calendar/v3 info: title: Calendar API description: Manipulates events and other calendar data. paths: /calendars: post: description: Creates a secondary calendar. parameters: -name: prettyPrint description: Returns response with indentations and line breaks. in: query required: false schema: type: boolean requestBody: content: application/json: schema: properties: summary: description: Title of the calendar. type: string timeZone: description: The time zone of the calendar. (Formatted as an IANA Time Zone Database name, e.g. "Europe/ Zurich".) Optional. type: string type: object security: - Oauth2: - https://www.googleapis.com/auth/calendar (a) OpenAPI specification (YAML) // Create a secondary calendar with summary "Example Calendar" and time zone "America/Los_Angeles". Pretty print the response. const axios = require(’axios’); axios.post(’https://www.googleapis.com/calendar/v3/ calendars’, { summary: ’Example Calendar’, timeZone: ’America/Los_Angeles’, }, { headers: { Authorization: ’Bearer <access_token>’ }, params: { prettyPrint: true, } }).then(response => { console.log(’Calendar created’, response.data); }); (b) API invocation (JavaScript) { "headers": { "Accept":"application/json, text/plain, */*", "Content-Type":"application/json", "Authorization":"Bearer <access_token>" }, "params": { "prettyPrint":true }, "method":"post", "url":"https://www.googleapis.com/calendar/v3/calendars ", "data": { "summary":"Example Calendar", "timeZone":"America/Los_Angeles" } } (c) Configuration object (JSON) Fig. 1: Example of a web API and its usage. Top: Excerpt from the Google Calendar OpenAPI specification. Left: JavaScript code to send a request to this API using the Axios library. Right: Configuration object that describes the request sent. For our evaluation, we pair the task description (comment in the JavaScript code) with the configuration to create an input–output sample. •Qwen/Qwen2.5-Coder-14B •Qwen/Qwen2.5-Coder-32B •Salesforce/codet5p-16b As can be seen, we use only base (i.e., non-instruction-tuned) models. While instruction-tuned models tend to perform better on coding tasks 4 , we considered them to be inappropriate for our setting, which is based on code completion. Performing 4 Cf., e.g., the EvalPlus leaderboard (https://evalplus . github . io/ leaderboard.html) code completion on instruction-tuned models leads to rather unnatural results, as the models often do not directly return the completion and instead start with some response (“Here is your completed code ...”) followed by a code snipped that may contain 1) only the completion, 2) the starter code followed by the completion, or 3) an arbitrarily modified version of the starter code and corresponding completion. These factors make the model output very hard to parse reliably. Therefore, we decided to focus our evaluation on base models. The following hyperparameters were used: •floating point precision = 16 bit •# beams = 1 •temperature = 0.0 E. Prompts Listing 1 shows the prompt used for creating the dataset with Gemini 1.5 pro. To avoid incomplete responses when generating the dataset based on OpenAPI specifications with more than 100 endpoints (Asana and Slack), we prompted it multiple times, asking for different subsets of endpoints. The Slack API required additional instructions to account for changes in the API that are not reflected in the OpenAPI specification 5 . Listing 2 shows the prompt used in our evaluation pipeline when generating API invocation code. In both cases, we decided against few-shot prompting to increase generalizability and to avoid associated pitfalls. Fewshot prompting techniques mainly help the model to understand the task it is supposed to solve and the desired response format. This is not the issue in the type of task we investigate. Rather, the main problem is that models select the wrong API endpoints and/or pass the wrong arguments to the endpoint. On the other hand, using few-shot prompts introduces new challenges, such as a high sensitivity to the examples provided while reducing the generalizability of the prompt and not guaranteeing inherently superior performance to that of zero-shot prompting 6 . Further optimization of model performance via prompt engineering would require either revealing information about the API’s endpoints and parameters—undermining the purpose of our evaluation of the models’ memorized knowledge—or tailoring prompts to each individual model (e.g., adjusting wording, formatting, etc.). However, this approach relies on trial-and-error and introduces high uncertainty and variance. F. Extended Results The full set of metrics calculated by our pipeline is explained in Table I. Tables II and III provide a comprehensive summary of these metrics for all evaluated models 7 (cf. Appendix D). Additionally, the results are broken down by API exemplarily for StarCoder2 and GPT-4o in Tables IV to VII. The best values are in bold. Note that some metrics can only be calculated either for full completion or for argument completion and are therefore not shown in all tables. Values marked with (t) are ratios relative to the 395 total samples, while values marked with (e) are relative to the subset of executable codes, which varies from experiment to experiment. The complete raw data these results are based on can be found in our artifact (cf. Appendix A). 5https://api.slack.com/changelog/2017-10-keeping-up-with-the-jsons 6 Cf. Reynolds & McDonell, 2021 (https://doi . org/10 . 1145/ 3411763.3451760) 7 While we also evaluated Gemini 1.5 Pro on WAPIIBench, we excluded it from all discussion—since it was the model that generated the dataset, its evaluation results might be skewed. Listing 1: Prompt for generating the test data for a given API. Words in curly braces are placeholders. Consider this OpenAPI specification: ‘‘‘yaml {spec} ‘‘‘ I want you to generate test data for this API. The data should be in JSON format and look like this: ‘‘‘json { "samples": [ { "task": "...", "config": { "url": "...", "method": "...", "headers": { ... }, "params": { ... }, "data": { ... } } }, ... ] } ‘‘‘ *‘samples‘ is an array containing all the test cases. *‘task‘ is a natural language description of a specific task that can be solved by sending a request to the given API. All information required to unambiguously identify and implement the corresponding API request must be contained in the task description. Therefore, state all expected argument values explicitly. The only exception is authentication keys and tokens, which must not be specified here (but you may specify the authentication method to be used if multiple ones are available). *‘config‘ is an object containing the expected configuration of the described request. If a property of ‘config‘ is empty, it can be omitted. *‘url‘ is the full URL of the endpoint (server URL + path) and may include path parameters but no query parameters. *‘method‘ is the HTTP method used. It can be one of ‘get‘, ‘put‘, ‘post‘, ‘delete‘, and ‘patch‘. *‘headers‘ is an object containing all expected header arguments. Note that the property ‘"Accept": " application/json, text/plain, */*"‘ is always present and if ‘data‘ is sent in the request body, ‘"ContentType": "mime/type"‘ is present as well (substitute " mime/type" for the respective media type). *‘params‘ is an object containing all expected query arguments. *‘data‘ is an object containing all data from the request body. It is only used for the methods put, post, delete , and patch. Remember that for authentication, an additional argument might be required. Take a look at an endpoint’s ‘ security‘ property and the ‘securitySchemes‘ section in the specification, to find out if and how to authenticate. I assume you know how the usual authentication schemes ‘apiKey‘, ‘http‘, and ‘oauth2‘ work. Use ‘<key>‘ as a placeholder for API keys (e.g., ‘"name": "<key>"‘) and ‘<token>‘ as a placeholder for authorization tokens (e.g., ‘"Authorization": "Bearer < token>"‘). {api_specific_instructions} Now, please give me a JSON object that matches my description and that contains diverse examples of requests to this API. {path_selection} TABLE I: Complete Evaluation Metrics for Correctness and Specification-Compliance of API Invocations Metric Description Executable codes Generated code is complete and contains no syntax or runtime error Correct codes Generated executable code matches the ground-truth configuration Illegal codes Generated code contains at least one violation of the API specification Correct URLs Generated URL matches the ground-truth URL Illegal URLs Generated URL is not defined in the API specification Correct methods Generated HTTP method matches the ground-truth HTTP method Illegal methods Generated HTTP method is not defined for the generated URL in the API specification Correct argument names Generated arguments are correct Correct argument values Generated argument values are correct Missing arguments Expected arguments are not generated Unexpected arguments Generated arguments are not expected Unnecessary arguments Redundant arguments are generated for an API endpoint Illegal arguments Generated arguments are not permitted for the generated API endpoint Mean argument precision Probability that the generated arguments are correct Mean argument recall Probability that the correct arguments are generated Mean argument Jaccard index Overlap between generated and correct arguments Mean argument value conditional accuracy Probability that an argument value is correct if the argument name is correct Total errors Any type of error prevented execution Incomplete codes Generated code did not contain a complete API invocation Runtime errors Generated code produced an error when trying to execute it Listing 2: Prompt for generating API invocations given a task description. Words in curly braces are placeholders. You are an AI programming assistant that helps users write API requests. You are given a comment that describes what the user wants to achieve and are supposed to implement it using the Axios library in JavaScript. For this, write a single call to Axios (with syntax ‘axios .method(url[, config])‘) that does exactly what was described in the comment. *Make sure to include all parameters in ‘config‘ that are required to solve the given task but don’t include any unnecessary parameters. *Insert all values directly into the place where they belong, rather than using intermediate variables. *If the API requires some form of authentication, use ‘< key>‘ as a placeholder for API keys or ‘<token>‘ as a placeholder for authorization tokens, respectively. *If a request body requires a media type other than ‘text/ json‘, explicitly set the ‘Content-Type‘ header to the respective type, and Axios will automatically serialize the request body accordingly. Your next task is about the {api} API. Complete the following code snippet{extra_instructions}: ‘‘‘javascript // {task} const axios = require(’axios’); axios.{method}(’{url}’, TABLE II: Complete Evaluation Results for Full Completion CodeT5+ (16B) StarCoder (15.5B) StarCoder2 (3B) StarCoder2 (7B) StarCoder2 (15B) DeepSeek-Coder (1.3B) DeepSeek-Coder (6.7B) DeepSeek-Coder (7B) DeepSeek-Coder (33B) DeepSeek-Coder-V2 (16B) Qwen2.5-Coder (0.5B) Qwen2.5-Coder (1.5B) Qwen2.5-Coder (3B) Qwen2.5-Coder (7B) Qwen2.5-Coder (14B) Qwen2.5-Coder (32B) Code Llama (7B) Code Llama (13B) Code Llama (70B) Llama 3.1 (8B) Llama 3.1 (70B) Gemini 1.5 Pro GPT-4o mini GPT-4o Executable implementations (t) 0.95 0.94 0.97 0.95 0.99 0.96 0.96 0.57 0.98 0.95 0.01 0.00 0.00 0.00 0.00 0.00 0.99 0.89 0.99 0.00 0.00 1.00 1.00 1.00 Correct implementations (t) 0.06 0.12 0.11 0.10 0.26 0.08 0.18 0.04 0.17 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.08 0.30 0.00 0.00 0.45 0.39 0.60 Correct implementations (e) 0.06 0.13 0.12 0.11 0.27 0.08 0.18 0.06 0.17 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.09 0.31 0.00 0.00 0.45 0.39 0.60 Correct URLs (t) 0.31 0.51 0.40 0.45 0.60 0.33 0.57 0.35 0.67 0.51 0.01 0.00 0.00 0.00 0.00 0.00 0.48 0.51 0.72 0.00 0.00 0.72 0.57 0.78 Correct URLs (e) 0.32 0.54 0.41 0.48 0.60 0.35 0.60 0.62 0.68 0.53 0.67 0.00 0.00 0.00 0.00 0.00 0.48 0.58 0.72 0.00 0.00 0.72 0.57 0.78 Illegal URLs (t) 0.37 0.17 0.22 0.27 0.22 0.32 0.19 0.10 0.13 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.21 0.15 0.00 0.00 0.19 0.23 0.09 Illegal URLs (e) 0.39 0.18 0.23 0.29 0.22 0.34 0.20 0.17 0.14 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.24 0.15 0.00 0.00 0.20 0.23 0.09 Correct methods (t) 0.81 0.83 0.78 0.88 0.86 0.79 0.84 0.53 0.86 0.81 0.01 0.00 0.00 0.00 0.00 0.00 0.73 0.75 0.88 0.00 0.00 0.91 0.83 0.91 Correct methods (e) 0.85 0.88 0.81 0.93 0.87 0.83 0.88 0.93 0.87 0.84 1.00 0.00 0.00 0.00 0.00 0.00 0.74 0.84 0.89 0.00 0.00 0.92 0.83 0.91 Illegal methods (t) 0.12 0.09 0.12 0.05 0.11 0.12 0.10 0.03 0.10 0.13 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.13 0.10 0.00 0.00 0.07 0.14 0.07 Illegal methods (e) 0.13 0.09 0.12 0.05 0.11 0.12 0.11 0.05 0.10 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.23 0.14 0.10 0.00 0.00 0.07 0.14 0.07 Correct argument names (t) 0.39 0.50 0.49 0.41 0.63 0.45 0.53 0.25 0.58 0.52 0.00 0.00 0.00 0.00 0.00 0.00 0.37 0.56 0.68 0.00 0.00 0.79 0.76 0.86 Correct argument names (e) 0.41 0.54 0.51 0.43 0.63 0.47 0.56 0.48 0.59 0.55 0.29 0.00 0.00 0.00 0.00 0.00 0.38 0.63 0.68 0.00 0.00 0.80 0.76 0.87 Correct argument values (t) 0.29 0.43 0.40 0.33 0.58 0.39 0.47 0.18 0.53 0.48 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.39 0.63 0.00 0.00 0.76 0.72 0.83 Correct argument values (e) 0.30 0.46 0.42 0.35 0.58 0.41 0.50 0.34 0.54 0.50 0.29 0.00 0.00 0.00 0.00 0.00 0.33 0.43 0.63 0.00 0.00 0.77 0.72 0.83 Missing arguments (t) 0.61 0.50 0.51 0.59 0.37 0.55 0.47 0.75 0.42 0.48 1.00 1.00 1.00 1.00 1.00 1.00 0.63 0.44 0.32 1.00 1.00 0.21 0.24 0.14 Missing arguments (e) 0.59 0.46 0.49 0.57 0.37 0.53 0.44 0.52 0.41 0.45 0.71 0.00 0.00 0.00 0.00 0.00 0.62 0.37 0.32 0.00 0.00 0.20 0.24 0.13 Unexpected arguments (t) 0.30 0.25 0.34 0.31 0.25 0.32 0.25 0.11 0.24 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.22 0.20 0.00 0.00 0.18 0.17 0.11 Unexpected arguments (e) 0.31 0.26 0.35 0.32 0.25 0.33 0.26 0.20 0.25 0.23 0.12 0.00 0.00 0.00 0.00 0.00 0.17 0.24 0.20 0.00 0.00 0.18 0.17 0.11 Mean argument precision (t) 0.50 0.61 0.53 0.49 0.67 0.53 0.62 0.36 0.59 0.63 0.01 0.00 0.00 0.00 0.00 0.00 0.56 0.65 0.75 0.00 0.00 0.82 0.82 0.89 Mean argument precision (e) 0.52 0.65 0.54 0.51 0.68 0.55 0.65 0.63 0.60 0.67 0.67 0.00 0.00 0.00 0.00 0.00 0.57 0.72 0.75 0.00 0.00 0.82 0.82 0.89 Mean argument recall (t) 0.41 0.51 0.48 0.42 0.63 0.47 0.54 0.28 0.56 0.55 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.58 0.69 0.00 0.00 0.81 0.78 0.87 Mean argument recall (e) 0.43 0.54 0.50 0.44 0.63 0.49 0.56 0.49 0.56 0.57 0.33 0.00 0.00 0.00 0.00 0.00 0.41 0.65 0.69 0.00 0.00 0.82 0.78 0.87 Mean arg. Jaccard index (t) 0.34 0.46 0.40 0.35 0.56 0.38 0.48 0.25 0.47 0.49 0.00 0.00 0.00 0.00 0.00 0.00 0.37 0.52 0.62 0.00 0.00 0.75 0.72 0.83 Mean arg. Jaccard index (e) 0.36 0.48 0.41 0.37 0.56 0.40 0.50 0.44 0.48 0.52 0.33 0.00 0.00 0.00 0.00 0.00 0.37 0.58 0.63 0.00 0.00 0.75 0.72 0.83 Mean arg. val. cond. acc. (t) 0.51 0.64 0.60 0.57 0.78 0.66 0.69 0.31 0.74 0.72 0.01 0.00 0.00 0.00 0.00 0.00 0.58 0.51 0.84 0.00 0.00 0.95 0.95 0.95 Mean arg. val. cond. acc. (e) 0.53 0.68 0.62 0.60 0.79 0.69 0.72 0.54 0.75 0.76 0.67 0.00 0.00 0.00 0.00 0.00 0.59 0.57 0.84 0.00 0.00 0.96 0.96 0.95 Total errors 18 22 13 21 3 16 16 169 6 18 392 395 393 395 395 390 5 42 2 395 395 111 Incomplete implementations 14 5 11 5 2 12 3 2 6 9 29 35 12 2 160 04514010 0 Runtime errors 4 17 2 16 1 4 13 167 09 363 360 381 393 235 390 1 37 1 391 395 01 1 TABLE III: Complete Evaluation Results for Argument Completion CodeT5+ (16B) StarCoder (15.5B) StarCoder2 (3B) StarCoder2 (7B) StarCoder2 (15B) DeepSeek-Coder (1.3B) DeepSeek-Coder (6.7B) DeepSeek-Coder (7B) DeepSeek-Coder (33B) DeepSeek-Coder-V2 (16B) Qwen2.5-Coder (0.5B) Qwen2.5-Coder (1.5B) Qwen2.5-Coder (3B) Qwen2.5-Coder (7B) Qwen2.5-Coder (14B) Qwen2.5-Coder (32B) Code Llama (7B) Code Llama (13B) Code Llama (70B) Llama 3.1 (8B) Llama 3.1 (70B) Gemini 1.5 Pro GPT-4o mini GPT-4o Executable implementations (t) 0.94 0.97 0.98 0.99 0.99 0.97 0.99 0.99 0.99 0.98 0.89 0.95 0.92 0.99 0.93 0.99 0.99 0.95 1.00 0.98 0.99 1.00 0.99 1.00 Correct implementations (t) 0.12 0.27 0.21 0.19 0.25 0.21 0.26 0.08 0.25 0.35 0.08 0.03 0.16 0.28 0.37 0.38 0.23 0.12 0.40 0.05 0.29 0.61 0.63 0.77 Correct implementations (e) 0.13 0.27 0.22 0.19 0.25 0.22 0.27 0.08 0.25 0.36 0.09 0.03 0.18 0.28 0.40 0.38 0.23 0.13 0.40 0.05 0.29 0.61 0.64 0.77 Illegal implementations (t) 0.59 0.27 0.48 0.47 0.35 0.51 0.33 0.21 0.38 0.31 0.58 0.50 0.44 0.41 0.32 0.32 0.11 0.28 0.23 0.30 0.26 0.17 0.20 0.09 Illegal implementations (e) 0.63 0.27 0.49 0.47 0.36 0.52 0.33 0.22 0.39 0.32 0.65 0.53 0.48 0.41 0.35 0.32 0.11 0.30 0.23 0.31 0.26 0.17 0.20 0.09 Correct argument names (t) 0.49 0.71 0.62 0.53 0.64 0.60 0.66 0.52 0.69 0.67 0.43 0.47 0.44 0.67 0.65 0.80 0.48 0.70 0.78 0.43 0.63 0.90 0.88 0.93 Correct argument names (e) 0.52 0.73 0.63 0.54 0.65 0.62 0.67 0.53 0.69 0.69 0.49 0.50 0.48 0.67 0.70 0.81 0.48 0.73 0.79 0.44 0.64 0.90 0.89 0.93 Correct argument values (t) 0.42 0.61 0.52 0.44 0.57 0.52 0.60 0.45 0.64 0.62 0.34 0.41 0.40 0.63 0.58 0.73 0.42 0.51 0.72 0.38 0.58 0.85 0.84 0.90 Correct argument values (e) 0.45 0.62 0.53 0.44 0.58 0.54 0.61 0.46 0.65 0.63 0.40 0.43 0.43 0.63 0.63 0.74 0.42 0.53 0.72 0.39 0.58 0.85 0.84 0.90 Missing arguments (t) 0.51 0.29 0.38 0.47 0.36 0.40 0.34 0.48 0.31 0.33 0.57 0.53 0.56 0.33 0.35 0.20 0.52 0.30 0.22 0.57 0.37 0.10 0.12 0.07 Missing arguments (e) 0.48 0.27 0.37 0.46 0.35 0.38 0.33 0.47 0.31 0.31 0.51 0.50 0.52 0.33 0.30 0.19 0.52 0.27 0.21 0.56 0.36 0.10 0.11 0.07 Unnecessary arguments (t) 0.03 0.05 0.08 0.07 0.07 0.05 0.07 0.06 0.07 0.03 0.03 0.03 0.03 0.06 0.03 0.07 0.05 0.05 0.05 0.03 0.04 0.04 0.02 0.01 Unnecessary arguments (e) 0.03 0.05 0.08 0.07 0.07 0.05 0.07 0.06 0.07 0.03 0.03 0.03 0.03 0.06 0.03 0.07 0.05 0.05 0.05 0.03 0.04 0.04 0.02 0.01 Illegal arguments (t) 0.24 0.13 0.21 0.20 0.14 0.23 0.15 0.08 0.13 0.13 0.29 0.24 0.23 0.17 0.18 0.11 0.06 0.13 0.09 0.16 0.12 0.09 0.09 0.05 Illegal arguments (e) 0.25 0.13 0.21 0.20 0.14 0.24 0.15 0.09 0.13 0.13 0.31 0.25 0.24 0.17 0.20 0.11 0.06 0.13 0.09 0.16 0.12 0.09 0.09 0.05 Mean argument precision (t) 0.55 0.79 0.60 0.61 0.71 0.64 0.73 0.73 0.69 0.76 0.48 0.59 0.58 0.71 0.69 0.78 0.66 0.75 0.83 0.63 0.78 0.88 0.89 0.93 Mean argument precision (e) 0.58 0.81 0.62 0.61 0.71 0.65 0.74 0.74 0.69 0.77 0.54 0.62 0.63 0.71 0.74 0.78 0.66 0.79 0.83 0.65 0.79 0.88 0.90 0.94 Mean argument recall (t) 0.50 0.72 0.61 0.53 0.64 0.63 0.66 0.52 0.66 0.70 0.46 0.46 0.46 0.68 0.67 0.78 0.53 0.70 0.78 0.42 0.66 0.90 0.90 0.93 Mean argument recall (e) 0.53 0.74 0.62 0.54 0.65 0.64 0.67 0.52 0.67 0.71 0.52 0.49 0.50 0.68 0.72 0.79 0.53 0.74 0.79 0.43 0.67 0.90 0.90 0.94 Mean arg. Jaccard index (t) 0.42 0.66 0.51 0.46 0.57 0.54 0.59 0.46 0.57 0.64 0.38 0.39 0.41 0.59 0.62 0.71 0.49 0.64 0.72 0.38 0.61 0.85 0.85 0.91 Mean arg. Jaccard index (e) 0.45 0.68 0.52 0.47 0.57 0.55 0.60 0.47 0.57 0.65 0.43 0.41 0.45 0.59 0.67 0.71 0.49 0.67 0.73 0.39 0.61 0.85 0.86 0.92 Mean arg. val. cond. acc. (t) 0.66 0.80 0.68 0.71 0.82 0.75 0.82 0.77 0.84 0.84 0.59 0.71 0.68 0.88 0.72 0.87 0.65 0.62 0.89 0.66 0.83 0.94 0.94 0.95 Mean arg. val. cond. acc. (e) 0.71 0.83 0.70 0.72 0.83 0.77 0.83 0.78 0.84 0.86 0.67 0.74 0.74 0.89 0.78 0.87 0.65 0.65 0.89 0.67 0.84 0.94 0.95 0.95 Total errors 23 11 9 4 3 11 4 5 3 6 45 19 32 2 28 2 3 19 195121 Incomplete implementations 22 5 7 4 1 11 4 3 3 6 44 17 31 2 27 2 3 5 1 6 1 1 0 0 Runtime errors 1 6 2 0200200121010 0 14 03402 1 TABLE IV: Evaluation Results of StarCoder2 by API for Full Completion Overall Asana Google Calendar Google Sheets Slack Executable implementations (t) 0.99 0.99 1.00 1.00 0.99 Correct implementations (t) 0.26 0.37 0.46 0.18 0.13 Correct implementations (e) 0.27 0.37 0.46 0.18 0.13 Correct URLs (t) 0.60 0.60 0.89 0.47 0.54 Correct URLs (e) 0.60 0.61 0.89 0.47 0.55 Illegal URLs (t) 0.22 0.25 0.03 0.18 0.22 Illegal URLs (e) 0.22 0.25 0.03 0.18 0.23 Correct methods (t) 0.86 0.90 0.89 0.47 0.86 Correct methods (e) 0.87 0.90 0.89 0.47 0.87 Illegal methods (t) 0.11 0.10 0.05 0.29 0.11 Illegal methods (e) 0.11 0.10 0.05 0.29 0.11 Correct argument names (t) 0.63 0.73 0.81 0.64 0.49 Correct argument names (e) 0.63 0.74 0.81 0.64 0.49 Correct argument values (t) 0.58 0.64 0.79 0.59 0.46 Correct argument values (e) 0.58 0.64 0.79 0.59 0.47 Missing arguments (t) 0.37 0.27 0.19 0.36 0.51 Missing arguments (e) 0.37 0.26 0.19 0.36 0.51 Unexpected arguments (t) 0.25 0.11 0.18 0.26 0.36 Unexpected arguments (e) 0.25 0.11 0.18 0.26 0.37 Mean argument precision (t) 0.67 0.88 0.76 0.60 0.46 Mean argument precision (e) 0.68 0.88 0.76 0.60 0.47 Mean argument recall (t) 0.63 0.76 0.75 0.66 0.47 Mean argument recall (e) 0.63 0.76 0.75 0.66 0.48 Mean argument Jaccard index (t) 0.56 0.71 0.68 0.52 0.38 Mean argument Jaccard index (e) 0.56 0.72 0.68 0.52 0.39 Mean argument value conditional accuracy (t) 0.78 0.85 0.87 0.84 0.70 Mean argument value conditional accuracy (e) 0.79 0.85 0.87 0.84 0.70 Total errors 3 1 002 Incomplete implementations 2 1 001 Runtime errors 1 0001 TABLE V: Evaluation Results of StarCoder2 by API for Argument Completion Overall Asana Google Calendar Google Sheets Slack Executable implementations (t) 0.99 0.99 1.00 1.00 0.99 Correct implementations (t) 0.25 0.37 0.46 0.24 0.10 Correct implementations (e) 0.25 0.37 0.46 0.24 0.10 Illegal implementations (t) 0.35 0.23 0.08 0.24 0.55 Illegal implementations (e) 0.36 0.23 0.08 0.24 0.55 Correct argument names (t) 0.64 0.69 0.86 0.67 0.55 Correct argument names (e) 0.65 0.69 0.86 0.67 0.55 Correct argument values (t) 0.57 0.59 0.81 0.61 0.49 Correct argument values (e) 0.58 0.59 0.81 0.61 0.50 Missing arguments (t) 0.36 0.31 0.14 0.33 0.45 Missing arguments (e) 0.35 0.31 0.14 0.33 0.45 Unnecessary arguments (t) 0.07 0.03 0.10 0.06 0.10 Unnecessary arguments (e) 0.07 0.03 0.10 0.06 0.10 Illegal arguments (t) 0.14 0.11 0.02 0.06 0.20 Illegal arguments (e) 0.14 0.11 0.02 0.06 0.20 Mean argument precision (t) 0.71 0.84 0.82 0.83 0.55 Mean argument precision (e) 0.71 0.85 0.82 0.83 0.55 Mean argument recall (t) 0.64 0.72 0.80 0.68 0.53 Mean argument recall (e) 0.65 0.72 0.80 0.68 0.54 Mean argument Jaccard index (t) 0.57 0.68 0.74 0.64 0.42 Mean argument Jaccard index (e) 0.57 0.69 0.74 0.64 0.43 Mean argument value conditional accuracy (t) 0.82 0.84 0.91 0.85 0.78 Mean argument value conditional accuracy (e) 0.83 0.84 0.91 0.85 0.79 Total errors 3 1 002 Incomplete implementations 1 0001 Runtime errors 2 1 001 TABLE VI: Evaluation Results of GPT-4o by API for Full Completion Overall Asana Google Calendar Google Sheets Slack Executable implementations (t) 1.00 0.99 1.00 1.00 1.00 Correct implementations (t) 0.60 0.56 0.89 0.59 0.57 Correct implementations (e) 0.60 0.57 0.89 0.59 0.57 Correct URLs (t) 0.78 0.75 1.00 0.76 0.75 Correct URLs (e) 0.78 0.76 1.00 0.76 0.75 Illegal URLs (t) 0.09 0.12 0.00 0.00 0.10 Illegal URLs (e) 0.09 0.12 0.00 0.00 0.10 Correct methods (t) 0.91 0.90 0.89 0.88 0.93 Correct methods (e) 0.91 0.90 0.89 0.88 0.93 Illegal methods (t) 0.07 0.10 0.00 0.00 0.07 Illegal methods (e) 0.07 0.10 0.00 0.00 0.07 Correct argument names (t) 0.86 0.83 1.00 0.88 0.86 Correct argument names (e) 0.87 0.84 1.00 0.88 0.86 Correct argument values (t) 0.83 0.79 1.00 0.81 0.83 Correct argument values (e) 0.83 0.80 1.00 0.81 0.83 Missing arguments (t) 0.14 0.17 0.00 0.12 0.14 Missing arguments (e) 0.13 0.16 0.00 0.12 0.14 Unexpected arguments (t) 0.11 0.11 0.00 0.06 0.13 Unexpected arguments (e) 0.11 0.11 0.00 0.06 0.13 Mean argument precision (t) 0.89 0.90 1.00 0.93 0.85 Mean argument precision (e) 0.89 0.90 1.00 0.93 0.85 Mean argument recall (t) 0.87 0.85 1.00 0.89 0.86 Mean argument recall (e) 0.87 0.86 1.00 0.89 0.86 Mean argument Jaccard index (t) 0.83 0.81 1.00 0.86 0.81 Mean argument Jaccard index (e) 0.83 0.81 1.00 0.86 0.81 Mean argument value conditional accuracy (t) 0.95 0.94 1.00 0.93 0.95 Mean argument value conditional accuracy (e) 0.95 0.95 1.00 0.93 0.95 Total errors 1 1 000 Incomplete implementations 00000 Runtime errors 1 1 000 TABLE VII: Evaluation Results of GPT-4o by API for Argument Completion Overall Asana Google Calendar Google Sheets Slack Executable implementations (t) 1.00 0.99 1.00 1.00 1.00 Correct implementations (t) 0.77 0.77 0.86 0.71 0.75 Correct implementations (e) 0.77 0.77 0.86 0.71 0.75 Illegal implementations (t) 0.09 0.07 0.14 0.06 0.11 Illegal implementations (e) 0.09 0.07 0.14 0.06 0.11 Correct argument names (t) 0.93 0.96 0.93 0.92 0.91 Correct argument names (e) 0.93 0.97 0.93 0.92 0.91 Correct argument values (t) 0.90 0.91 0.93 0.89 0.88 Correct argument values (e) 0.90 0.92 0.93 0.89 0.88 Missing arguments (t) 0.07 0.04 0.07 0.08 0.09 Missing arguments (e) 0.07 0.03 0.07 0.08 0.09 Unnecessary arguments (t) 0.01 0.02 0.00 0.00 0.02 Unnecessary arguments (e) 0.01 0.02 0.00 0.00 0.02 Illegal arguments (t) 0.05 0.02 0.07 0.02 0.07 Illegal arguments (e) 0.05 0.02 0.07 0.02 0.07 Mean argument precision (t) 0.93 0.96 0.93 0.98 0.90 Mean argument precision (e) 0.94 0.97 0.93 0.98 0.90 Mean argument recall (t) 0.93 0.97 0.93 0.93 0.91 Mean argument recall (e) 0.94 0.97 0.93 0.93 0.91 Mean argument Jaccard index (t) 0.91 0.95 0.91 0.92 0.88 Mean argument Jaccard index (e) 0.92 0.95 0.91 0.92 0.88 Mean argument value conditional accuracy (t) 0.95 0.95 1.00 0.96 0.94 Mean argument value conditional accuracy (e) 0.95 0.96 1.00 0.96 0.94 Total errors 1 1 000 Incomplete implementations 00000 Runtime errors 1 1 000