Cross Language A2A Agent Benchmarking with Antigravity CLI

agy

git clone https://github.com/xbill9/a2a-benchmark
cd a2a-benchmark

xbill@penguin:~/a2a-benchmark$ source init.sh
Project ID file found, skipping.
--- Authentication Method ---
Do you want to use a Gemini API Key for authentication? (y/n): n
WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
Updated property [core/project].
--- Setup complete ---

xbill@penguin:~/a2a-benchmark$ source set_env.sh
--- Setting Google Cloud Environment Variables ---
Checking gcloud authentication status...
gcloud is authenticated.
Exported PROJECT_ID=comglitn
Exported PROJECT_NUMBER=1056842563084
Exported SERVICE_ACCOUNT_NAME=1056842563084-compute@developer.gserviceaccount.com
Exported GOOGLE_CLOUD_PROJECT=comglitn
Exported GOOGLE_GENAI_USE_VERTEXAI=TRUE
Exported GOOGLE_CLOUD_LOCATION=us-central1
Exported REPO_NAME=
Exported REGION=us-central1
--- Environment setup complete ---
xbill@penguin:~/a2a-benchmark$

gcloud auth login
gcloud auth application-default login

cd ~/adk-hello-world-a2a
source set_env.sh

cd ~
git clone https://github.com/a2aproject/a2a-inspector

Running the Go project...
{"time":"2025-11-25T13:47:18.071713651-05:00","level":"INFO","msg":"Using Model ","model":"gemini-2.5-flash"}
{"time":"2025-11-25T13:47:18.07195886-05:00","level":"INFO","msg":"Starting A2A mersenne prime server","port":"8102"}
{"time":"2025-11-25T13:47:18.072116547-05:00","level":"INFO","msg":"Starting the web server: &{port:8102 writeTimeout:15000000000 readTimeout:15000000000 idleTimeout:60000000000}"}
{"time":"2025-11-25T13:47:18.072119644-05:00","level":"INFO","msg":""}
{"time":"2025-11-25T13:47:18.072121732-05:00","level":"INFO","msg":"Web servers starts on http://localhost:8102"}
{"time":"2025-11-25T13:47:18.072123829-05:00","level":"INFO","msg":" a2a: you can access A2A using jsonrpc protocol: http://localhost:8102"}
{"time":"2025-11-25T13:47:18.072125402-05:00","level":"INFO","msg":""}

"id": "adk-88a8aefa-3070-48ff-b8d1-631af65ea57e",
  "name": "generate_primes",
  "response": {
    "result": "Elapsed time: 307.932µs"
  }

what do you do

I can calculate Mersenne primes using the Lucas-Lehmer primality test. I can find the list of the first N Mersenne primes.

find the first 10 primes

It took 0.0002677440643310547 seconds to find the first 10 Mersenne primes.

xbill@penguin:~/a2a-benchmark$ source bench-python.sh

INFO: Started server process [10495]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8101 (Press CTRL+C to quit)

/home/xbill/a2a-benchmark/benchmark-node
staring a2a bench node generate prime

what do you do
message
Found first 5 Mersenne primes in 0.09ms.

✅
find 10 primes
message
Found first 10 Mersenne primes in 0.81ms.

✅

xbill@penguin:~/a2a-benchmark$ source test-rust.sh 
Checking if Rust A2A agent is running on port 8104...
Rust agent is already running.
Running Test 1: Fetching Agent Card...
Agent Card Response:
{
  "name": "Mersenne Prime Agent Rust",
  "description": "A rust agent that builds a list of the first n Mersenne primes and reports the elapsed time. Configured with model: Not specified.",
  "protocolVersion": "0.3.0",
  "version": "0.1.0",
  "url": "http://0.0.0.0:8104/",
  "skills": [
    {
      "id": "find-mersenne-rust",
      "name": "Find Mersenne Primes in rust",
      "description": "Finds the list of the first n Mersenne primes in Rust",
      "tags": [
        "math",
        "benchmark"
      ]
    }
  ],
  "capabilities": {},
  "defaultInputModes": [],
  "defaultOutputModes": []
}
✅ Test 1 Passed: Agent Card is valid.
Running Test 2: Sending message/send request to calculate 5 Mersenne primes...
Response:
{
  "id": 1,
  "jsonrpc": "2.0",
  "result": {
    "contextId": "test-context-123",
    "kind": "message",
    "messageId": "f20c704c-bafa-4081-a18d-8d7984d4213c",
    "parts": [
      {
        "kind": "text",
        "text": "Found first 5 Mersenne primes in 0.11ms."
      }
    ],
    "role": "agent"
  }
}
✅ Test 2 Passed: Successfully calculated primes.
Running Test 3: Sending invalid RPC method...
Error Response:
{
  "error": {
    "code": -32601,
    "message": "Method not found: invalid/method"
  },
  "id": 2,
  "jsonrpc": "2.0"
}
✅ Test 3 Passed: Invalid method was rejected successfully.
All tests passed successfully!

╭──────────────────────────────────────────────────────────────────────────────╮                        
                       │ │                        
                       │ ▄▀▀ ▄▀█ █▀▀ ▀█▀ █▀▄▀█ █▀▀ █▀█ │                        
                       │ █▀ █▀█ ▄▄█ █ █ ▀ █ █▄▄ █▀▀ │                        
                       │ │                        
                       │ FastMCP 2.13.1 │                        
                       │ │                        
                       │ │                        
                       │ 🖥 Server name: benchmark │                        
                       │ │                        
                       │ 📦 Transport: HTTP │                        
                       │ 🔗 Server URL: http://127.0.0.1:8100/mcp │                        
                       │ │                        
                       │ 📚 Docs: https://gofastmcp.com │                        
                       │ 🚀 Hosting: https://fastmcp.cloud │                        
                       │ │                        
                       ╰──────────────────────────────────────────────────────────────────────────────╯                        


[11/25/25 14:17:36] INFO Starting MCP server 'benchmark' with transport 'http' on [http://127.0.0.1:8100/mcp](http://127.0.0.1:8100/mcp) server.py:2055
INFO: Started server process [21826]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on [http://127.0.0.1:8100](http://127.0.0.1:8100) (Press CTRL+C to quit)

xbill@penguin:~/a2a-benchmark/.agents$ more mcp_config.json 
{
  "mcpServers": {
    "bench": {
      "serverUrl": "http://127.0.0.1:8100/mcp",
      "serverURL": "http://127.0.0.1:8100/mcp"
    }
  }
}

  I have completed the benchmark run across all four language agents for primes 1 to 22.

  ### Summary of Actions Taken:

  1. Agent Startup & Verification: Automatically checked and started all four target language agents (Node.
  js/TypeScript, Rust, Go, Python) and verified they were active and listening on their respective A2A JSON-RPC ports.
  2. Benchmark Scripting: Created a customized benchmarking script run_benchmark_1_22.py that targets prime calculations
from
  $N = 1$ to $N = 22$.
  3. Execution: Ran the benchmark and recorded the exact computation time (within the tool call) and the total Agent-to-
  Agent (A2A) network/orchestration latency (RTT) for every prime count $N$.
  4. Data Plotting: Generated visualizations for both calculation times ( prime_calculation_times_1_22.png ) and
  overall latency ( a2a_latency_times_1_22.png ).
  5. Reporting: Compiled all calculations and visualizations into the analysis_results.md artifact.

  Please view analysis_results.md to inspect the table and embedded plots showing the comparison of calculation speeds
and
  latency characteristics!

  Formulating Key Findings
  Here is a summary of the key findings from the benchmark results for primes 1 to 22:

  ### 1. Core Computation Performance (Lucas-Lehmer Prime Calculation)

  • Rust is the fastest: Computes $N=22$ in 195.96 ms.
  • Node.js (TS) is extremely close: Computes $N=22$ in 214.06 ms.
  • Go performs well: Computes $N=22$ in 279.54 ms.
  • Python is the slowest: Computes $N=22$ in 572.48 ms (roughly 3x slower than Rust).

  ### 2. A2A Framework & Protocol Overhead

  • Minimal Overhead (Rust & Node.js):
      • Rust and Node.js use lightweight custom HTTP wrappers. Their framework overhead is practically negligible (< 2 
      ms). For example, Node's total round-trip time (RTT) for $N=22$ is 215.78 ms (only 1.72 ms higher than the raw
      calculation time).
  • Significant Base Overhead (Go & Python):
      • Both Go and Python use the official google-adk library. This introduces a fixed baseline overhead per request
      (schema validation, logging, and metadata context processing).
      • Go incurs a consistent baseline RTT of ~1.2s to 1.6s.
      • Python incurs a consistent baseline RTT of ~2.3s to 3.0s.

  Analyzing Agent Initialization
  Based on an analysis of the codebase and execution paths, here is the breakdown of why the benchmark produced these
  results:
  ──────
  ### Factor 1: The Massive Latency (RTT) Difference (Framework vs. Mocking)

  The most striking difference is that Rust and Node.js have near-zero overhead, whereas Go takes ~1.5s and Python     
  takes
  ~2.5s. This is entirely due to how the agents process the benchmark messages:

  1. Rust and Node.js (Mocked Agent Logic / Direct Execution):
      • Both main.rs and server.ts implement custom JSON-RPC endpoints that intercept incoming queries and
      bypass the LLM entirely.
      • They use simple regex to extract the number of primes directly from the text "Calculate the first {N} Mersenne
      primes" , execute the math, and return a mock agent response message. Since there are no LLM API calls, the
      overhead is practically zero.
  2. Go and Python (Real Agentic Execution via ADK):
      • Both Go (main.go) and Python (agent.py) use the official Agent Development Kit ( google-adk ) and
      hook up the Gemini 2.5 Flash model.
      • When they receive a message, the ADK runner executes a full agent reasoning loop. This involves two sequential 
      remote LLM calls (one to determine which tool to call, and another to formulate the final natural language
      response).
      • This remote LLM invocation via the internet accounts for the 1.5–3.0 second latency overhead.

  ──────
  ### Factor 2: Mathematical Computation Speeds (Lucas-Lehmer Loop)

  At higher values (e.g., $N=22$, where the Mersenne number $2^{9941}-1$ has 2,993 decimal digits and the Lucas-Lehmer
  test requires 9,939 iterations of squaring and modular reductions), we see clear performance tiers:

  1. Rust (195.96 ms):
      • Rust is compiled to optimized native machine code (using --release ). The num-bigint crate executes the big
      integer arithmetic without garbage collection overhead, making it the most efficient compiled loop.
  2. Node.js (214.06 ms):
      • V8's JIT compiler is extremely good at compiling JavaScript loops to native machine instructions. Additionally,
      V8's native BigInt implementation is heavily optimized at the engine level (written in assembly/C++).
  3. Go (279.54 ms):
      • While Go compiles to native code, its implementation in main.go performs heap allocations inside the hot
      loop:
        a := new(big.Int).Rsh(s, uint(p))
        b := new(big.Int).And(s, m)
      Allocating new big.Int objects inside a loop that runs thousands of times puts extra pressure on Go's runtime
      allocator and garbage collector, dragging down performance compared to Rust.
  4. Python (572.48 ms):
      • Python handles arbitrary-precision integers natively, but executing the loop in interpreted bytecode has a lot
      of interpreter overhead compared to compiled or JIT-compiled languages. 9,939 iterations of the loop in bytecode
      is the bottleneck.