All Comparisons
AgentsKB
+
Bun
Real Test - December 2025

Bun 1.3: How Well Do AI Models Know It?

50 advanced questions on Redis, S3, WebSocket, SQLite, Workers, and more.
3 Claude models. With and without web search.

50 questions Bun 1.3 APIs Opus, Sonnet, Haiku
THE FINDING

All 3 Models Got the Same Question Wrong

Even with web search, they misread the documentation.

Question #4

What environment variable does Bun check first for Redis connection URL?

All 3 Models Said

VALKEY_URL first, then REDIS_URL

Correct Answer

REDIS_URL first, then VALKEY_URL

The information was in the search results. The models simply misread the order. With pre-researched answers, the interpretation happens once, correctly.

Results Summary

Same 50 questions. Same models. Different knowledge sources.

AgentsKB
AgentsKB
Pre-researched answers
50/50
100% accuracy
With Web Search
Opus, Sonnet, Haiku
47-49/50
94-98% accuracy
Training Data Only
No search, no tools
6-21/50
12-42% accuracy

Why This Matters

Training Data Gaps

Bun 1.3 features (Redis client, S3) aren't in training data yet. Without search, models couldn't answer most questions.

Search Finds, But Misreads

Web search found the right pages. But all three models misinterpreted the Redis URL priority order from the docs.

Three Advantages Over Search

Speed: ~500ms vs 10-15s. Efficiency: ~500 tokens vs 10,000+. Focus: We research once with full attention; agents search while distracted by the task at hand.

The 50 Test Questions

Advanced Bun 1.3 specifics — click to expand each category

Redis Client 5 questions
  1. 3. What is the minimum Redis server version supported by Bun's built-in Redis client?
  2. 4. What environment variable does Bun check first for Redis connection URL?
  3. 5. What is the default connectionTimeout for Bun Redis client?
  4. 14. What is the default idleTimeout for Bun Redis client?
  5. 15. What is the default maxRetries for Bun Redis client?
S3 Client 3 questions
  1. 11. What is the default AWS region if none is specified in Bun S3?
  2. 12. What environment variable does Bun check first for S3 access key ID?
  3. 13. How much faster is Bun S3 client vs @aws-sdk/client-s3?
WebSocket 2 questions
  1. 1. What is the default maxPayloadLength for Bun WebSocket?
  2. 2. What is the default idleTimeout for Bun WebSocket?
SQLite 1 questions
  1. 6. How much faster is bun:sqlite compared to better-sqlite3?
Workers & Spawn 5 questions
  1. 17. What does smol: true do for Bun workers?
  2. 31. How much faster is Bun.spawnSync() compared to Node.js?
  3. 32. What is the default killSignal for Bun.spawn() timeout?
  4. 33. What is the default stdout value for Bun.spawn()?
  5. 34. What is the default stderr value for Bun.spawn()?
Other APIs 34 questions
  1. 7. What is the default port for Bun.serve() HTTP server?
  2. 8. What is the .env file loading order in Bun?
  3. 9. What is bun install speed compared to npm?
  4. 10. What is the default test timeout in bun test?
  5. 16. How much faster is Bun's postMessage compared to Node.js?
  6. 18. What is the lockfile format in Bun 1.2+?
  7. 19. What is the default linker strategy for workspaces in Bun?
  8. 20. What is the default network concurrency for bun install?
  9. 21. What is the default password hashing algorithm in Bun?
  10. 22. What is the default non-cryptographic hash algorithm in Bun.hash()?
  11. 23. What is the DNS cache limit in Bun?
  12. 24. What is the default DNS cache TTL in Bun?
  13. 25. How much faster is Bun.semver compared to node-semver?
  14. 26. What is the default value of onlyFiles in Bun.Glob scan options?
  15. 27. What happens when a password exceeds 72 bytes with Bun's bcrypt?
  16. 28. Does Bun shell auto-escape interpolated variables?
  17. 29. What built-in shell commands does Bun provide cross-platform?
  18. 30. What is the bcrypt cost parameter range in Bun?
  19. 35. What does Bun.nanoseconds() return?
  20. 36. How much faster is Bun.stringWidth() compared to npm's string-width?
  21. 37. What does Bun.peek() do?
  22. 38. What values can Bun.peek.status() return?
  23. 39. What compression levels does Bun's Zstandard support?
  24. 40. What is Bun's HTMLRewriter based on?
  25. 41. What import attribute types does Bun support?
  26. 42. Is bun:ffi production-ready?
  27. 43. What languages can bun:ffi call?
  28. 44. How much faster is Bun's FFI compared to Node.js FFI?
  29. 45. What integer types does bun:ffi support?
  30. 46. Is memory automatically managed in bun:ffi?
  31. 47. Does Bun implement the DOM API?
  32. 48. What stream APIs does Bun implement?
  33. 49. Does Bun support alert(), confirm(), and prompt()?
  34. 50. Does Bun implement ShadowRealm?

Source: bun.sh

Methodology & FAQ

How was this test conducted?

Each model was asked the same 50 questions about Bun 1.3 in three configurations:

  • AgentsKB: Pre-researched answers from our knowledge base
  • With Web Search: Claude with web search enabled
  • Training Only: Claude without any tools

Test date: December 5, 2025 • Models: Claude Opus 4.5, Sonnet 4.5, Haiku 4.5

How accurate is Claude AI on Bun 1.3 questions?

With web search enabled: 94-98%. Without web search (training data only): 12-42%. The massive gap shows that Bun 1.3 features (Redis, S3 clients) aren't in the models' training data yet.

Does Claude know about Bun's built-in Redis and S3 clients?

Without web search, Claude models have limited knowledge of Bun 1.3's newer features like the built-in Redis client (Bun.redis) and S3 client (Bun.s3). With web search, models can find this info but may still misinterpret details.

Is AI web search reliable for technical documentation?

AI web search can find correct documentation but may misinterpret details. In this test, all three Claude models found the right Redis documentation but misread the environment variable priority order. Pre-researched answers avoid this issue.

Researched Once. Available Instantly.

Stop waiting for web search. Get pre-researched answers.

Try the Playground See All Tests