Search Engine Limitations

DuckDuckGo Result Limit

Problem

DuckDuckGo often limits the number of search results to 10 results, regardless of how many results are requested. This is a known limitation of the DuckDuckGo API/DDGS library.

Configuration

In config.json, you can configure the maximum number of results:

{
  "search": {
    "max_results": 25,  // For normal web searches
    ...
  },
  "osint": {
    "max_results": 25,  // For OSINT/site: searches
    ...
  }
}

Note: Even if you set max_results: 25, DuckDuckGo may still only return 10 results.

Solutions

1. Fallback to Other Search Engines

If more results are needed, you can use alternative search providers:

{
  "search": {
    "provider": "brave",  // or "serper"
    "fallback_providers": ["duckduckgo"]
  }
}

Brave Search and Serper support more results:

• Brave: up to 20+ results
• Serper: up to 100+ results (with API key)

2. Multiple Searches with Different Keywords

Instead of a single search, you can perform multiple searches with more specific keywords:

site:example.com product
site:example.com service
site:example.com contact

3. Pagination (Not Supported)

DuckDuckGo does not support true pagination via the DDGS API.

Current Implementation

CrawlLama automatically logs a warning when fewer results are returned than requested:

⚠️ DuckDuckGo returned only 10 results (requested: 25). This is a known limitation.

Recommendation

For OSINT analyses with many results:

1. Set "provider": "brave" in config.json
2. Or use Serper with an API key
3. Or perform multiple specific searches

Setting Up API Keys

Serper (Recommended for Many Results)

1. Register at https://serper.dev/
2. Get a free API key (2500 requests/month)
3. Set the environment variable: SERPER_API_KEY=your_key_here

Brave Search

1. Register at https://brave.com/search/api/
2. Get a free API key (2000 requests/month)
3. Set the environment variable: BRAVE_API_KEY=your_key_here

Status

• ✅ DuckDuckGo: Free, no API keys, but only ~10 results
• ✅ Brave: Free (with API key), up to 20+ results
• ✅ Serper: Free (with API key), up to 100+ results

Last Updated: October 24, 2025