BylinerBot

BylinerBot is our automated web crawler that helps power fact-checking and content analysis features in byliner.ai. We use it to gather information from publicly available web sources to help journalists verify facts and enhance their reporting.

User-Agent Information

What We Do

• Respect robots.txt files and website crawling policies
• Identify ourselves clearly with our User-Agent string
• Cache content to minimize repeated requests
• Limit request frequency to be respectful of server resources
• Extract only textual content for analysis purposes

Crawling Behavior

• We only crawl publicly accessible content
• We respect rate limits and don't overwhelm servers
• We cache results to avoid unnecessary repeated requests
• We limit the size of content we download (1MB maximum)
• We extract only the main textual content, ignoring ads and navigation

Contact & Blocking

If you need to block BylinerBot from crawling your site, you can:

• Add our User-Agent to your robots.txt file
• Contact us at support@byliner.ai
• Use standard HTTP blocking methods if needed

Purpose

BylinerBot is used exclusively to support fact-checking and content analysis features that help journalists verify information and improve the quality of their reporting. We believe in transparent, responsible web crawling that respects website owners and contributes to better journalism.