Using Crawl4AI to Prepare Web Content for Amazon Bedrock Knowledge Bases

When developing Retrieval Augmented Generation (RAG) applications, answer accuracy relies heavily on ingested data quality. AWS detailed an ingestion pipeline using the open-source Python tool Crawl4AI to preprocess public web content before ingesting it into Amazon Bedrock Knowledge Bases. While Bedrock provides a built-in web crawler connector, complex sites often require custom preprocessing such as boilerplate removal, CSS-targeted extraction, and custom metadata tagging. In this reference workflow, Amazon EventBridge triggers an Amazon ECS task on AWS Fargate. Crawl4AI extracts target content, prunes boilerplate into structured Markdown, and outputs metadata sidecars up to Bedrock's 10 KB limit. The task uploads these files to an Amazon S3 bucket and triggers Bedrock's ingestion job via the API, which parses, chunks, and indexes data into a managed vector store. AWS released the complete implementation in an open-source sample repository.

Key takeaways

• AWS demonstrated using open-source Crawl4AI to preprocess web content, outputting structured Markdown and JSON metadata sidecars to Amazon S3 for Bedrock Knowledge Bases.
• The workflow runs as an AWS Fargate task triggered by Amazon EventBridge, performing bounded crawling with CSS selectors and content filters.
• After S3 upload, the task calls Bedrock ingestion APIs to execute automatic parsing, chunking, embedding, and indexing into managed vector stores.
• Each metadata sidecar must match the source filename with a 10 KB size limit, enabling custom filtering on attributes like source URLs and crawl timestamps.