Extracting insights from dozens of internet sites usually means manually checking every one, a course of that rapidly turns into overwhelming. Design groups want to trace competitor merchandise, advertising groups wish to monitor content material traits, and product managers want to remain on high of market intelligence. However doing this manually means somebody has to go to websites, copy content material, and set up info earlier than actual evaluation can start. Rule-based scrapers supply some automation, however they’re tightly coupled to web page construction. A web site redesign or a migration to a JavaScript-rendered frontend can silently break the pipeline for days earlier than the crew notices.
Amazon Bedrock AgentCore is a platform to construct, join, and optimize brokers at scale, with any framework or mannequin. The answer on this publish makes use of the AgentCore Browser, a functionality of Amazon Bedrock AgentCore. This absolutely managed browser service renders JavaScript-heavy pages reliably, so your pipeline is extra resilient when web sites change.
This publish demonstrates deploy an automatic perception extraction answer utilizing Amazon Bedrock AgentCore Browser, Amazon Bedrock for AI-powered evaluation, Amazon OpenSearch Serverless for semantic search, and AWS Lambda for orchestration. You construct a system that displays RSS feeds, retrieves net content material utilizing the AgentCore managed browser, extracts insights with AI, and makes every part searchable by means of an internet interface.
Use circumstances
This answer was constructed for design and product groups who want to trace trade traits, however the structure applies to broader eventualities:
Aggressive intelligence: Monitor competitor blogs, product bulletins, and press releases. The AI can establish new options, pricing modifications, or strategic shifts.
Market analysis: Monitor trade information, analyst stories, and commerce publications. Seek for rising traits or applied sciences related to your enterprise.
Content material curation: Combination content material from a number of sources and let the AI establish essentially the most related items to your viewers.
Compliance monitoring: Watch regulatory web sites and information sources for modifications which may have an effect on your enterprise.
What you’ll be taught
On the finish of this publish, you’ll perceive how the event-driven structure separates content material assortment from AI processing, how the browser automation in Amazon Bedrock AgentCore handles JavaScript-heavy pages, and the way vector embeddings in Amazon OpenSearch Serverless energy semantic search throughout your collected insights. The entire implementation is on the market on this GitHub repository.
If you’re new to the companies used on this answer, the next sources present foundational context:
Answer structure
The structure follows an event-driven sample that separates content material assortment from processing. The next diagram illustrates the end-to-end system, organized into three practical layers.
Determine 1 — Structure diagram for the automated net perception extraction system
How the answer works
RSS feed assortment: An Amazon EventBridge schedule triggers an AWS Lambda operate each quarter-hour to examine configured RSS feeds for brand new articles and deduplicate towards Amazon Easy Storage Service (Amazon S3).
Browser-based content material retrieval: For every new article, the Lambda operate opens a browser session by means of Amazon Bedrock AgentCore and connects to it utilizing Playwright over the Chrome DevTools Protocol (CDP). Not like commonplace HTTP requests, the distant browser renders the complete web page together with JavaScript-heavy content material, waits for dynamic parts to load, takes a screenshot, and downloads photos. That is the step that makes the remainder of the pipeline potential: with out dependable web page rendering, the AI extraction downstream would obtain incomplete or damaged content material. The artifacts are uploaded to Amazon S3 in a structured format.
Occasion-driven processing: Amazon S3 add occasions publish messages to an Amazon Easy Queue Service (Amazon SQS) queue, and a second Lambda operate extracts clear textual content from the uncooked HTML.
AI-powered perception extraction: Amazon Bedrock generates summaries, identifies themes and entities, extracts actionable insights, and creates vector embeddings from the cleaned content material.
Indexing and semantic search: The enriched outcomes are listed into Amazon OpenSearch Serverless, which helps each key phrase and vector search.
Person and API entry: Finish customers authenticate by means of Amazon Cognito and entry a React-based frontend on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. For programmatic entry, a Mannequin Context Protocol (MCP) server on Amazon ECS with Fargate exposes the system by means of Amazon CloudFront. MCP is an open commonplace that lets AI assistants and instruments hook up with exterior knowledge sources by means of a unified interface.
Answer parts
The next sections describe the primary parts of the answer.
Content material assortment and browser automation
The RSS sync Lambda operate does greater than fetch feeds. It parses every RSS feed, filters for articles revealed inside a configurable time window (24 hours by default), and deduplicates utilizing a URL hash as a singular ID in Amazon S3. For brand new articles, the operate opens a distant browser session by means of Amazon Bedrock AgentCore and connects through Playwright over CDP. This isn’t a headless browser working in Lambda. It’s a managed browser service: AgentCore hosts the distant browser, and Playwright controls it by means of a WebSocket connection. The browser renders the complete web page, waits for JavaScript to load, and captures the entire rendered output. The operate then extracts photos from the web page, downloads them, and uploads every part to Amazon S3 in a structured format:
s3://bucket/
└── instance.com/
└── abc123def456/ # URL hash
├── article.html # Full HTML
├── screenshot.png # Web page screenshot
├── metadata.json # Article metadata
└── photos/ # Captured photos
├── image1.jpg
└── image2.jpg
The metadata.json file consists of the unique URL, title, timestamp, and references to downloaded property. When the metadata file is uploaded to Amazon S3, it triggers the processing pipeline mechanically.
This separation means content material assortment and AI processing scale independently. When you’re monitoring dozens of RSS feeds, the gathering Lambda operate handles them concurrently whereas the Amazon SQS queue with a dead-letter queue supplies automated retries for processing failures.
AI-powered content material evaluation
The concept extraction Lambda operate preprocesses HTML to scale back token consumption, a step that considerably improves the consistency of AI-generated output.
When it receives a message from Amazon SQS, it pulls the HTML from Amazon S3 and runs it by means of a preprocessing step. Massive HTML recordsdata (over 1 MB) get simplified utilizing html-to-text to keep away from token limits. Smaller recordsdata use Mozilla’s Readability library, which does a greater job of extracting the primary content material and figuring out the first picture.
The cleaned content material then goes to Amazon Bedrock with a customizable immediate. You need to use fashions supported by Amazon Bedrock. For mannequin availability by AWS Area, consult with Supported fashions by AWS Area in Amazon Bedrock. The immediate asks the AI to extract:
A concise abstract.
Key themes and subjects.
Important entities (individuals, firms, merchandise).
Related classes.
Actionable insights.
The Lambda operate additionally generates vector embeddings for semantic search, so yow will discover associated content material even when the precise phrases don’t match. The operate writes every part to Amazon OpenSearch Serverless: the unique content material, AI-extracted metadata, and the embedding vector. The index is configured for each key phrase and vector search. A question like “what are rising design traits” returns related outcomes even when these actual phrases don’t seem within the supply content material.
Accountable AI controls
As a result of this pipeline sends third-party net content material to a basis mannequin and publishes the AI-generated output to your crew, manufacturing deployments ought to embrace safeguards across the extraction step. Amazon Bedrock Guardrails helps you to apply these controls with out altering the extraction immediate:
Content material filtering blocks dangerous or inappropriate materials which will seem in scraped net pages earlier than it reaches your searchable index.
Denied subjects and phrase filters preserve the extracted insights throughout the scope your groups anticipate, which issues when supply feeds are exterior your management.
Contextual grounding checks validate that generated summaries and insights are grounded within the supply article, decreasing the danger of hallucinated claims being listed as info.
As a result of the pipeline ingests content material from exterior web sites, deal with the scraped textual content as untrusted enter: guardrails act as a management level between uncooked net content material and the insights your group consumes.
Net interface and programmatic entry
The React-based frontend with Amazon Cognito authentication supplies a search interface the place groups can browse and discover insights. The MCP server supplies an API layer for integration with different AI instruments, supporting programmatic search and retrieval by means of Amazon CloudFront. MCP is an open commonplace that defines how AI assistants uncover and name exterior instruments, so different AI brokers in your group can question the perception database immediately. Processing parts run inside an Amazon Digital Non-public Cloud (Amazon VPC), with Amazon CloudWatch offering observability and AWS Id and Entry Administration (IAM) implementing least privilege entry.
Classes discovered
Constructing a real-world perception extraction pipeline entails greater than wiring collectively managed companies. A couple of architectural selections had outsized affect on value, reliability, and output high quality, and are price calling out explicitly.
AgentCore Browser classes are highly effective however costly. In our take a look at, every Playwright render by means of Amazon Bedrock AgentCore takes 10–30 seconds and prices considerably greater than a plain HTTP request. URL hashing for deduplication (checking Amazon S3 earlier than triggering a brand new AgentCore session) is crucial to maintain prices manageable at scale. The tradeoff is price it: AgentCore reliably captures pages that will break an ordinary scraper, which is the inspiration the remainder of the pipeline will depend on.
Uncooked HTML is a poor enter for LLMs. Clear HTML earlier than you ship it to Amazon Bedrock to scale back token consumption and enhance the consistency of AI-extracted output. This preprocessing step just isn’t an optimization. It’s a prerequisite for dependable outcomes.
Semantic search broadens what yow will discover. A key phrase seek for “furnishings traits” returns actual matches, whereas vector search surfaces conceptually associated content material no matter terminology. For aggressive intelligence or market analysis, embeddings are what make the search interface genuinely helpful.
SQS decoupling is what makes the pipeline dependable. With out it, a processing failure means the article is rarely analyzed. An Amazon SQS dead-letter queue between Amazon S3 and the extraction Lambda supplies automated retries, a small architectural change with important reliability affect.
Amazon OpenSearch Serverless has an actual value flooring. The service requires minimal capability models no matter utilization, which fits manufacturing workloads with constant question quantity. For lower-volume deployments, Amazon Relational Database Service (Amazon RDS) with pgvector helps the identical vector search capabilities at a decrease baseline value and is price contemplating as a place to begin. See Amazon OpenSearch Serverless pricing particulars.
Clear up
To keep away from ongoing expenses, delete the sources you created.
Empty the Amazon S3 content material bucket.
Run cdk destroy from the venture listing (or delete the stack within the AWS CloudFormation console).
Conclusion
On this publish, you discovered construct an automatic perception extraction answer that transforms guide net monitoring right into a searchable, AI-enriched data base. The event-driven structure decouples content material assortment from processing, Amazon Bedrock AgentCore handles JavaScript-heavy web sites, and Amazon OpenSearch Serverless with vector embeddings powers semantic search past key phrase matching.
The entire supply code, deployment information, and configuration directions can be found within the GitHub repository.
Concerning the authors
Louisa Liu
Louisa is a Options Architect at AWS, the place she works immediately with prospects starting from startups to established companies. She helps them perceive their enterprise wants and guides initiatives from early structure selections by means of to manufacturing, placing the newest cloud and AI applied sciences to work in easy, scalable architectures.
Jia Ting
Jia is a Senior Options Architect at AWS Prototyping and AI Buyer Engineering (PACE), specializing in massive knowledge and agentic AI prototyping options throughout numerous industries, with deep expertise in gaming and automotive sectors.
Michelle Hong
Michelle, PhD, works as a Web3 Business Options Architect at Amazon Net Providers, the place she helps prospects construct modern purposes utilizing a wide range of AWS parts. She demonstrates her experience in machine studying, notably in pure language processing and agentic AI, to develop data-driven options that optimize enterprise processes and enhance buyer experiences.
