Building a Price Monitoring System: A Case-Study Architecture

Price monitoring is one of the most common commercial uses of web data. Retailers track competitor prices, brands watch for unauthorized discounting, and marketplaces monitor seller behavior. It's also a perfect case study in what separates a toy scraper from a production system: matching products correctly, capturing prices accurately, detecting real changes and alerting the right people.

This article walks through the design of a price monitoring system for a mid-size e-commerce business tracking a few thousand products across a dozen competitor sites.

#Key takeaways

• Product matching is the hardest and most important problem. Wrong matches produce confidently wrong insights.
• Capture the full price context: list price, sale price, currency, shipping, availability, seller and timestamp.
• Store price history as append-only observations, and detect meaningful changes, not noise.
• Build data quality checks per source: a scraper that silently returns stale or wrong prices is worse than none.
• Stay within terms, robots.txt and polite rate limits, and prefer official feeds or partnerships where they exist.

#Requirements

Functional

• Track ~5,000 of our products against matching items on ~12 competitor sites.
• Refresh key products several times a day; the long tail daily.
• Alert category managers when a competitor undercuts us by more than a threshold, or when stock status changes on key items.
• Dashboards: price position by category, price history per product, competitor promotion activity.

Non-functional

• Data freshness targets per product tier.
• High accuracy: a false undercut alert erodes trust fast.
• Polite crawling within each site's rules (responsible web scraping).
• Auditability: show the source page behind any price.

#Architecture overview

textProduct catalog ─▶ Matching service ─▶ Match table (our SKU ↔ competitor URL, confidence)
                                             │
                          Scheduler (tiers, politeness) ─▶ Fetchers ─▶ Raw store
                                                                          │
                                           Extractors (JSON-LD / selectors / LLM fallback)
                                                                          │
                                   Validation & normalization ─▶ price_observations (append-only)
                                                                          │
                              Change detection ─▶ Alerts (Slack/email) · Dashboards · Warehouse

This follows the general pattern in architecture of a large-scale scraping pipeline, specialized for pricing.

#1. Product matching

You can't compare prices until you know that competitor product X is your product Y.

Matching signals, strongest first:

1. Global identifiers: GTIN/UPC/EAN, manufacturer part numbers. Often present in JSON-LD or spec tables.
2. Brand + model number extracted and normalized.
3. Title and attribute similarity: embeddings plus attribute comparison (size, color, capacity) (text embeddings explained).
4. Image similarity as a supporting signal.

Process:

• Candidate generation via competitor site search or category crawls.
• Scoring with the signals above, where identifier matches are near-certain.
• Human review for mid-confidence matches, using a simple approve/reject UI.
• Store matches with confidence and provenance, and re-verify periodically, because listings change.

Variant handling matters: a 128 GB and a 256 GB phone are different products. Pack sizes ("pack of 6") require unit price normalization.

#2. Scheduling and crawling

• Tiered refresh: top sellers and volatile categories several times a day; the rest daily or less.
• Per-domain politeness enforced centrally: rate limits, concurrency caps, crawl-delay and robots.txt.
• Conditional requests and change hashes so unchanged pages cost little.
• HTTP first; a headless browser only for sources that require it (scraping with Playwright).
• Stop-on-block: if a site starts rejecting requests, pause that source and escalate to a human to review access, rather than trying to get around it.

#3. Extraction

Price pages are good candidates for structured data: many retailers publish Product and Offer JSON-LD with price, currency and availability. Priority order:

1. JSON-LD / embedded JSON.
2. Source-specific selectors, with tests against saved fixtures.
3. LLM extraction as a fallback for new or changed layouts (LLMs for web scraping).

Extract the full price context:

json{
  "competitor": "shop-a",
  "url": "https://shop-a.example/p/12345",
  "observed_at": "2026-09-28T09:14:03Z",
  "list_price": 129.99,
  "sale_price": 109.99,
  "currency": "USD",
  "in_stock": true,
  "shipping_cost": 0,
  "seller": "shop-a",
  "promo_text": "Save 15% this week",
  "extractor": "jsonld@v3",
  "raw_ref": "s3://raw/shop-a/2026/09/28/ab12cd.html.gz"
}

#4. Validation and normalization

Pricing errors are costly, so validation is strict:

• Type and range checks: positive prices, known currencies.
• Plausibility: flag prices that deviate more than, say, 60% from the product's recent median; hold them for review instead of alerting.
• Sale vs list sanity: the sale price shouldn't exceed the list price.
• Currency and tax normalization, including VAT-inclusive vs exclusive by market.
• Unit price for multipacks.
• Staleness detection: if the page content hash hasn't changed in a long time for a normally volatile product, verify the crawl is actually working.

#5. Storage

• price_observations: an append-only table of every validated observation. It's the source of truth for history.
• current_prices: a materialized latest view per match for fast dashboards.
• A warehouse copy for analytics: price elasticity studies, promotion calendars and competitive positioning (data modeling with star schemas).

Append-only history makes it trivial to answer "what was their price on Black Friday?" and to reprocess if normalization logic changes.

#6. Change detection and alerts

Not every change deserves an alert. Rules based on business meaning:

• Undercut: competitor effective price (price + shipping) below ours by more than X% on a tracked product.
• Stock changes on key products: a competitor out of stock is an opportunity.
• Promotion detected: sale price appears or disappears.
• Sustained change: require two consecutive observations to confirm, which filters out transient glitches and A/B-tested prices.

Route alerts to the owning category manager, with a link to the evidence (the raw page snapshot and the history chart). Digest low-priority changes daily instead of pinging constantly.

#7. Monitoring the monitor

Per competitor source, track:

• Fetch success rate and status codes.
• Percentage of matches with a fresh valid price within the SLA.
• Extraction fill rate and validation rejects.
• Alert volume anomalies, since a spike often signals a parsing bug rather than a price war.

These metrics are the difference between a system people trust and one they quietly stop using (data quality guide).

#8. Compliance and relationships

• Review each competitor site's terms and robots.txt before adding it, and document the review.
• Collect only product and price data; no personal data.
• Use polite rates, honest identification and a stop-on-block policy.
• Where possible, use official channels: affiliate feeds, partner data programs or licensed pricing data providers can replace scraping entirely for some sources (web scraping vs APIs, is web scraping legal?).

#Build vs buy

Commercial price intelligence tools exist and can be the right choice for standard retail categories. A custom build tends to win when you need unusual sources, custom matching logic (complex variants, B2B catalogs), tight integration with your pricing engine, or control over data and costs.

#FAQ

#How often should competitor prices be checked?

Match frequency to price volatility and business impact: several times a day for key, volatile products; daily or weekly for the long tail. Higher frequency increases load on target sites and your costs, so justify it by the decisions it supports.

#How accurate does product matching need to be?

Very. A wrong match generates misleading comparisons and bad pricing decisions. Use global identifiers when available, require human review for uncertain matches, and periodically re-validate existing matches.

#Should prices update automatically based on competitors?

Automated repricing is possible but risky without guardrails: floors and ceilings, margin protection, rate limits on changes and human oversight. Many businesses start with alerts and recommendations, then automate narrow, well-understood cases.

#What about sites that show different prices by location or user?

Prices can vary by region, currency, membership and even experiment group. Define the context you're measuring (region, logged-out, default currency), keep it consistent, and record it with each observation.