## Introduction
For data and analytics teams in startups and fast-growing companies, monitoring competitor pricing is critical to making informed pricing strategies and staying competitive. Manually checking competitor websites daily for price changes is inefficient and error-prone. Automating this process not only saves time but allows teams to react faster to market changes.
This article provides a step-by-step guide on how to build an automation workflow using n8n to scrape competitor pricing daily. We’ll cover how to extract pricing data from competitor websites, store the results, and alert your team if significant changes occur.
**Who Benefits:**
– Data & Analytics teams who track pricing data for competitive intelligence.
– Product managers and marketing teams needing real-time pricing insights.
– Automation engineers aiming to build reliable, scalable workflows.
**Problem Solved:**
– Eliminates manual monitoring of competitor prices.
– Provides consistent, automated data collection.
– Enables timely updates to pricing strategies.
—
## Tools and Services Integrated
– **n8n:** Open-source workflow automation tool.
– **HTTP Request Node:** To retrieve competitor web pages.
– **HTML Extract Node:** To parse and extract pricing data.
– **Google Sheets Node:** To store pricing data for easy access and historical comparison.
– **Slack Node:** To send alerts on significant price changes.
– **Cron Node:** To schedule daily scraping.
Optional:
– **Webhook Node:** To trigger workflows externally if needed.
—
## Workflow Overview
The workflow triggers daily via the Cron node.
1. The HTTP Request node fetches the HTML content of competitor pricing pages.
2. The HTML Extract node parses the HTML to find pricing data.
3. The workflow compares the current data with stored historical prices in Google Sheets.
4. If price changes exceed defined thresholds, the Slack node sends an alert.
5. Updated pricing data is appended or updated in Google Sheets.
—
## Step-by-Step Technical Tutorial
### Step 1: Preparing n8n and Access Credentials
– Set up an n8n instance (cloud or self-hosted).
– Configure Google Sheets credentials (via OAuth) in n8n.
– Configure Slack credentials with a webhook URL or Slack App credentials.
### Step 2: Setting up the Cron Trigger
– Add a **Cron** node.
– Configure it to run once every day at a preferred time (e.g., 8 AM UTC).
### Step 3: Fetch Competitor Pricing Pages
– Add an **HTTP Request** node connected to the Cron node.
– Configure the node with:
– HTTP Method: GET
– URL: Competitor pricing page URL (e.g., https://competitor.com/pricing)
– Optional: Set appropriate headers (User-Agent) to avoid scraping blocks.
_If multiple competitors or pages exist, use the **SplitInBatches** node or generate dynamic URLs via Function nodes._
### Step 4: Extract Pricing Data
– Add an **HTML Extract** node connected to the HTTP Request node.
– Configure CSS selectors or XPath to target pricing elements on the page.
For example, if prices are within `$123.45`, use selector `.price`.
– Extract prices and relevant product identifiers.
_Use the **Function** node if data transformation is required to normalize formatting._
### Step 5: Retrieve Historical Data from Google Sheets
– Add a **Google Sheets** node to read the existing price data.
– Use the ‘Lookup’ or ‘Read Rows’ operation to get historical prices for comparison.
– Configure it to retrieve data for the relevant competitor and product entries.
### Step 6: Compare Current Prices with Historical Prices
– Add a **Function** node to perform price comparison logic.
– Logic example:
– For each fetched product price, check against stored price.
– Calculate price difference and percentage change.
– Flag products with price changes above a threshold (e.g., 5%).
### Step 7: Send Slack Alerts on Significant Price Changes
– Add a **Slack** node connected via conditional routing from the Function node.
– If flagged price changes exist, construct a message summarizing:
– Product name
– Old price
– New price
– Percentage change
– Send the alert to your #pricing-alerts Slack channel.
### Step 8: Update Google Sheets with New Pricing Data
– Add a **Google Sheets** node to append or update rows with the scraped prices.
– Ensure to replace outdated prices or add new product entries.
### Step 9: Error Handling and Retry Logic
– Enable error triggers in critical nodes (HTTP Request, Google Sheets).
– Use **IF** nodes and **Error Workflow** to catch errors and send notifications.
– Consider adding exponential backoff or retry logic if HTTP requests fail.
—
## Common Errors and Tips for Robustness
– **Blocked requests:** Competitor websites might block scraping. To mitigate:
– Rotate user-agent headers.
– Respect robots.txt and terms of service.
– Use proxies if necessary.
– **Page structure changes:** HTML markup changes will break CSS selectors.
– Regularly monitor selector validity.
– Log extraction errors and notify maintenance team.
– **Rate limiting:** Avoid sending too many requests in short period.
– Use n8n’s scheduling and batching features.
– **API quotas:** Google Sheets and Slack have usage limits.
– Optimize data reads/writes.
– Batch updates.
– **Data consistency:** Ensure time zone consistency when scheduling and storing timestamps.
—
## Scaling and Adaptation
– **Multiple competitors:** Use a data list of URLs and loop through them dynamically using **SplitInBatches** or **IF** nodes.
– **Multiple products per page:** Parse all products using advanced selectors or JSON extraction if API is available.
– **Data enrichment:** Integrate with BI tools or databases for further analytics.
– **Alert customization:** Add thresholds per competitor or product category.
– **Upgrade storage:** Switch from Google Sheets to databases (e.g., PostgreSQL) via n8n nodes for large datasets.
– **UI Dashboards:** Feed results into visualization tools like Metabase or Grafana.
—
## Summary
Automating daily competitor pricing scraping with n8n empowers data teams to maintain real-time visibility on market pricing. This workflow harnesses n8n’s flexible nodes to fetch, extract, compare, and report pricing data efficiently.
Key takeaways:
– Start with a clear job schedule via Cron.
– Fetch dynamic webpages reliably using HTTP Request and handle potential blocking.
– Extract data with precise selectors and process comparisons in Function nodes.
– Use Slack for timely alerts and Google Sheets for accessible data storage.
– Build in error handling and design modular workflows to scale easily.
—-
## Bonus Tip: Version Control and Deployment
Use n8n’s workflow JSON export/import features along with Git version control to keep track of workflow changes. Combine this with containerized deployment for consistent, stable automation environments across development and production.
This practice will massively improve your team’s collaboration and workflow maintainability as your automation grows.