New Homes Calgary: Builder Scraping Engine & AI Property Intelligence
Automated Multi-Builder Pipeline, Mapbox Geocoding, Gemini Vision Enrichment & AI Recommender Chatbot
Operating Context & Friction Points
Prospective homebuyers and real estate professionals in Calgary struggled to navigate a fragmented new-construction market. Over 9 major home builders (including Shane Homes, Truman, Homes by Avi, Excel, and Genesis) hosted their quick-possession and presale inventory across proprietary websites with incompatible data schemas, missing coordinates, unstandardized property types, and zero centralized comparison. Buyers had no unified way to evaluate homes against critical life criteria like school catchments and neighborhood safety.
Architectural Scope & Objectives
As Lead Pipeline Architect for the SAIT Capstone project (newhomeincalgary.ca), design and implement an end-to-end automated data and AI system: build an extensible Python scraping engine across 9 builders, normalize and geocode properties to Calgary communities and quadrants, persist fresh data to Supabase with automated stale listing removal, enrich properties using Gemini AI vision, monitor operations via Slack webhooks in GitHub Actions, and deploy an AI chatbot recommending homes based on family demographics, nearby schools, and Calgary crime statistics.
Extensible Multi-Builder Scraping Engine & Data Integrity
Engineered an object-oriented Python scraping architecture built on an extensible BaseScraper abstract class, extracting fresh listings across 9 prominent Calgary builders.
- Implemented BaseScraper with robust error logging, request rate limiting, exponential backoff retries, and user-agent rotation across Shane Homes, Truman, Homes by Avi, Excel, Genesis, and more.
- Engineered delta filtering (filter_new_properties.py) against Supabase to isolate truly new inventory and minimize downstream processing load.
- Constructed automated stale data removal (delete_missing_properties.py) that detects off-market listings and purges them from the database to ensure high inventory freshness.
Mapbox Spatial Geocoding & Supabase Relational Persistence
Developed automated spatial normalization connecting physical street addresses with municipal community boundaries and Calgary quadrants.
- Integrated Mapbox API to forward-geocode scraped street addresses into high-precision latitude/longitude coordinates.
- Programmatically mapped coordinates to official Calgary communities and assigned regional quadrants (NE, NW, SE, SW).
- Modeled relational database architecture in Supabase PostgreSQL, configuring Properties (with coordinates JSONB, image arrays, and needs-enrichment flags) and Communities tables.
Gemini Multimodal Enrichment, Dual GitHub Actions & Slack Monitoring
Automated two-stage daily orchestration in GitHub Actions pairing multimodal AI classification with operational observability.
- Orchestrated pipeline.yml (daily 7:00 AM MST) for raw extraction, geocoding, and Supabase upserts, followed by enrichment.yml (8:00 AM MST) for AI processing.
- Utilized Google Gemini multimodal vision to classify property types (Single Family, Townhome, Laned Home, Front Drive, Duplex, Condo, Villa) directly from hero images.
- Synthesized compelling marketing descriptions combining bedroom/bathroom counts, square footage, address, and garage configurations.
- Integrated Slack incoming webhooks delivering execution duration, processed counts, success/failure statuses, error logs, and direct links to newly enriched properties.
Consumer AI Chatbot: School Catchment & Community Crime Recommender
Built an interactive conversational recommendation chatbot on newhomeincalgary.ca to transform property discovery for prospective families.
- Conversational intake analyzed customer family size, lifestyle criteria, and whether they had school-aged children.
- Cross-referenced property coordinates against nearby school zones and catchment boundaries to highlight education options for families.
- Ingested official Calgary open data community crime statistics to provide transparent safety context alongside listing recommendations.
Unified PropTech Ecosystem & Capstone Distinction
The automated scraping, enrichment, and chatbot platform successfully powered newhomeincalgary.ca, serving as a flagship capstone demonstration at SAIT's School for Advanced Digital Technology.