From Manual Price Surveys to a Digital Price Statistics Pipeline
19
Product groups digitised
3
Integrated data sources
10+
Index methods supported
V4
International presentation
The Organisation
A National Statistics Office Modernising Price Data Collection
The Statistical Office of Slovakia (SOS) is the national authority responsible for producing official price statistics, including the Consumer Price Index (CPI) — a key input into Slovakia's national inflation figures. The organisation operates under rigorous methodological standards aligned with Eurostat guidelines and serves government, central banks, and the wider public with accurate, timely economic data.
As digital commerce reshaped retail across Europe, SOS faced a growing challenge: the methods used to collect price data had not kept pace with how and where goods are actually sold.
The Challenge
A Data Collection Process Built for a Pre-Digital Era
Price data collection for SOS relied on three parallel methods. Scanner data from large retail chains covered food and groceries. National accounts handled energy and fuel. Everything else — consumer electronics, clothing, furniture, sports equipment, and more — was gathered the traditional way: field employees visiting brick-and-mortar stores each month, manually recording prices.
That third method, covering the broadest range of product groups, carried significant limitations:
High and growing cost of maintenance
Subjective sample selection by the employee conducting the visit
No sales volume data — only price points available
No digitised product parameters, preventing data-driven product clustering
Scaling up collection frequency would have multiplied costs proportionally
The mandate was clear: digitise price data collection for those manually-covered product groups, reduce dependency on field visits, and lay the groundwork for a more scalable, statistically robust approach that could eventually integrate into official production.
The Solution
Building a Digital Price Data Pipeline on AWS
The Dynamic Price Model (DCM) project set out to replace manual field collection with automated digital data gathering for 19 product groups at the ECOICOP 5 level — including footwear, clothing, sports equipment, consumer electronics, furniture, mobile phones, toys, and watches.
Heureka — a leading price comparison platform — was identified as the primary data source. A partnership was established that exceeded expectations: in addition to prices and product parameters, Heureka provided sold quantities, enabling weighted statistical methods for the first time in this category. The resulting dataset covered product IDs and names, sold quantities aggregated across all connected retailers, revenue data, a list of contributing retailers, and full product parameter sets enabling downstream clustering.
This was complemented by daily price data collected from Google Ads via a third-party API service, and weekly web scraping of selected e-commerce retailer pages for current prices and additional product parameters.
The full technical solution — developed on AWS with Python and later migrated to SAS as the defined destination environment for SOS — covered automated data collection, multi-step data cleaning, machine learning-based product clustering, and a full index calculation suite aligned with Eurostat methodology.
Roadmap Highlights
How We Built It
Evaluated 19 product group data sources across sample size, parameter richness, and technical feasibility
Established data feed partnership with Heureka, unlocking sold quantities alongside prices
Integrated Google Ads API and targeted web scraping as complementary sources
Built the DCM data model on AWS: collection, cleaning, clustering, index calculation
Implemented 10 index calculation methods (bilateral and multilateral, weighted and non-weighted)
Migrated solution from AWS/Python into SAS — the production environment of SOS
Deployed Power BI dashboards to visualise system outputs throughout development
Stakeholders engaged
Key Milestone
Unlocking Weighted Statistics Through a Single Partnership
When Heureka agreed to share sold quantities alongside prices, the project gained sales-weighted data from dozens of retailers through a single integration — transforming DCM from a price-collection exercise into a foundation capable of supporting Eurostat-aligned consumer price statistics.
Outcome
A Foundation for the Future of Price Statistics in Slovakia
The DCM project delivered a functioning, end-to-end digital price data pipeline covering 19 product groups — automating collection, cleaning, clustering, and index calculation in a single integrated system. What began as a feasibility and design exercise grew into a full technical implementation, developed on AWS and migrated into the SOS production environment in SAS.
The solution proved capable of producing price indices using both bilateral and multilateral methods, with and without sales weighting — a significant step beyond what the manual process could ever support. Power BI dashboards provided clear visibility into system behaviour at every stage of the pipeline.
The project also had an early impact beyond Slovakia's borders. Before final handover, the development team was invited to present the solution at a V4 national statistics conference in Bratislava in March 2023, introducing the DCM approach to representatives from the Czech Republic, Poland, and Hungary.
