StratoData Logo
    Content/Case Study
    Case StudyData Collection & Automation

    From Manual Price Surveys to a Digital Price Statistics Pipeline

    Statistical Office of Slovakia·Public Sector / Price Statistics·DCM Project·EU-Funded

    19

    Product groups digitised

    3

    Integrated data sources

    10+

    Index methods supported

    V4

    International presentation

    The Organisation

    A National Statistics Office Modernising Price Data Collection

    The Statistical Office of Slovakia (SOS) is the national authority responsible for producing official price statistics, including the Consumer Price Index (CPI) — a key input into Slovakia's national inflation figures. The organisation operates under rigorous methodological standards aligned with Eurostat guidelines and serves government, central banks, and the wider public with accurate, timely economic data.

    As digital commerce reshaped retail across Europe, SOS faced a growing challenge: the methods used to collect price data had not kept pace with how and where goods are actually sold.

    The Challenge

    A Data Collection Process Built for a Pre-Digital Era

    Price data collection for SOS relied on three parallel methods. Scanner data from large retail chains covered food and groceries. National accounts handled energy and fuel. Everything else — consumer electronics, clothing, furniture, sports equipment, and more — was gathered the traditional way: field employees visiting brick-and-mortar stores each month, manually recording prices.

    That third method, covering the broadest range of product groups, carried significant limitations:

    High and growing cost of maintenance

    Subjective sample selection by the employee conducting the visit

    No sales volume data — only price points available

    No digitised product parameters, preventing data-driven product clustering

    Scaling up collection frequency would have multiplied costs proportionally

    The mandate was clear: digitise price data collection for those manually-covered product groups, reduce dependency on field visits, and lay the groundwork for a more scalable, statistically robust approach that could eventually integrate into official production.

    The Solution

    Building a Digital Price Data Pipeline on AWS

    The Dynamic Price Model (DCM) project set out to replace manual field collection with automated digital data gathering for 19 product groups at the ECOICOP 5 level — including footwear, clothing, sports equipment, consumer electronics, furniture, mobile phones, toys, and watches.

    Heureka — a leading price comparison platform — was identified as the primary data source. A partnership was established that exceeded expectations: in addition to prices and product parameters, Heureka provided sold quantities, enabling weighted statistical methods for the first time in this category. The resulting dataset covered product IDs and names, sold quantities aggregated across all connected retailers, revenue data, a list of contributing retailers, and full product parameter sets enabling downstream clustering.

    This was complemented by daily price data collected from Google Ads via a third-party API service, and weekly web scraping of selected e-commerce retailer pages for current prices and additional product parameters.

    The full technical solution — developed on AWS with Python and later migrated to SAS as the defined destination environment for SOS — covered automated data collection, multi-step data cleaning, machine learning-based product clustering, and a full index calculation suite aligned with Eurostat methodology.

    AWSPythonSASPower BIMachine LearningHeureka APIGoogle Ads APIWeb Scraping

    Roadmap Highlights

    How We Built It

    1

    Evaluated 19 product group data sources across sample size, parameter richness, and technical feasibility

    2

    Established data feed partnership with Heureka, unlocking sold quantities alongside prices

    3

    Integrated Google Ads API and targeted web scraping as complementary sources

    4

    Built the DCM data model on AWS: collection, cleaning, clustering, index calculation

    5

    Implemented 10 index calculation methods (bilateral and multilateral, weighted and non-weighted)

    6

    Migrated solution from AWS/Python into SAS — the production environment of SOS

    7

    Deployed Power BI dashboards to visualise system outputs throughout development

    Stakeholders engaged

    Price Statistics MethodologyData EngineeringSOS ManagementEurostat (methodology)Heureka (data partner)

    Key Milestone

    Unlocking Weighted Statistics Through a Single Partnership

    When Heureka agreed to share sold quantities alongside prices, the project gained sales-weighted data from dozens of retailers through a single integration — transforming DCM from a price-collection exercise into a foundation capable of supporting Eurostat-aligned consumer price statistics.

    Outcome

    A Foundation for the Future of Price Statistics in Slovakia

    The DCM project delivered a functioning, end-to-end digital price data pipeline covering 19 product groups — automating collection, cleaning, clustering, and index calculation in a single integrated system. What began as a feasibility and design exercise grew into a full technical implementation, developed on AWS and migrated into the SOS production environment in SAS.

    The solution proved capable of producing price indices using both bilateral and multilateral methods, with and without sales weighting — a significant step beyond what the manual process could ever support. Power BI dashboards provided clear visibility into system behaviour at every stage of the pipeline.

    The project also had an early impact beyond Slovakia's borders. Before final handover, the development team was invited to present the solution at a V4 national statistics conference in Bratislava in March 2023, introducing the DCM approach to representatives from the Czech Republic, Poland, and Hungary.