Executive summary
Public energy data in Ontario is plentiful. IESO publishes market and system reports. OEB publishes distributor reporting. Anyone with a browser can find a file. That abundance is not a usable analytical foundation. Each publisher encodes a different job, grain, vocabulary, and idea of “current.” A directory of links invites every analyst to reconstruct the same interpretation: hour-ending conventions, forecast versus actual, demand versus consumption, customer versus connection, system megawatts versus distributor kilowatts.
AskBhalo treats that interpretation work as the product. Authoritative public sources are catalogued first; a smaller set of governed data products is published second. Each product has a stated purpose, native grain, lineage, freshness, and a quality status that may differ from commercial visibility. The catalogue is the consumer surface, not a flattened warehouse table. This case study is grounded in the repository: IESO as the first operational foundation, OEB as a second publisher, and an architecture that can add further public publishers without collapsing their semantics.
1. The problem
Energy analysts do not lack files. They lack a shared answer to five questions: what the measure is; at what grain it was observed; who published it and after what transformation; how current it is against that publisher’s cadence; and whether the series is healthy enough to use. Those questions become expensive when two series both contain the word “demand,” when a blank in a regulatory file has already been replaced with zero, or when today’s incomplete hourly file is treated as a missing-data failure.
Public sources differ in structure, cadence, history, and language. IESO hour-ending reports are not OEB annual RRR extracts. Ontario Demand is not Market Demand, not FSA kWh, and not a distributor’s billed kW. Treating those as interchangeable columns is a modelling error dressed up as convenience.
A publisher’s data directory is a discovery surface for *their* reports, not a product catalogue. Cataloguing a report does not mean it has been ingested. Ingesting it does not mean it is commercially visible. Users should not have to learn every publisher-specific implementation detail before they can work. They also should not be protected from provenance.
2. Why a file catalogue was not enough
The first architectural decision is older than any commercial SKU: catalogue before ingestion. IESO lists dozens of reports on its Data Directory. Ingesting all of them would delay the semantic layer and conflate “we have seen this report” with “we can stand behind a product.” Every publicly listed report can be registered. Ingestion stays selective. A catalogued dataset does not automatically create an adapter or a curated table. The same rule was applied to OEB Open Data. Catalog completeness is not ingestion completeness, and ingestion completeness is not productization.
Four registry objects stay distinct (ADR-008): the source dataset the publisher emits; the curated asset at one grain; the metric as a governed measure, not a chart label; and the data product as the consumer bundle of assets, metrics, quality, lineage, and interfaces. “The IESO demand file” is not “the Ask Bhalo demand product.”
A file catalogue also cannot express commercial readiness. Commercial visibility is not data availability, and neither is API or download availability. A product can appear as Data Contracted or Coming Soon without implying that rows or a stable history window exist.
3. Designing the catalogue around data products
The opposite failure mode from a file dump is a giant universal energy table: one row grain, many renamed columns, and a hope that “demand” means the same thing everywhere. The catalogue rejects that. Analytical domains are distinct products, grouped into families that match jobs rather than publisher folders: Ontario Demand & Adequacy; Generation & Renewables; Market Prices & Congestion; Operating Reserve; Interties & Transmission; and Ontario Electricity Distributor Intelligence.
Families are taxonomy and discovery, not a licence to join every member on a common key. FSA consumption, industrial load by sector, and transmission-tariff peak series are related family concepts around Ontario demand; they are not required dependencies of the current Ontario Electricity Demand contract (ADR-021). Activating a planned source later is a product-version change, not a silent widening of today’s grain.
Ontario Electricity Demand (`energy.ontario.electricity_demand`) is the worked example of a composed product (ADR-014). The parent is a consumer bundle. Child assets keep their own grains and measurement types—actual, forecast, preliminary, final. Hourly actuals, a forecast vintage at issue time × target hour, FSA × hour, and a tariff reporting month cannot share one physical table without destroying meaning.
Increment 01 delivers the reusable lifecycle plus that demand product, sourced from the IESO Hourly Ontario and Market Demand report. The adapter interface is publisher-agnostic so later publishers need not subclass IESO types. Where a job cannot yet be certified, the product stays in discovery, data-contracted, building, or validating.
4. What makes a product governed
A governed product is a contract with named obligations. In this codebase those obligations are first-class metadata, not footnotes on a dashboard.
Purpose and intended use. Every consumer product states the problem it answers and who it is for, and what it will not do: no reconstructed participant settlement statements, no composite utility scores, no uncertified cross-publisher ratios.
Source and publisher. Role, dependency lifecycle, and `required_for_current_release` are independent facts. Real-time Totals can remain semantically primary for demand while staying planned and not required for the current BETA actuals contract.
Native grain and canonical schema. Demand actuals are one row per Ontario geography, market date, and hour ending. IESO hours are Eastern Standard Time year-round (ADR-005). Distributor products remain annual.
Business and metric definitions. Glossary terms keep source wording and canonical meaning separate (ADR-010). Metrics are registered objects; only validated and certified metrics are treated as trusted (ADR-011). Measures are classified as published `[P]`, official equation `[E]`, deterministic derived `[D]`, or modelled `[M]` (ADR-037).
Lineage, freshness, quality, history, and versioning. Lineage is bidirectional (ADR-013). Freshness is explicit against publisher cadence. Historical coverage is loaded and entitled coverage. Raw artifacts are immutable and content-hashed. Source version is the file hash; pipeline version is the transformation; data-product version is the consumer contract. Commercial lifecycle is separate from operational health. A Beta product can be healthy.
Access and ownership. The commercial API is a common product contract. Source routes, raw RRR (Future Product version), and internal canonical slugs are not customer products (`commercial_exposed = false`). Metadata ownership is accountability for definitions, not custody of the public series.
5. Preserving grain and source semantics
The most expensive mistakes in this domain are false joins. Ontario Demand is energy withdrawn from the IESO-controlled grid to serve Ontario load, including losses, excluding exports. Market Demand typically adds export schedules. FSA consumption is kWh for a restricted customer population. OEB distributor demand is billing or distribution-scope kW, usually annual, often at generic rate class. Increment 04 certifies no IESO↔OEB transformation. The allowed relationship is `RELATED_NOT_COMPARABLE` or `NOT_COMPARABLE`, never `EXACTLY_COMPARABLE` unless an unquestionable shared definition exists (ADR-035).
The same discipline applies inside one publisher. Day-ahead or pre-dispatch Total Load is not the Ontario Demand forecast-error pair; Adequacy Average, Peak, and Forecast Ontario Demand paired with real-time Ontario Demand is. Industrial load leaves are not summed into a total the publisher already publishes under a different aggregation role. Customers are not connections. CAIDI, where derived, is SAIDI/SAIFI when SAIFI is non-zero, and is labelled derived when the source does not publish CAIDI.
Publisher transformations are first-class. When OEB states that blanks were populated with zeros, those zeros are retained with `UNKNOWN_ZERO_ORIGIN` and a blank-to-zero publisher transformation (ADR-032, ADR-033). They are not converted to NULL and not labelled as reported zeros. Physical fields map to canonical elements through explicit mappings (ADR-009); ambiguous names are not auto-merged. Organization identity is exact legal name, known alias, or historical predecessor—never fuzzy auto-resolution. Unresolved names go to a resolution queue. Historical observations join the organization valid in that reporting period.
These rules are product boundaries. They keep the catalogue honest when the consumer experience becomes simpler.
6. Lineage, freshness and quality
Governance here is operational, not a PDF produced after launch. Quality results persist per ingestion. Completeness is evaluated against the product grain and a publication-expectation record, not against “today × 24 hours” (ADR-015). An open current market day is `EXPECTED_PARTIAL` and does not degrade product health. A closed period with a publisher gap is `MISSING`. The 2025-05-01 hour-ending 01 omission in IESO’s `PUB_Demand_2025.csv` is stored as a source-missing observation. The value is not imputed.
`quality_origin` separates inherited publisher issues from platform issues (ADR-016): `SOURCE`, `BHALO` (including processing lag), `MIXED`, or `UNKNOWN`. Product health is operational trust of the *current published contract only* (ADR-020). Planned child assets do not make a usable Beta look unhealthy. Coverage is how much of the intended scope is implemented. Maturity is contract readiness. Mixing those three fields was an early mistake.
Ingestion lineage (file hash → pipeline run) remains necessary. Product-level lineage answers which catalogue SKU is affected when a source lags. Critical quality failures block curated publication. Downloads are generated from curated rows. Raw source files are not the commercial interface. Freshness is stated in publisher time: hourly IESO actuals have a daily rhythm and a 36-hour product target; annual OEB RRR is not “stale” while waiting on next year’s filing. The catalogue must not claim five paid years if three are loaded.
7. From one publisher to a multi-publisher architecture
IESO is the first practical foundation: catalogued Data Directory, selective ingestion, immutable raw capture, standardized then curated zones, and a commercial demand product with derived daily, monthly, and annual actual-demand metrics. The IESO adapter does not own the platform. New publishers implement a source adapter.
OEB entered the same way: catalogue and profile first. Priority RRR families are inspected against live payload schemas before commercial modelling. Field mappings require a verification status. Republications such as `rrr/2024-2/` are first-class identity, not “the latest URL.” Historical organization names and generic rate classes are master data, not inferred tariffs.
The commercial OEB family is Ontario Electricity Distributor Intelligence: several SKUs—profile, customer and connection, demand and consumption, reliability, service quality, interruption cause, and peer benchmarks—not one giant RRR table and not a weighted utility score. Internal canonical assets are not customer products and are not exposed as `/oeb` or `/rrr` routes.
The point of a multi-publisher architecture is a shared factory, not a joined “Ontario energy” fact table. The graph may record that a dataset supports a product or that two metrics are related. It does not mint equivalence. Other Canadian public publishers are anticipated in the adapter design. They are not invented here as live catalogue members.
8. What the catalogue enables
Once the governed layer exists, analysis starts on a documented grain with known units. Sourcing and cleaning are not eliminated, publishers still revise files, but they are done once, under a contract. BI tools, notebooks, and APIs consume the same product identifiers. A later AI system pointed at these products inherits definitions instead of scraping column headers.
The commercial value is reuse and trust, not a measured time-saved claim. Usage analytics were not instrumented in Increment 01. The platform records ingestion success, freshness, quality failures, and whether data, API, and download flags are actually true. Product pages, downloads, the commercial API, and operator admin share one layer. They do not each reinterpret IESO or OEB.
9. Product and governance lessons
“Available” does not mean “usable.” A public URL is only a starting condition. It confirms that a publisher has made something accessible; it does not establish what each field means, whether the file is complete, how revisions are handled, or whether the latest version is analytically comparable with its history. Usability requires a documented grain, units, definitions, lineage, transformation history, freshness expectation and quality state that a professional can defend. Ask Bhalo therefore treats source discovery as the beginning of product work, not its completion.
A data catalogue is not a file directory. A directory organizes reports according to how a publisher releases them. A governed catalogue organizes products according to the analytical jobs consumers need to perform. It distinguishes the original source dataset, the curated asset, the governed metric and the commercial data product instead of allowing one file name to stand in for all four. This distinction also permits honest lifecycle states: a source may be catalogued but not ingested, ingested but not productized, or represented by a visible product whose data contract is still being validated.
Grain is a product decision. Grain determines what one row represents and therefore what comparisons, aggregations and joins are legitimate. Hourly Ontario system demand in megawatts, annual distributor demand in kilowatts and FSA-level consumption in kilowatt-hours may all relate to electricity demand, but they describe different populations, time scales and measurement concepts. Flattening them into one universal energy table would conceal those differences rather than resolve them. Ask Bhalo preserves each native grain and composes products from related assets without forcing them into one misleading physical structure.
Lineage is part of the user experience. Lineage should not be reserved for engineers investigating a failed pipeline. A consumer needs to know which publisher, report, file version and transformation produced a metric—and which commercial product is affected when a source arrives late, changes structure or fails a quality rule. Ask Bhalo therefore treats lineage as bidirectional: users can move from a product back to its supporting sources, while operators can move from a source event forward to the products and interfaces it may affect. Provenance becomes part of using the product, not an explanation assembled after something goes wrong.
Freshness must be explicit. “Current” has no useful meaning without the publisher’s cadence and the product’s own service expectation. An annual OEB filing is not stale because a new record has not appeared this month. An open IESO market day is not incomplete merely because all 24 hourly observations are not yet available. Ask Bhalo distinguishes an expected partial period from a genuine publisher gap and separates source delay from platform-processing delay. Freshness is evaluated against what should reasonably exist at that moment—not against an indiscriminate rule that newer is always healthier.
Product boundaries matter. Related products are not automatically equivalent products. Membership in the same analytical family helps consumers discover relevant information, but it does not create a valid join key, common denominator or shared definition. Ontario Demand, Market Demand, distributor consumption and transmission-tariff peaks can belong to a broader demand-and-adequacy family while remaining analytically distinct. Ask Bhalo records relationships such as RELATED_NOT_COMPARABLE where appropriate instead of manufacturing comparability. Commercial visibility is also separate from data availability, operational health and service-level commitments.
Governance should be operationalized. Governance loses value when definitions live in a wiki while production pipelines, APIs and downloads operate under different assumptions. In Ask Bhalo, definitions, classifications, metric formulas, publisher transformations, lineage, completeness rules and quality outcomes sit within the same operating model that determines whether rows can be published. Critical failures can prevent curated publication rather than merely generate a warning for someone to read later. Governance therefore becomes an executable product control: it influences what enters the catalogue, what reaches consumers and what must remain validating.
Derived intelligence must remain traceable. A calculated metric should never become indistinguishable from a value published by the source. Ask Bhalo classifies measures according to their evidentiary basis: [P] for publisher-reported values, [E] for calculations using a verified official equation, [D] for deterministic derived measures with disclosed formulas, and [M] for modelled outputs. A [D] metric must preserve its inputs, formula, aggregation logic and zero-denominator treatment. An [E] classification requires authoritative evidence for the equation. [M] cannot be used as a convenient label for an undocumented score whose logic cannot be defended.
Do not certify what you have not verified. A product roadmap creates pressure to move capabilities from validating to available, but commercial urgency cannot substitute for semantic proof. Intertie direction, sign conventions and similar publisher-specific rules must remain validating until authoritative documentation or reproducible evidence establishes their meaning. Holding a product back under those conditions is not engineering delay; it is responsible data-product management. A trustworthy catalogue must be willing to say “not yet verified” because every certification becomes a promise that consumers may use in consequential analysis.
Conclusion
Ask Bhalo is a governed energy data-product catalogue and a product factory. It is not a reporting interface over someone else’s ZIP file, and it is not a lake of renamed columns. The job is to move a professional user from “I found the data” to “I understand this product well enough to use it”: what the number means, who published it, at what grain, after which transformation, how current it is, and whether the current contract is healthy. IESO made that factory real. OEB proved it can hold a second publisher without erasing the first. Further public sources can enter the same way—catalogued first, productized only when the contract can be told truthfully. The catalogue will keep growing. The constraint that should not grow is ambiguity.
How A18 Delivers BI Modernization and Governance Engineering
A18 tailors each engagement to the client’s business priorities, regulatory context, internal capability, and existing technology environment. A typical engagement may include:
- Metadata-estate and source-system assessment;
- Connector and ingestion planning;
- Catalogue and governance-model design;
- Ownership, stewardship and classification;
- Scan-orchestration deployment;
- Lineage and impact-analysis implementation;
- Schema-drift operating procedures;
- AI-enrichment controls and approval workflows;
- Security, credentials, role-based access and production hardening.
A18 can lead the full modernization, provide an independent assessment and roadmap, establish the governance and architecture, or work alongside the client’s internal teams and implementation partners.
Partner With A18 Analytics
If your organization has too many reports, conflicting KPIs, slow regulatory reconciliation, unclear ownership, or low confidence in executive information, A18 Analytics can help you design and implement a governed BI environment that works with your technology landscape.
Website: www.a18analytics.com
Inquiries: info@a18analytics.com