Executive summary
Enterprise metadata is rarely missing. It is scattered, across relational databases, cloud warehouses, business intelligence (BI) semantic layers, transformation projects, and code repositories, each exposing structure through different native catalogues, APIs, and conventions. Conventional data dictionaries maintained in documents or spreadsheets decay quickly because they are updated separately from the systems they describe.
MetaOps Studio, an independent product of A18 Analytics Inc., addresses this as an enterprise metadata operations platform. It connects to source systems, extracts native technical metadata, normalizes it into a canonical catalogue, detects structural change, supports governance workflows, exposes lineage, and applies reviewable artificial intelligence (AI) intelligence. The platform is automation-first: operational evidence is created continuously through scheduled and on-demand scans, not typed manually into blank forms.
This case study describes the platform’s demonstrated operating model, its architecture, metadata lifecycle, governance boundary, and current engineering maturity, as evidenced by the MetaOps Studio repository.
The enterprise metadata problem
Metadata operations become difficult at scale for predictable reasons. Every platform exposes metadata differently: PostgreSQL through `information_schema` and `pg_catalog`, SQL Server through `INFORMATION_SCHEMA` and `sys.*`, Snowflake through account-level views, BigQuery through per-dataset catalogues, and Databricks through Unity Catalog. Technical metadata changes continuously as schemas evolve, pipelines are refactored, and BI models are republished.
Manually maintained catalogues fall behind production. Schema drift often surfaces only when downstream jobs fail or reports break. Technical context, column types, constraints, view definitions, frequently lives apart from business meaning such as domain assignment, ownership, and sensitivity classification. Lineage across databases, dbt projects, and BI tools is hard to reconstruct after the fact. AI-generated documentation without structured context and human review introduces governance risk rather than reducing it.
MetaOps Studio is designed around these constraints: automate extraction, preserve evidence, normalize for comparison, govern business meaning separately, and treat AI outputs as suggestions subject to steward approval.
Why MetaOps Studio is automation-first
MetaOps Studio is not primarily a form for stewards to document assets by hand. Its operating model begins with connected systems and proceeds through a repeatable pipeline:
Native connector → raw metadata snapshot → normalization → drift comparison → catalogue upsert → governance → lineage → AI intelligence
The `ScanOrchestrator` in `apps/api/app/services/scan_orchestrator.py` coordinates this flow. Connectors extract platform-native payloads; `RawMetadataSnapshot` records immutable evidence; `MetadataNormalizer` maps to canonical contracts; Drift Detection Service compares against prior catalogue state; `CatalogService` upserts technical fields; optional lineage ingestion and AI enrichment follow. Structured scan events (`queued`, `running`, `completed`, `failed`) and audit records give teams observable progress rather than opaque batch jobs.
Scans are dispatched asynchronously by default. The API returns HTTP 202 when a scan is queued (`POST /v1/connectors/{id}/scan`, `POST /v1/scans`), and Celery workers execute `metaops.run_metadata_scan` against Redis. A synchronous development mode (`SCAN_EXECUTION_MODE=sync`) exists for local troubleshooting but is not the intended production pattern.
A universal metadata query across all engines fails in practice. MetaOps Studio therefore implements platform-specific adapters under `apps/api/app/connectors/`, each with native `queries.py`, a connector class implementing `extract_raw_metadata()`, and a dialect normalizer.
The `ConnectorFactory` registry includes seven implemented database platforms, all registered in `IMPLEMENTED_TYPES`:
| Platform | Native catalogues (evidence) | Status |
| PostgreSQL | Present | Implemented and operational |
| SQL Server | Present | Implemented with limitations |
| MySQL | Present | Implemented with limitations |
| Oracle | Present | Implemented with limitation |
| Snowflake | Present | Implemented with limitations |
| BigQuery | Present | Implemented with limitations |
| Databricks | Present | Implemented with limitations |
Each non-PostgreSQL adapter follows the same contract as PostgreSQL, connection test, schema discovery, column extraction, and lineage candidate harvesting from view and procedure definitions, but operational maturity varies by environment. Connector credentials, network access, and platform-specific configuration (for example Snowflake account and warehouse, BigQuery project credentials, Databricks HTTP path) remain deployment concerns. The repository’s automated test surface is limited, end-to-end scan validation is primarily demonstrated through the seeded PostgreSQL path and manual integration.
The architectural benefit is separation of concerns: adapters understand source platforms; normalizers translate; downstream drift, catalogue, and governance services consume canonical metadata without dialect-specific logic.
Raw metadata before interpretation
Before normalization, MetaOps Studio persists `Raw Metadata Snapshot` rows in the metadata_snapshots table. Each snapshot stores a JSONB payload exactly as extracted, together with connector type, source-system identity, extraction timestamp, and a deterministic `payload_hash` computed via `app/core/hashing.py`.
This design supports audit, replay, and diagnosis. Normalization reads from the stored snapshot rather than repeatedly querying live sources during downstream processing. If mapping logic changes, teams can reason about what was extracted independently of how it was interpreted. The scan detail API exposes snapshots through `GET /v1/scans/{id}/raw-snapshot`.
Canonical normalization without erasing source context
`MetadataNormalizer` routes raw payloads to dialect-specific normalizers—`PostgresMetadataNormalizer`, `SQLServerMetadataNormalizer`, and counterparts for MySQL, Oracle, Snowflake, BigQuery, and Databricks—producing `CanonicalAsset` and column structures mapped to shared type categories (`STRING`, `INTEGER`, `TIMESTAMP`, and others) through `type_mapping.py`.
Validation in `normalization/validation.py` logs warnings before catalogue upsert; invalid assets are not hard-blocked in the current beta behaviour. Drift detection compares normalized structures only, keeping comparison logic engine-agnostic. Source-platform nuance is preserved in raw snapshots and dialect normalizers rather than flattened prematurely at extraction time.
Background scan orchestration and observability
The scan lifecycle matches the orchestrator implementation: queue scan, worker pickup, connection test, native extraction, snapshot persistence, normalization, optional lineage ingest (`ENABLE_LINEAGE_ON_SCAN`), drift detection, catalogue upsert, optional AI enrichment dispatch, completion metrics, and audit.
The frontend provides scan list and detail views with live polling (`ScanLiveView`, `ScanTimeline`) rather than WebSocket streaming. The observability API aggregates scan metrics, drift counts, connector health indicators, and AI enrichment statistics. Celery tasks support soft time limits, late acknowledgement, and configurable retries via `scan_task_max_retries` and `scan_task_timeout_seconds` in application settings.
Detecting schema drift before catalogue upsert
Operation order matters. `DriftDetectionService.detect()` runs after normalization and before `CatalogService.upsert_from_snapshot()`, comparing current normalized assets against `CatalogService.load_previous_catalog_state()`. On a connector’s first completed scan, drift returns empty, there is no prior baseline.
Detected events include asset addition and removal, column addition and removal, data type changes, nullability changes, and view definition changes. Each `DriftEvent` carries severity informed by `DriftRulesEngine`, previous and current values, and a review status from `pending_review`, `accepted`, `dismissed`, or `remediated`. Missing objects are deactivated (`is_active = false`) rather than deleted, preserving history.
This is structural drift detection. The repository does not implement semantic rename inference: a column renamed in the source appears as a removal plus an addition, not as a linked rename event.
The governed business layer
MetaOps Studio separates scanned technical metadata from steward-managed business meaning. Technical fields, schema names, data types, view definitions, metadata hashes, are updated during scans via `CatalogService`. Business fields, `business_name`, `business_description`, `domain`, `subject_area`, `owner_user_id`, `steward_user_id`, `criticality`, `sensitivity_level`, and documentation status are updated through `DataDictionaryService` and related APIs.
`Metadata Completeness Service` calculates zero-to-one-hundred completeness scores from weighted field presence. `Documentation Gap Service` surfaces queues for assets without owners, missing business descriptions, sensitive assets without stewards, and incomplete mission-critical documentation. Tags and classifications are managed through governed taxonomy services with junction tables rather than ad hoc strings alone.
Scans update technical fields only; catalogue upsert may still refresh source-derived `description`, `owner`, `tags`, and `classifications` from normalized extraction. Stewards should treat business-layer fields as authoritative for governance meaning. AI suggestions remain in separate `ai.*` tables until explicitly approved through `AISuggestionReviewService`.
Lineage, external ingestion and semantic models
Lineage capabilities extend MetaOps beyond database cataloguing into the analytical ecosystem, with implementation status varying by source.
SQL-derived lineage (implemented). During scans, connectors collect lineage candidates from view and procedure definitions. `LineageService` uses sqlglot for dialect-aware parsing and persists nodes and edges. The interactive lineage graph is available at `/lineage` via `LineageGraphView`, backed by `GET /v1/lineage/graph` and neighbourhood expansion at `GET /v1/lineage/graph/neighborhood/{node_id}`.
Dbt manifest ingestion (implemented). `POST /v1/ingestion/dbt/manifest` ingests manifest JSON, creates lineage nodes and edges from `parent_map`, and optionally links to catalogue assets by fully qualified name.
Power BI and Tableau ingestion (implemented with limitations). `POST /v1/ingestion/powerbi/scan` and `POST /v1/ingestion/tableau/scan` accept structured payloads through scanner services, register semantic models, and create lineage relationships. These are ingestion APIs for supplied metadata rather than live platform connectors that authenticate to vendor APIs directly.
Semantic models (implemented). The `lineage.semantic_models` registry and `/semantic-models` UI list BI semantic layers with optional auto-linking to catalogue assets by name.
MetaOps Studio does not claim exhaustive end-to-end lineage for every transformation platform. Coverage depends on what scans, manifests, and ingestion payloads supply.
AI metadata intelligence, but with control
MetaOps Studio’s AI layer operates on structured metadata context. It is not positioned as a generic chatbot. Post-scan, `AIEnrichmentOrchestrator` can queue Celery task `metaops.run_ai_enrichment` when `ENABLE_AI_ENRICHMENT_ON_SCAN` is enabled.
Verified capabilities include AI-generated asset and column summaries, classification suggestions (personally identifiable information, financial, regulatory heuristics), governance insights (undocumented critical assets, missing owners, drift hotspots), plain-language lineage explanations, semantic search over hash-based embeddings, metadata-grounded retrieval-augmented generation (RAG) at `POST /v1/ai/rag/query`, glossary term suggestions, and approve/reject workflows before applying outputs to the catalogue.
The default provider is deterministic `metaops-heuristic-v1` via `HeuristicLLMClient`. Optional OpenAI integration activates when `OPENAI_API_KEY` is configured. Semantic embeddings use `metaops-hash-v1` hash vectors stored as JSON in `ai.ai_semantic_embeddings`; pgvector remains a documented roadmap item, not current production storage.
One operational metadata lifecycle
The platform’s value emerges from integrating capabilities into a continuous lifecycle:
Connect → Test → Scan → Preserve raw evidence → Normalize → Detect drift → Update catalogue
→ Add business meaning → Assign accountability → Classify → Measure completeness → Map lineage
→ Generate governed AI suggestions → Review → Monitor continuously
Individual screens, catalogue browse, drift review, AI insights, documentation gaps—support this lifecycle but do not define it. The lifecycle is the operating model.
Enterprise value
Without inventing customer metrics, MetaOps Studio is designed to deliver organizational value through:
- Reduced repeated manual metadata collection across platforms
- Earlier visibility into structural change via drift events and review queues
- Traceable extraction records with timestamps and payload hashes
- Clearer ownership and stewardship through assignments and gap detection
- Measurable documentation coverage via completeness scoring
- Easier impact analysis through lineage graph exploration
- Stronger auditability via structured scan events and audit logs
- A shared technical and business metadata foundation for data teams
- Safer AI assistance because outputs are grounded in catalogue context and require review
- Extensibility through new connector adapters without rewriting catalogue logic
These outcomes depend on deployment discipline: running workers, securing credentials, and maintaining connector connectivity.
Current boundaries and product maturity
An honest maturity assessment is essential. Several documentation files predate recent implementation and should not override code evidence.
Connectors. All seven database adapters are implemented in code; PostgreSQL has the strongest demonstrated path through seed data and local development workflows. Other connectors require environment-specific validation.
Security. Credentials are stored as plaintext `secret_value` in development (`ConnectorCredentialReference`). `CredentialVault` is scaffolding only. Role-based access control (RBAC) decorators exist but `require_permission` is a pass-through. Authentication uses an `X-Actor` header placeholder, not JSON Web Token (JWT) enforcement. Multi-tenancy is not implemented.
Drift and lineage. Drift is structural, not semantic. Lineage from SQL parsing depends on available definitions; external scanners require supplied payloads. Parser caches are in-memory.
AI and search. Hash-vector semantic search suits demonstration; production semantic search would require embedding infrastructure. RAG retrieves structured context and optionally calls an LLM, it is steward-oriented intelligence, not unconstrained chat.
Observability. Prometheus, OpenTelemetry, dead-letter queues, and worker heartbeat persistence remain roadmap items.
Documentation conflicts. `README.md` still lists several connectors as placeholders; `ARCHITECTURE_GAPS.md` marks Power BI, Tableau, lineage visualization, and glossary as gaps despite subsequent Phase 5–6.5 implementations in code and `docs/ROADMAP.md`. The code and migrations are the authoritative source for this case study.
Conclusion
MetaOps Studio reframes metadata management from a periodic documentation exercise into an observable, governed, and increasingly intelligent operational process. By extracting native evidence, normalizing it once at a shared boundary, detecting structural change before catalogue updates, separating business governance from technical scans, and applying AI only through reviewable workflows, the platform gives data leaders a practical control environment for the modern data estate.
How A18 Helps Organizations Operationalize Metadata
Then change its content to focus on:
- metadata-estate and source-system assessment;
- connector and ingestion planning;
- catalogue and governance-model design;
- ownership, stewardship and classification;
- scan-orchestration deployment;
- lineage and impact-analysis implementation;
- schema-drift operating procedures;
- AI-enrichment controls and approval workflows;
- security, credentials, role-based access and production hardening.
That gives the full article a clean progression:
A18 can lead the full modernization, provide an independent assessment and roadmap, establish the governance and architecture, or work alongside the client’s internal teams and implementation partners.
Partner With A18 Analytics
Website: www.a18analytics.com
X: @A18AnalyticsInc
Inquiries: info@a18analytics.com