Ask a data scientist what clin_sig, gda_score or pt_dob means, and you'll often get an answer. Ask whether it's safe to use, and you'll get a pause.
Life sciences organizations sit on enormous amounts of data: clinical records, variant calls, protein annotations, drug-target references, trial registries. The problem is rarely having the data. It's knowing what each column actually contains, whether it matches an industry standard, and whether it carries patient identifiers that should never leave a controlled environment.
That knowledge usually lives in data dictionaries that are out of date, in the heads of a few data stewards, or nowhere at all. So every new analysis starts with the same detective work, and every audit starts with the same scramble.
You can't govern, share or trust data you can't describe.
At Clovertex, we built a Metadata Catalog Agent to do that describing automatically, and, just as importantly, to know when it isn't sure.
The problemWhy metadata falls behind
- Too many columns, too few stewards. A single program can span hundreds of columns across clinical, genomics and pharma sources. Labeling them by hand never keeps up.
- Every source speaks its own dialect. The same idea appears as
clin_sig,clinical_significanceandCLNSIGin different datasets. - Standards mapping is specialist work. Linking columns to CDISC or OMOP takes expertise that's scarce and expensive.
- Privacy risk hides in plain sight. A column of birth dates or record identifiers looks harmless until it ends up somewhere it shouldn't.
- Labels go stale. When an ETL job changes, yesterday's description may quietly stop being true.
The approachFrom raw columns to a governed catalog
The agent works across a wide range of clinical, genomics and pharma reference data, including clinical EHR data (Synthea), genomics references (GnomAD, Ensembl), protein annotation (UniProt), clinical research (ClinicalTrials.gov), drug discovery and pharmacogenomics (ChEMBL, PharmGKB) and disease genomics (DisGeNET). Each column passes through four stages:
- Pre-flight. The pipeline scans Amazon S3 for data files, extracts column metadata, infers data types, classifies each column's domain and fetches any prior enrichment history.
- Agent enrichment. An AI agent powered by Amazon Bedrock studies each column using several sources of evidence: the source schema and ETL logic, pipeline run logs, existing canonical labels, and knowledge-base lookups against standards such as CDISC and OMOP.
- Post-flight. Confidence thresholds are applied, results are written to Amazon DynamoDB, summary reports go to S3 and metrics are published to CloudWatch.
- Human review. Results appear in a dashboard where data stewards approve, or edit and approve, anything the agent wasn't sure about.
The fast path
Each column's derivation logic in the source ETL script is fingerprinted. If the fingerprint hasn't changed since a label was validated, the pipeline simply reuses that validated label and skips the AI entirely. That keeps runs fast and cheap, and means a label is only re-examined when the data behind it actually changes.
Runs can be triggered on demand or on a schedule through AWS Lambda and EventBridge, with AWS Glue handling data movement.
What gets addedMore than a label
For every column, the agent produces a complete, catalog-ready record:
Enriched label & description
A human-readable name and plain-language description, so clin_sig becomes "Clinical Significance".
Canonical label
A normalized identifier like CLINICAL_SIGNIFICANCE that stays consistent across every dataset.
Category & sub-category
For example Genomics → Variant Info, or Pharmacogenomics → Gene-Disease Association.
Standards mapping
A reference to the matching CDISC (CDISC.DOMAIN.VARIABLE) or OMOP (OMOP.table.field) variable.
PII flag
Columns that may contain HIPAA identifiers are flagged before anyone builds on them.
Source & lineage
Where the label came from, when it was last updated and its confidence history over time.
The key ideaAn agent that knows what it doesn't know
Most AI labeling tools give you an answer. This one also tells you how much to trust it. Every enrichment carries a confidence score from 0.00 to 1.00, computed from multiple sources of evidence, and that score decides what happens next.
High · 0.70 and above
Auto-validatedReliable labels go straight into the catalog, without a review step, unless someone chooses to override them.
Medium · 0.60 to 0.70
Spot-checkProbably right, but queued for a reviewer to confirm or edit.
Low · below 0.60
Manual reviewHigh risk. A person decides the right label, guided by the agent's reasoning and evidence.
Crucially, the agent explains itself. A low-confidence label comes with reasoning such as: "No existing canonical labels found. Knowledge-base relevance below 75%. Label inferred from column name and domain context." A reviewer can see at a glance why the agent hesitated, rather than having to second-guess a black box.
Automation where it's confident. People where it isn't.
Human in the loopReview that respects people's time
Because high-confidence labels are validated automatically, the review queue contains only the columns that genuinely need a human: the medium and low tiers. Reviewers can filter that queue by data source, category or confidence tier, start with the riskiest items first, and approve a label as suggested or edit it before approving.
Every enriched column ends up in one of three clear states:
- High Confidence: auto-validated by the agent
- Pending: waiting for a person to review it
- Overridden: approved by a reviewer, possibly with edits; the final, human-validated state
The outcomeA business catalog people actually use
The result is a searchable business catalog of every column, with its label, canonical name, category, data source, confidence and status. Anyone can inspect a column to see its description, standard mapping, confidence reasoning and lineage, or export a filtered view for downstream use. An enrichment summary shows overall confidence, quality by category and data source, and how much of the catalog is already mapped to industry standards: the gaps worth closing next.
Faster data understanding
Analysts find and understand data in minutes instead of chasing down data owners.
Privacy risk surfaced early
PII-flagged columns are visible before data is shared or joined.
Standards alignment
CDISC and OMOP mappings make submission work and cross-study analysis easier.
Audit-ready by default
Every label has a confidence score, a reason, a reviewer and a history.
What's nextMetadata as the foundation for AI
Well-described data is the foundation for everything else on the life sciences AI roadmap. The analytics agents in our precision medicine work depend on exactly this kind of semantic layer: plain-language descriptions of every table and field are what let an AI agent query data accurately instead of guessing.
The Metadata Catalog Agent makes that layer practical to build and keep current: AI does the labeling at scale, confidence scores route the hard cases to people, and every decision is recorded.
If your teams spend more time figuring out what data means than using it, or you're preparing for an audit and dreading the data dictionary, we'd be glad to show you the agent on your own datasets.
