Skip to content

SDRF Conversion

When SDRF is ingested into QPX, bracket-style column names (e.g. characteristics[organism]) are mapped to clean snake_case field names (e.g. organism) and the data is split into separate Parquet files -- one for biological samples and one for data acquisition runs. This page documents the column mapping registry, the multi-valued metadata convention, and the end-to-end conversion process.

SDRF Column Name Mapping

When SDRF is converted to QPX, bracket-style column names are mapped to snake_case field names according to the following registry. This mapping is deterministic and applied automatically during ingestion.

Sample Columns (→ sample.parquet)

Sample columns describe biological properties of the specimen. These are deduplicated and stored in sample.parquet.

SDRF Column QPX Field Name Target File
source name sample_accession sample.parquet
characteristics[organism] organism sample.parquet
characteristics[organism part] organism_part sample.parquet
characteristics[disease] disease sample.parquet
characteristics[cell line] cell_line sample.parquet
characteristics[cell type] cell_type sample.parquet
characteristics[sex] sex sample.parquet
characteristics[age] age sample.parquet
characteristics[developmental stage] developmental_stage sample.parquet
characteristics[ancestry category] ancestry sample.parquet
characteristics[individual] individual sample.parquet

Run Columns (→ run.parquet)

Run columns describe technical and instrument-related properties of the data acquisition. These are stored in run.parquet.

SDRF Column QPX Field Name Target File
assay name run_accession run.parquet
comment[data file] run_file_name run.parquet
comment[label] label (inside samples list) run.parquet
comment[fraction identifier] fraction run.parquet
characteristics[biological replicate] biological_replicate (inside samples list) run.parquet
comment[technical replicate] technical_replicate (inside samples list) run.parquet
comment[instrument] instrument (as ONTOLOGY_TERM) run.parquet
comment[cleavage agent details] enzymes (as list of ONTOLOGY_TERM) run.parquet
comment[proteomics data acquisition method] acquisition_method (plain string; also resolved into additional_terms as ONTOLOGY_TERM) run.parquet
comment[dissociation method] dissociation_method (as ONTOLOGY_TERM) run.parquet
comment[modification parameters] modification_parameters (as list of MODIFICATION) run.parquet

Factor Columns (planned factor.parquet)

Factor columns capture experimental design variables from factor value[X] columns in the SDRF.

Planned view -- not yet serialized

factor.parquet is a planned view. QPX 1.1 does not ship a factor.yaml schema or write a factor.parquet file. Experimental-design / factor information is currently carried in the differential-expression view (factor_names in the DE AnnData uns), and factor values remain available in the source .sdrf.tsv preserved alongside the Parquet files. The mapping below documents the intended field naming for a future release.

SDRF Column QPX Field Name Status
factor value[disease] factor_disease planned
factor value[organism part] factor_organism_part planned

Note

Factor columns are open-ended. Any factor value[X] column in the SDRF is intended to map to factor_X (with spaces replaced by underscores). Replicate identifiers (biological_replicate, technical_replicate) are stored in the samples struct of run.parquet, so replicate structure is already self-contained there.

Multi-valued Metadata

In rare cases a single SDRF row may describe a sample derived from multiple organisms (e.g. a spike-in experiment) or multiple cell lines (pooled samples). QPX handles this natively using list columns.

Convention

All biological property columns in sample.parquet use list[string] types. During SDRF ingestion, pipe-delimited values in the source TSV (e.g. Homo sapiens|Saccharomyces cerevisiae) are split and stored as proper list elements:

# Single-valued (most samples)
organism = ["Homo sapiens"]

# Multi-valued (spike-in / pooled samples)
organism = ["Homo sapiens", "Saccharomyces cerevisiae"]

This avoids delimiter conventions in the stored data and enables native list operations in query engines.

SQL Queries

When querying sample.parquet with DuckDB, use UNNEST to explode list columns or list_contains to filter:

-- All unique organisms in the project
SELECT DISTINCT UNNEST(organism) AS organism
FROM 'PXD014414.sample.parquet';

-- Samples with a specific disease
SELECT sample_accession, disease
FROM 'PXD014414.sample.parquet'
WHERE list_contains(disease, 'breast cancer');

SDRF Conversion Process

At ingestion time, the SDRF TSV is split into sample.parquet and run.parquet:

  1. Load the SDRF TSV file
  2. Rename columns using the field registry above
  3. Extract sample-level columns -- deduplicate rows by sample_accession (= source name) to produce sample.parquet
  4. Extract run-level columns -- one row per run_accession (= assay name), with the samples list linking to sample accessions, labels, and replicate numbers, to produce run.parquet
  5. Parse ontology terms -- instrument, enzyme, modification parameter cells are parsed into ONTOLOGY_TERM structs
  6. Preserve provenance -- the original .sdrf.tsv file is kept alongside the Parquet files

Project Layout

PXD014414/
  PXD014414.sdrf.tsv             # Original SDRF (provenance)
  PXD014414.sample.parquet       # Biological samples (deduplicated)
  PXD014414.run.parquet          # Data acquisition runs with ontology terms
  PXD014414.dataset.parquet      # Project-level metadata
  PXD014414.feature.parquet
  PXD014414.psm.parquet
  ...