Scan Numbers¶
The scan field in QPX identifies a specific MS/MS spectrum within a raw or converted spectra file. Because different mass spectrometer vendors use different internal identifiers, QPX defines a consistent encoding convention based on the HUPO-PSI USI (Universal Spectrum Identifier) standard.
A scan value is always stored as an array of int32 values. For simple instruments with a single scan number, the array contains one element. For instruments with multi-component identifiers, each component is stored as a separate element in the array.
Instrument-specific formats¶
Each instrument vendor uses a different native identifier structure. QPX encodes all numeric components in their native order, including repeated values. The default Thermo controller components are the exception described below. The examples are not an exhaustive list of native ID formats.
Thermo¶
Thermo instruments use controllerType, controllerNumber, and scan components. Since controllerType=0 and controllerNumber=1 are the defaults for mass spectra, only the scan number is stored.
| Native ID | QPX scan value |
|---|---|
controllerType=0 controllerNumber=1 scan=43920 |
[43920] |
Note
In rare cases where controllerType is not 0 or controllerNumber is not 1 (e.g., referencing a PDA spectrum), the full nativeId form must be used: controllerType=5 controllerNumber=1 scan=7 becomes [5, 1, 7].
Bruker¶
Bruker TIMS instruments use a two-component identifier combining frame and scan.
| Native ID | QPX scan value |
|---|---|
frame=120 scan=475 |
[120, 475] |
frame=120 scan=475 precursor=3 |
[120, 475, 3] |
frame=120 windowGroup=2 scan=475 |
[120, 2, 475] |
merged=0 frame=120 scanStart=4 scanEnd=8 |
[0, 120, 4, 8] |
Waters¶
Waters instruments use a three-component identifier: function, process, and scan.
| Native ID | QPX scan value |
|---|---|
function=10 process=1 scan=345 |
[10, 1, 345] |
AB Sciex¶
AB Sciex instruments use a four-component identifier: sample, period, cycle, and experiment.
| Native ID | QPX scan value |
|---|---|
sample=1 period=1 cycle=2740 experiment=10 |
[1, 1, 2740, 10] |
The scan_format metadata field¶
Because the scan array alone does not tell a reader how to interpret the integer components, a producer can declare scan_format in the Parquet file footer when it knows the format used throughout the file. A one-element array alone cannot distinguish scan from index.
scan_format value |
Meaning | Example scan value |
|---|---|---|
scan |
Simple Thermo-style scan number (1 element) | [43920] |
index |
Zero-based spectrum index in the file (1 element) | [0], [1], [2] |
nativeId |
Multi-component native identifier (Bruker, Waters, AB Sciex) | [120, 475] |
Array length varies by scan_format
scanandindexformats always produce a single-element array.nativeIdformat produces arrays of 2 to 4 elements, depending on the instrument vendor.- The
scan_formatmetadata field tells the reader how many components to expect and how to interpret them.
The OpenMS consensusXML converter declares this metadata for PSM output when
all encountered spectrum references have the same recognized format. It uses
the original native ID keys, including explicit scan= and index=, rather
than guessing from array length. Mixed or unknown formats remain undeclared.
Both the in-memory and streaming conversion paths follow this rule.
On PyArrow 14–16, a streaming writer that has already flushed batches cannot
update its footer declaration: conversion warns and leaves scan_format
unset. PyArrow 17 or newer supports the late update. A declaration known before
the first batch is written works on all supported versions.
The mzIdentML converter declares scan or index from the spectrum references
in the same way. With --mgf-path, index=N references become the scan numbers
the MGF records (SCANS= or a scan= title), and the PSM view declares scan;
without it they stay file positions and the view declares index. qpxc joins of
PSMs to spectra follow the declaration, and the mz view declares its own format.
A value that is not a scan number must be declared: a file without scan_format
is read as holding scan numbers.
For a declared PSM format, validation checks non-empty scan arrays against the
lengths above. A mismatch is a warning by default and an error under strict
validation, including qpxc validate. Legacy files without scan_format remain
valid; an unknown historical value produces a warning and skips this check.
This check currently covers standalone local files, not unions of multiple
Parquet shards or partitioned datasets with potentially different declarations.
Feature arrays can contain components from several supporting spectra. This PSM cardinality check does not apply to Feature output, and the consensusXML converter does not declare one native-ID format for those combined arrays.
When to use nativeId vs scan¶
flowchart TD
A["Is the scan a single integer<br/>from a Thermo instrument?"] -->|Yes| B["Use scan_format: scan<br/>Value: [43920]"]
A -->|No| C["Is it a zero-based index?"]
C -->|Yes| D["Use scan_format: index<br/>Value: [0]"]
C -->|No| E["Is it a multi-component<br/>vendor ID?"]
E -->|Yes| F["Use scan_format: nativeId<br/>Value: [120, 475]"]
style B fill:#e8f5e9
style D fill:#e8f5e9
style F fill:#e8f5e9
DIA¶
In a DIA run no scan belongs to one precursor. scan on a DIA feature is the MS2 scan at the precursor's apex: the report's MS2.Scan (Ms2.Scan in a 2.x report written with --export-quant) for DIA-NN, else the MS2 scan nearest to rt when --mzml-info-folder gives the scan list. Spectronaut reports have no scan, so it is empty.
Where scan is used¶
The scan field appears in the following QPX views:
| View | Field name | Notes |
|---|---|---|
PSM (psm_file) |
scan |
Scan of the identified MS/MS spectrum |
Feature (feature_file) |
scan |
Scan components reported by the feature producer |
MZ (mz_file) |
id, scan |
id is the string spectrum identifier retaining the native ID; scan holds the same components as PSM/feature scan for joins |
Tip
The OpenMS consensusXML converter can combine components from multiple supporting spectra in feature.scan. Use the linked PSM rows to retrieve each complete spectrum identifier; do not treat the combined feature array as one native ID. id_run_file_name records the direct identification's run when that origin is known.
File metadata example¶
The scan_format is stored in the Parquet file metadata alongside other QPX metadata fields.
import pyarrow.parquet as pq
file_metadata = {
"qpx_version": "1.3",
"scan_format": "scan",
"file_type": "psm_file",
"software_provider": "qpx",
"creator": "quantms",
"project_accession": "PXD012345",
"creation_date": "2024-06-15",
}
# Attach metadata to the Arrow schema before writing the Parquet file.
encoded_metadata = {key.encode(): value.encode() for key, value in file_metadata.items()}
table = table.replace_schema_metadata({**(table.schema.metadata or {}), **encoded_metadata})
pq.write_table(table, "output.psm.parquet")
Further reading¶
- Scores & CV Terms -- additional metadata attached to PSMs and features
- QPX Format Overview -- full list of views and concepts