Command line¶

manifold-genetics <command> --help

Every command prints its own options and a worked example.

Starting from nothing¶

manifold-genetics acquire synthetic  # simulate a cohort and a config beside it
manifold-genetics acquire hgdp       # fetch and prepare the real HGDP+1KGP cohort
manifold-genetics acquire custom     # a config for genotypes you already have
manifold-genetics acquire aou        # fetch All of Us (workbench only)

pip install downloads no data and no example config, so acquire is what gives you something to run.

synthetic needs no network and takes under a minute. It places 2,000 samples along a branching tree — eight branches, some separated by unsampled gaps — then draws 1,000 variants whose allele frequencies drift along it. No single variant carries the tree; it is recoverable only from all of them together, which is the situation the pipeline exists for. It writes the PLINK triples, labels, colormap and config.yaml, plus dla_tree_ground_truth.png — the tree itself, to check the embedding against.

hgdp downloads about 183 MB: a variant-processed cohort of 4,151 samples over 172,152 SNPs. From it, two plink2 --keep calls select the 4,094 that pass QC and the 3,400 of those that are also unrelated, using the flags in the archive's metadata.csv. Sample selection only — no MAF, missingness, indel or LD filtering, because the archive arrives with that already done. It needs internet, so on a cluster run it on a login node.

Labels carry both Population and Genetic_region_merged, and the colormap ships the published colours for each — 78 population colours plus the 7 region ones — so a figure from acquire hgdp is comparable with the published ones. The public archive's metadata.csv also has coordinates, so geographic.csv is written too (Americas and ACB/ASW/CEU excluded, as they don't preserve geography), and the config carries the published admixture settings — k_min/k_max and admix_group_column — behind skip: admixture: true until you install the admixture extra and are on a GPU node.

You can also point acquire at archived data:

manifold-genetics acquire hgdp --archive hgdp_1kgp_full.tar.gz

--no-download means never fetch: it still unpacks an archive already present.

custom takes --fit-plink and labels, plus optionally --project-plink, --preset and --n-pcs. Labels are either one --labels describing both sets, or --fit-labels and --project-labels when they differ, as in every UK Biobank config. It writes a colour for every label value -- 22 of them for UK Biobank's self_described_ancestry -- and refuses if fewer than half the genotyped samples appear in the label file, which is the failure that otherwise shows up as a figure colouring some of its points.

aou fetches the All of Us array genotypes of the release attached to the workspace (from $CDR_STORAGE_PATH/microarray/plink, or --bucket-root) and labels every sample by self-reported race and ethnicity from the CDR, which needs the aou extra (pip install 'manifold-genetics[aou]'). It only works inside the Researcher Workbench: it checks the environment first — GOOGLE_PROJECT, WORKSPACE_CDR, gsutil, bq — and names everything missing at once. What it writes is the cohort alone; the manuscript's figures projected it onto HGDP+1KGP, which is acquire hgdp --archive gs://…/1KGPHGDP.tar.gz followed by preprocess ref/config.yaml aou/config.yaml --preset harmonise --fit-has-chr-prefix (see Preprocessing). It is a port of the shell script that produced the published figures, and has run inside the workbench on the v9 release (via the aou_prepare workflow).

All four take --out DIR, and none overwrites an existing config.yaml without --force.

Running a whole pipeline¶

manifold-genetics run config.yaml [--dry-run] [--output DIR] [--skip-...] [-v]

The entry point for real work. Reads a config file, applies a few command-line overrides, runs the pipeline.

--dry-run prints the resolved call and exits. Use it before anything long.
--output DIR overrides output_dir, so one config can drive a verification run without being edited.
--memory-gb GB overrides the PCA memory budget, because how much memory you have is a property of the machine rather than of the analysis. Above the budget the fit streams: bounded memory, about nineteen times the wall clock. Lower it if a run is killed; raise it on a large node to keep a big cohort in memory.
--skip-pca, --skip-admixture, --skip-metrics add to the config's own skip settings.
-v enables debug logging for this package only. It does not turn on debug for numba and every other library in the process, which is what it used to do.

manifold-genetics pipeline --fit-plink data/fit --project-plink data/project \
    --labels labels.csv --colormap colors.json --output results/ \
    --n-pcs 50 --k-min 2 --k-max 10 --embedding phate --knn 100

pipeline is the same thing with every setting on the command line. It predates run and remains useful for one-off experiments; for anything you will want to repeat or review, a config file is the better record.

Individual stages¶

Each stage reads and writes the standard CSV — sample_id then dim_1 … dim_n — so they compose, and you can start from the middle if you already have components.

command	in	out
`pca`	PLINK prefixes	PCA CSVs and a projectable model
`admixture`	PLINK prefixes	Q matrices per K
`embed`	PCA CSV	embedding CSV (2-D by default, `--n-components 3` for 3-D)
`plot`	embedding CSV + labels + colormap	scatter per label column
`plot-3d`	3-D embedding CSV + labels + colormap	rotatable HTML and MP4 per label column
`plot-pca`	PCA CSV	PC-pair grid
`plot-projection`	fit and project embeddings	both cohorts on one figure
`plot-admixture`	Q matrices	stacked bar plots
`plot-admixture-embedding`	embedding + Q	embedding coloured by component
`plot-knn-composition`	embedding + labels	neighbour label composition
`metrics-geographic`	embedding + coordinates	JSON
`metrics-admixture`	embedding + Q	JSON

# PCA, choosing the backend explicitly
manifold-genetics pca --fit-plink data/fit --project-plink data/project \
    --fit-output out/fit_pca.csv --project-output out/project_pca.csv \
    --n-pcs 50 --pca-backend python

# An embedding of components you already have
manifold-genetics embed --method phate --fit-input out/project_pca.csv \
    --project-output out/phate_2d.csv --knn 100 --t 3

# The same in three dimensions, then a rotatable figure of it
manifold-genetics embed --method phate --fit-input out/project_pca.csv \
    --project-output out/phate_3d.csv --knn 100 --t 3 --n-components 3

manifold-genetics plot-3d --input out/phate_3d.csv \
    --labels data/labels.csv --colormap data/colormap.json \
    --output out/figures/

--n-components works on embed and pipeline, for every method. It defaults to 2, so existing commands are unchanged.

At biobank scale pass --n-landmark 10000 --random-landmarking. Without landmarks PHATE needs dense n × n memory, and a run that cannot fit is refused before it starts.

Three-dimensional embeddings¶

A 3-D scatter is only useful if you can turn it, so plot-3d writes a standalone interactive HTML file per label column rather than a PNG — open it in a browser and drag to rotate; hovering a point names its sample and label.

It needs the optional interactive extra (plotly, and ffmpeg for the video):

uv sync --extra interactive     # or: pip install 'manifold-genetics[interactive]'

Every point is written into the file itself, so --max-points caps how many are included (default 100,000) and reports the subsample it took. Pass --max-points 0 to keep all of them, bearing in mind that a biobank-scale cohort then produces a document a browser opens slowly.

The figure opens looking straight down dim 3 — the plane of the 2-D figure — with an orthographic camera, so it starts as the picture you already know and rotation reveals what the third axis adds. By default (--aspect match) dims 1 and 2 are stretched to equal length, exactly as the 2-D figures are, and dim 3 is left at its true length relative to dim 1; --aspect true shows every axis at true scale. Depth is never inflated: on a curved manifold the third axis is usually the shortest, and stretching it to fill a cube exaggerates the curvature two- to three-fold.

When a pipeline run uses n_components: 3, the embedding-visualisation step writes these HTML figures alongside the PNGs automatically (<method>_3d_by_<column>.html), provided plotly is installed; otherwise it logs a warning naming the extra.

Beside each HTML, plot-3d writes a video of the embedding making one full turn (.mp4; --video-format gif for a GIF), for slides and supplementary material. --fps (24) and --seconds (12) set its length, --elev (20°) the camera's tilt, and --no-video skips it. Points stack as in the other figures: the colormap's first entries on top, Unknown underneath.

Hover labels name the sample by default. --no-hover-ids leaves the identifiers out of the document altogether and shows only the label, which is what you want before a figure of a controlled-access cohort leaves the environment holding it.

Admixture colouring works the same way. plot-admixture-embedding always writes its PNG grid; given a 3-D embedding it also writes <output>_3d.html, a single rotatable figure in which a dropdown selects the K and component that colour the points, so you keep your viewing angle while switching between them. It takes --component-colormap and --no-hover-ids like its 2-D counterpart and plot-3d, and --subsample caps how many points go into the file. A pipeline run with n_components: 3 writes it as figures/admixture/project_admixture_colored_embedding_3d.html.

The HTML files are self-contained pages that load plotly.js from a CDN, so they can be embedded in a website as they are:

<iframe src="phate_3d_by_population.html" width="100%" height="700" style="border:0"></iframe>

For a page, export with a smaller --max-points (a few tens of thousands) and with --no-hover-ids.

--pca-backend takes python (in process, the default) or flashpca (the external binary). They agree to 1.5e-7 and write the same artefacts, so a model fitted by either is readable by the other.

preprocess¶

manifold-genetics preprocess cohort/config.yaml --out filtered/
manifold-genetics preprocess ref/config.yaml biobank/config.yaml --out proj/ --preset harmonise

Optional. Filters the SNPs of one cohort, or intersects two, into a new cohort directory in the same layout, so the result can go to run, subsample, or another preprocess. With one config, the cohort's own fit and project sets are the two sides; with two, the first supplies the fit set and the output is a projection. Samples are never removed here. It runs the shell script that produced the published figures, shipped inside the package, and needs bash, plink2 and plink v1.9.

preset	what runs
`--preset intersect-only`	indels, missingness, intersection — no external references
`--preset harmonise`	WRayner/TOPMed, GIAB, HLA, dedup, MAF on the reference side only (`--skip-project-maf`), LD pruning, `--cleanup`
(none)	the shell defaults: every step on, MAF on both sides; add `--skip-*` flags

--skip-wrayner, --skip-giab, --skip-hla, --skip-ld-prune, --skip-dedup, --skip-maf, --skip-geno, --skip-project-maf turn off one step each, on top of any preset. The UK Biobank flow is --skip-wrayner --skip-project-maf.
--maf, --geno, --ld-window, --ld-step, --ld-r2 set the thresholds (shell defaults 0.01, 0.05, 150 kb, 1, 0.05).
--fit-has-chr-prefix when the fit set's chromosomes are already chr1, not 1 — true of the workbench HGDP+1KGP panel acquire hgdp --archive writes.
--threads, --memory (MB, default 100000), --temp-dir, --cleanup for resources; --tools-dir for where the harmonise references live; --min-common-snps (50,000) is the floor below which the run aborts.

harmonise downloads its references at run time unless manifold-genetics setup --preprocessing has fetched them — and one of those downloads is currently broken upstream. See Preprocessing, which also has the worked UK Biobank and All of Us examples.

subsample¶

manifold-genetics subsample proj/config.yaml --out 10k/ \
    --group "race_ethnicity=White|European:10000" \
    --group "race_ethnicity=Black or African American:10000" --include-rest

Reads a cohort directory and writes one whose fit set is a chosen subset of its project set; the project set is linked, not copied (a biobank .bed can be tens of GB). The output uses the subsample preset.

Choose the fit samples by exactly one of:

--group COLUMN=PATTERN:COUNT — take COUNT samples whose COLUMN matches PATTERN (case-insensitive regex); repeatable, and a sample is taken at most once across groups. --include-rest also adds every sample matched by no group.
--fit-samples FILE — a FID IID list chosen elsewhere.
--geosketch N — take N samples via geometric sketching (Hie et al. 2019), with no labels involved. Sketching runs in PC space, not on genotypes: PCA is fitted on --sketch-pool samples drawn at random (default 100,000; the whole cohort if it is smaller), every sample is projected in, and the sketch is taken in those --n-pcs PCs (default 20). They are saved as OUT/sketch_pca.csv and reused if the command is rerun. --memory-gb sets the fit and projection memory budget (default 8). These PCs only choose the sample; run refits PCA on it, as for any subsample.
--geosketch N --pca CSV — sketch in PCA coordinates computed elsewhere (sample_id, dim_1, dim_2, ...), restricted to the samples in the project .fam first. --n-pcs limits how many of the CSV's columns are used (default: all).

Setup¶

manifold-genetics setup [--skip-plink1] [--preprocessing]

Pre-fetches plink2, plink and flashpca into the per-user cache (~/.cache/manifold-genetics/bin, or the checkout's bin/ when you are running from one). They are also fetched on first use, so this is optional — what it is for is fetching them before submitting a job, because compute nodes usually have no internet. Needed for data preparation, not for the pipeline itself; see Install.

--preprocessing also places the GIAB, WRayner and TOPMed references that preprocess --preset harmonise needs (about 4 GB, almost all of it TOPMed) in the cache's preprocessing/ subdirectory. The WRayner checker ships with the package; the other two are downloaded. Every reference is attempted even when one fails, and re-running fetches only what is still missing.

Exit codes and logging¶

Commands return 0 on success and non-zero on failure, with the reason on stderr rather than a traceback for the errors a user can actually act on — a missing config, an unknown key, absent input files.

Logging goes to stderr at INFO by default and DEBUG under -v, scoped to the manifold_genetics logger.