Command line¶

manifold-genetics <command> --help

Every command prints its own options and a worked example.

Starting from nothing¶

manifold-genetics acquire synthetic  # simulate a cohort and a config beside it
manifold-genetics acquire hgdp       # fetch and prepare the real HGDP+1KGP cohort
manifold-genetics acquire custom     # a config for genotypes you already have
manifold-genetics acquire aou        # fetch All of Us (workbench only)

pip install downloads no data and no example config, so acquire is what gives you something to run.

synthetic needs no network and takes under a minute. It places 2,000 samples along a branching tree — eight branches, some separated by unsampled gaps — then draws 1,000 variants whose allele frequencies drift along it. No single variant carries the tree; it is recoverable only from all of them together, which is the situation the pipeline exists for. It writes the PLINK triples, labels, colormap and config.yaml, plus dla_tree_ground_truth.png — the tree itself, to check the embedding against.

hgdp downloads about 183 MB: a variant-processed cohort of 4,151 samples over 172,152 SNPs. From it, two plink2 --keep calls select the 4,094 that pass QC and the 3,400 of those that are also unrelated, using the flags in the archive's metadata.csv. Sample selection only — no MAF, missingness, indel or LD filtering, because the archive arrives with that already done. It needs internet, so on a cluster run it on a login node.

Labels carry both Population and Genetic_region_merged, and the colormap ships the published colours for each — 78 population colours plus the 7 region ones — so a figure from acquire hgdp is comparable with the published ones. The public archive's metadata.csv also has coordinates, so geographic.csv is written too (Americas and ACB/ASW/CEU excluded, as they don't preserve geography), and the config carries the published admixture settings — k_min/k_max and admix_group_column — behind skip: admixture: true until you install the admixture extra and are on a GPU node.

You can also point acquire at archived data:

manifold-genetics acquire hgdp --archive hgdp_1kgp_full.tar.gz

--no-download means never fetch: it still unpacks an archive already present.

custom takes --fit-plink and labels, plus optionally --project-plink, --preset and --n-pcs. Labels are either one --labels describing both sets, or --fit-labels and --project-labels when they differ, as in every UK Biobank config. It writes a colour for every label value -- 22 of them for UK Biobank's self_described_ancestry -- and refuses if fewer than half the genotyped samples appear in the label file, which is the failure that otherwise shows up as a figure colouring some of its points.

aou fetches the All of Us V8 array genotypes from their bucket and labels every sample by self-reported race and ethnicity from the CDR, which needs the aou extra (pip install 'manifold-genetics[aou]'). It only works inside the Researcher Workbench: it checks the environment first — GOOGLE_PROJECT, WORKSPACE_CDR, gsutil, bq — and names everything missing at once. What it writes is the cohort alone; the manuscript's figures projected it onto HGDP+1KGP, which is acquire hgdp --archive gs://…/1KGPHGDP.tar.gz followed by preprocess ref/config.yaml aou/config.yaml --preset harmonise --fit-has-chr-prefix (see Preprocessing). It is a port of the shell script that produced the published figures and has not yet been run inside the workbench itself (issue #124).

All four take --out DIR, and none overwrites an existing config.yaml without --force.

Running a whole pipeline¶

manifold-genetics run config.yaml [--dry-run] [--output DIR] [--skip-...] [-v]

The entry point for real work. Reads a config file, applies a few command-line overrides, runs the pipeline.

--dry-run prints the resolved call and exits. Use it before anything long.
--output DIR overrides output_dir, so one config can drive a verification run without being edited.
--memory-gb GB overrides the PCA memory budget, because how much memory you have is a property of the machine rather than of the analysis. Above the budget the fit streams: bounded memory, about nineteen times the wall clock. Lower it if a run is killed; raise it on a large node to keep a big cohort in memory.
--skip-pca, --skip-admixture, --skip-metrics add to the config's own skip settings.
-v enables debug logging for this package only. It does not turn on debug for numba and every other library in the process, which is what it used to do.

manifold-genetics pipeline --fit-plink data/fit --project-plink data/project \
    --labels labels.csv --colormap colors.json --output results/ \
    --n-pcs 50 --k-min 2 --k-max 10 --embedding phate --knn 100

pipeline is the same thing with every setting on the command line. It predates run and remains useful for one-off experiments; for anything you will want to repeat or review, a config file is the better record.

Individual stages¶

Each stage reads and writes the standard CSV — sample_id then dim_1 … dim_n — so they compose, and you can start from the middle if you already have components.

command	in	out
`pca`	PLINK prefixes	PCA CSVs and a projectable model
`admixture`	PLINK prefixes	Q matrices per K
`embed`	PCA CSV	2-D embedding CSV
`plot`	embedding CSV + labels + colormap	scatter per label column
`plot-pca`	PCA CSV	PC-pair grid
`plot-projection`	fit and project embeddings	both cohorts on one figure
`plot-admixture`	Q matrices	stacked bar plots
`plot-admixture-embedding`	embedding + Q	embedding coloured by component
`plot-knn-composition`	embedding + labels	neighbour label composition
`metrics-geographic`	embedding + coordinates	JSON
`metrics-admixture`	embedding + Q	JSON

# PCA, choosing the backend explicitly
manifold-genetics pca --fit-plink data/fit --project-plink data/project \
    --fit-output out/fit_pca.csv --project-output out/project_pca.csv \
    --n-pcs 50 --pca-backend python

# An embedding of components you already have
manifold-genetics embed --method phate --fit-input out/project_pca.csv \
    --project-output out/phate_2d.csv --knn 100 --t 3

--pca-backend takes python (in process, the default) or flashpca (the external binary). They agree to 1.5e-7 and write the same artefacts, so a model fitted by either is readable by the other.

preprocess¶

manifold-genetics preprocess cohort/config.yaml --out filtered/
manifold-genetics preprocess ref/config.yaml biobank/config.yaml --out proj/ --preset harmonise

Optional. Filters the SNPs of one cohort, or intersects two, into a new cohort directory in the same layout, so the result can go to run, subsample, or another preprocess. With one config, the cohort's own fit and project sets are the two sides; with two, the first supplies the fit set and the output is a projection. Samples are never removed here. It runs the shell script that produced the published figures, shipped inside the package, and needs bash, plink2 and plink v1.9.

preset	what runs
`--preset intersect-only`	indels, missingness, intersection — no external references
`--preset harmonise`	WRayner/TOPMed, GIAB, HLA, dedup, MAF on the reference side only (`--skip-project-maf`), LD pruning, `--cleanup`
(none)	the shell defaults: every step on, MAF on both sides; add `--skip-*` flags

--skip-wrayner, --skip-giab, --skip-hla, --skip-ld-prune, --skip-dedup, --skip-maf, --skip-geno, --skip-project-maf turn off one step each, on top of any preset. The UK Biobank flow is --skip-wrayner --skip-project-maf.
--maf, --geno, --ld-window, --ld-step, --ld-r2 set the thresholds (shell defaults 0.01, 0.05, 150 kb, 1, 0.05).
--fit-has-chr-prefix when the fit set's chromosomes are already chr1, not 1 — true of the workbench HGDP+1KGP panel acquire hgdp --archive writes.
--threads, --memory (MB, default 100000), --temp-dir, --cleanup for resources; --tools-dir for where the harmonise references live; --min-common-snps (50,000) is the floor below which the run aborts.

harmonise downloads its references at run time unless manifold-genetics setup --preprocessing has fetched them — and one of those downloads is currently broken upstream. See Preprocessing, which also has the worked UK Biobank and All of Us examples.

subsample¶

manifold-genetics subsample proj/config.yaml --out 10k/ \
    --group "race_ethnicity=White|European:10000" \
    --group "race_ethnicity=Black or African American:10000" --include-rest

Reads a cohort directory and writes one whose fit set is a chosen subset of its project set; the project set is linked, not copied (a biobank .bed can be tens of GB). The output uses the subsample preset.

Choose the fit samples by exactly one of:

--group COLUMN=PATTERN:COUNT — take COUNT samples whose COLUMN matches PATTERN (case-insensitive regex); repeatable, and a sample is taken at most once across groups. --include-rest also adds every sample matched by no group.
--fit-samples FILE — a FID IID list chosen elsewhere.
--geosketch N --pca CSV — take N samples via geometric sketching (Hie et al. 2019) on the PCA coordinates in CSV (sample_id, dim_1, dim_2, ...), restricted to the samples in the project .fam first. --n-pcs limits how many of the CSV's columns are used (default: all). Needs the geosketch extra: pip install 'manifold-genetics[geosketch]'.

Setup¶

manifold-genetics setup [--skip-plink1] [--preprocessing]

Pre-fetches plink2, plink and flashpca into the per-user cache (~/.cache/manifold-genetics/bin, or the checkout's bin/ when you are running from one). They are also fetched on first use, so this is optional — what it is for is fetching them before submitting a job, because compute nodes usually have no internet. Needed for data preparation, not for the pipeline itself; see Install.

--preprocessing also fetches the GIAB, WRayner and TOPMed references that preprocess --preset harmonise needs (about 2 GB, most of it TOPMed) into the cache's preprocessing/ subdirectory. Every reference is attempted even when one fails. The WRayner URL currently returns 404 upstream, so today it places GIAB and TOPMed and then exits non-zero, naming the failed URL and the path where a hand-placed HRC-1000G-check-bim.pl goes; Preprocessing has the details. Re-running is safe: it fetches only what is still missing.

Exit codes and logging¶

Commands return 0 on success and non-zero on failure, with the reason on stderr rather than a traceback for the errors a user can actually act on — a missing config, an unknown key, absent input files.

Logging goes to stderr at INFO by default and DEBUG under -v, scoped to the manifold_genetics logger.