velvet-pine-22/PULSE-code

PULSE benchmark code

Reference code of the paper "PULSE: A Synchronized Five-Modality Dataset for Sensorimotor Coordination in Long-Horizon Daily Activities" (NeurIPS 2026 Evaluations and Datasets Track, spotlight). It loads the public PULSE dataset, defines the volunteer-stratified 5-fold cross-validation splits, trains every baseline of the benchmark (T1-T7, F5), runs the F0 case study, and regenerates the benchmark tables, figures and numbers of the paper from the run results.

Paper	https://openreview.net/forum?id=AoU576r9O2
Dataset	https://huggingface.co/datasets/velvet-pine-22/PULSE (CC BY-NC 4.0), DOI 10.57967/hf/11314
Small sample of the dataset	https://huggingface.co/datasets/velvet-pine-22/PULSE-sample
Code (this repository)	https://huggingface.co/velvet-pine-22/PULSE-code (MIT, see LICENSE)
Contact	Yang Gao (corresponding author, gaoyang2025@scut.edu.cn); maintainer Wolin Liang (202610196903@mail.scut.edu.cn)

The benchmark data are the dataset's tag v1.1 (commit e27b1bb6bb35f714705ae8685507f71773c065eb, 3,001 files, 22.0 GB: sensor streams, annotations, metadata). All numbers in the paper were computed on that tag with the commands in section 3.

Contents

pulse/            Python package: paths, data layer, splits, taxonomy, metrics, statistics,
                  provenance, aggregation, SLURM launcher, models/, tasks/ (one module per task)
configs/          tables.yaml (registry of every table and its rows), slurm.yaml, per-task configs
splits/           pool33.json, pool27.json (volunteer-disjoint 5-fold CV with inner validation)
taxonomy/         taxonomy_v1.1.json (T2 / F5 label vocabulary)
manifests/        hf_PULSE_v1.1_files.tsv (file list of dataset tag v1.1), availability_v1.1.json,
                  expected_v1.1.json
scripts/          run_table, aggregate, build_outputs, smoke_suite, verify_data, fetch_hf_manifest,
                  make_splits, build_taxonomy, build_availability, build_manifests
figures/          figure scripts (T1, T2, T5, F0, F5)
prompts/          VLM annotation prompt templates used to draft the action labels (see prompts/README.md)
slurm/            sbatch templates
tests/            unit tests (synthetic mini dataset + optional checks on the real data)
environment.yml   the environment every number was produced with

1. Setup

git clone https://huggingface.co/velvet-pine-22/PULSE-code && cd PULSE-code
conda env create -f environment.yml      # Python 3.12.13, torch 2.4.1+cu121
conda activate pulse

# dataset, pinned to the benchmark tag (22.0 GB, no videos)
hf download velvet-pine-22/PULSE --repo-type dataset --revision v1.1 --local-dir PULSE
export PULSE_ROOT=$PWD/PULSE

# check every file against the tag's file list (sha256 for LFS files, git blob id otherwise)
python scripts/verify_data.py --root $PULSE_ROOT --manifest manifests/hf_PULSE_v1.1_files.tsv --workers 8
python -m pytest -q tests                # unit tests on CPU; a few also read $PULSE_ROOT

hf is the Hugging Face CLI (installed with huggingface_hub, which the environment contains); with older versions use huggingface-cli download with the same arguments. Tasks read the dataset from --root or $PULSE_ROOT. Run every command from the repository root (or pip install -e . to import pulse from elsewhere; the configs, splits and manifests are still read from the checkout).

Dataset revisions. The main branch of the dataset contains everything in v1.1 plus the first-person videos (622 files, about 63 GB), the S9 motion-primitive labels (annotations/vX/s9.json, 30 files, and annotations_flat/segments_s9.csv) and updated documentation. The sensor streams and the S1-S8 annotations are identical; metadata/recordings.csv differs only in the line endings and in marking the scene video of v5s8 and v18s2 as missing (not a benchmark modality). The benchmark needs only v1.1; F5 additionally needs the S9 labels (section 3, F5). Against a full download of main, verify_data.py reports those differences and the extra files; it passes with:

python scripts/verify_data.py --root $PULSE_ROOT --manifest manifests/hf_PULSE_v1.1_files.tsv --workers 8 \
    --expect-different README.md --expect-different DATASHEET.md --expect-different LICENSE \
    --expect-different CODE_LICENSE --expect-different croissant.json \
    --expect-different metadata/recordings.csv --expect-different docs/upload_to_huggingface.md \
    --ignore-extra '*.mp4' --ignore-extra 'annotations/*/s9.json' --ignore-extra annotations_flat/segments_s9.csv

2. Getting started

Load one recording and its action segments, and iterate the folds of a split (save as a file in the repository root and run it there):

# Run from the root of a PULSE-code checkout, with $PULSE_ROOT = the dataset (revision v1.1).
from pulse.paths import DataRoot
from pulse.splits import load_split
from pulse.data import load_recording
from pulse.data.annotations import load_segments
from pulse.data.availability import Availability
from pulse.data.manifest import expected_manifest

dr = DataRoot.from_arg()                                  # reads $PULSE_ROOT
av = Availability.load("manifests/availability_v1.1.json")  # usable streams per recording
mods = ["mocap", "emg", "eyetrack", "imu", "pressure"]
split = load_split("splits/pool27.json")                  # volunteer-disjoint 5-fold CV

# (b) iterate the folds: which recordings each partition uses for these modalities
for k in range(split.n_folds):
    fold = split.fold(k)
    man = expected_manifest(av, fold, mods, require_annotation=True)
    print(f"fold {k}: test volunteers {list(fold.test)}; {man.summary()}")

# (a) one test recording of the last fold: aligned 100 Hz arrays + action segments
rid = man.partitions["test"][0]
rec = load_recording(dr, rid, mods, align_trimmed=True)
print(rid, {m: x.shape for m, x in rec.arrays.items()})   # (T, features), common T
for s in load_segments(dr, rid)[:3]:
    print(f"{s.start_sec:7.2f} {s.end_sec:7.2f}  {s.verb} {s.noun} ({s.hand})")

Output on v1.1:

fold 0: test volunteers ['v25', 'v28', 'v11', 'v38', 'v5', 'v35']; pool27 fold 0 [mocap+emg+eyetrack+imu+pressure; s1,s2,s3,s4,s5,s6,s7,s8; labelled]: train 128 / inner_val 24 / test 40 recordings (excluded 11/0/8)
fold 1: test volunteers ['v3', 'v32', 'v2', 'v13', 'v36', 'v23']; pool27 fold 1 [...]: train 123 / inner_val 24 / test 45 recordings (excluded 16/0/3)
fold 2: test volunteers ['v26', 'v33', 'v22', 'v41', 'v24']; pool27 fold 2 [...]: train 128 / inner_val 24 / test 40 recordings (excluded 19/0/0)
fold 3: test volunteers ['v37', 'v29', 'v12', 'v39', 'v27']; pool27 fold 3 [...]: train 141 / inner_val 23 / test 28 recordings (excluded 11/1/7)
fold 4: test volunteers ['v1', 'v40', 'v31', 'v34', 'v30']; pool27 fold 4 [...]: train 134 / inner_val 19 / test 39 recordings (excluded 13/5/1)
v1s2 {'mocap': (5645, 620), 'emg': (5645, 8), 'eyetrack': (5645, 24), 'imu': (5645, 180), 'pressure': (5645, 50)}
   0.00    5.00  open box (both)
  11.00   14.00  close box (both)
  14.00   16.00  open box (both)

Every benchmark run has the same interface and writes one results.json (metrics, per-recording statistics, provenance):

python -m pulse.tasks.<task> --root $PULSE_ROOT --split splits/<pool>.json \
    --fold <k> --seed <s> --config configs/<...>.yaml [--out DIR] [--set key=value] [--device auto|cpu|cuda]
python -m pulse.tasks.<task> ... --print-manifest      # list the recordings a configuration uses

A quick CPU check of the whole chain (nearest-centroid scenario classifier, not in the paper):

python -m pulse.tasks.datacheck --root $PULSE_ROOT --split splits/pool33.json \
    --fold 0 --seed 42 --config configs/datacheck/emg_pressure.yaml

3. Reproducing the paper

configs/tables.yaml registers every table: its task, split, seeds and rows (one config per row). scripts/run_table.py turns a table into one run per (row, fold, seed) and scripts/aggregate.py turns the runs into LaTeX tables and number macros:

python scripts/run_table.py --table 3                # print the commands (135 runs)
python scripts/run_table.py --table 3 --local        # run them here, one after another
python scripts/run_table.py --table 3 --slurm        # or as a SLURM job array (section 7)
python scripts/aggregate.py --table 3                # -> tables/tab3.tex, tab3_common.tex, tab3_paired.tex, numbers.tex
python scripts/build_outputs.py --runs runs --out out --f0 --fig7   # every table, report and figure at once

Runs whose results.json exists are skipped, so an interrupted table can be resumed. The registry ids below are internal names; the registry titles, config comments and docstrings use the table numbers of the submitted version of the paper, and the table below gives the camera-ready numbers.

Camera-ready paper	Registry table(s)	Task	Split	Runs	Outputs
Table 3, Fig. 3 (T1 scenario recognition, each method's own modalities)	3	t1	pool33	135	tab3*.tex; Fig. 3: python figures/t1/fig3_confusion.py --runs runs --table 3 --row syncfuse
Table 4 (T1 matched inputs: IMU only, 4 modalities, 5 modalities, 4 -> 5 modalities on the same recordings)	3imu, 3m4, 3m5, 3gain, 3gain-as	t1	pool33	120	tab3imu*.tex, ... (rows marked reuse: read the runs of the identical row of another table)
Table 5 (T1 on the right-wrist accelerometer: harnet5 vs TinyHAR)	3harnet	t1	pool33	45	tab3harnet*.tex
Table 6, Table 27 (T2 action recognition, sliding windows; full per-head metrics; training-majority row)	4, 4maj	t2	pool27	120	tab4*.tex, tab4maj*.tex
Table 7 (T2, segment-aligned protocol)	4seg (4segmaj: majority rates, not in the paper)	t2	pool27	120	tab4seg*.tex
Table 8 (T2 matched inputs: all five / IMU only)	4m5, 4imu	t2	pool27	240	tab4m5*.tex, tab4imu*.tex
Table 9, Fig. 4 (F5 motion-primitive pre-training -> T2)	5 (5seg: segment-aligned variant, not in the paper)	f5	pool27	150	tab5*.tex; Fig. 4: python figures/f5/plot_f5.py --table 5 --tables tables --out figures/out
Table 10 (T3 grasp-onset anticipation)	6	t3	pool27	90	tab6*.tex; per-fold counts: python -m pulse.tasks.t3_report --table 6
Table 11 (T4 missing modalities, 4 modalities)	7	t4	pool33	30	tab7*.tex
Table 12 (T4 with five modalities; 5 vs 4 modalities on the same recordings)	7m5, 7m5-matched	t4	pool33	45	tab7m5*.tex, tab7m5-matched*.tex
Tables 13, 14, 15, 28; Figs. 7, 8 (T5 grasp state, EMG vs EMG + pressure)	8	t5	pool27	105	python -m pulse.tasks.t5_report --table 8 --table 8full --runs runs --out tables -> t5_tab8_lift.tex (13), _classes.tex (14), _events.tex (15), _backbones.tex (28); Fig. 7: python figures/t5/fig_label_derivation.py --root $PULSE_ROOT --out figures/t5; Fig. 8: python figures/t5/fig_sustained_lift.py --report tables/t5_tab8.json --out figures/t5
Table 16 (T5 with the full non-tactile stack +- pressure)	8full	t5	pool27	90	t5_tab8full_lift.tex (same report command)
Table 17 (T6a / T6b pressure prediction, single-seed and within-fold 3-seed ensemble columns; Ghorbani GRU with four modalities, marked with a dagger)	11a, 11b, 11a-ens, 11b-ens, 11b-b19	t6, t6ens	pool27	120 GPU + 35 CPU	tab11a*.tex, tab11b*.tex, tab11a-ens*.tex, tab11b-ens*.tex, tab11b-b19*.tex
Table 18 (T6a budget-matched: EMG vs four modalities on the same recordings)	11a-b7-pham, 11a-b7-transformer, 11a-b7-bao	t6	pool27	90	tab11a-b7-*.tex
Table 19 (T7 EMG -> right-hand pose)	12	t7	pool33	75	tab12*.tex
Table 23 (appendix backbone ablation, 1 seed)	16	t1	pool33	45	tab16*.tex
Table 24 (appendix fusion ablation, 1 seed)	17	t1	pool33	45	tab17*.tex
Figs. 5, 6 and the F0 numbers (anticipatory coordination case study, CPU)	--	f0	all recordings with EMG + pressure + MoCap	1	python -m pulse.tasks.f0 --root $PULSE_ROOT --out runs/f0 --workers 16, then python figures/f0/plot_f0.py --in runs/f0 --out figures/out --tables tables (f0_grasp_phase_timing.pdf = Fig. 5, f0_delay_distributions.pdf = Fig. 6, numbers_f0.tex); see configs/f0/README.md
Table 25 (verb rollup), Table 26 (test recordings per input configuration)	--	--	--	--	taxonomy/taxonomy_v1.1.json (scripts/build_taxonomy.py); manifests/expected_v1.1.json (python scripts/build_manifests.py --check)

Main tables use 5 folds x 3 seeds (42, 43, 44); the appendix ablations one seed. The T2 confusion-matrix script (figures/t2/confusion.py) draws a figure that is not in the paper.

Result provenance. This repository is published as a single snapshot; its development history is not public. The numbers in the paper come from runs of development commits e792638 (T1-T6 and the appendix ablations), cb55d98 (T7 and F0) and 487a753 (F5), all from clean checkouts on data tag v1.1 (each run records its commit id in its results.json). Between e792638 and 487a753 the T1-T6 code paths are unchanged except the label of one T6a baseline (the multi-branch CNN of Hajian et al. 2021) and documentation. The benchmark code in this snapshot is identical to 487a753; the snapshot only adds documentation, packaging metadata, the licence and prompts/. Every results.json records the commit (and whether the checkout had uncommitted changes), the data tag and the md5 of the split file, taxonomy, availability table and expected manifest, so a rerun can be compared run by run. GPU runs are not bit-reproducible across hardware and library versions; expect small differences from the paper's numbers within the reported fold-to-fold spread.

F5 needs the S9 labels, which are not in tag v1.1. They are published on the dataset's main branch (identical to the reviewed labels the paper's F5 runs used):

hf download velvet-pine-22/PULSE --repo-type dataset --revision main \
    --include "annotations/*/s9.json" --local-dir PULSE_S9
export PULSE_S9_ANNOTATIONS=$PWD/PULSE_S9/annotations
python scripts/run_table.py --table 5 --local          # 10 rows x 5 folds x 3 seeds

Pre-training uses volunteers v9 and v16-v21 (none of them is in pool27); v20s9's IMU fails the |x| > 1e6 rule, so 6 recordings are used. Details: configs/f5/README.md.

harnet5 rows (3harnet, and the harnet5 row of 3imu) load OxWearables/ssl-wearables at commit 150550ea with torch.hub (University of Oxford Academic Use Licence, not vendored; checkpoint SHA-256 verified). On machines without internet access fetch it once and point $PULSE_SSL_WEARABLES_DIR at it:

python -m pulse.models.t1.harnet --fetch ~/ext/ssl-wearables     # once, with internet
export PULSE_SSL_WEARABLES_DIR=~/ext/ssl-wearables/ssl-wearables-150550ea5d41800229c95e36f88f5bf0d2e7cf04

Launch paper runs from a checkout nobody commits to and keep --runs outside it: every task refuses to write results if the checkout changes while it runs, and the aggregator refuses to merge runs from different commits, configurations, --set overrides, device types or torch versions.

4. Protocol

Volunteer pools and folds (splits/). Two split files, volunteer-disjoint 5-fold CV:

File	Volunteers	Test folds	Used by
splits/pool33.json	33 (all with EMG + skeleton MoCap)	7/7/7/6/6, round-robin by volunteer number	T1, T4, T7, appendix ablations
splits/pool27.json	27 (all with EMG + pressure)	6/6/5/5/5, seed-42 random partition	T2, T3, T5, T6, F5

Within a configuration only recordings that have all its modalities are used (e.g. 26 of the 33 pool33 volunteers have pressure; v5 has no MoCap); paper Table 26 lists the resulting test recordings per input configuration.

Inner validation. Each fold also lists inner-validation volunteers: round(0.15 x n_train) training volunteers (pool33: 4, pool27: 3), chosen deterministically (strata by modality coverage, full S1-S8 coverage required, each volunteer at most once across folds; the exact rule is stored in the split file). Epoch selection, early stopping and LR scheduling use only the inner-validation volunteers; the test fold is evaluated once.

Seeds and statistics. Main tables: seeds 42, 43, 44. Per fold the metric is averaged over seeds; tables report mean +- SD (ddof = 1) over the 5 fold means. Rows are compared on the common test set (per fold, the recordings every compared row loaded) with paired t-tests over fold means (n = 5) and over fold x seed pairs (n = 15), plus the number of folds where the row is better. The T6 within-fold ensemble averages all 3 seeds.

Labels (T2, F5). v1.1 human-reviewed annotations, frozen in taxonomy/taxonomy_v1.1.json by scripts/build_taxonomy.py: 17 fine verbs (the single support segment is dropped), 6 composite verbs, 33 nouns with

= 50 segments over all 36 annotated volunteers (cup as named in v1.1), 3 hands; 7,076 of 7,356 segments kept. Uniform chance: verb@1 5.9 %, noun@1 3.0 %, action@5 (top-5 verbs x top-5 nouns) 25/561 = 4.5 %.

T1 (activity-scenario recognition) and T4 (missing modalities). One sample = one whole S1-S8 recording, label = its scenario (8 classes; no action annotations), pool33. The configured modalities are cut to their common length (the three re-trimmed recordings v2s6, v13s2, v13s3 aligned first, align_trimmed), concatenated in config order, decimated to 20 Hz (longest sequence 4,844 steps; every positional encoding holds 8,192) and z-scored with training-frame statistics; batches are zero-padded with a frame mask. Training (configs/t1/base.yaml, configs/t4/base.yaml): Adam lr 1e-3, weight decay 1e-4, batch 16 (ST-GCN / CTR-GCN 4), cross-entropy with inverse-frequency class weights and label smoothing 0.1, gradient clipping 1.0, ReduceLROnPlateau on the inner-validation loss, early stopping after 15 epochs; the epoch is selected on inner-validation macro F1 (T1, at most 80 epochs) or inner-validation loss (T4 and the appendix ablations, at most 100 epochs); the test fold is evaluated once. SyncFuse is pulse/models/t1/syncfuse.py; its modality dropout (p = 0.3) is drawn per sample, and its parameter count (about 3.1M with five modalities) is stored as n_params in every results.json. "Transformer+Late" (per-modality Transformers, mean of logits) is trained from scratch. LIMU-BERT is the supervised-only variant. ST-GCN and CTR-GCN average over real frames only (the frame mask follows the stride-2 blocks), so a recording's prediction does not depend on the other recordings in its evaluation batch. InceptionTime has no frame mask, so it is evaluated one recording at a time (train.eval_batch_size: 1). T1/T4 refuse a split file whose used_by does not list them. T4 trains the late-fusion Transformer with per-sample input dropout (p = 0 / 0.3) and evaluates it on full, leave-one-out and singleton inputs (zeroed z-scored inputs), all on the same test recordings; gate_modalities: [pressure] restricts a 4-modality run to the 5-modality recordings (tables 3gain*, 7m5-matched). harnet5 (3harnet): right-wrist (WT8) acceleration in g resampled to 30 Hz, 5 s windows every 5 s, recording prediction = mean of window logits; AdamW lr 1e-3, batch 64, 80 epochs, patience 15.

T2 (fine-grained action recognition). One config per model (configs/t2/<model>.yaml) serves every T2 table. Primary protocol (paper Table 6, registry 4, protocol: sliding): 4 s windows at 1 s stride over each recording; a window takes the labels of the kept segment that contains its centre and is skipped otherwise; models are trained, inner-validated and tested on such windows. Secondary (paper Table 7, registry 4seg, --set protocol=segment): one sample per kept segment, cut at its annotated [start, end) (the 63 of 5,312 pool27 segments longer than 12.8 s are centre-cropped to 256 frames). Signals are decimated to 20 Hz and z-scored with training-sample statistics. Windows and segments lie in the recording's evaluation span -- the frames covered by every usable benchmark modality, computed from the availability table -- so two rows that load a recording score identical samples; the re-trimmed EMG / eye-tracking / pressure files of v2s6, v13s2 and v13s3 are placed at their release offset (mocap_padding.front_samples) on the annotation timeline. Training: AdamW (lr 3e-4, weight decay 1e-4), batch 32, at most 40 epochs, 5-epoch linear warm-up then cosine decay (fixed schedule), label smoothing 0.05, no class weighting, loss weights verb 1 / composite 0.5 / noun 1 / hand 0.5; the epoch is selected and training stopped (patience 25) on inner-validation act_vn@5; the test fold is evaluated once. Every T2 input set contains IMU or MoCap, so v5 (72-column IMU, no MoCap) never enters T2: the effective pool is 26 volunteers (of the 211 annotated S1-S8 pool27 recordings, 198 have IMU, 198 MoCap, 192 all five modalities). Metrics: act_vn@k = verb in the top-k verbs and noun in the top-k nouns (chance at k = 5: 25/561 = 4.5 %), verb / noun top-1 and top-5, composite-verb and hand top-1, per-head macro-F1, and the accuracy of the training-majority class (4maj, 4segmaj). DAF = DailyActFormer: per-modality multi-scale conv stems, whole-modality dropout p = 0.3 drawn once per batch, concat + 1x1 fusion, 3-layer Transformer, learnable-query pooling, joint verb-noun head (17 x 33). "ActionLLM" is a sensor surrogate trained from scratch (conv stem + 2-layer Transformer; no language model). The matched-input tables (paper Table 8) are 4m5 (all five modalities) and 4imu (IMU only); rows marked reuse: in configs/tables.yaml read the runs of the identical row of table 4.

T3 (grasp-onset anticipation). Protocol in the docstring of pulse/tasks/t3.py and configs/t3/base.yaml; per-fold sample counts and the positive fraction: python -m pulse.tasks.t3_report --table 6.

T5 (tactile-driven grasp-state recognition). At anchors every 0.25 s the past 1 s of the inputs (20 Hz) predicts the state of the next 0.5 s: NoGrasp unless >= 50 % of it lies in v1.1 segments with a grasp verb (grasp, hold, pick_up, move, place, put_down, pull, rotate, insert, remove); otherwise SustainedGrasp if its longest contact run is >= 0.3 s (contact = >= 3 of the 50 pressure cells > 10 g), else AttemptedGrasp. Anchors are also stratified by past / future contact (non-contact, pre-contact, steady-grip, release; paper Table 15). All rows of a table load the same streams -- table 8: EMG + pressure (pressure defines the labels; the EMG-only rows do not see it), table 8full: all five -- so all rows score identical anchors (checked by the report); test recordings over the 5 folds: 210 (8; v5s8 has no pressure) and 192 (8full). Backbones DAF, FUTR, DeepConvLSTM (pulse/models/t5). Training: each class capped at 10,000 anchors, class-weighted cross-entropy, AdamW (wd 1e-3), fixed cosine schedule, d_model 64, dropout 0.3; DAF / FUTR lr 3e-4, 30 epochs, patience 6; DeepConvLSTM lr 1e-4, 50 epochs, patience 12; the epoch is chosen on inner-validation macro F1 and the test fold is evaluated once. Lift = with - without pressure, paired per (fold, seed); per backbone paired t over the 5 fold means and the 15 fold x seed runs; 3-backbone aggregate over fold means (n = 5) and the 15 fold x backbone pairs. In v2s6, v13s2 and v13s3 the labels are shifted by the re-trim offset (mocap_padding.mocap_sq.front_samples = 1048 / 393 / 336 frames): their annotations are on the full timeline (the last segments end 1.5 / 3.8 / 0.3 s before the end of the re-trimmed streams on that timeline, but 9.0 / 0.1 / 3.1 s after it without the shift). Details: docstrings of pulse/tasks/t5.py and pulse/tasks/t5_data.py.

T6 (cross-modal pressure prediction). T6a regresses the right-hand pressure sum (R1..R25, 1-D; the Myo side), T6b the 50 cells; metrics are pooled over all scored test frames of a fold, in grams (T6a pearson, T6b pearson_max = r_max, the best of the 50 per-cell r, and pearson_mean). Only active segments are trained on and scored: pressure total > 100 g, dilated by 1.0 s, 1.0 s context on each side, overlapping segments merged. Inputs are z-scored per subject from a calibration window only -- the first 30 s of the subject's recordings, S1 first (std floored at 0.1 x the pooled training std, clipped at +-10); calibration frames are not scored; no statistic of the rest of a test recording is used. EMG is a 200 ms RMS envelope; T6b MoCap is the 460-column hand set (both hands and forearms, wrist-relative, + velocity), the T6a 4-modality rows use the full skeleton. Reported model = SWA average of epochs 30-60 (ReduceLROnPlateau on the inner-validation Huber loss before that); the test fold is evaluated once. The ensemble tables (pulse.tasks.t6ens, CPU) average the per-frame test predictions of seeds 42, 43 and 44 of a row. The budget-matched rows (paper Table 18) are trained and tested on identical recordings (the EMG-only rows are gated on the 4-modality recordings); 11b-b19 is the 4-modality Ghorbani GRU row of paper Table 17. A T6 run refuses to write results if the checkout changes while it runs; t6ens refuses a subset of the table's seeds and a predictions.npz that does not reproduce its member's results.json. Protocol details: docstring of pulse/tasks/t6.py.

T7 (EMG -> right-hand pose). Input: the 8 Myo channels, pooled to 20 Hz by block RMS (5 frames per block); target: the right hand as 21 joints x 3 = 63-D in the wrist frame (mm) from the skeleton TSV -- wrist + per finger RightHand{f}1 (MCP), {f}2, {f}3 and Right{f}Tip (the usual 21-keypoint layout). MPJPE is over the 21 joints, fingertip MPJPE over the 5 tips, and the angle error is the bone-direction error (atan2(|u x v|, u . v)) over the 20 hand-skeleton bones; finger_angle_err_deg (15 finger bones, without the 5 palm bones that are fixed in the wrist frame) is also written to results.json. MoCap dropout frames (zero wrist quaternion, NaN) are masked. Definitions: pulse/tasks/t7_pose.py. Loss (train.loss): mpjpe (default; masked MPJPE in mm, fingertips weighted train.tip_weight = 2), or huber / mse on the z-scored target. Rows: "MS-CNN+Transformer" (multi-scale CNN + Transformer; named "EMG2Pose" in the submitted version, but it is not the TDS + LSTM vemg2pose of the emg2pose paper), NeuroPose, Transformer, TDS (the emg2pose featurizer) and the training-set mean pose (reference row). MS-CNN+Transformer uses its position head: the velocity-integration head drifts over whole recordings (fold-0 inner-validation MPJPE 138.5 mm after 12 epochs vs 21.6 mm; configs/t7/mscnn_transformer_velocity.yaml, not in the table). Training batches are whole recordings padded to the longest one; the BatchNorm layers of MS-CNN+Transformer and NeuroPose take their statistics from real frames only, and NeuroPose resets padded positions to 0 after every block. RMS pooling is used instead of decimation (emg[::5]): on fold-0 inner-validation volunteers a ridge probe on decimated EMG equals the mean pose (21.95 vs 21.95 mm) while RMS blocks give 20.99 mm. Both choices (position head, RMS pooling) were made once, on the fold-0 inner-validation volunteers v16/v25/v33/v36, who are test volunteers in folds 1-4; they are fixed for all folds.

python -m pulse.tasks.t7 --root $PULSE_ROOT --split splits/pool33.json \
    --fold 0 --seed 42 --config configs/t7/mscnn_transformer.yaml   # one run

5. Data layer (pulse/paths.py, pulse/data/)

• Modalities: emg (Myo, 8 ch, right forearm for every volunteer), imu (10 IMUs x 18 = 180 columns; subsets imu_wrist, imu_finger), eyetrack (24 Dikablis core columns; subsets eyetrack_gaze, eyetrack_ocular), pressure (25 cells per hand), mocap (skeleton TSV: hip-relative positions

  • position velocity, 620 columns), mocap_markers, myo_quat. All streams are on the dataset's common 100 Hz timebase.

• The skeleton TSV aligned_vXsY_s_Q.tsv is in data/vX/sY/raw/ for 237 recordings and at the recording root for 10 (v1s5, v2s2, v2s6, v3s1, v3s2, v3s5, v3s6, v11s8, v13s2, v13s3); both are supported.

• Nothing is skipped silently. A missing file raises MissingFileError; a file that does not provide the modality raises ModalityUnavailableError (v5's IMU file has 72 columns -> treated as a missing modality; one-eye eye-tracking exports lack core columns); a corrupted stream raises DataQualityError. Select recordings through the availability table (as in section 2) rather than loading every recording id.

• manifests/availability_v1.1.json (built by scripts/build_availability.py) records, per recording and modality, whether it is usable. For each task configuration and fold, pulse.data.manifest.expected_manifest lists exactly which recordings train / inner-val / test must load; every task checks what it loaded against it before writing results. manifests/expected_v1.1.json (scripts/build_manifests.py) freezes these lists for reference configurations, e.g. pool33 MoCap+EMG+eye+IMU: 235 test recordings over the 5 folds; with pressure (or gated on pressure): 192.

• Documented exclusions found by the scan of v1.1 (no data errors):

  Recording(s)	Effect
  v5 (s1-s9)	IMU file has 72 columns -> no imu; no MoCap
  v14s1, v20s3-s9, v29s4-s9	IMU values with abs > 1e6 -> no imu (quality:extreme_values)
  v1s1, v14s8	one-eye eye-tracking export -> no eyetrack
  v23s3	scene-camera gaze columns > 90 % zero -> no eyetrack_gaze (full eyetrack is fine)
  v11s6	marker CSV corrupted -> no mocap_markers (skeleton mocap is fine)
  v4s1, v4s4	listed in recordings.csv but ship no files

  Notes: v17s6 has an aligned skeleton TSV although recordings.csv lists no mocap (that column tracks the marker CSV), so it is used; for 7 of the 10 root-level TSVs alignment_metadata.json does not list MoCap but the streams have exactly the recording's sample count; in v2s6, v13s2 and v13s3 the EMG, eye-tracking and pressure streams are shorter than the others (re-trimmed at release) and multi-modal inputs are cut to the shortest stream. The short files cover frames [front, front + n) of the IMU / MoCap timeline (mocap_padding.mocap_sq.front_samples / n_frames in alignment_metadata.json), so cutting from the start shifts IMU / MoCap against EMG / eye / pressure by 10.5 s, 3.9 s and 3.4 s. load_recording(..., align_trimmed=True) slices the full-timeline streams to that window first (off by default in the loader; T1 / T4 turn it on through data.align_trimmed, T3 always uses it). T6 and T7 use the same window (release_trim_window); T2, T5, F5 and F0 place the short streams at their release offset on the annotation timeline (recording_timeline).

6. Results files

results.json (schema pulse-results/1, see pulse/provenance.py) holds the test metrics, per-recording sufficient statistics (confusion matrices, top-k hits, Pearson sums, ...), from which every metric can be recomputed on any recording subset (the aggregator does this for the common test set; per-volunteer metrics can be computed the same way), and provenance: git commit (+ dirty flag), data tag v1.1, split file md5, taxonomy md5, availability md5, manifest md5, seed, device, train / inner-val / test volunteers, loaded recording ids and the full config. scripts/aggregate.py writes tables/tab<N>.tex, tab<N>_common.tex, tab<N>_paired.tex, tab<N>.json and tables/numbers.tex (macros \pulsenum{<table>/<row>/<metric>/<stat>}).

7. Running on a SLURM cluster

scripts/run_table.py --slurm writes <runs>/_jobs/<name>-<time>/{commands.txt,job.sbatch,job.json} and submits one job array (one GPU, 8 CPUs and 60 GB per task, 8 h limit; configs/slurm.yaml); repeat --table to put several tables into one array. The launcher was written for the authors' cluster: the GPU partition name is hard-coded in pulse/slurm.py (GPU_PARTITION) and slurm/gpu_array.sbatch.in, and the CPU partition is set in configs/slurm.yaml; change these for another cluster, or use --local / --dry-run (writes the job files without submitting). The user's running and pending GPUs on that partition are counted first and the array is throttled to stay within a cap (default 4, MAXGPU=1..8). --after JOBID[,JOBID] adds --dependency=afterany:.... GPU arrays are refused from a checkout with uncommitted changes, and every --set key (CLI and the rows' set:) must exist in the row's config (--allow-new-keys for smoke-test keys such as train.max_train_batches). The variables the rows need are checked before submission and exported in the job script: PULSE_ROOT, PULSE_SSL_WEARABLES_DIR (harnet rows) and PULSE_S9_ANNOTATIONS (F5 rows with S9 pre-training). The account is $PULSE_SLURM_ACCOUNT, else account in configs/slurm.yaml, else the user's default. --cpu runs on the CPU partition. Set PULSE_PYTHON (the environment's python) before submitting.

# CPU smoke suite: every run row of every table, fold 0, seed 42, 1 short epoch (configs/smoke.yaml)
python scripts/smoke_suite.py --runs SMOKE/runs --submit
# GPU smoke before the full runs: GPU tables only, 2 full epochs
python scripts/smoke_suite.py --runs GPUSMOKE/runs --gpu --submit
# every table, numbers.tex, report and figure from one runs directory (with a LaTeX build check)
python scripts/build_outputs.py --runs SMOKE/runs --out SMOKE/out --allow-incomplete --latex

8. Compute

The 1,515 GPU runs behind the T1-T7 tables of the paper (including the appendix ablations and the matched-input comparisons) took about 53 GPU-hours on NVIDIA A800 80 GB GPUs, one GPU per run (median about 2 minutes, at most 22 minutes per run); the F5 runs (150 for paper Table 9) are not included in that figure. The T6 seed ensembles and F0 run on CPU. Peak host memory of a run was about 34 GB (T4). On CPU, MulT at batch 16 needs more than 64 GB (no fused attention over 4,844 steps); the CPU smoke suite runs it at batch 4.

9. Known issues and naming

• v5: IMU has 4 sensors (72 columns) and is treated as missing; no MoCap.
• 10 skeleton TSVs sit at the recording root (handled, see section 5).
• F5 reads the S9 labels from --s9-annotations / $PULSE_S9_ANNOTATIONS (vX/s9.json, published on the dataset's main branch); a missing label file raises MissingFileError. v20s9's IMU fails the |x| > 1e6 rule, so pre-training uses the other 6 recordings.
• "Transformer+Late" (T1) is trained from scratch; the earlier name "Transformer+Late+Pretrain" is dropped because no pretraining is performed.
• metadata/recordings.csv column split is a legacy fixed hold-out and is not used; all results use the 5-fold split files in splits/.
• T7: the TSV fingertip points Right{f}Tip were added in post-processing (alignment_metadata.json: tips_added, tip_scale 0.75) and the MCP joints are rigid in the wrist frame (fixed bone lengths per volunteer).
• Registry titles, config comments and docstrings refer to the table numbers of the submitted version (e.g. "Table 4" for T2, "Tables 16/17" for the appendix ablations) and to review-round row groups (R1-R3, B7, B13, B19, B21); section 3 maps every registry table to the camera-ready paper.

10. Annotation prompts

The action labels were drafted by a vision-language model and then reviewed by a human (every released segment was confirmed or corrected). The prompt templates for S1-S8 and for the S9 primitive pass, the tables filled into them and a script that renders the exact prompt are in prompts/ (see prompts/README.md).

License

Code: MIT (LICENSE). Dataset: CC BY-NC 4.0 (see the dataset card). The harnet5 weights are loaded from OxWearables/ssl-wearables under its own licence and are not distributed here.

Citation

@inproceedings{liang2026pulse,
  title     = {{PULSE}: A Synchronized Five-Modality Dataset for Sensorimotor Coordination in Long-Horizon Daily Activities},
  author    = {Liang, Wolin and Gao, Yang and Yan, Peiyu and Hu, Yuexiang and Xiao, Yingjing and Ying, Xiongfeng and Guo, Yingnian and Zhang, Cheng and Jin, Zhanpeng},
  booktitle = {NeurIPS 2026 Evaluations and Datasets Track},
  year      = {2026},
  url       = {https://openreview.net/forum?id=AoU576r9O2}
}

Dataset DOI: 10.57967/hf/11314.