Technical Report & Project Page

TouchFormer

Evaluating Transfer Learning & Self-Supervised MAE for Pressure-Based Human Pose Estimation

Subhranil Mondal Independent Researcher

Figure 1: 3D Human Pose and Shape Estimation from a Pressure Image. TouchFormer predicts the 3D human mesh and pose coordinates directly from 2D tactile matrix inputs.

Abstract

Tactile and physical-contact perception has historically been restricted to task-specific CNN architectures. We present TouchFormer, a Vision Transformer (ViT-B/16) framework for 3D human pose estimation from physical pressure maps. Through two independent experiments, we investigate whether transformer pretraining strategies — (A) cross-domain transfer from ImageNet, and (B) self-supervised Masked Autoencoder (MAE) pretraining on unlabeled pressure data — can substantially outperform the supervised CNN baseline. Both strategies yield significant improvements over the prior CNN baseline, with the MAE result being particularly notable given its training under severe compute constraints.

Related Work & Scientific Novelty

Self-supervised learning has gained significant traction in tactile representation learning. Most notably, Sparsh (Meta AI, 2024) demonstrated large-scale general-purpose touch representations for vision-based tactile sensors (e.g., GelSight, DIGIT), pre-training on 460k+ optical gel-deformation images using masking objectives and self-distillation in pixel and latent spaces.

TouchFormer addresses a different sensing modality: non-optical, macro-scale physical pressure grids (64×27 force-resistive matrices, e.g., smart mattresses). Unlike optical tactile sensors which capture RGB images of gel deformation, these grids output low-resolution numerical force distributions across a large area. This work provides empirical evidence that transformer pretraining strategies — both ImageNet transfer and domain-specific MAE — are highly effective for this modality, complementing work like Sparsh rather than competing with it.

Model Architecture & Methodology

TouchFormer processes raw 2D tactile arrays as visual patches using Google's Vision Transformer (ViT-B/16). We conducted two independent experiments to evaluate distinct pretraining strategies: Experiment A uses ImageNet-pretrained weights via transfer learning, while Experiment B builds a custom Masked Autoencoder (MAE) trained from scratch on unlabeled pressure data.

Figure 2: The TouchFormer Architecture. Phase 1 (top): Self-supervised MAE pretraining masks 75% of the tactile map patches and reconstructs them using an encoder-decoder architecture. Phase 2 (bottom): For pose estimation, the encoder is initialized with pre-trained weights (MAE or ImageNet) and receives the entire, unmasked tactile map to regress 3D keypoints.

Masked Pretraining

By masking 75% of the pressure map inputs during pretraining, the model is optimized to reconstruct physical weight distributions, teaching it body shapes and weight distribution physics in a self-supervised manner.

Google ViT backbone

Leverages Google's 12-layer Vision Transformer (ViT-B/16). Initializing from ImageNet weights provides robust visual feature abstractions that accelerate downstream task learning on tactile grids.

Evaluation & Results

TouchFormer was evaluated on the 10GB PressurePose corpus. We analyzed different pretraining and weight initialization strategies to understand whether self-supervised pretraining on pressure maps from scratch compares to or outperforms standard ImageNet representations. Notably, due to academic compute limits, our custom MAE model (Experiment B) was only pre-trained for 20 epochs (~5 hours), yet it achieved highly competitive results close to the visual foundation model.

Model Configuration	Backbone Initialization	Pretraining Objective	MPJPE (cm) ↓	PCK @ 15cm (%) ↑
CNN Baseline (Clever et al., CVPR 2020)	Random Init	None (Supervised Only)	11.18	—
TouchFormer (Custom MAE - Exp B)	Random Init	MAE (Pressure Maps Only)	6.91	90.8%
TouchFormer (ImageNet Transfer - Exp A - Ours)	ImageNet-1k Weights	None (Direct Fine-Tuning)	6.29	92.8%

Table 1: Backbone pretraining comparison. Mean Per Joint Position Error (MPJPE) and Percentage of Correct Keypoints (PCK) calculated over the real test dataset partition.

Key Takeaways

1. Transfer Learning Benefits: Initializing the tactile encoder with weights from Google's ImageNet-1k trained ViT-B/16 (Experiment A) yields the best error rate of 6.29 cm MPJPE, showing that visual features transfer exceptionally well to physical contact matrices.

2. Self-Supervised Tactile Priors: Training purely on pressure maps from scratch using the Masked Autoencoder objective (Experiment B) yields a highly competitive 6.91 cm MPJPE, significantly outperforming the supervised CNN baseline (11.18 cm MPJPE) without using any labeled 3D keypoint data.

3. Compute & Pretraining Efficiency: Due to academic compute constraints, our custom MAE model (Experiment B) was pre-trained for only 20 epochs (~5 hours on a single GPU), compared to standard MAE schedules of 800–1600 epochs. Getting so close to the fully pre-trained ImageNet model with a fraction of the compute highlights the extreme sample efficiency of self-supervised learning for tactile grids.

Dataset & Baseline Acknowledgment

This project utilizes the PressurePose dataset and baseline methodologies from the following foundational work. If you build upon this project, please consider citing the original authors:

@inproceedings{clever2020bodies,
  title={Bodies at Rest: 3D Human Shape and Pose Estimation from a Pressure Image using Synthetic Data},
  author={Clever, Henry M and Erickson, Zackory and Kapusta, Ariel and Turk, Greg and Liu, C Karen and Kemp, Charles C},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}