MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fields

arXiv preprint

Paul Friedrich¹, Florentin Bieder¹, Julian McGinnis², Julia Wolleb³, Daniel Rueckert^2,4, Philippe C. Cattin¹

¹University of Basel ²Technical University of Munich

³Yale University ⁴Imperial College London

arXiv Code Video (soon) MedNF Datasets (soon)

Our proposed MedFuncta framework generalizes medical neural fields from single-instances to dataset-level neural representations. Single data points are encoded into a unified representation, namely a 1D latent vector, which enables consistent and modality-agnostic downstream processing.

Abstract

Research in medical imaging primarily focuses on discrete data representations that poorly scale with grid resolution and fail to capture the often continuous nature of the underlying signal. Neural Fields (NFs) offer a powerful alternative by modeling data as continuous functions. While single-instance NFs have successfully been applied in medical contexts, extending them to large-scale medical datasets remains an open challenge. We therefore introduce MedFuncta, a unified framework for large-scale NF training on diverse medical signals. Building on Functa, our approach encodes data into a unified representation, namely a 1D latent vector, that modulates a shared, meta-learned NF, enabling generalization across a dataset. We revisit common design choices, introducing a non-constant frequency parameter \(\omega\) in widely used SIREN activations, and establish a connection between this \(\omega\)-schedule and layer-wise learning rates, relating our findings to recent work in theoretical learning dynamics. We additionally introduce a scalable meta-learning strategy for shared network learning that employs sparse supervision during training, thereby reducing memory consumption and computational overhead while maintaining competitive performance. Finally, we evaluate MedFuncta across a diverse range of medical datasets and show how to solve relevant downstream tasks on our neural data representation. To promote further research in this direction, we release our code, model weights and the first large-scale dataset - MedNF - of > 500 k latents for multi-instance medical NFs.

Medical Data as Neural Functions

This work introduces MedFuncta, a framework that generalizes medical Neural Fields from isolated, single-instance models to dataset-level neural representations. The central idea is to meta-learn a shared neural representation across the dataset, in which each signal is represented by a unique, signal-specific parameter vector that conditions a shared network. This structure enables the model to capture and reuse redundancies across different signals, drastically improving computational efficiency and scalability. Unlike prior methods that rely on patch-based representations, our proposed framework represents each signal, from 1D time series to 3D volumetric data, with a single 1D latent vector. This abstraction enables consistent downstream processing across diverse data types, and is especially advantageous in medical applications, where the ability to unify multiple data modalities under a common representation is desirable, and where the inherent capability of Neural Fields to handle irregularly sampled, heterogeneous data provides further benefits.

The proposed network architecture with an \(\omega\)-schedule, that implicitly controls the layer's effective learning rates.

Architecture with \(\omega\)-schedule

We argue that most sets of signals (datasets) contain large amounts of redundant information or structure that we can learn over the entire set. This is particularly true in medicine, where patients exhibit broadly similar yet slightly varying anatomies. We therefore define a neural network \(f_{\theta,\phi^{(i)}}:\mathbb{R}^C\rightarrow\mathbb{R}^D\) with shared network parameters \(\theta\) that represent this redundant information and additional signal-specific parameters \(\phi^{(i)}\in\mathbb{R}^{P}\) that condition the base network to represent a specific signal \(s_i\). We apply a \(K\)-layer MLP architecture with a hidden dimension of \(L\) and FiLM modulated SIREN activations. While recent research treats SIRENs frequency parameter \(\omega\) as a single hyperparameter that remains constant over all network layers, we identify this as a main restriction when being applied in a generalization setting. Following recent work demonstrating that shallow network layers learn the low-frequency content of a signal and deeper layers add more and more high-frequency information, we introduce an \(\omega\)-schedule that linearly increases from \(\omega_{1}\) to \(\omega_{K}\). We find that this is equivalent to a layer-wise learning rate schedule, which positively influences the network's learning dynamics. For further information on the relationship between an SIREN layer's \(\omega\)-parameter and it's effective learning rate, we refer to our paper.

An ablation study demonstrating the effect of our proposed \(\omega\)-schedule, with \(\omega_1\) and \(\omega_K=\omega_1\delta\).

Meta-Learning Shared Model Parameters (with Context Reduction)

To efficiently create a set of NFs, we aim to meta-learn the shared parameters \(\theta\) so that we can fit a signal \(s_i\) by only optimizing \(\phi^{(i)}\) for very few update steps. We follow a CAVIA meta-learning approach. Before we take a single meta-update step with respect to the shared parameters \(\theta\), we update \(\phi\) for \(G\) inner-loop update steps. Since performing a single meta-update step requires backpropagating through the entire inner-loop optimization, the computational graph must be retained in GPU memory to compute the required second-order gradients. This is a resource-intensive task that does not scale well to high-dimensional signals. While first-order approximations or auto-decoder training approaches that do not rely on second-order optimization exist, recent research has shown that this results in severe performance drops or unstable training.

(Left) The proposed meta-learning framework with context reduction. (Right) The test-time optimization process after meta-learning.

To circumvent this issue, we present a context-reduction approach that reduces the amount of pixel-value pairs used to optimize \(\phi\) in the inner-loop. We randomly sample \(\gamma|\mathcal{C}|\) coordinate-value pairs from the full context set \(\mathcal{C}\). This significantly reduces the computational burden, while resulting in marginal perfromance drops.

An ablation study demonstrating the effectiveness of our proposed context reduction scheme.

Test-Time Adaptation

Given the meta-learned model parameters \(\theta^{*}\), we fit a Neural Field to each signal \(s_1, ..., s_N\). We start with initializing the signal-specific parameters \(\phi^{(i)}:=\mathbf{0}\) and optimize \(\phi^{(i)}\) for \(H\) steps. As no second-order optimization is required at test-time, we can make use of the full context set. A set of NFs representing the signals \(s_1, ..., s_N\) is therefore defined by the network architecture, the shared model parameters \(\theta^{*}\), and the signal-specific parameters \(\phi^{(1)}, ..., \phi^{(N)}\).

Learning on this Representation

To assess whether the learned representation captures relevant information about the underlying signal, we perform classification experiments on the signal-specific parameters \(\phi\), using a \(k\)-Nearest-Neighbor (\(k\)-NN) classifier, or a 3-layer MLP with ReLU activations and dropout.

We find that solving the two classification tasks (binary and multi-class) on our proposed representation \(\phi\) generally works well. We outperform both ResNet50 and EfficientNet-B0, applied to the original images, in terms of accuracy and can demonstrate competitive F1 scores, while requiring less training time and model parameters.

Reconstruction Results

A Comparison to Functa

A visual comparison to Functa, as well as PSNR (left) and SSIM (right) curves for the validation performance throughout a training run.

BibTeX

      
@article{friedrich2025medfuncta,
         title={MedFuncta: A Unified Framework for Learning Efficient Medical Neural Fields},
         author={Friedrich, Paul and Bieder, Florentin and McGinnis, Julian and Wolleb, Julia and Rueckert, Daniel and Cattin, Philippe C},
         journal={arXiv preprint arXiv:2502.14401},
         year={2025}
        }

Acknowledgements

This work was financially supported by the Werner Siemens Foundation through the MIRACLE II project. JM is supported by Bavarian State Ministry for Science and Art (Collaborative Bilateral Research Program Bavaria – Québec: AI in medicine, grant F.4-V0134.K5.1/86/34).