All publications

Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture

Robert Metzger, Mattes Ohlenbusch, Christian Rollwage and Simon Doclo

16th ITG Conference on Speech Communication
Berlin, Germany · 24–26 September 2025

How can smaller speech-enhancement models learn from a larger model? A comparison of five knowledge-distillation methods, and the trade-off between model size and speech quality.

The question

Neural networks can reduce background noise in microphone recordings, but large models are difficult to deploy on devices with limited resources. Simply shrinking a model often reduces speech quality. We investigated whether knowledge distillation — training a smaller “student” with guidance from a larger “teacher” — can soften that trade-off.

What we compared

We used the Frequency-Time Joint Non-linear Filter (FT-JNF) architecture with a 1.4-million-parameter teacher. Five methods transferred information from the output mask, the linear layer, the frequency-processing LSTM, the time-processing LSTM, or a combination of layers. Self-similarity matrices allowed us to compare internal representations even when teacher and student had different numbers of hidden units.

Students first learned from the teacher, then trained against clean speech. We compared the methods at approximately 44,000 parameters and subsequently tested the three effective methods across student sizes from approximately 13,000 to 365,000 parameters.

What we found

  • The transfer point mattered. Distillation from the linear layer, time-processing LSTM, or combined layers improved PESQ over a same-size student trained without distillation. Mask matching performed similarly to the baseline; frequency-LSTM-only distillation performed worse.
  • About a quarter of the parameters could be enough. At 0 dB signal-to-noise ratio, the largest distilled students achieved PESQ scores comparable to the teacher with roughly 25% of its parameters.
  • Further compression involved a trade-off. The paper reports a 96% reduction in model size with a decrease of about 0.1 PESQ relative to the teacher. The smallest tested students did not benefit from distillation over their non-distilled counterparts.

What these results mean — and what they do not

For this architecture, later-layer guidance was more useful than copying the earliest recurrent representation. Distillation helped preserve quality as models became smaller, but did not eliminate the limits of model capacity.

The evaluation used simulated, reverberant five-microphone recordings with German speech and directional noise. Method comparisons covered −5 to 15 dB SNR; model scaling was evaluated at 0 dB. PESQ is an objective speech-quality metric, not a listener study. These results do not by themselves demonstrate real-time operation, battery savings or performance on a particular physical device.

This overview summarises the paper, especially Sections 4–6 and Figures 3–4. Please consult the original for the full methods and results.

Reference

Robert Metzger, Mattes Ohlenbusch, Christian Rollwage and Simon Doclo. Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture. 16th ITG Conference on Speech Communication, 2025.

Find the conference paper on IEEE Xplore or through VDE. The freely accessible preprint is available on arXiv.