LenslessMic

Code arXiv Hugging Face Collection

With society’s increasing reliance on digital data sharing, the protection of sensitive information has become critical. Encryption serves as one of the privacy-preserving methods; however, its realization in the audio domain predominantly relies on signal processing or software methods embedded into hardware. In this paper, we introduce LenslessMic, a hybrid optical hardware-based encryption method that utilizes a lensless camera as a physical layer of security applicable to multiple types of audio. We show that LenslessMic enables robust authentication of audio recordings and encryption strength greater than or equal to that of AES-256 with high-quality signals and minimal loss of content information. The approach is validated with a low-cost Raspberry Pi prototype and is open sourced together with datasets to facilitate research in the area.

Demo Samples

Below we show example audio from the collected Librispeech and SongDescriber datasets for different models: Learned (with different \(g, r\) variations), R-Learned, No-PSF, and ADMM-100.

All models and train/test datasets are published on Huggingface Collection.

Audio Reconstruction

Below we provide recordings from test-clean set of Librispeech. We recommend listening to the ground-truth and codec versions only after listening to the reconstructions to avoid phonetic-restoration effect, i.e. unintelligible audio may become meaningful after you know what is the actual content of the speech. Methods with \(g \ge 2\) increase LenslessMic robustness to ensure that this effect will not allow hearing speech content from No-PSF reconstructions.

Learned and Codec recordings sound identical, showcasing high quality reconstruction abilities of LenslessMic. Since it may be possible to understand some part of the speech content for No-PSF, we provide LenslessMic variants with improved robustness that operate on group frames. The examples are provided below. \(g=3\) case provides the best balance between security robustness and reconstruction quality.

We test whether LenslessMic is applicable to other neural audio codecs and collect a dataset with X-codec instead of DAC. The results are presented below. While some utterances sound almost as good as DAC-ones, there are several recordings that have severe reverberation-like or filtering-like effect that decreases intelligibility. In general, LenslessMic is applicable on other codecs and even in cross-codec scenario, however, training on the actual data is required for high-quality reconstruction. R-Learned achieves better quality because it is trained on test-random data and is not overfitted towards any single codec.

Besides, LenslessMic works on music data too (we downsampled dataset to 16kHz, so some instruments are a bit distorted):

Audio Encryption and Authentication

LenslessMic reconstruction results in noise if PSF is wrong. This is a core property behind LenslessMic authentication robustness and accuracy. Below you can find examples:

LenslessMic encryption strength is greater than or equal to that of AES-256. Below we provide audio samples for different \(W\), i.e., the ratio of correctly determined pixels in the PSF. \(W=7\%\) and \(W=4\%\) correspond to AES-256 and AES-128 strength, respectively.

Results for \(g=3\):

Results for \(g=2\):

\(g=3\) is more robust, however, leads to slightly worse reconstruction quality.

Captured Video Representation

The video below shows how reconstruction of the same frame enhances from \(W=0\%\) to \(W=100\%\):

Extra Experiments