The Phantom Menace in Crypto-Based PET-Hardened Deep Learning Models : Invisible Configuration-Induced Attacks

The increasing use of deep learning (DL) models has given rise to significant privacy concerns regarding training and inference data. To address these concerns, the community has increasingly adopted crypto-based privacy-enhancing technologies (CPET) like homomorphic encryption (HE), secure multi-party computation (MPC), and zero-knowledge proofs (ZKP). The integration of CPET with DL, often referred to as CPET-DL, is commonly facilitated by specialized frameworks like CrypTen, TenSEAL, and EZKL. These frameworks offer configurable parameters to balance model accuracy and computational efficiency during privacy-preserving operations. However, these configurations, while seemingly harmless, can introduce subtle vulnerabilities. The stealthy attacks induced by misconfigurations are hard to detect because 1) the plaintext models remain vulnerability-free, and 2) existing auditing tools are hardly applicable to CPET-hardened models. This creates a paradox: tools intended to protect privacy can be undermined through configuration manipulation. We present ConPETro, the first attack on CPET-hardened models by manipulating the CPET-DL framework configurations. We show that well-crafted configurations allow attackers to create CPET-hardened models that function similarly to benign plaintext models under normal inputs, but exhibit significantly reduced robustness for malicious inputs embedded with triggers. ConPETro strategically selects triggers to maximize behavioral deviations with benign models and uses gradient consistency to guide configuration exploration, effectively finding malicious configurations that bypass standard plaintext model auditing. Evaluations across three mainstream CPET-DL frameworks (HE, MPC, and ZKP) demonstrate ConPETro's effectiveness in both semantic and non-semantic triggers. ConPETro achieves an average maximum attack success rate (ASR) of 72.27% in CPET-hardened models with non-semantic triggers; the accuracy only drops by 4%, thus maintaining stealthiness. It also achieves a maximum ASR of 94.74% with semantic triggers across three datasets. We also demonstrate that our stealthy attacks can bypass advanced defense and detection tools.