Identifying Suspected Mislabeled Apps in Google Play Application Removal Prediction: An Empirical Comparison of Label Noise Detection Methods
Researchers compare three label-noise detectors on 870,514 Google Play apps used to predict removals.
A cs.CR paper studies label noise in models that predict Google Play app removals, where both voluntary withdrawals and policy takedowns are labeled removed and uncaught spam can remain labeled stable. Isolation Forest, Neighborhood Disagreement, and Prediction Inconsistency were applied to 870,514 apps from a 2022 dataset. Their overlap, 7,598 apps at default settings, is treated as the strongest mislabeling set, but removing flagged apps never beat the baseline and performance worsened as more were removed. A model trained on 3,021 trainable overlap apps reached a test AUC of 0.2518, and VirusTotal and Quark Engine confirmation found flagged apps underrepresented among confirmed removals.
- Store presence labels do not record why an app disappeared.
- Three detectors overlap on 7,598 candidate mislabeled apps.
- Removing flagged apps does not beat the baseline model.
- An overlap-only model scores test AUC 0.2518, below chance.
Full article298 words · extracted from arxiv.org · click to collapse
Models that predict which Google Play apps will be removed are trained on labels that record only whether an app was still in the store at a later observation. A disappeared app is labeled removed and a present one stable, but neither records why. A voluntary withdrawal and a policy takedown both produce removed, and an uncaught spam app keeps stable. This work calls that mismatch label noise. Three detectors from different methodological families are applied to the 870,514 apps of Mohsen, Karastoyanova, and Azzopardi (2022): Isolation Forest, flagging apps unusual in the feature space, Neighborhood Disagreement, flagging apps whose nearest neighbors carry the opposite label, and Prediction Inconsistency, flagging apps a classifier labels differently from the data. The apps flagged by all three, the overlap, number 7,598 at default settings and are the strongest mislabeling candidates. Two questions follow. First, does removing flagged apps improve the model? It does not. No detector, overlap, or union beats the baseline, and the loss grows with the number removed. Second, do flagged apps appear less often than expected among apps whose label VirusTotal and Quark Engine confirm? Among confirmed removals they do, falling to 0.43 times the expected rate as the threshold tightens, while an excess on the stable side disappears once the age of the scanned apps is accounted for. A model trained on only the 3,021 trainable overlap apps reaches a test AUC of 0.2518, far below chance, so the relationship between features and labels there runs opposite to the rest of the data. The flagged apps run wrong in both directions: abandoned apps that resemble spam carry stable, while apps that look healthy carry removed. The value of the detectors lies in characterizing this label noise. They locate a small set of candidates they cannot profitably remove.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.25487