Candidate Set Query for AL

Abstract

This paper introduces a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query. Unlike traditional AL queries requiring the oracle to examine all possible classes, our method narrows down the set of candidate classes likely to include the ground-truth class, significantly reducing the search space and labeling cost. Moreover, we leverage conformal prediction to dynamically generate small yet reliable candidate sets, adapting to model enhancement over successive AL rounds. To this end, we introduce an acquisition function designed to prioritize data points that offer high information gain at lower cost. Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet64x64 demonstrate the effectiveness and scalability of our framework. Notably, it reduces labeling cost by 42% on ImageNet64x64.

Motivation

Figure 1. (left) Comparison between the conventional query (CQ) and the proposed candidate set query (CSQ). By narrowing the size of candidate set, CSQ significantly reduces the labeling time. (right) The result of our user study. Narrowing the search space of annotators not only reduces the annotation cost (i.e., labeling time), but also improves the annotation quality (i.e., accuracy). This potentially results in a better dataset construction.

Active Learning

Figure 2. Active learning aims at reducing the labeling burden by annotating only necessary samples until the desired model performance is achieved. From a small initially labeled dataset, it repeatedly selects the most informative samples and asks the oracle to label them. Traditional active learning focuses on the selection of informative samples, driven by an acquisition function, whereas our work focuses both on the design of an effective acquisition function (selection) and the process of getting the selected samples labeled (query).

Experiment Results

Main Results

Figure 3. Accuracy (%) versus relative labeling cost (%) for conventional query (CQ) and the proposed candidate set query (CSQ) combined with different sampling strategies. CSQ strategies combined with the proposed cost-efficient sampling (blue lines) consistently outperform CQ baselines (red lines) by a significant margin across various budgets, acquisition functions, and datasets. You can find the zoomed-in version in the paper.

Ablation Studies

Figure 4. (a) Contribution of each component of our method, measured by accuracy (%) versus relative labeling cost (%) (left), and relative labeling cost (%) versus AL round (right). All components of our method lead to steady performance improvement over successive AL rounds. (b) Relative labeling cost (%) at fifth round with varying calibraiton set sizes $n_\text{cal}$. Our method shows robust performance with varying calibraiton set sizes.

Figure 5. Impact of the candidate set design. We compare Conventional (all classes), Top1 (top-1 most confident class), Top10 (top10 most confident classes), and Conformal(our method with fixed $\alpha=0.1$), and Oracle(the smallest candidate set that contains the ground-truth class). Our method reliably surpasses the baselines and delivers a substantial reduction in labeling cost compared to the baselines. We note that Oracle is an unattainable upper bound requiring knowledge of the ground truth. Also, our candidate set effectively includes the ground-truth class in over 90% of cases, even when the model accuracy is low.

Qualitative Results

Figure 6. Example results of constructed candidate sets along with input images on ImageNet64x64. The ground-truth class is highlighted in red (best viewed in color). For easy samples, the candidate set is small to minimize the labeling cost, while for hard samples, the candidate set is expanded to include the ground-truth class.

BibTeX

@article{gwon2025enhancing,
    title={Enhancing Cost Efficiency in Active Learning with Candidate Set Query},
    author={Gwon, Yeho and Hwang, Sehyun and Kim, Hoyoung and Ok, Jungseul and Kwak, Suha},
    journal={arXiv preprint arXiv:2502.06209},
    year={2025},
    primaryClass={cs.LG},
    url={https://arxiv.org/abs/2502.06209}
}