Although Domain Generation Algorithm (DGA) detectors often achieve
strong performance under closed-set evaluation, deployed systems must
also handle previously unseen DGA families and heterogeneous domains
that differ substantially from the training distribution. Existing
evaluations, however, rarely study these two forms of novelty under a
common protocol. To address this gap, we introduce
\textbf{OpenSetDGA}, a benchmark of 549,999 normalized domain names
with predefined splits for known-class classification, unseen-family
out-of-distribution (OOD) detection using five held-out DGA families,
and real-world OOD detection using 100,000 malicious and uncommon
legitimate domains from multiple public sources. We evaluate
feature-based LightGBM classifiers and character-level CNN and BiLSTM
models with multiple OOD scoring methods over five independent runs.
On unseen-family OOD, BiLSTM with ReAct achieves the highest mean
AUROC of $0.8636 \pm 0.0116$, whereas BiLSTM with KNN-k5 performs
best on real-world OOD with an AUROC of $0.7224 \pm 0.0021$. This
contrast shows that family-level novelty and heterogeneous real-world
OOD pose distinct detection challenges. We further identify a
\emph{benign-overfit} failure mode in which uncommon legitimate
domains and domains from several malicious sources receive high benign
confidence, highlighting limitations of benign distributions learned
primarily from popular-domain data.