Beyond the Prompt: Assessing Domain Knowledge Strategies for High-Dimensional LLM Optimization in Software Engineering
This program is tentative and subject to change.
Background/Context: Large Language Models (LLMs) demon- strate strong performance on low-dimensional software engineer- ing optimization tasks (≤11 features) but consistently underperform on high-dimensional problems where Bayesian methods dominate. A fundamental gap exists in understanding how systematic inte- gration of domain knowledge (whether from humans or automated reasoning) can bridge this divide. Objective/Aim: We compare human versus artificial intelli- gence strategies for generating domain knowledge. We systemati- cally evaluate four distinct architectures to determine if structured knowledge integration enables LLMs to generate effective warm starts for high-dimensional optimization. Method: We evaluate four approaches on MOOT1 datasets strat- ified by dimensionality: (1) Human-in-the-Loop Domain Knowledge Prompting (H-DKP), utilizing asynchronous expert feedback loops; (2) Adaptive Multi-Stage Prompting (AMP), implementing sequen- tial constraint identification and validation; (3) Dimension-Aware Progressive Refinement (DAPR), conducting optimization in pro- gressively expanding feature subspaces; and (4) Hybrid Knowledge- Model Approach (HKMA), synthesizing statistical scouting (TPE) with RAG-enhanced prompting. Performance is quantified via Chebyshev dis- tance to optimal solutions and ranked using Scott-Knott clustering against random and Bayesian (UCB_GPM) baselines. Note that all human studies conducted as part of this study will comply with the policies of our local Institutional Review Board.