Formulate a question that evidence can answer

A research question should emerge from a concrete uncertainty. "Make agents better" is too broad to falsify. "Does evidence-aware routing reduce unsupported answers on unseen support tickets under the same model-call budget?" identifies a mechanism, an outcome, a population, and a resource constraint. State what evidence would count against the proposed explanation before seeing the final results.

Break the question into an intervention and a claim. The intervention might add a rule that requests another retrieval when evidence coverage is low. The claim might predict improved factual support without exceeding two model calls per ticket. The engineering system can work correctly even if the scientific claim fails. Keeping those outcomes separate allows a negative result to teach something: perhaps the coverage signal is noisy, the baseline already retrieves enough, or the added call changes latency without improving the relevant cases.

An evaluation needs a unit of analysis. Multiple turns from one conversation or paraphrases of one ticket are correlated and should not be treated as independent evidence without justification. If deployment concerns complete support resolutions, a token-level or answer-level metric may miss failed handoffs or duplicated actions. Define whether the unit is a request, episode, user, document, or organization, then align sampling and uncertainty estimates with that unit.

Specify the population the result is intended to describe. A curated set of English database questions does not establish performance on multilingual customer support. A synthetic fixture can test a causal mechanism under controlled conditions while still having limited external validity. Write the sampling procedure, inclusion criteria, and known exclusions. These details determine how far the conclusion can travel, often more than the number of decimal places in the reported score.

Build a related-work search around several formulations: the task, the suspected failure, the proposed mechanism, and the evaluation method. A retrieval-routing idea may connect to selective prediction, budgeted inference, active information acquisition, and confidence calibration even if those papers never use the phrase agentic system. Read original papers and official implementation documentation where available. Use surveys to locate candidates, then verify the claims you rely on against their primary sources.

Create a comparison table with problem setting, available information, baseline, resource budget, evaluation unit, and limitations. Follow references backward to the method's origins and citations forward to replications or critiques. Save the search date and queries because the literature changes. A title match or abstract summary does not establish equivalence: inspect the actual assumptions and experimental setup before saying that prior work does or does not solve your problem.

Useful contributions can include a new method, a better measurement, an analysis that explains a failure, a reproducible comparison under a missing constraint, or a replication that reveals where a result does not transfer. Combining existing components can be valuable engineering without automatically constituting a new scientific method. State the difference at the level you can support, and avoid claims such as first or universal unless the necessary evidence has actually been established.

Suppose an existing router uses token uncertainty, while your proposed router uses missing citations. The contribution is not merely that both are wrapped in a new interface. A testable difference is whether citation coverage predicts unsupported claims under a fixed retrieval budget. Compare to the strongest relevant simple alternative, perhaps always retrieving once more. If that baseline achieves the same outcome at lower complexity, the experiment has identified a limitation of the proposed mechanism.

Write an experiment card with the hypothesis, primary metric, important guardrails, baseline, data split, tuning budget, stop rule, and falsifier. An invented plan might require a positive paired change in supported-answer rate while allowing no increase in unauthorized action acceptance and no more than a specified latency budget. These are proposed criteria, not achieved results. If exploratory analysis suggests a better metric, document the change and evaluate it on fresh data when a confirmatory claim is intended.

Start with a small pilot to find broken instrumentation, ambiguous labels, or impossible resource assumptions. Use pilot data for design decisions, then separate it from final evaluation. Practical methodology guidance emphasizes starting from a workable baseline; the original value in your project comes from the question and the disciplined comparison. An experiment that can clearly tell you no is usually more informative than one whose success definition can change after every run.

The card validator checks a few structural research requirements. It cannot judge whether the hypothesis is important, whether the metric is valid, or whether related examples share hidden lineage. Add group-level checks and human review of the falsifier for a real study.