Researchers propose DS-Lighting, a toolkit to make agent harnesses explicit for data-science automation

LLM-based agents have demonstrated potential in automating end-to-end data-science workflows, but their performance is heavily dependent on the agent harness—the system component that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback.

Existing data-science agents often leave this harness implicit, which makes it difficult to reproduce results, compare performance across heterogeneous tasks, and attribute outcomes reliably.

To address this, researchers introduce DS-Lighting, a unified harness toolkit designed to make harness design explicit for data-science automation.

DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation. This modular design supports both predefined pipelines and adaptive search strategies by representing diverse agents as executable operator programs.

The toolkit further integrates multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparisons under a shared task interface, sandboxed runtime, and standardized metric protocol.

In experiments spanning agents, harnesses, models, and ablations, the researchers report that explicit harness design improves reproducibility, comparability, and reliability while reducing avoidable system-level failures in end-to-end data-science workflows.

The DS-Lighting codebase is publicly available at https://github.com/usail-hkust/dslighting.