WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation
Published in arXiv preprint arXiv:2609.12171, 2026
We introduce an automated pipeline for constructing synthetic datasets of workplace emails that reflect realistic enterprise scenarios, paired with both long-form and concise questions and reference answers grounded in the underlying data. The pipeline simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. Evaluating standard agentic baselines built on strong language models, we find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating substantial room for improvement in enterprise question-answering systems.
Keywords: Enterprise Question Answering, Synthetic Data Generation, Agentic Evaluation, Information Retrieval, Large Language Models
Recommended citation: Varhade, A., Sutradhar, A., Krishnaswamy, R., & Goyal, N. (2026). "WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation." arXiv preprint arXiv:2609.12171.
Download Paper