\section{Analysis}
\label{sec:analysis}


\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{images/owl-exp.pdf}
  \caption{\textit{Left}: Average scores on GAIA validation categorized by different agent capabilities. \textcolor{blue}{Blue} compares different agent frameworks, while \textcolor[RGB]{190,184,181}{brown} compares different planner models. \textit{Right}: Changes in Workforce performance as the number of replanning iterations increases.}
  \label{fig:owl-capabiltiy—ttt}
\end{figure}

% \subsection{Performance Across Capability Types}
\noindent \textit{\textbf{Performance Across Capability Types.}}
% To evaluate model performance across different agent capabilities, we categorized evaluation cases into five types. 
Each evaluation case may require one or more types of agent capabilities (e.g., Web Browsing, Coding, Multimodal Processing). 
% We analyze OWL's performance across these dimensions and observe consistent improvements across all categories, highlighting the effectiveness of our approach regardless of the specific capability requirements.
As shown in Figure~\ref{fig:owl-capabiltiy—ttt}(a), our experimental results demonstrate that:
\textit{(i)} The \workforce approach consistently outperforms both Role Playing and Single Agent approaches across different capabilities.
\textit{(ii)} After \shortowl training, we observe consistent improvements across all capability types.

% \subsection{}
\noindent \textit{\textbf{Test-time Scaling.}} 
As discussed in Section~\ref{sec:workforce}, we introduced a \textit{replanning} mechanism, which enables \textit{test-time scaling} in multi-agent systems.
As shown in Figure~\ref{fig:owl-capabiltiy—ttt}(b), both GPT-4o and Claude-3.7-Sonnet workforces demonstrate improved performance as the number of replanning iterations increases.
Notably, the models have no access to ground truth in this process, which indicates that the workforce possesses inherent self-correction and self-evolving capabilities during test time.

\noindent \textit{\textbf{Robustness Across Capability Requirements.}}
We analyzed robustness by grouping tasks based on required capabilities (1, 2, or $\geq$ 3). Figure~\ref{fig:error_analysis_training_configuration_cap_requrements}(b) shows that baselines degrade significantly on multi-capability tasks (Role Playing drops from 62.3\% to 34.6\%), while \workforce maintains consistent performance across all complexity levels. This stability comes from our modular design, where specialized workers handle focused subtasks, making complex tasks more manageable than when a single agent must handle everything.
Other robustness metrics can be found in Appendix~\ref{app:robustness_details}.

\begin{figure}[tb]
    \centering
    \includegraphics[width=\linewidth]{images/error_analysis_and_ablation_study.pdf}
    \caption{Manual error analysis and performance comparison of our multi-agent system.}
    \label{fig:error_analysis_training_configuration_cap_requrements}
    % \vspace{-0.5cm}
\end{figure}

% \subsection{Whole Trajectory Data vs. Planner Only}

\noindent \textit{\textbf{Planner vs. Worker Training.}}
Our ablation study in Figure~\ref{fig:error_analysis_training_configuration_cap_requrements}(c) reveals that training only the planner (45.45\%) significantly outperforms training only workers (31.51\%). 
Training both components together offers minimal additional gains (46.68\%) while substantially increasing computational costs. 
This confirms our design choice to prioritize planner optimization, as effective task decomposition is more crucial than enhancing individual worker capabilities. 
% With a well-trained planner, even base worker models can achieve strong performance, providing a more efficient approach to system improvement.

\noindent \textit{\textbf{Error Analysis.}}
To deeply investigate the failure modes of \workforce, we conducted a manual error analysis on Claude-3.7-Sonnet results.
As shown in Table~\ref{tab:error-hierarchy}, approximately half of the errors stem from limitations in the foundation models themself or from tool-related issues.
Among agent-specific errors, the highest proportion is attributed to planner failures (21.15\%), further highlighting the significance of planner optimization.
% In 21.15\% of failed cases, the error occurred because the planner did not produce correct planning.
% This further demonstrates the critical importance of optimizing the planner for improving the overall multi-agent system performance.
Examples for each error category and detailed error distribution can be found in Appendix~\ref{app:error_analysis}. We also provide more qualitative analysis in Appendix~\ref{app:case_studies}.

