\section{Contributions}

\noindent \textbf{Paper Writing.}
Mengkang Hu and Yuhang Zhou finished most part of the paper. 
Mengkang Hu, Yuhang Zhou, Bowei Xia, and Yuzhou Nie finished the appendix part. 
Mengkang Hu and Ziyu Ye collaborated on the related work part. 
Yingru Li and Ziyu Ye helped refining the abstract and introduction sections for multiple rounds.
Yingru Li, Ziyu Ye, and Qiguang Chen carefully reviewed the paper and gave helpful feedback. 

\noindent \textbf{Experiments.}
Mengkang Hu, Yuhang Zhou, and Bowei Xia conducted experiments of Table~\ref{tab:main-results} (including baseline experiment).
Mengkang Hu and Yuhang Zhou conducted analysis experiments.
Mengkang Hu, Yuhang Zhou, Yuzhou Nie, and Ziyu Ye conducted experiments of Table~\ref{tab:planner_comparison}, including model training and evaluation.

\noindent \textbf{Dataset Construction.}
Mengkang Hu designed the data collection schema and pipeline. 
Mengkang Hu, Yuhang Zhou, Bowei Xia, Tao Sun, Yuzhou Nie, and Zeyu Zhang collaborated on data collection, with each one responsible for an individual part of the dataset.

\noindent \textbf{Code Implementation.}
Mengkang Hu and Yuhang Zhou implemented the code base of the \shortowl project and experiment pipeline based on the CAMEL~\citep{camel} framework. The agent infrastructure by CAMEL sped up the implementation.
Yuzhou Nie, Mengkang Hu and Yuhang Zhou collaborated on the implementation of the model training pipeline. 
Mengkang Hu, Yuhang Zhou, and Yuzhou Nie designed and implemented the data collection pipeline, including SFT and DPO datasets.
Mengkang Hu and Yuhang Zhou designed and implemented the essential toolkits for generalist ai assistants, including a browser toolkit, document processing toolkit, multimodal analysis toolkit (for audio, video, and image), excel toolkit, and so on.
Qianshuo Ye helped with refining the multimodal analysis toolkit.
Zhaoxuan Jin implemented the initial version of the workforce. Mengkang Hu and Yuhang Zhou collaborated on refinement (including adding new features and debugging), instantiation, and implementation of the general multi-agent assistant.

\noindent \textbf{Open-source Maintenance.}
Wendong Fan (Lead), Mengkang Hu (Co-Lead), Yifeng Wang, Tao Sun, Qianshuo Ye, Yuhang Zhou, Bowei Xia, Guohao Li, and other open-source community members helped maintain the open-source implementation of \shortowl. 
Their efforts in improving code stability, adding auxiliary features, and managing community feedback have helped enhance the usability and visibility of the project. While the research, design, and writing of this paper were conducted independently, the continued maintenance work by the team played a supportive role in sustaining the project's public engagement.

\noindent \textbf{Error Analysis and case study.}
Mengkang Hu, Yuhang Zhou, Bowei Xia, and Guohao Li designed and conducted the error analysis, including the evaluation results of role playing and workforce.

\noindent \textbf{Methodology Proposal.}
The initial ideation of the Workforce framework was led by Guohao Li.
The Optimized Workforce Learning approach was jointly proposed by Guohao Li and Mengkang Hu. Yuzhou Nie, Yuhang Zhou, Ziyu Ye, and Yingru Li also participated in the discussion regarding the methodology proposal.

\noindent \textbf{Project Leadership.}
The overall direction of the project was led by Guohao Li and Mengkang Hu.
Guohao Li also supervised the coordination and integration of all research and engineering components.
Mengkang Hu led the implementation and experimental evaluation of the research project.

\section{Limitations} \label{sec:limitation}

Despite our modular architecture's plug-and-play capabilities, performance remains dependent on the availability of high-quality domain-specific tools. In domains lacking reliable toolkits, the system may face execution bottlenecks. Additionally, reinforcement learning from real-world feedback is time-consuming due to latency issues (such as delays in online searches).


% \section{Preliminary}


\section{Formalization}
\label{app:formalization}

\subsection{Large Language Model-based Agents}

Formally, an LLM-based agent can be defined as a tuple $\mathcal{A} = (M, \mathcal{O}, \mathcal{T}, \pi)$, 
where $M$ is the underlying model, $\mathcal{O}$ represents the observation space, $\mathcal{T}$ is the set 
of available tools, and $\pi: \mathcal{O} \rightarrow \mathcal{T}$ is the policy function. 
The agent operates in a perception-reasoning-action loop, where at each step $t$, it receives an observation 
$o_t \in \mathcal{O}$, processes it through the language model $M$ to determine the appropriate action $a_t 
= \pi(o_t) \in \mathcal{T}$, and executes this action to achieve its goals.
Multi-agent systems extend this paradigm by enabling multiple LLM-based agents to collaborate. A multi-agent 
system can be represented as $\mathcal{S} = \{\mathcal{A}_1, \mathcal{A}_2, ..., \mathcal{A}_n, \mathcal{C}\}
$, where each $\mathcal{A}_i$ is an individual agent and $\mathcal{C}$ defines the communication protocol 
between agents. Frameworks such as CAMEL~\citep{camel} and MetaGPT~\citep{metagpt} have demonstrated that 
Collaborative multi-agent approaches can outperform single-agent systems on complex tasks requiring diverse 
expertise.

% \subsection{MDP Environment}
% To rigorously characterize the interaction process, we model the environment as a discrete-time Markov Decision Process (MDP), defined by the tuple
% \[
%   \mathcal{E} = (\mathcal{S}, \mathcal{T}, P, R, \gamma),
% \]
% where:
% \begin{itemize}
%   \item $\mathcal{S}$ is the \textbf{state space}: $s_t \in \mathcal{S}$ denotes the complete environment state at time $t$, which may include the world state, the agent's internal state, historical context, etc.
%   \item $\mathcal{T}$ is the \textbf{action space}: each action $a_t \in \mathcal{T}$ corresponds to invoking a particular tool or executing a function available to the agent.
%   \item $P: \mathcal{S} \times \mathcal{T} \times \mathcal{S} \rightarrow [0,1]$ is the \textbf{state transition probability function}: $P(s_{t+1} \mid s_t, a_t)$ gives the probability of the environment transitioning to state $s_{t+1}$ given the current state $s_t$ and action $a_t$.
%   \item $R: \mathcal{S} \times \mathcal{T} \rightarrow \mathbb{R}$ is the \textbf{reward function}: $R(s_t, a_t)$ denotes the immediate scalar reward received after the agent executes action $a_t$ in state $s_t$.
%   \item $\gamma \in [0,1)$ is the \textbf{discount factor}: it determines the relative weighting of immediate and future rewards when computing the total expected return.
% \end{itemize}

% In many practical scenarios, the agent does not have access to the full environment state $s_t$, but instead receives a partial or processed observation. To capture this, we define an \textbf{observation function}:
% \[
%   o_t = \phi(s_t), \qquad \phi: \mathcal{S} \rightarrow \mathcal{O},
% \]
% where $o_t \in \mathcal{O}$ is the observation available to the agent at time $t$. The agent's decision-making is thus based solely on the observation sequence $(o_1, o_2, \ldots, o_t)$.

% The agent's policy $\pi$ is a mapping from observations to actions:
% \[
%   \pi: \mathcal{O} \rightarrow \mathcal{T}.
% \]
% At each timestep, the agent chooses an action according to its policy, i.e., $a_t = \pi(o_t)$. The process then unfolds according to the following loop:
% \begin{enumerate}
%   \item The environment is in state $s_t$;
%   \item The agent receives observation $o_t = \phi(s_t)$;
%   \item The agent selects action $a_t = \pi(o_t)$;
%   \item The environment transitions to $s_{t+1} \sim P(\cdot \mid s_t, a_t)$ and emits reward $r_t = R(s_t, a_t)$.
% \end{enumerate}

% The overall objective of the agent is to learn a policy $\pi$ that maximizes the expected cumulative discounted reward:
% \[
%   J(\pi) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R(s_t, a_t)\right],
%   \qquad \text{where}\;\; a_t = \pi(o_t),\; s_{t+1} \sim P(\cdot \mid s_t, a_t).
% \]

\section{More Details on Workforce}
\label{app:workforce_details}

\subsection{Workforce processing flow}
\label{app:workforce processing flow}

\makeatletter
\newenvironment{breakablealgorithm}
  {% \begin{breakablealgorithm}
   \begin{center}
     \refstepcounter{algorithm}% New algorithm
     \hrule height.8pt depth0pt \kern2pt% \@fs@pre for \@fs@ruled
     \renewcommand{\caption}[2][\relax]{% Make a new \caption
       {\raggedright\textbf{\ALG@name~\thealgorithm} ##2\par}%
       \ifx\relax##1\relax % #1 is \relax
         \addcontentsline{loa}{algorithm}{\protect\numberline{\thealgorithm}##2}%
       \else % #1 is not \relax
         \addcontentsline{loa}{algorithm}{\protect\numberline{\thealgorithm}##1}%
       \fi
       \kern2pt\hrule\kern2pt
     }
  }{% \end{breakablealgorithm}
     \kern2pt\hrule\relax% \@fs@post for \@fs@ruled
   \end{center}
  }
\makeatother

Define the following symbols:
$\mathcal{F}$ denotes the failure count, $\mathcal{I}$ represents the failure information set, $\mathcal{B}$ indicates the task failure status, $\mathcal{P}$ represents the planner agent, $\mathcal{C}$ represents the coordinator agent, and $\mathcal{T}$ represents the trajectory set.
The overall processing flow of \workforce is shown in Algorithm~\ref{alg:workforce}.

\begin{breakablealgorithm}
\caption{Workforce Framework}
\label{alg:workforce}
\begin{algorithmic}[1]
\Require Task $T$, Worker Registry $\mathcal{W}$, Task Channel $\mathcal{C}$, Max Replanning Tries $K$
\Ensure Final Output $O$

\State \textbf{Initialize:}
\State $\mathcal{F} \gets 0$
\State $\mathcal{I} \gets \{\}$
\State $\mathcal{B} \gets \text{False}$

\While{$\mathcal{F} \le K$}
    \State \textbf{Planning}
    \State $S \gets \{\}$ \Comment{Initialize subtask set}
    \If{$|\mathcal{I}| > 0$}
        \State $S \gets \mathcal{P}.\text{replan}(T, \mathcal{W}, \mathcal{I})$
    \Else
        \State $S \gets \mathcal{P}.\text{decompose}(T, \mathcal{W})$
    \EndIf

    \State \textbf{Coordinating and Processing}
    \State $R \gets \{\}$ \Comment{Initialize result set}
    \For{each subtask $s_i \in S$}
        \State $w_i \gets \mathcal{C}.\text{find\_assignee}(s_i, \mathcal{W})$ \Comment{Worker assignment}
        \State $\mathcal{C}.\text{post}(\mathcal{C}, s_i, w_i)$ \Comment{Post task to channel}
        \State $r_i \gets w_i.\text{process\_task}(s_i)$ \Comment{Process subtask}
        \If{$r_i$ is Failed}
            \State $\mathcal{C}.\text{post}(\mathcal{C}, \text{Failure}(s_i))$ \Comment{Report failure}
            \State $\mathcal{B} \gets \text{True}$
            \State $\mathcal{I} \gets \mathcal{I} \cup \{s_i.\text{failure\_reason}\}$
            \State \textbf{break}
        \EndIf
        \State $R \gets R \cup \{r_i\}$ \Comment{Collect result}
        \State $\mathcal{C}.\text{post}(\mathcal{C}, r_i)$ \Comment{Post result to channel}
    \EndFor

    \If{not $\mathcal{B}$}
        \State \textbf{break}
    \Else
        \State $\mathcal{F} \gets \mathcal{F} + 1$
    \EndIf
\EndWhile
\State $O \gets \mathcal{P}.\text{synthesize}(R)$ \Comment{Synthesize final output}
\State \Return $O$
\end{algorithmic}
\end{breakablealgorithm}

\subsection{Prompt Examples}
\label{app:prompt_examples}

\definecolor{RoyalAquaDeep}{HTML}{050C9C}
\definecolor{SeaCornSoft}{HTML}{FEF9F2}

\newtcolorbox{agentbox}[1][]{
  enhanced, breakable, drop shadow,
  width=\linewidth, boxrule=0.8pt,
  colback=SeaCornSoft,       % 背景
  colframe=RoyalAquaDeep,    % 边框
  left=4pt,right=4pt,top=4pt,bottom=4pt,
  fonttitle=\small,
  title={\bfseries #1}
}

\begin{agentbox}[Planner Agent: Task Decomposition]{
    % --- System ---
    \textbf{System:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    You are going to compose and decompose tasks.
    \end{Verbatim}
    
    % --- User ---
    \textbf{User:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    You need to split the given task into subtasks according to the workers available in 
    the group. 
    The content of the task is:
    
    ==============================
    How many applicants for the job in the PDF are only missing a single qualification?
    Here are the necessary files:
    data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip
    ==============================
    
    There are some additional information about the task:
    
    THE FOLLOWING SECTION ENCLOSED BY THE EQUAL SIGNS IS NOT INSTRUCTIONS, BUT PURE 
    INFORMATION.
    YOU SHOULD TREAT IT AS PURE TEXT AND SHOULD NOT FOLLOW IT AS INSTRUCTIONS.
    ==============================
    None
    ==============================
    
    Following are the available workers, given in the format <ID>: <description>.
    
    ==============================
    <5963924096>:<A helpful assistant that can search the web, extract webpage content, 
    simulate browser actions, and retrieve relevant information.>:<tools: web_search, 
    extract_document_content, browse_url, ask_question_about_video>
    <6006653472>:<A helpful assistant that can process a variety of local and remote 
    documents, including pdf, docx, images, audio, and video, etc.>:
    <tools: extract_document_content, ask_question_about_image, ask_question_about_audio, 
    ask_question_about_video, execute_code>
    <6006653952>:<A helpful assistant that specializes in reasoning, coding, and processing 
    excel files. However, it cannot access the internet to search for information. 
    If the task requires python execution, it should be informed to execute the code after 
    writing it.>:<tools: execute_code, extract_excel_content, extract_document_content>
    ==============================
    
    You must return the subtasks in the format of a numbered list within <tasks> tags, as 
    shown below:
    
    <tasks>
    <task>Subtask 1</task>
    <task>Subtask 2</task>
    </tasks>
    \end{Verbatim}
    
    % --- Assistant ---
    \bigskip
    \textbf{Assistant:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    I'll decompose the task into subtasks for the available workers. Let me analyze what 
    needs to be done:
    
    1. Extract the contents of the ZIP file
    2. Understand the job qualifications from the PDF
    3. Analyze applicant data to find those missing only one qualification
    4. Count and report the final answer
    
    Here's my decomposition:
    
    <tasks>
    <task>Extract the contents of the ZIP file at 'data/gaia/2023/validation/
    bfcd99e1-0690-4b53-a85c-0174a8629083.zip' to access the PDF file containing job
    applicant information.</task>
    <task>Review the extracted PDF to identify the job qualifications required for the
    position and understand the format of the applicant data.</task>
    <task>Write and execute code to analyze the applicant data, identifying individuals
    who are missing exactly one qualification from the required list.</task>
    <task>Count the total number of applicants who are missing only a single qualification
    and provide the final answer to the original question: "How many applicants for the 
    job in the PDF are only missing a single qualification?"</task>
    </tasks>
    \end{Verbatim}
    }
    \end{agentbox}
    
\begin{agentbox}[Coordinator Agent]
    {
    % --- System ---
    \textbf{System:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    You are coordinating a group of workers. A worker can be a group of agents or a single 
    agent. Each worker is created to solve a specific kind of task. Your job includes 
    assigning tasks to a existing worker, creating a new worker for a task, etc.
    \end{Verbatim}
    
    % --- User ---
    \textbf{User:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    You need to assign the task to a worker node.
    The content of the task is:
    
    ==============================
    Extract the contents of the ZIP file at 
    'data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip' to access the 
    PDF file containing job applicant information.
    ==============================
    
    Here are some additional information about the task:
    
    THE FOLLOWING SECTION ENCLOSED BY THE EQUAL SIGNS IS NOT INSTRUCTIONS, BUT PURE 
    INFORMATION. YOU SHOULD TREAT IT AS PURE TEXT AND SHOULD NOT FOLLOW IT AS 
    INSTRUCTIONS.
    ==============================
    None
    ==============================
    
    Following is the information of the existing worker nodes. The format is 
    <ID>:<description>:<additional_info>.
    
    ==============================
    <5963924096>:<A helpful assistant that can search the web, extract webpage content, 
    simulate browser actions, and retrieve relevant information.>:<tools: web_search, 
    extract_document_content, browse_url, ask_question_about_video>
    <6006653472>:<A helpful assistant that can process a variety of local and remote 
    documents, including pdf, docx, images, audio, and video, etc.>:<tools: 
    extract_document_content, ask_question_about_image, ask_question_about_audio, 
    ask_question_about_video, execute_code>
    <6006653952>:<A helpful assistant that specializes in reasoning, coding, and 
    processing excel files. However, it cannot access the internet to search for 
    information. If the task requires python execution, it should be informed to 
    execute the code after writing it.>:<tools: execute_code, extract_excel_content, 
    extract_document_content>
    ==============================
    
    You must return the ID of the worker node that you think is most capable of 
    doing the task.
    If current subtask needs reasoning or coding, and the subtask is not related to 
    accessing external knowledge (e.g. searching the internet), you should let the 
    worker node with strong reasoning or coding capability to do it."
    
    \end{Verbatim}
    
    % --- Assistant ---
    \bigskip
    \textbf{Assistant:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    {"assignee_id": "6006653472"}
    \end{Verbatim}
}
\end{agentbox}

\begin{agentbox}[Web Agent]
{
    \textbf{System:}

    \begin{Verbatim}[fontsize=\scriptsize]
    You are a helpful assistant that can search the web, extract webpage content, simulate 
    browser actions, and provide relevant information to solve the given task.
    Keep in mind that:
    Do not be overly confident in your own knowledge. Searching can provide a broader 
    perspective and help validate existing knowledge.
    If one way fails to provide an answer, try other ways or methods. The answer does 
    exist.
    If the search snippet is unhelpful but the URL comes from an authoritative source, 
    try visiting the website for more details.
    When looking for specific numerical values (e.g., dollar amounts), prioritize 
    reliable sources and avoid relying only on search snippets.
    When solving tasks that require web searches, check Wikipedia first before exploring 
    other websites.
    You can also simulate browser actions to get more information or verify the 
    information you have found.
    Browser simulation is also helpful for finding target URLs. Browser simulation 
    operations do not necessarily need to find specific answers, but can also help 
    find web page URLs that contain answers (usually difficult to find through simple 
    web searches). You can find the answer to the question by performing subsequent 
    operations on the URL, such as extracting the content of the webpage.
    Do not solely rely on document tools or browser simulation to find the answer; 
    you should combine document tools and browser simulation to comprehensively 
    process web page information. Some content may need browser simulation to get, 
    or some content is rendered by JavaScript.
    In your response, you should mention the URLs you have visited and processed.
    
    Here are some tips that help you perform web search:
    Never add too many keywords in your search query! Some detailed results need to 
    perform browser interaction to get, not using search toolkit.
    If the question is complex, search results typically do not provide precise 
    answers. It is not likely to find the answer directly using search toolkit only; 
    the search query should be concise and focus on finding official sources rather 
    than direct answers.
    For example, as for the question "What is the maximum length in meters of #9 in 
    the first National Geographic short on YouTube that was ever released according 
    to the Monterey Bay Aquarium website?", your first search term must be 
    coarse-grained like "National Geographic YouTube" to find the YouTube website 
    first, and then try other fine-grained search terms step-by-step to find more URLs.
    The results you return do not have to directly answer the original question; 
    you only need to collect relevant information.
    \end{Verbatim}
    
    \textbf{User:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    We are solving a complex task, and we have split the task into several subtasks.

    You need to process one given task. Don't assume that the problem is 
    unsolvable. The answer does exist. If you can't solve the task, please 
    describe the reason and the result you have achieved in detail.
    
    The content of the task that you need to do is:
    
    <task>  
    Access the Collins Spanish-to-English dictionary online and locate the 1994 
    example sentence for the identified Spanish word  
    </task>
    
    
    Here is the overall task for reference, which contains some helpful 
    information that can help you solve the task:
    
    <overall_task>
    The Latin root of the Yola word "gimlie" shares a spelling with a Spanish 
    word. What is the Google translation of the source title for the 1994 example 
    sentence for that word in the Collins Spanish-to-English dictionary online? 
    Answer in plain text, without punctuation.
    </overall_task>
    
    Here are results of some prerequisite results that you can refer to (empty if 
    there are no prerequisite results):
    
    <dependency_results_info>
    
    The Latin Root of the Yola Word "Gimlie"
    
    Etymology
    
    According to the Wiktionary entry for "gimlie", this Yola word has the 
    following etymology:
    
    The Yola word "gimlie" (meaning "chimney") comes from Middle English 
    "chimeney", which in turn comes from Old French "cheminee". The ultimate 
    Latin root of the word is Late Latin "caminata".
    
    Latin Root
    
    The Latin word "caminata" is the source from which the Yola word "gimlie" 
    derives. This Latin root shares a spelling with the Spanish word "caminata", 
    which means "a (long) walk", "hike", "trek", or "ramble" in modern Spanish.
    
    Etymology Chain
    - Yola: "gimlie" (meaning "chimney")
    - Middle English: "chimeney"
    - Old French: "cheminee"
    - Late Latin: "caminata"
    - Modern Spanish cognate: "caminata"
    
    Semantic Evolution
    
    The Latin root "caminata" is related to the Latin word "caminus" (meaning 
    "fireplace" or "forge"), which itself comes from Greek "kauivoc" (kaminos, 
    meaning "oven" or "furnace"). The semantic shift from a heating structure to 
    a chimney occurred during the evolution of these words.
    
    Conclusion
    
    This research confirms that the Latin root of the Yola word "gimlie" is 
    "caminata", which shares its spelling with the modern Spanish word "caminata".
    
    Sources
    1.https://en.wiktionary.org/wiki/gimlie
    2.https://dictionary.cambridge.org/us/dictionary/spanish-english/caminata
    3.https://www.collinsdictionary.com/dictionary/spanish-english/caminata
    4.http://web.archive.org/web/20161113033419/http://www.collinsdictionary.
    com:80/dictionary/spanish-english/caminata
    
    Finding a Spanish Word that Shares the Same Spelling as the Latin Root of 
    "Gimlie"
    
    Answer: caminata
    
    According to the dependency results provided, the Latin root of the Yola word 
    "gimlie" is "caminata", which shares the same spelling as the Spanish word 
    "caminata" (meaning "a (long) walk", "hike", "trek", or "ramble" in modern 
    Spanish).
    
    Verification Process:
    1.From the dependency results, I confirmed that the Latin root of "gimlie" is 
    "caminata", which evolved through this chain:
    - Yola: "gimlie" (meaning "chimney")
    -Middle English: "chimeney"
    - Old French: "cheminee"
    - Late Latin: "caminata"
    2.The modern Spanish word that shares the same spelling (without the macron 
    over the 'i') is "caminata".
    3.I verified this information by checking the Collins Spanish-English 
    Dictionary, which confirms that "caminata" is indeed a Spanish word meaning 
    "long walk", "hike", or "ramble".
    
    Sources Consulted:
    - The dependency results provided in the task
    - Collins Spanish-English Dictionary via web.archive.org
    - Cambridge Dictionary
    
    The Spanish word that shares the same spelling as the Latin root of "gimlie" 
    is "caminata".
    
    </dependency_results_info>
    
    Here are some additional information about the task (only for reference, and 
    may be empty):
    <additional_info>
    None
    </additional_info>
    
    Now please fully leverage the information above, try your best to leverage 
    the existing results and your available tools to solve the current task.
    \end{Verbatim}
    
    \bigskip
    \textbf{Assistant(tool\_calls):}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    {
        "id": "toolu_012yD77v37BkHufLmKPdgBTg",
        "type": "function",
        "function": {
            "name": "web_search",
            "arguments": "{\"question\": \"Collins Spanish-English dictionary
            caminata online\"}"
        }
    }
    \end{Verbatim}

    \bigskip
    \textbf{tool:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    Here are some relevant resources for the Collins Spanish-English Dictionary 
    online:
    1. **Collins English-Spanish Dictionary**\n   - **URL**: [Collins English-
    Spanish Dictionary](https://www.collinsdictionary.com/dictionary/english-
    spanish)   
    - **Information**: This online dictionary provides complete English-Spanish 
    translations with accurate pronunciations in both languages, usage examples, 
    and grammar guides.
    2. **Collins Spanish-English on the App Store**
    - **URL**: [Collins Spanish-English on the App Store]
    (https://apps.apple.com/us/app/collins-spanish-english/id292239107)
    - **Information**: This app offers a dependable Spanish and English reference 
    with over 100,000 translations, verb conjugations, usage notes, and examples. 
    It functions without requiring an internet connection.
    These resources should help you access the Collins Spanish-English Dictionary online.
    \end{Verbatim}
}
\end{agentbox}

\begin{agentbox}[Document Processing Agent]
{
    \textbf{System:}

    \begin{Verbatim}[fontsize=\scriptsize]
    You are a helpful assistant that can process documents and multimodal data, 
    such as 
    images, audio, and video.
    \end{Verbatim}
    
    \textbf{User:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    We are solving a complex task, and we have split the task into several 
    subtasks.

    You need to process one given task. Don't assume that the problem is 
    unsolvable. 
    The answer does exist. If you can't solve the task, please describe the 
    reason 
    and the result you have achieved in detail.
    The content of the task that you need to do is:
    
    <task>
    Review the extracted PDF to identify the job qualifications required for the 
    position 
    and understand the format of the applicant data.
    </task>
    
    Here is the overall task for reference, which contains some helpful 
    information that 
    can help you solve the task:
    
    <overall_task>
    How many applicants for the job in the PDF are only missing a single 
    qualification? 
    Here are the necessary files: 
    data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip
    </overall_task>
    
    Here are results of some prerequisite results that you can refer to (empty if 
    there 
    are no prerequisite results):
    
    <dependency_results_info>
    # ZIP File Extraction Report
    
    ## Summary
    I have successfully extracted the contents of the ZIP file at 
    `data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip`. 
    The extraction was performed to a directory named `extracted_files`.
    
    ## Extracted Files
    The ZIP file contained two files:
    1. **Job Listing.pdf** - A PDF file that likely contains information about 
    the 
    job requirements
    2. **Applicants.xlsx** - An Excel file that likely contains information about 
    the 
    job applicants
    
    ## File Locations
    - The PDF file is located at `extracted_files/Job Listing.pdf`
    - The Excel file is located at `extracted_files/Applicants.xlsx`
    
    ## Next Steps
    Based on the overall task, which is to determine "How many applicants for the 
    job in the PDF are only missing a single qualification?", these extracted 
    files will be essential for the next steps. The PDF file likely contains the 
    job qualifications, while the Excel file likely contains the applicant 
    information and their qualifications.
    
    ## Status
    The task has been completed successfully, and the files are now accessible 
    for further processing.
    
    </dependency_results_info>
    
    Here are some additional information about the task (only for reference, and 
    may be empty):
    <additional_info>
    None
    </additional_info>
    
    Now please fully leverage the information above, try your best to leverage 
    the existing results and your available tools to solve the current task.
    \end{Verbatim}
    
    \bigskip
    \textbf{Assistant(tool\_calls):}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    {
        "id": "toolu_01V6RB7mNUYD5h7Ut4Kf5v9g",
        "type": "function",
        "function": {
            "name": "extract_document_content",
            "arguments": "{\"document_path\": \"extracted_files/Job 
            Listing.pdf\", \"query\": \"What are the job qualifications required 
            for the position?\"}"
        }
    }
    \end{Verbatim}

    \bigskip
    \textbf{tool:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    Biologist at ABC Biotech Research Company
    
    Job Title: Biologist
    Company: ABC Biotech Research Co.
    Job Type: Full-time
    
    About the Company:
    ABC Biotech is a leading biotechnology company dedicated to advancing ...
    \end{Verbatim}
    
}
\end{agentbox}


\begin{agentbox}[Reasoning and Coding Agent]
{
    \textbf{System:}

    \begin{Verbatim}[fontsize=\scriptsize]
    You are a helpful assistant that specializes in reasoning and coding, and can 
    think step by step to solve the task. When necessary, you can write python 
    code to solve the task. If you have written code, do not forget to execute 
    the code. Never generate codes like 'example code', your code should be able 
    to fully solve the task. You can also leverage multiple libraries, such as 
    requests, BeautifulSoup, re, pandas, etc, to solve the task. For processing 
    excel files, you should write codes to process them.
    \end{Verbatim}
    
    \textbf{User:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    We are solving a complex task, and we have split the task into several 
    subtasks.

    You need to process one given task. Don't assume that the problem is 
    unsolvable. The answer does exist. If you can't solve the task, please 
    describe the reason and the result you have achieved in detail.
    The content of the task that you need to do is:
    
    <task>
    Write and execute code to analyze the applicant data, identifying individuals 
    who are missing exactly one qualification from the required list.
    
    </task>
    
    Here is the overall task for reference, which contains some helpful 
    information that can help you solve the task:
    
    <overall_task>
    How many applicants for the job in the PDF are only missing a single 
    qualification? Here are the necessary files: 
    data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip
    </overall_task>
    
    Here are results of some prerequisite results that you can refer to (empty if 
    there are no prerequisite results):
    
    <dependency_results_info>
    # ZIP File Extraction Report
    
    ## Summary
    I have successfully extracted the contents of the ZIP file at 
    `data/gaia/2023/validation/bfcd99e1-0690-4b53-a85c-0174a8629083.zip`. The 
    extraction was performed to a directory named `extracted_files`.
    
    ## Extracted Files
    The ZIP file contained two files:
    1. **Job Listing.pdf** - A PDF file that likely contains information about 
    the job requirements
    2. **Applicants.xlsx** - An Excel file that likely contains information about 
    the job applicants
    
    ## File Locations
    - The PDF file is located at `extracted_files/Job Listing.pdf`
    - The Excel file is located at `extracted_files/Applicants.xlsx`
    
    ## Next Steps
    Based on the overall task, which is to determine "How many applicants for the 
    job in the PDF are only missing a single qualification?", these extracted 
    files will be essential for the next steps. The PDF file likely contains the 
    job qualifications, while the Excel file likely contains the applicant 
    information and their qualifications.
    
    ## Status
    The task has been completed successfully, and the files are now accessible 
    for further processing.
    # Job Qualifications and Applicant Data Analysis
    
    Based on my review of the extracted PDF and the applicant data, I've 
    identified the job qualifications and the format of the applicant data.
    
    ## Job Qualifications Required for the Position
    
    From the "Job Listing.pdf" file, I've identified the following qualifications 
    required for the Biologist position at ABC Biotech Research Company:
    
    1. **Education**: Masters Degree or higher in biology, biochemistry, or 
    biotechnology
    2. **Experience**: 3+ years of experience
    3. **Training**: Training with laboratory equipment
    4. **Publications**: 3+ publications in the field of biotechnology
    5. **Citizenship**: Citizenship in X Country
    6. **Programming**: C++, C#, or Fortran experience
    7. **Language**: 1+ second language
    
    ## Applicant Data Format
    
    The "Applicants.xlsx" file contains information about job applicants with the 
    following structure:
    
    1. **Name**: Applicant's full name
    2. **Degree Field**: Field of study (Biology, Biochemistry, Biotechnology, 
    etc.)
    3. **Degree Level**: Education level (Bachelor, Master, Ph.D., etc.)
    4. **Experience (Years)**: Number of years of experience
    5. **Publications**: Number of publications
    6. **Lab Trained (Y/N)**: Whether the applicant has laboratory training (Y = 
    Yes, N = No)
    7. **Citizen (Y/N)**: Whether the applicant is a citizen (Y = Yes, N = No)
    8. **Programming Lang**: Programming language experience
    9. **Second Language**: Second language knowledge (NaN/missing values 
    indicate no second language)
    
    ## Data Observations
    
    - The Excel file contains data for 50 applicants
    - All columns are complete except for "Second Language" which has 15 missing 
    values
    - The missing values in "Second Language" likely indicate applicants who 
    don't have a second language
    - The data format aligns with the job qualifications, allowing for direct 
    comparison between requirements and applicant qualifications
    
    This information will be essential for determining which applicants are 
    missing only a single qualification for the job, which is the overall task 
    objective.
    \end{Verbatim}
    
    \bigskip
    \textbf{Assistant(tool\_calls):}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    {
        "id": "toolu_01B1DAYBLEnfU82qPf5jEsMY",
        "type": "function",
        "function": {
            "name": "execute_code",
            "arguments": "{\"code\": \"import pandas as pd\\nimport numpy as 
            np\\n\\n# Load the Excel file\\napplicants_df = 
            pd.read_excel('extracted_files/Applicants.xlsx')\\n\\n# Define a 
            function to check if an applicant meets each requirement\\ndef 
            check_requirements(row):\\n    requirements_met = []\\n    
            requirements_failed = []\\n    \\n    # 1. Education: Masters Degree 
            or higher in biology, biochemistry, or biotechnology\\n    
            valid_degrees = ['Master', 'Ph. D.']\\n    valid_fields = ['Biology', 
            'Biochemistry', 'Biotechnology']\\n    if row['Degree Level'] in 
            valid_degrees and row['Degree Field'] in valid_fields:\\n        
            requirements_met.append('Education')\\n    
            ...
        }
    }
    \end{Verbatim}

        \bigskip
    \textbf{tool:}
    
    \begin{Verbatim}[fontsize=\scriptsize]
    Executed Results:

    Number of applicants missing exactly one qualification: 17
    
    Applicants missing exactly one qualification:
    Nabil Bates: Missing Publications
    Safwan Sanders: Missing Education
    ...
    \end{Verbatim}
}
\end{agentbox}


\subsection{Worker Implementation and Toolkit Design}
\label{app:worker_agent_details}

 \workforce employs specialized worker agents, each equipped with specific tools to handle different aspects of complex tasks. The implementation follows a modular design where each worker agent is instantiated with a tailored prompt, a foundation model, and a set of tools optimized for its domain of expertise.

As shown in Table \ref{tab:worker-agents}, when solving tasks from GAIA, \workforce consists of three primary worker agents:

\begin{enumerate}
    \item \textbf{Web Agent}: Specializes in web-based interactions, including search, content extraction, and browser simulation.
    \item \textbf{Document Processing Agent}: Handles multimodal data processing including documents, images, audio, and video, etc.
    \item \textbf{Reasoning and Coding Agent}: Focuses on logical reasoning, code execution, and structured data analysis.
\end{enumerate}

The tools assigned to each agent are carefully selected to align with their specialized functions while avoiding unnecessary overlap.

\begin{table}[h]
  \centering
  \small
  \caption{Worker Agent Implementation Details}
  \label{tab:worker-agents}
  \begin{tabular}{p{4cm}p{4cm}p{4cm}}
      \toprule
      \textbf{Worker Agent} & \textbf{Tools} & \textbf{Model Backend} \\
      \midrule
      \multirow{7}{*}{Web Agent} & search\_google & \multirow{7}{*}{GPT-4o / Claude-3.7-Sonnet}  \\
      & search\_wiki & \\
      & search\_wiki\_revisions & \\
      & search\_archived\_webpage & \\
      & extract\_document\_content & \\
      & browse\_url & \\
      & ask\_question\_about\_video & \\
      \midrule
      \multirow{5}{*}{Document Processing Agent} & extract\_document\_content & \multirow{5}{*}{GPT-4o / Claude-3.7-Sonnet} \\
      & ask\_question\_about\_image & \\
      & ask\_question\_about\_audio & \\
      & ask\_question\_about\_video & \\
      & execute\_code & \\
      \midrule
      \multirow{3}{*}{Reasoning \& Coding Agent} & execute\_code & \multirow{3}{*}{o3-mini / Claude-3.7-Sonnet} \\
      & extract\_excel\_content & \\
      & extract\_document\_content & \\
      \bottomrule
  \end{tabular}
\end{table}

\begin{table}[h]
    \centering
    \small
    \caption{Toolkit Design and Implementation Details}
    \label{tab:toolkit-design}
    \begin{tabular}{p{3cm}p{3.5cm}p{6cm}}
        \toprule
        \textbf{Toolkit Category} & \textbf{Tools} & \textbf{Toolkit Design and Implementation Details} \\
        \midrule
        \multirow{4}{*}{Web Search} & search\_google & \multirow{4}{*}{API-based web search services} \\
        & search\_wiki & \\
        & search\_wiki\_revisions & \\
        & search\_archived\_webpage & \\
        \midrule
        \multirow{2}{*}{Browser Simulation} & \multirow{2}{*}{browse\_url} & GPT-4o (For action execution)\\
        & & O3-mini (For navigation planning) \\
        \midrule
        \multirow{2}{*}{Document Processing} & extract\_document\_content & Document parsing API services and libraries  \\
        & extract\_excel\_content &  e.g. html2text, Unstructured, Firecrawl, chunkr \\
        \midrule
        \multirow{3}{*}{Multimodal Analysis} & ask\_question\_about\_image & GPT-4o (For images)\\
        & ask\_question\_about\_audio & Whisper-1 + O3-mini (For audio)\\
        & ask\_question\_about\_video & Gemini 2.0-Flash (For video) \\
        \midrule
        \multirow{1}{*}{Code Execution} & execute\_code & Python Executor \\
        \bottomrule
    \end{tabular}
\end{table}

As is shown in Table \ref{tab:toolkit-design}, our tool design is carefully configured based on downstream tasks. 
Each area is implemented with appropriate safeguards and optimized for specific use cases:

\begin{itemize}
    \item \textbf{Web Search}: Provides structured access to search engines, Wikipedia, and archived web pages with rate limiting and content filtering
    \item \textbf{Browser Simulation}: Enables web page interaction through a specialized dual-model approach where one model observes and acts while another plans navigation strategy
    \item \textbf{Document Processing}: Handles various document formats with efficient parsing and information extraction capabilities
    \item \textbf{Multimodal Analysis}: Processes different media types using specialized models for each modality
    \item \textbf{Code Execution}: Provides Python code execution using sandbox or subprocess module
\end{itemize}

This modular design allows for easy extension of the workforce with additional specialized agents and tools as needed, while maintaining a clean separation of concerns between different functional domains.

In addition, the toolkit can also be extended to support more specialized agents and tools as needed according to the task requirements.

\subsection{Synthesis Trajectories Statistics}
\label{app:trajectory_details}

\begin{table}[]
    \centering
    \caption{More details on the statistics of our synthesis-based supervised fine-tuning dataset.}
      \begin{tabular}{lcccccc}
        \toprule
        \textbf{Dataset}    & \textbf{\# Total} & \textbf{\# Filtering} & \textbf{Avg. Subtasks} & \textbf{Avg. Steps} & \textbf{Agent Capabilities} \\
        \midrule
        HotpotQA            &  998       & 495          & 3.49          & 26.92      & 
          \agenticon{images/multi_agent_training_pdf/Web-Browsing}
          \agenticon{images/multi_agent_training_pdf/Reasoning}
          \agenticon{images/multi_agent_training_pdf/multimodal1} \\
        WikiTableQuestions  &  869       & 577          & 3.03          & 20.81      & 
          \agenticon{images/multi_agent_training_pdf/Document_Processing}
          \agenticon{images/multi_agent_training_pdf/Reasoning} \\
        Math-related        & 1100       & 487          & 3.87          & 23.32      & 
          \agenticon{images/multi_agent_training_pdf/Reasoning}
          \agenticon{images/multi_agent_training_pdf/coding} \\
        Infinity-MM         &  499       & 40           & 2.88          & 16.56      & 
          \agenticon{images/multi_agent_training_pdf/Reasoning}
          \agenticon{images/multi_agent_training_pdf/multimodal1} \\
        \midrule
        Total/Average       & 3466       & 1599         & 3.41          & 22.75      & \\
        \bottomrule
      \end{tabular}%
    \label{tab:more_synthesis_stats}
\end{table}

Table~\ref{tab:more_synthesis_stats} presents detailed statistics of our synthesis dataset. 
We collected 3,466 trajectories in total in four different datasets, with 1,599 remaining after data cleaning. 
On average, each trajectory contains 3.41 subtasks and 22.75 execution steps. 
Among these datasets, Math-related tasks require the most subtasks (3.87) while HotpotQA tasks involve the most execution steps (26.92). 
Each data set requires different agent capabilities, and reasoning is a common requirement across all data sets.


% \subsection{Comparison between Agent Frameworks}

% \begin{table}[thbp]
% \centering
% \caption{Comparison of LLM-based Multi-Agent Frameworks. \mk{should move to appendix}}
% \label{tab:llm-mas-comparison}
% \begin{tabular}{lccccc}
% \toprule
% \textbf{Framework} & \textbf{Hierarchy} & \textbf{Communication} & \textbf{Task Types} & \textbf{Scalability} & \textbf{Topology} \\
% \midrule
% CAMEL           & Flat          & Dialogue           & Diverse  & High      & Flexible \\
% SWE-Agent       & Partial       & Feedback loop      & Code fixing       & Low       & Loop     \\
% ChatDev         & Phase-based   & Sequential dialog  & Software dev.     & Moderate  & Chain    \\
% MetaGPT         & Yes           & Centralized        & Software dev.     & Moderate  & Star     \\
% AppWorld        & Yes           & Centralized        & App integration   & Moderate  & Star     \\
% AutoGen         & Flexible      & Flexible           & General tasks     & High      & Flexible \\
% Magentic-One    & Yes           & -        & Web/File/Coding   & Moderate      & Star     \\
% \midrule
% \textbf{Workforce} & Yes & Task Channel & Diverse & High & Star/Tree \\
% \bottomrule
% \end{tabular}
% \end{table}



% \section{Workforce Application}

\section{Error Analysis}
\label{app:error_analysis}

\subsection{Error Distribution}

\begin{table}[t]
  \centering
  \small
  \caption{Categorization and distribution of failure modes on GAIA benchmark.}
  \resizebox{\linewidth}{!}{%
    \begin{tabular}{lp{7.5cm}l}
      \toprule
      \textbf{Error Type} & \textbf{Explanation} & \textbf{Proportion (\%)} \\
      \midrule
      \textbf{Planner Error} & Errors stemming from the planner agent. & \textbf{21.15\%} \\
      ~~~~Incorrect Plan & Planner produced entirely incorrect task decomposition. & ~~~~19.23\% \\
      ~~~~Subtask Ambiguity & Plans were correct but ambiguous. & ~~~~1.92\% \\
      \midrule
      \textbf{Worker Error} & Execution failures by worker agents during subtask handling. & \textbf{9.62\%} \\
      ~~~~Tool Selection Error & Wrong tool selected for subtask execution. & ~~~~9.62\% \\
      \midrule
      \textbf{Limited Tool Capability} & Failures caused by toolkits' limitations. & \textbf{32.69\%} \\
      ~~~~Web Toolkit Failure & Failures due to inability of web agent to complete tasks. & ~~~~13.46\% \\
      ~~~~Document Toolkit Failure & Specific to document processing tools. & ~~~~1.92\% \\
      ~~~~Multimodal Toolkit Failure & Errors when handling multimodal inputs. & ~~~~17.31\% \\
      \midrule
      \textbf{Responding Error} & Final response generation failed after successful subtasks. & \textbf{7.69\%} \\
      ~~~~Response Format & Output did not conform to expected format. & ~~~~3.85\% \\
      ~~~~Reasoning Error & Incorrect answer despite access to correct information. & ~~~~3.84\% \\
      \midrule
      \textbf{Language/Question Ambiguity} & Ambiguity in user input led to misunderstanding. & \textbf{9.62\%} \\
      \midrule
      \textbf{Limited Model Capability} & Model-internal limitations. & \textbf{19.23\%} \\
      ~~~~Hallucination & Model fabricated non-existent facts. & ~~~~1.92\% \\
      ~~~~Context Exceed & Model failed due to context length limitations. & ~~~~5.77\% \\
      ~~~~Limited Coding Capability & Model failed to generate usable code. & ~~~~11.54\% \\
      \bottomrule
    \end{tabular}
  }
  \label{tab:error-hierarchy}
\end{table}


Table~\ref{tab:error-hierarchy} shows the categorization and distribution of different error types encountered in the GAIA benchmark. 
The errors are broadly classified into six main categories: Planner Error, Worker Error, Limited Tool Capability, Responding Error, Language/Question Ambiguity, and Limited Model Capability. 
Among these, Limited Tool Capability accounts for the largest proportion (32.69\%) of failures, particularly in web toolkit and multimodal toolkit operations. 
Planner Error is the second most common type (21.15\%), mainly due to incorrect task decomposition. 
Limited Model Capability represents 19.23\% of errors, with coding capability limitations being a significant factor. 
Worker Error and Language/Question Ambiguity each account for 9.62\% of failures, while Responding Error makes up 7.69\% of the total errors.

\subsection{Error Distribution by Levels}

\begin{table}[h]
  \centering
  \small
  \caption{Detailed error counts by type and GAIA difficulty level.}
  \begin{tabular}{lccc|c}
    \toprule
    \textbf{Error Type} & \textbf{Level 1} & \textbf{Level 2} & \textbf{Level 3} & \textbf{Sum} \\
    \midrule
    \textbf{Planning Error} \\
    ~~~~Incorrect Plan & 2 & 4 & 4 & 10 \\
    ~~~~Subtask Ambiguity & 0 & 0 & 1 & 1 \\
    \textbf{Worker Error} \\
    ~~~~Tool Selection Error & 0 & 1 & 4 & 5 \\
    \textbf{Limited Tool Capability} \\
    ~~~~Web Toolkit Failure & 1 & 6 & 0 & 7 \\
    ~~~~Document Toolkit Failure & 0 & 1 & 0 & 1 \\
    ~~~~Multimodal Toolkit Failure & 2 & 6 & 1 & 9 \\
    \textbf{Responding Error} \\
    ~~~~Response Format & 0 & 2 & 0 & 2 \\
    ~~~~Reasoning Error & 0 & 1 & 1 & 2 \\
    \textbf{Language/Question Ambiguity} & 1 & 2 & 2 & 5 \\
    \textbf{Limited Model Capability} \\
    ~~~~Hallucination & 0 & 1 & 0 & 1 \\
    ~~~~Context Exceed & 0 & 0 & 3 & 3 \\
    ~~~~Limited Coding Capability & 2 & 4 & 0 & 6 \\
    \bottomrule
  \end{tabular}
  \label{tab:error-detailed}
\end{table}

Table~\ref{tab:error-detailed} presents a detailed breakdown of errors across different difficulty levels in the GAIA benchmark. 
Level 2 tasks show the highest number of errors overall, particularly in web toolkit failures (6 cases) and multimodal toolkit failures (6 cases). 
Level 3 tasks demonstrate increased complexity with more tool selection errors (4 cases) and context exceed errors (3 cases) compared to lower levels. 
Level 1 tasks have relatively fewer errors across categories, though they still show some issues with incorrect planning (2 cases) and limited coding capability (2 cases). 
This distribution suggests that error patterns vary significantly with task complexity, with higher levels generally showing more sophisticated failure modes.


\subsection{Error Cases Examples}

\tcbset{
  colframe=blue!50!black, colback=white,
  fonttitle=\bfseries, nobeforeafter, center title
}

%—— Template macros ——%
\newcommand{\ErrorCaseBox}[6]{%
  \begin{tcolorbox}[title={#1: Error Case Level#2~\##3}]
    \textbf{Question}: #4 \\[0.5em]
    \textbf{Answer}: #5 \\[0.5em]
    \textbf{Annotation steps}: 
    #6
  \end{tcolorbox}
}

\newcommand{\ErrorAnalysisBox}[3]{%
  \begin{tcolorbox}[
    colframe=orange!60!black,    % 边框：深橙
    coltitle=orange!80!black,     % 标题文字：橙黑混合
    colbacktitle=orange!20!white, % 标题栏背景：更淡的橙
    title={#1 Error Analysis}, 
    fonttitle=\bfseries,
    nobeforeafter, center title,
    enhanced,breakable
  ]
    \textbf{Analysis}: #2 \\[0.5em]
    \textbf{Agent Content}: \\ #3
  \end{tcolorbox}
}

%—— Incorrect Plan ——%
\ErrorCaseBox{Incorrect Plan}{3}{12}
{I was referencing each of the tables in the file from papers that were cited by the ``Trans fatty acid contents in chocolates and chocolate wafers in Turkey'' paper. I lost my own reference sheet and need to know which of the papers each table came from. The file may not use the full table caption. If the references in the ``Trans fatty acid'' paper bibliography were numbered starting with 1, give me the numbers in the order that they would be used to fill the cells in the Excel file from top to bottom, as a comma separated list.}
{8, 29, 22, 1, 8, 26}
{1. Searched ``Trans fatty acid contents in chocolates and chocolate wafers in Turkey'' on Google. 2.Opened https://www.researchgate.net/publication/234034780
\_Trans\_fatty\_acid\_contents\_in\_chocolates\_
and\_chocolate\_wafers\_in\_Turkey. 3.Opened the Excel file. 4.Searched each reference in the paper on Google. 5.Checked any free-to-access reference for a table similar to the titles in the Excel file. 6.Added the numbers of the references to the Excel file. 7.Copied the numbers into a comma-separated list.
}
\ErrorAnalysisBox{Incorrect Plan}
{The subtask only lets the agent match the table content with the reference paper, without further extracting the content of the reference paper itself.}
{
\{``subtask'': ``Write and execute code to analyze the Excel file content and \textbf{\textcolor{red}{match the tables with the references from the paper}}.'', \\
``result'': ``Table-to-Reference Matching AnalysisBased on my analysis of the Excel file and the bibliography from the ``Trans fatty acid contents in chocolates and chocolate wafers in Turkey'' paper, I've matched each table to its corresponding reference number. \\
\}
}

%—— Subtask Ambiguity ——%
\ErrorCaseBox{Subtask Ambiguity}{3}{18}
{The year is 2022. I am at the National Air and Space Museum east of the Potomac River. I want to go to Fire Station 301 DCA ARFF using the metro. I go in the wrong direction and end up at the station closest to Cleveland Elementary School. How many metro stations am I away from my original destination if I don't change lines? Your answer should be a numerical integer value.}
{8}
{1. Google search ``National Air and Space Museum''. 2. Note there are two National Air and Space Museums. One in Virginia, the other in Washington D.C. 3. Google map search ``Potomac River'' and zoom out. 4. See that Washington DC is east of the Potomac River. 5. Determine that the National Air and Space Museum refers to the one in Washington D.C. 6. Google search ``Metro Station National Air and Space Museum Washington D.C.''. 7. Clicked on the first result: Getting Here | National Air and Space Museum, https://airandspace.si.edu/visit/museum-dc/directions. 8. Read on the website, ``The closest Metrorail stop is at L'Enfant Plaza.'' Note this location. 6. Google map search ``Fire Station 301 DCA ARFF''. 7. Zoom out to look for nearby metro stations. 8. The closest station is Ronald Reagan Washington National Airport. 9. Google map search ``Cleveland Elementary School''. 10. The closest metro station to Cleveland Elementry School is Shaw-Howard Univ Station. 11. Google search ``DC Metro Station Map''. 12. Clicked on the second result: 2022 System Map, https://www.wmata.com/schedules/maps/upload/2022-System-Map.pdf. 13. Locate L'Enfant Plaza station. It is the transfer station for all color lines. 14. Locate Shaw-Howard Univ stations 4 stops above L'Enfant Plaza station. 15. Locate Ronald Reagan National Airport station on the blue/yellow line. 16. Recall the current location: Shaw-Howard Univ station's yellow/green line. 17. Since the question says no line changes, we deduce the line must be one that Shaw-Howard Univ and Ronald Reagan National Airport stations have in common: yellow line. 18. Begin at Shaw-Howard Univ station and follow the yellow line. 19. Count the number of stops until it reaches Ronald Reagan National Airport station. 20. Final answer: 8.}
\ErrorAnalysisBox{Subtask Ambiguity}
{The subtask asks how many subway stations there are between the stations, but the question actually requires calculating how many subway stations you need to pass to reach your destination, which is different.}
{
\{
``subtask'': ``Calculate how many metro stations are \textbf{\textcolor{red}{between the station nearest to Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF}}, without changing lines. Provide the final answer as a numerical integer value.'', \\
``result'': ``Calculating Metro Stations Between Shaw-Howard University and National Airport Based on the information provided in the dependency results and my analysis of the Washington DC Metro system, I need to calculate how many metro stations are between the station nearest to Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF, without changing lines. Identifying the Stations From the dependency results:- The station nearest to Cleveland Elementary School is **Shaw-Howard University** on the Green Line The station nearest to Fire Station 301 DCA ARFF is **Ronald Reagan Washington National Airport** (also called ``National Airport'') on the Blue and Yellow Lines  Determining the Connecting LineAccording to the dependency results, the Yellow Line connects both stations. This means a person can travel between these two stations without changing lines by using the Yellow Line. Calculating the Number of StationsI analyzed the Yellow Line stations from north to south and found:- Shaw-Howard University is station 9 on the Yellow Line- National Airport is station 17 on the Yellow LineThe stations between them (excluding the starting station) are: 1. Mount Vernon Square 2. Gallery Place 3. Archives 4. L'Enfant Plaza 5. Pentagon 6. Pentagon City 7. Crystal City Therefore, there are **7** metro stations between Shaw-Howard University and National Airport on the Yellow Line without changing lines. Final Answer The number of metro stations between the station nearest to Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF, without changing lines, is **7**.'', \\
\}
}
%—— Tool Selection Error ——%
\ErrorCaseBox{Tool Selection Error}{3}{8}
{I was trying to remember how well the Cheater Beater performed in comparison to the Cheater when James tested it on his channel. I know that the Cheater still outperformed the Cheater Beater in terms of CFM. Could you please look that up for me, and report the CFM of both the Cheater and the Cheater Beater? I'm not sure if he made any changes to his testing, but this was back in season 4, so just report the value from that season. Please format your response like this: CFM number for Cheater, CFM number for Cheater beater.}
{101.376, 84.348}
{1. Using a web browser, navigate to a search engine and conduct a search: ``James Cheater Cheater Beater CFM Season 4'' 2. Finding no relevant result, navigate to a search engine and conduct another search: ``Cheater Beater Season 4'' 3. Navigate to the first search result, https://www.youtube.com/watch?v=2vq3COPZbKo 4. Evaluate the YouTube page, noting that the video description identifies the video content comparing the performance of computer fans to a fan referred to as the ``cheater'' 5. Follow the link to the YouTube channel Major Hardware, https://www.youtube.com/@MajorHardware 6. Navigate to the About tab link, https://www.youtube.com/@MajorHardware/about 7. Evaluate the content, noting that the page identifies the operator of the channel as James 8. Navigate to a search engine and conduct a search, ``James Major Hardware Cheater Beater'' 9. Navigate to the first result, identical to the result from step 3 above, https://www.youtube.com/watch?v=2vq3COPZbKo 10. Search the page for CFM, finding no result 11. Load the video content and review it 12. Note an onscreen text element identifying a fan as ``CALL SIGN: CHEATER BEATER'' at timestamp 224 13. Note an onscreen table identifying the performance of various fans tested during season four, at timestamp 485 14. Evaluate the table content, identifying an entry for a fan named ``Cheater'' and a fan named ``Cheater Beater'' 15. Evaluate the table content, identifying that the data for both fans were recorded in season 4, S4E1 for Cheater, S4E6 for Cheater Beater 16. Record the data from the CFM column for the two fans, ``Cheater: 101.376'', and ``Cheater Beater: 84.348'' 17. Report the correct response to my user: ``Cheater: 101.376 Cheater Beater: 84.348''.}
\ErrorAnalysisBox{Tool Selection}
{The agent traverses season4's videos, but only stops at searching and does not open the video page for searching and processing.}
{
\{
  ``role'': ``assistant'', 
  ``tool\_calls'': [
    \{
      ``type'': ``function'', \\
      ``function'': \{
        ``name'': ``\textbf{\textcolor{red}{web\_search}}'', \\
        ``arguments'': ``\{``question'': ``Major Hardware Fan Showdown \textbf{Season 4 Episode 6 Cheater Beater}''\}''
      \}
    \}
  ],
\}, \\
\{
  ``role'': ``assistant'', 
  ``tool\_calls'': [
    \{
      ``type'': ``function'', \\
      ``function'': \{
        ``name'': ``\textbf{\textcolor{red}{web\_search}}'', \\ 
        ``arguments'': ``\{``question'': ``Major Hardware Fan Showdown Season 4 Episode 7 Cheater Beater''\}''
      \}
    \}
  ],
\}, \\
\{
  ``role'': ``assistant'', 
  ``tool\_calls'': [
    \{
      ``type'': ``function'', \\
      ``function'': \{
        ``name'': ``web\_search'', \\
        ``arguments'': ``\{``question'': ``Major Hardware Fan Showdown Season 4 Episode 8 Cheater Beater''\}''
      \}
    \}
  ]
\}
}

%—— Web Toolkit Failure ——%
\ErrorCaseBox{Web Toolkit Failure}{1}{38}
{On Cornell Law School website's legal information institute, under the fifth section of federal rules alphabetically, what word was deleted in the last amendment to the first rule in the article that has ``witnesses'' in the most titles as of 2021?}
{inference}
{1. Searched ``Cornell Law School legal information institute'' on Google. 2. Opened https://www.law.cornell.edu/. 3. Clicked Get The Law - Federal Rules - Federal Rules of Evidence (fourth section down). 4. Found the article that has ``witnesses'' in the most titles (VII). 5. Opened the first rule (701). 6. Scrolled to the last amendment as of 2021 (2011 amendment). 7. Found the word that was deleted (inference).}
\ErrorAnalysisBox{Web Toolkit Failure}
{The question asked to count the titles containing the exact word ``witnesses''. As a result, the browser also counted the titles containing the singular ``witness'' (such as ``Who May Impeach a Witness'').}
{
\{
``role'': ``assistant'', \\
``tool\_calls'': [
\{
``type'': ``function'', \\
``function'': \{
    ``name'': ``browse\_url'', \\
    ``arguments'': ``\{``start\_url'': ``https://www.law.cornell.edu/rules/fre/article\_vi'', ``task\_prompt'': ``List all the rule titles in Article VI of the Federal Rules of Evidence to count how many contain the word 'witnesses'''\}''
\}
\}
]
\}, \\
\{
    ``role'': ``tool'', \\
    ``content'': ``The task was to list all the rule titles in Article VI of the Federal Rules of Evidence and count how many contain the word ``witnesses.'' Here's the final answer:The following rule titles contain the word ``witnesses'': 1. Rule 605. Judge's Competency as a \textbf{Witness} 2. Rule 606. Juror's Competency as a \textbf{Witness} 3. Rule 607. Who May Impeach a \textbf{Witness} 4. Rule 608. A \textbf{Witness}'s Character for Truthfulness or Untruthfulness 5. Rule 611. Mode and Order of Examining Witnesses and Presenting Evidence 6. Rule 612. Writing Used to Refresh a \textbf{Witness} 7. Rule 613. \textbf{Witness}'s Prior Statement 8. Rule 614. Court's Calling or Examining a \textbf{Witness} 9. Rule 615. Excluding Witnesses. Total count: 9 rule titles contain the word ``witnesses.'' '',
\}
}

%—— Document Toolkit Failure ——%
\ErrorCaseBox{Document Toolkit Failure}{2}{44}
{The work referenced in footnote 397 of Federico Lauria's 2014 dissertation is also the source for the titles of two paintings in the Smithsonian American Art Museum's collection, as of August 2023. What is the absolute difference between the chapter numbers of the chapters that the titles of these two paintings quote?}
{8}
{1. Use search engine to search for ``Federico Lauria's 2014 dissertation''. 2. Open the result from philarchive.org and open the PDF file for the full paper. 3. Search for footnote 397 to find that the referenced work is Thomas Hobbes's ``Leviathan''. 4. Use search engine to search for ``Smithsonian American Art Museum collection search''. 5. Go to the museum's search webpage. 6. Enter ``Hobbes Leviathan'' into the search box and submit the search. 7. Open the two results, one by Jan Stussy (``A free man...'') and one by Leon Karp (``Hereby it is manifest...''). 8. Verify from the full titles of these works that the titles are quotes from ``Leviathan''. 9. Use search engine to search for ``Thomas Hobbes Leviathan full text''. 10. Open any result that contains the full text, like the Project Gutenberg version. 11. Search the text for the titles of each painting, using different substrings from the titles as needed to account for variations in spelling and punctuation. 12. Find that the ``A free man...'' quote is from Chapter XXI (21) and that the ``Hereby it is manifest...'' quote is from Chapter XIII (13). 13. Calculate the absolute difference of the chapter numbers: 21 - 13 = 8.}
\ErrorAnalysisBox{Document Toolkit Failure}
{The PDF address of the paper has been determined, but the Page limit exceeded.}
{
\{ ``role'': ``assistant'', 
``tool\_calls'': [ \{ ``function'': \{ ``name'': ``extract\_document\_content'', \\ ``arguments'': ``\{ ``document\_path'': ``https://philarchive.org/go.pl?id=LAUQLO\&proxyId=\&u=
https\%3A\%2F\%2Fphilpapers.org\%2Farchive\%2FLAUQLO.pdf'', 
``query'': ``footnote 397 Moby Dick chapter titles'' \}'' \} ] \}, \\ 
\{ ``role'': ``tool'', \\ ``content'': ``(True, `\textbf{\textcolor{red}{Error while processing document: Page limit exceeded}}')'' \}
}

%—— Multimodal Toolkit Failure ——%
\ErrorCaseBox{Multimodal Toolkit Failure}{3}{17}
{Which of the fruits shown in the 2008 painting ``Embroidery from Uzbekistan'' were served as part of the October 1949 breakfast menu for the ocean liner that was later used as a floating prop for the film ``The Last Voyage''? Give the items as a comma-separated list, ordering them in clockwise order based on their arrangement in the painting starting from the 12 o'clock position. Use the plural form of each fruit.}
{pears, bananas}
{1. Use search engine to search for ``2008 painting Embroidery from Uzbekistan''. 2. Open the top result, a link to the painting's page on the Dayton Art Institute website, and verify that the painting has the specified title and year. 3. Identify the fruits in the painting as watermelon, pear, lemon, and banana, which can be verified by either watching the video on the page or reading its linked transcript. 4. Use search engine to search for ``ocean liner floating prop The Last Voyage''. 5. Note from the results that this ocean liner was the SS \u00cele de France. 6. Use search engine to search for ``October 1949 breakfast menuSS \u00cele de France''. 7. Go to the result that shows the vintage SS \u00cel de France breakfast menu for October 1949. 8. Search the menu for each of the four fruits from the painting, finding ``Pear'' and ``Bananas'' but no matches for ``lemon'' or ``watermelon''. 9. Check the positions of the fruits in the painting to find that the pears come before the bananas in clockwise order starting from the 12 o'clock position. 10. Format the final answer as specified using the correct ordering: pears, bananas.}
\ErrorAnalysisBox{Multimodal Toolkit Failure}
{ImageAnalysisToolkit recognizes that the banana is placed at 12 o'clock and the pear is placed at 9 o'clock, but their actual positions are 11 o'clock and 7 o'clock respectively.}
{
\{
  ``subtask'': ``Create the final answer by listing the matching fruits in a comma-separated list, ordered clockwise from the 12 o'clock position as they appear in the painting, using the plural form of each fruit.'',\\
  ``result'': ``I'll solve this task by creating a final answer that lists the matching fruits in a comma-separated list, ordered clockwise from the 12 o'clock position as they appear in the painting, using the plural form of each fruit. Based on the dependency results, I need to: 1. Identify which fruits from the painting ``Embroidery from Uzbekistan'' were also served on the SS Île de France's October 1949 breakfast menu 2. List these matching fruits in clockwise order starting from the 12 o'clock position 3. Use the plural form of each fruit From the comparison analysis in the dependency results, I can see that the fruits that appear in both the painting and the breakfast menu are: - Bananas - Pears Now, I need to arrange these in clockwise order starting from the 12 o'clock position as they appear in the painting. According to the dependency results, the clockwise order of fruits in the painting starting from 12 o'clock is: \textbf{\textcolor{red}{1. Bananas (12 o'clock) 2. Watermelon (3 o'clock) 3. Pears (6 o'clock) 4. Lemons (9 o'clock) }}Since only bananas and pears match with the breakfast menu, and they are already in plural form, the final answer is: bananas, pears This is the comma-separated list of matching fruits in clockwise order from the 12 o'clock position as they appear in the painting, using the plural form of each fruit.''
\}
}

%—— Response Format ——%
\ErrorCaseBox{Response Format}{2}{85}
{At the two-minute mark in the YouTube video uploaded by the channel GameGrumps on May 14, 2017 as part of their playthrough of the game Mario Kart 8 Deluxe, the show hosts are competing on one of the games racetracks. What was the world record time for that track in the games 150cc mode as of June 7, 2023? Express your answer in minutes and seconds, rounding the seconds to the nearest hundredth, e.g. 1:01.001.}
{1:41.614}
{1. Search the web for ``gamegrumps mario kart 8 deluxe may 14 2017''. 2. Click on the YouTube video result. 3. Navigate to two minutes into the video. 4. Scroll further back until I see the name of the racecourse, Yoshi Circuit. 5. Search the web for ``mario kart 8 deluxe yoshi circuit world record 150cc''. 6. Scroll down until I find a reliable world record listing site. 7. Navigate through the site until I find the record that meets the specified criteria. 8. Read the date the record was set to confirm that it applies to the question's specified date.}
\ErrorAnalysisBox{Response Format}
{This question requires rounding the seconds to the nearest hundred, such as 1:01.001, but the agent finally gives the result of **1:41.61**.}
{
\{
``role'': ``assistant'', \\
``content'': ``I need to analyze the results from all subtasks to determine the final answer to the original question.Looking at the results: 1. The first subtask identified the correct GameGrumps video from May 14, 2017: ``Mario Kart 8 Deluxe: The Grand Prix - PART 7'' (https://www.youtube.com/watch?v=nvaLkvUkW0w). 2. The second subtask determined that at the two-minute mark of this video, the hosts are racing on **Yoshi Circuit**. 3. The third subtask found that the world record time for Yoshi Circuit in 150cc mode as of June 7, 2023 was **1:23.964**. 4. The fourth subtask formatted this time as **1:23.96** (rounding to the nearest hundredth). 5.However, the verification task found a different time: **1:41.614** from an archived version of the Mario Kart 8 Deluxe World Records website from May 29, 2023. There's a discrepancy between the two reported world record times. The verification task provides more specific details, including: - The record holder: ``Army'' from France Date set: March 10, 2023 - Character/kart setup used: Birdo (Black) with Mr. Scooty kart, Azure Roller tires, and Paper Glider The verification task also specifically mentions ``GCN Yoshi Circuit,'' which is the full name of the track (GCN indicating it originated in GameCube's Mario Kart: Double Dash).Given that the verification task provides more detailed information and cites a specific archive date (May 29, 2023) that is closer to our target date (June 7, 2023), I will consider this the more reliable result.Therefore, \textbf{the final answer to the original question is: 1:41.61}(1 minute, 41.61 seconds)This is the world record time for Yoshi Circuit in Mario Kart 8 Deluxe's 150cc mode as of June 7, 2023, expressed in minutes and seconds with the seconds rounded to the nearest hundredth.''
\}
}

%—— Reasoning Error ——%
\ErrorCaseBox{Reasoning Error}{3}{25}
{In NASA's Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon.}
{White; 5876}
{
1.Use search engine to search for ``NASA's Astronomy Picture of the Day 2006 January 21''. 2. Open the link to the image. 3. Read the explanation to find that the image is of astronaut Charles ``Pete'' Conrad reflected in the helmet of astronaut Alan Bean. 4. Observe that the smaller astronaut in the image is the one reflected in the other's helmet, so the smaller astronaut must be Charles ``Pete'' Conrad. 5. Go to the Wikipedia page for Charles ``Pete'' Conrad. 6. Search for ``Astronaut Group'' to find that Conrad was a member of NASA Astronaut Group 2. 7. Open the Wikipedia pages for each member of NASA Astronaut Group 2. 8. For those who are not deceased, go to View history and select the latest version of their Wikipedia page as of August 2023. 9. Compare the times listed in the infobox of each astronaut's Wikipedia page under ``Time in space'', observing that Ed White has the least time in space with 4d 01h 56m, but also that Elliott See does not have a listed ``Time in space''. 10. Read through Elliott See's Wikipedia article to find that he died in an accident before his first space flight, so he should be excluded, making Ed White's 4d 01h 56m the least amount of time in space. 11. Convert 4d 01h 56m to minutes: 4d * 24h/d * 60m/h + 1h * 60m/h + 56m = 5,876m. 12. Format the final answer as specified: White; 5,876.
}
\ErrorAnalysisBox{Reasoning Error}
{The accumulated time in space for NASA's ``New Nine'' had been correctly captured, but the people who had not been to space were incorrectly taken into account.}
{
\{
  ``subtask'': ``Research the space time duration for each astronaut in this group. Use web search to find reliable sources about each astronaut's missions and total time spent in space. Create a comprehensive list with their names and space time durations.'',\\
  ``result'': ``Space Time Durations for NASA Astronaut Group 2 Members''\\
  1. **Neil Armstrong** – Total time in space: 8 days, 14 hours, 12 minutes – Missions: Gemini 8 (1966), Apollo 11 (1969)\\
  2. **Frank Borman** – Total time in space: 19 days, 21 hours, 35 minutes – Missions: Gemini 7 (1965), Apollo 8 (1968)\\
  3. **Charles Pete Conrad** – Total time in space: 49 days, 3 hours, 38 minutes – Missions: Gemini 5 (1965), Gemini 11 (1966), Apollo 12 (1969), Skylab 2 (1973)\\
  4. **Jim Lovell** – Total time in space: 29 days, 19 hours, 5 minutes (715 hours, 5 minutes) – Missions: Gemini 7 (1965), Gemini 12 (1966), Apollo 8 (1968), Apollo 13 (1970)\\
  5. **James McDivitt** – Total time in space: 14 days, 2 hours, 56 minutes – Missions: Gemini 4 (1965), Apollo 9 (1969)\\
  \textbf{6. **Elliot See** – Total time in space: 0 minutes – Note: Selected but never flew; died before Gemini 9 mission.}\\
  7. **Thomas P. Stafford** – Total time in space: 21 days, 3 hours, 43 minutes (507 hours, 43 minutes) – Missions: Gemini 6A (1965), Gemini 9A (1966), Apollo 10 (1969), Apollo-Soyuz Test Project (1975)\\
  8. **Ed White** – Total time in space: 4 days, 1 hour, 56 minutes – Missions: Gemini 4 (1965); first American spacewalk; died in Apollo 1 fire.\\
  9. **John Young** – Total time in space: 34 days, 19 hours, 39 minutes – Missions: Gemini 3 (1965), Gemini 10 (1966), Apollo 10 (1969), Apollo 16 (1972), STS-1 (1981), STS-9 (1983)
\}
}

%—— Language/Question Ambiguity ——%
\ErrorCaseBox{Language/Question Ambiguity}{2}{73}
{I'm curious about how much information is available for popular video games before their release. Find the Wikipedia page for the 2019 game that won the British Academy Games Awards. How many revisions did that page have before the month listed as the game's release date on that Wikipedia page (as of the most recent entry from 2022)?}
{60}
{1. Search the web for British Academy Video Games Award for Best Game 2019. 2. Find the answer, Outer Wilds. 3. Find the Wikipedia page for Outer Wilds. 4. Go to the last revision from 2022. 5. Note the release date, May 29, 2019. 6. View the page history. 7. Count how many edits were made to the page before May 2019. 8. Arrive at the answer, 60.}
\ErrorAnalysisBox{Language/Question Ambiguity}
{There is an ambiguity in interpreting the phrase ``2019 game that won the British Academy Games Awards.'' One interpretation refers to the game that won the ``Best Game'' award at the 2019 BAFTA Games Awards (British Academy Games Awards), regardless of when it was actually released. The other interpretation requires the game to have been both released in 2019 and won the BAFTA Game Award. This difference leads to inconsistent results, as the agent prefers the former (based solely on the award year), while the ground truth requires both conditions (release year and award) to be met.}
{
\{``subtask'': ``Search for the \textbf{\textcolor{red}{2019 game that won the British Academy Games Awards (BAFTA)}} and identify the specific game.'',\\
  ``result'': ``2019 BAFTA Games Awards Winner God of War  Based on the information from the 15th British Academy Games Awards held on April 4, 2019, **God of War** (2018) was the winner of the Best Game award. Game Details - **Developer**: Santa Monica Studio - **Publisher**: Sony Interactive Entertainment Europe - **Platform**: PlayStation 4 - **Release Date**: April 2018 Awards Won (5 total) 1. Best Game 2. Audio Achievement 3. Music 4. Narrative 5. Performer (Jeremy Davies for his role as ``The Stranger'') Additional Information - God of War received 10 nominations in total, making it the most nominated game at the ceremony - The 15th BAFTA Games Awards honored games released in 2018, though the ceremony was held in 2019''\}
}

%—— Context Exceed ——%
\ErrorCaseBox{Context Exceed}{3}{5}
{What animals that were mentioned in both Ilias Lagkouvardos's and Olga Tapia's papers on the alvei species of the genus named for Copenhagen outside the bibliographies were also present in the 2021 article cited on the alvei species' Wikipedia page about a multicenter, randomized, double-blind study?}
{mice}
{1. Searched ``alvei copenhagen'' on Google. 2. Opened https://en.wikipedia.org/wiki/
Hafnia\_(bacterium). 3. Searched ``Ilias Lagkouvardos hafnia alvei'' on Google. 4. Opened https://www.mdpi.com/2076-2607/11/1/123?type=check\_update\&version=2. 5. Opened a new tab. 6. Searched ``Olga Tapia hafnia alvei'' on Google. 7. Opened https://pubmed.ncbi.nlm.nih.gov/36080356/. 8. Found all animals mentioned in the first paper. 9. Searched each animal from the first paper in the second paper. 10. Noted the animals mentioned in both outside the bibliographies. 11. Went back to the Wikipedia article. 12. Opened the link in the references to ``The Probiotic Strain H. alvei HA4597\textregistered{} Improves Weight Loss in Overweight Subjects under Moderate Hypocaloric Diet: A Proof-of-Concept, Multicenter Randomized, Double-Blind Placebo-Controlled Study''. 13. Opened the PDF. 14. Found the animals shared by all three papers.}
\ErrorAnalysisBox{Context Exceed}
{The content of Olga Tapia'research has been extracted correctly, but the mice mentioned in the text have not been extracted from the content successfully.}
{
\{
  ``role'': ``assistant'',\\
  ``tool\_calls'': [ \{\\
      ``type'': ``function'',\\
      ``function'': \{\\
        ``name'': ``extract\_document\_content'',\\
        ``arguments'': ``\{``document\_path'': ``https://www.mdpi.com/1420-3049/27/17/5585'', ``query'': ``Olga Tapia research on Hafnia alvei with mentions of animals''\}'' \\
      \} \} ] \}
}

%—— Limited Coding Capability ——%
\ErrorCaseBox{Limited Coding Capability}{1}{36}
{Bob was invited to participate in a game show, and he advanced to the final round. The final round offered Bob the chance to win a large sum by playing a game against the host. The host has 30 shiny prop coins, each of which is worth \$1,000 if Bob manages to win them by playing the game. The host hides the coins in three different prize boxes and then shuffles their order. The only rule restricting the host's coin placement is that one box must contain at least 2 coins, and one box must contain 6 more coins than another box. In order to play, Bob must submit three guesses, one guess for the number of coins in each box. The box is then opened and the number of coins is revealed. If Bob's guess is a number greater than the number of coins in the box, Bob earns no coins. If Bob guesses a number equal to or less than the number of coins in the box, Bob wins a number of coins equal to his guess.If Bob plays uses the optimal strategy, what's the minimum amount of money he can win from the game?}
{16000}
{Step 1: Evaluate the problem statement provided by my user, storing the relevant information: 30 coins with a value of \$1,000 distributed between 3 boxes. Each box must contain at least 2 coins. One box must contain 6 more coins than another. Step 2: Evaluate the base distribution: 2-8-20, noting that two boxes must contain at least 8 coins. Step 3: Evaluate the most even allowable distribution: 8,8,14, noting that two boxes must contain at least 8 coins. Step 4: Evaluate a case where Bob guesses 8 for each box in the outlier distributions. Step 5: For the worst case 2-8-20 distribution, Bob wins 0+8+8 = 16 coins. Step 6: For the 8-8-14 distribution, Bob wins 8+8+8 = 24 coins. Step 7: Convert the worst-case coin count to a prize value, 16*\$1,000 = \$16,000. Step 8: Report the correct answer to my user: ``\$16,000''}
\ErrorAnalysisBox{Limited Coding Capability}
{The agent ignored the hard constraint that each box must contain at least 2 coins when writing the code, and mistakenly considered the distribution of 0 and 1 coins (such as (0,6,24), (1,7,22), etc.) as legal.}
{
\{``subtask'': ``Implement a solution using code to systematically explore all valid coin distributions, calculate Bob's optimal guesses for each distribution, and determine the minimum guaranteed winnings.'',\\
  ``result'': ``Optimal Strategy for the Game Show Problem  I've implemented a solution to systematically explore all valid coin distributions, calculate Bob's optimal guesses for each distribution, and determine the minimum guaranteed winnings. Let me explain my approach and findings. Solution Approach  1. **Generate Valid Distributions**: I created a function to generate all valid distributions of 30 coins among 3 boxes that satisfy the constraints: - Total of 30 coins - \textbf{One box must contain at least 2 coins} - One box must contain 6 more coins than another box  2. **Calculate Winnings**: For each distribution and guessing strategy, I calculated the maximum possible winnings by trying all permutations of the guesses (since Bob can choose which guess to apply to which box).  3. **Find Optimal Strategy**: I implemented a systematic search to find the strategy that maximizes the minimum guaranteed winnings across all possible distributions. Key Findings  1. **Valid Distributions**: The code identified 12 valid distributions of 30 coins that satisfy all constraints.  2. **Optimal Strategy**: \textbf{\textcolor{red}{The optimal strategy for Bob is to guess **[0, 6, 12]** for the three boxes.}}  3. **Minimum Guaranteed Winnings**: With this strategy, Bob is guaranteed to win at least **\$18,000**, regardless of how the host distributes and shuffles the coins.  4. **Worst Case Scenario**: The worst-case distribution for Bob is **(0, 6, 24)**, where one box has 0 coins, one has 6 coins, and one has 24 coins.  5. **Verification**: I verified that with the optimal strategy against the worst-case distribution, Bob indeed wins exactly \$18,000.  Comparison with Other Strategies  I also tested several other strategies mentioned in the dependency results: - Strategy (6, 6, 6): Minimum guaranteed \$12,000 - Strategy (2, 6, 10): Minimum guaranteed \$16,000 - Strategy (1, 5, 12): Minimum guaranteed \$17,000 - Strategy (7, 7, 7): Minimum guaranteed \$7,000 - Strategy (8, 8, 8): Minimum guaranteed \$8,000 - Strategy (6, 6, 7): Minimum guaranteed \$13,000 - Strategy (6, 7, 7): Minimum guaranteed \$13,000  This confirms that [0, 6, 12] is indeed the optimal strategy, guaranteeing \$18,000 in winnings. How the Optimal Strategy Works  With the [0, 6, 12] strategy: 1. Bob guesses 0 for the box with the fewest coins 2. Bob guesses 6 for the box with the middle number of coins 3. Bob guesses 12 for the box with the most coins  This strategy works because: - It ensures Bob always wins from at least two boxes - Even in the worst-case distribution (0, 6, 24), Bob can match his guesses optimally to win \$18,000 - No other strategy can guarantee more than \$18,000 across all possible distributions  Therefore, \$18,000 is the minimum amount Bob can win using the optimal strategy.''
\}
}




\section{Case Studies}
\label{app:case_studies}

% Challenges associated to relying on the web. Designing questions can be delicate when a source of truth is
% hosted on the web. First, the evidence might change over time. For example, a Wikipedia article could be
% updated between the moment the question is created and the moment it is asked to an AI assistant, potentially
% removing the evidence required to answer. For such questions, it is often important to specify a version of
% the evidence, such as the page's date. In practice, we find our benchmark to be robust to these changes
% since we try to rely as much as possible on evidence that will likely pass the test of time. Second, some
% website owners wish to prevent access to parts or totality of their website from bots via their robots.txt files.
% While this is rather a demand than a constraint, it is obviously desirable to comply. For example, OpenAI
% provides instruction to website owners wishing to forbid access to GPT4 on how to modify their robots.txt
% accordingly. Hence, we verify that accessing the part of the website hosting the evidence is not restricted.

\tcbset{colframe = blue!50!black, colback = white,
        fonttitle = \bfseries,
        nobeforeafter, center title}

The real environment is often more complex than the pre-built simulation environment. The GAIA benchmark is built in real-life environments, which brings many challenges. Below we will introduce the challenges that agents may face in solving problems in real environments.



\subsection{Differences in Information Sources}
In real-world scenarios, multiple information sources may provide different answers to the same query. 
This discrepancy creates a challenge where tasks might have several valid responses, each considered ``correct'' depending on the reference source consulted.

To illustrate this phenomenon, consider the following task: 

\begin{tcolorbox}[title = {Task from GAIA Benchmark}]
  \textbf{Question}: Of the authors (First M. Last) that worked on the paper ``Pie Menus or Linear Menus, Which Is Better?'' in 2015, what was the title of the first paper authored by the one that had authored prior papers?\\

  \textbf{Answer}: Mapping Human Oriented Information to Software Agents for Online Systems Usage.\\

  \textbf{Annotation steps}:
  1. Searched ``Pie Menus or Linear Menus, Which Is Better?'' on Google.
  2. Opened ``Pie Menus or Linear Menus, Which Is Better?'' on https://oda.oslomet.no/oda-xmlui/handle/10642/3162.
  3. Clicked each author's name.
  4. Noted the name that had no other papers listed.
  5. Searched ``Murano, Pietro'' on Google.
  6. Opened http://www.pietromurano.org/.
  7. Clicked ``Publications''.
  8. Found the earliest paper he contributed to.
\end{tcolorbox}

as shown in Figure \ref{figure:pietro-homepage}, consulting the author's personal homepage yields ``Mapping Human-Oriented Information to Software Agents for Online Systems Usage'' as the answer. 
However when referencing Google Scholar, the first publication listed is ``A new software agent 'learning' algorithm'' as shown in Figure \ref{figure:pietro-google-scholar}. 
This inconsistency demonstrates how different authoritative sources can lead to divergent yet equally defensible answers.

\begin{figure}[htbp]
    \centering
    \includegraphics[scale=0.2]{images/appendix/pietro-homepage.jpg}
    \caption{Pietro Murano's personal homepage showing publication history (accessed on 2025-05)}
    \label{figure:pietro-homepage}
\end{figure}


\begin{figure}[htbp]
    \centering
    \includegraphics[scale=0.55]{images/appendix/pietro-google-scholar.jpg}
    \caption{Pietro Murano's Google Scholar profile displaying a different publication order (accessed on 2025-05)}
    \label{figure:pietro-google-scholar}
\end{figure}


\subsection{Outdated information}

\textbf{Outdated information} is another common issue, especially when retrieving data via web search and information retrieval tools. The agent might encounter outdated or incorrect content that misguides its reasoning process, ultimately impacting the quality and effectiveness of task execution. For example, the original task may require visiting someone's YouTube channel or searching for some content in the channel. However, by now, the content may have been modified by the channel user, which will directly lead to error in the ground truth.

Here is an example from GAIA benchmark:

\begin{tcolorbox}[title = {Task from GAIA Benchmark}]
  \textbf{Question}: Eva Draconis has a personal website which can be accessed on her YouTube page. What is the meaning of the only symbol seen in the top banner that has a curved line that isn't a circle or a portion of a circle? Answer without punctuation.\\

  \textbf{Answer}: War is not here this is a land of peace.\\

  \textbf{Annotation steps}:
  1. By googling Eva Draconis youtube, you can find her channel.
  2. In her about section, she has written her website URL, orionmindproject.com.
  3. Entering this website, you can see a series of symbols at the top, and the text ``see what the symbols mean here'' below it.
  4. Reading through the entries, you can see a short description of some of the symbols.
  5. The only symbol with a curved line that isn't a circle or a portion of a circle is the last one.
  6. Note that the symbol supposedly means ``War is not here, this is a land of peace.''
\end{tcolorbox}

According to the annotation, the agent should first go to Eva Draconis youtube channel, and check her personal website URL from about section. However, there's nothing in the channel now, and the personal webpage cannot be opened either.

% \begin{figure}[htbp]
%     \centering
%     \includegraphics[scale=0.25]{images/appendix/eva_draconis.jpg}
%     \caption{The youtube webpage that is used to solve the given task \notes{There is a human face. Is it allowed to put a human face here?}}
%     \label{figure:eva_draconis}
% \end{figure}


\subsection{Language Ambiguity}

\textbf{Language ambiguity} occurs when the agent encounters unclear or imprecise language in the user query, leading to incorrect reasoning. For example, a question might contain terms with multiple meanings or lack the necessary context to disambiguate the intended meaning.

% For example, in validation set of GAIA benchmark, there is a question ``If Eliud Kipchoge could maintain his record-making marathon pace indefinitely, how many thousand hours would it take him to run the distance between the Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary'', the model may produce ``17000'' (17000 hours) as the final answer instead of ``17'' (17 thousand hours), and ``17000'' may also be considered correct when interpreting the question as asking ``how many thousand hours''.

For example, consider the following task:

\begin{tcolorbox}[title = {Task from GAIA Benchmark}]
  \textbf{Question}: I'm curious about how much information is available for popular video games before their release. Find the Wikipedia page for the 2019 game that won the British Academy Games Awards. How many revisions did that page have before the month listed as the game's release date on that Wikipedia page (as of the most recent entry from 2022)?\\

  \textbf{Answer}: 60\\

  \textbf{Annotation steps}:
  1. Search the web for British Academy Video Games Award for Best Game 2019
  2. Find the answer, Outer Wilds
  3. Find the Wikipedia page for Outer Wilds
  4. Go to the last revision from 2022.
  5. Note the release date, May 29, 2019
  6. View the page history
  7. Count how many edits were made to the page before May 2019
  8. Arrive at the answer, 60
\end{tcolorbox}

There are two possible interpretations of the question:
(1) ``2019 award winner'': refers to the game that won the ``Best Game'' award at the 2019 BAFTA Games Awards (British Academy Games Awards), regardless of when it was actually released.
(2) ``2019 released and won the game'': refers to the game that was released in 2019 and has won the BAFTA Game Award.


\subsection{Network Instability \& Permission Deny} 

\textbf{Network instability} is a critical factor impacting task evaluation. Since web search tools rely on internet connection, network instability can result in the agent being unable to access required resources or cause task execution to be interrupted.

\textbf{Permission deny} is also a critical factor, which refers to scenarios where the agent attempts to access resources that require (1) authentication or (2) human verification (such as downloading a paper from a journal that requires login access or human verification). In some cases, IP region restriction could also prevent agent from futher processing the task. This could cause the agent to fail in completing the task, even if it has found the correct URL containing the final answer.


\section{Broader Impacts}
\label{app:broader_impacts}

\subsection{Data Pollution}

\textbf{Data pollution} refers to a situation where the agent might directly retrieve answers from a source (such as GAIA's Huggingface) instead of performing its own reasoning. This issue may lead to the agent ``cheating'' by relying on pre-existing answers from the training data, rather than conducting the reasoning required for the task. A simple way to solve this problem is restrict the agent's access to certain sources (e.g., by filtering out urls containing ``huggingface'' to prevent the agent from retrieving answers directly from specific datasets).


\subsection{Potential Dangerous Behaviors}

While using tools, the agent might display \textbf{dangerous behaviors}, such as accessing harmful content, pushing inaccurate information, generating inappropriate interactions, etc. If the agent can execute code/use the terminal locally, some operations of the agent may destroy some of the original local environment. These behaviors can pose a threat to user safety and system stability. 


% \section{Experimental Statistical Significance}

% \subsection{Model Comparison}
% \label{app:significance_comparison}

% \noindent \textbf{\textit{Multi-agent Inference}}. 
% To further understand the impact of architectural design, we conducted pairwise significance testing between models. Notably, the \workforce framework, significantly outperforms both the \textbf{Single Agent} and \textbf{Role Playing} baselines. 
% Specifically, it exceeds the Single Agent by a wide margin (\textit{t = -6.16}, \textit{p < 0.0001}), and achieves statistically significant improvements over Role Playing as well (\textit{t = -2.19}, \textit{p = 0.0294}). 
% These results indicate that our modular architecture yields measurable and robust gains across diverse tasks.

% \noindent \textbf{\textit{Multi-agent Training}}. 

\section{Statistical Significance Analysis}
\label{app:statistical_significance}

To validate the robustness of our empirical findings, we conducted Wilcoxon signed-rank tests to compare system-level performance across key baselines and planner configurations. 

First, we observe that \workforce significantly outperforms both the Single Agent and Role Playing baselines, with $p$-values of $< 0.0001$ and $0.0203$ respectively. These results confirm that the observed improvements are statistically significant rather than due to random variability.

Furthermore, we evaluate the impact of our proposed training method, \textit{Optimized Workforce Learning (OWL)}, by comparing OWL-trained Qwen2.5-32B-Instruct against the base Qwen2.5-32B-Instruct model. The Wilcoxon test yields a $p$-value of $0.0018$, indicating a significant improvement in task planning effectiveness after OWL-based reinforcement learning.


\section{More Details on Analysis}

\subsection{More Details on Robustness}
\label{app:robustness_details}

\begin{table}[h]
  \centering
  \begin{tabular}{lcccccc}
  \toprule
  \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c}{\textbf{Number of Capabilities}} & \multirow{2}{*}{\textbf{Mean} $\uparrow$} & \multirow{2}{*}{\textbf{Std} $\downarrow$} & \multirow{2}{*}{\textbf{IQR} $\downarrow$} \\
  \cmidrule(lr){2-4}
  & \textbf{1} & \textbf{2} & \textbf{$\geq$3} & & & \\
  \midrule
  Single Agent & 38.36 & 35.14 & 44.44 & 39.31 & 3.86 & 4.65 \\
  Role Playing & 62.34 & 51.14 & 34.62 & 49.36 & 11.39 & 13.86 \\
  \rowcolor{blue!10} Workforce (Ours) & 63.01 & 59.46 & 55.56 & 59.34 & 3.05 & 3.73 \\
  \bottomrule
  \end{tabular}
  \caption{Performance across different capability requirements, as well as overall robustness indicators including standard deviation and interquartile range (IQR)}
  \label{tab:capability-stability}
\end{table}


The complete table of performance across different capability requirements is presented in Table~\ref{tab:capability-stability}.

\subsection{Planner vs. Worker Training}

\begin{table}[h]
  \centering
  \begin{tabular}{lcccccc}
      \toprule
      \multirow{2}{*}{\textbf{Variant}} & \multicolumn{2}{c}{\textbf{Trainable Components}} & \multirow{2}{*}{\textbf{Level 1}} & \multirow{2}{*}{\textbf{Level 2}} & \multirow{2}{*}{\textbf{Level 3}} & \multirow{2}{*}{\textbf{Average}} \\
      \cmidrule(lr){2-3}
      & \textbf{Planner} & \textbf{Worker} & & & & \\
      \midrule
      Qwen-32B-Instruct & \texttimes & \texttimes & 49.05 & 33.72 & 19.23 & 36.36 \\
      ~~~\textit{w.} Worker  & \texttimes & \checkmark & 43.39 & 30.23 & 11.54 & 31.51 \\
      ~~~\textit{w.} Both & \checkmark & \checkmark & 60.38 & 47.67 & 15.38 & 46.68 \\
      ~~~\textit{w.} Planner (Ours) & \checkmark & \texttimes & 60.38 & 45.34 & 15.38 & 45.45 \\
      \bottomrule
  \end{tabular}
  \caption{Performance comparison between different training configurations of planner and workers. Training the planner alone achieves nearly the same performance as training both components, while training only workers degrades performance.}
  \label{tab:trainable-components}
\end{table}

Table~\ref{tab:trainable-components} presents a detailed comparison of different training configurations for our multi-agent system. The results clearly demonstrate that training only the planner (45.45\% average accuracy) significantly outperforms training only the workers (31.51\%), and even performs comparably to training both components together (46.68\%). Interestingly, training only workers actually degrades performance below the baseline (36.36\%), suggesting that worker specialization without proper task decomposition can be counterproductive. These findings strongly support our design decision to prioritize planner optimization, as effective task planning and decomposition are more crucial than enhancing individual worker capabilities. The minimal performance gain from training both components (only +1.23\%) does not justify the substantial increase in computational costs, making planner-focused training the most efficient approach for improving overall system performance.