D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

# Create and activate environment
conda create -n dgara python=3.10 -y
conda activate dgara

# Clone the D-GARA repository and enter it
git clone https://github.com/sen0609/D-GARA.git
cd D-GARA

# Install dependencies
pip install -r requirements.txt

Preparing to Connect a Mobile Device via ADB

1. Install Android Debug Bridge (ADB).
2. Enable USB Debugging in developer options on your Android device.
3. Connect your device via USB and select File Transfer mode.
4. Verify connection:

   adb devices

   If your device appears in the list as device, you are ready to proceed.

Install ADB Keyboard (Optional)

1. Install ADB Keyboard on the device.
2. Switch the default input method to ADB Keyboard when programmatic typing is required.

Preparing to Connect an Android Emulator via ADB

1. Use Android Studio’s AVD Manager or a third-party emulator (e.g., MuMuPlayer).
2. Create an emulator with the desired API level (API 31 is recommended) and start it.
3. Confirm detection by ADB:

   adb devices
   # Example: emulator-5554    device
   

Extend Interference Modes

Extend the benchmark by designing new interference types (e.g., single-button, dual-button, banner, or image-based pop-ups). The process involves two layers: UI implementation and packaging, and logical registration within interference_library.yaml.

1. Implement the UI (APK Layer)

Edit our interference_template project in Android Studio to create the visual and interactive behavior of your new interference. Then rebuild and install the updated adbsmarttest.apk.

1. Clone & import interference_template into Android Studio.
2. Design the interference layout (popup, banner, toast, etc.) with unique, stable resource IDs.
3. Implement triggers (e.g., broadcast intent or activity entry point).
4. Build & install the release APK:

2. Register the Logic (YAML Layer)

Once the UI layer is ready, register its logic in interference_library.yaml. Each interference definition specifies its activity entry, parameters, and corresponding post actions.

Once registered, the new interference type becomes reusable within the D-GARA framework. You can later reference its id (e.g., data_warning_double_en) in task configurations or interference rules to trigger it dynamically during evaluation.

After completing these steps, your new interference mode can be directly integrated when creating or extending tasks in the Extend Tasks section below.

Extend Tasks

1. Pick app & draft tasks

Choose the target app and define tasks. Quick variants (e.g., “search iPhone 17” → “search PS5”) scale your set rapidly.

Example of a basic task definition — used in the following data collection process:

[
  {
    "id": 1,
    "description": "打开哔哩哔哩，搜索“稻香”，点击第一个视频，并点击“点赞”按钮",
    "app": "tv.danmaku.bili",
    "rule_key": "task_bili_search_and_like_video",
    "target_rule_ids": ["rule_bili_data_warning_double_1"]
  }
]

2. Collect pages (screenshots + UI XML) the agent may encounter in an app:

# Windows PowerShell:
python tools/datacollector.py ^
  --device 127.0.0.1:7555 ^
  --tasks ./configs/tasks.json ^
  --out ./Collected_trajs ^
  --max-steps 30

# macOS/Linux:
python tools/datacollector.py \
  --device 127.0.0.1:7555 \
  --tasks ./configs/tasks.json \
  --out ./Collected_trajs \
  --max-steps 30

Press S to save (screenshot + XML) or N to skip. Files are placed under Collected_trajs/<id>_<description>/.

3. Map interference points on meaningful pages to inject distractions deterministically:

- id: rule_bili_data_warning_double_1
  description: "Trigger data-usage warning on video page"
  priority: 500
  conditions:
    all:
      - type: semantic_element_exists
        match_threshold: 0.5
        keywords: ["简介","评价","关注"]
  actions:
    - type: inject_interference
      interference_id: "data_warning_double"
      trigger_probability: 1

4. Define success rules using terminal-state XML (tolerant to layout variants):

task_bili_search_and_like_video:
  description: "Unified validation of Bilibili like status (landscape & portrait)"
  conditions:
    - type: any_of
      sub_conditions:
        - type: element_property_contains
          selector: "resource-id='tv.danmaku.bili:id/frame_like'"
          attribute: "content-desc"
          value: "已点赞"
        - type: element_property
          selector: "resource-id='tv.danmaku.bili:id/like_icon'"
          attribute: "selected"
          value: "true"

Why human verification? Real-world apps evolve constantly, and some pages cannot be captured via ADB. Therefore, rule-based validation may not always be reliable. Until GUI agents can reliably detect task completion themselves, we rely on a lightweight human verification step to ensure accuracy.