Automating Android XR Before the Tooling Existed

A technical retrospective on UIAutomator2, TestBridge, and a custom reporter

When we started working on QA for an Android XR application, Android XR was not yet a mature or publicly established platform. It was closer to an idea becoming a product than to a conventional mobile ecosystem.

That context shaped every technical decision we made.

We were not simply choosing between existing automation frameworks. We were trying to discover whether the usual automation model could be applied to applications that lived in spatial environments, while the operating system, the devices, and the testing tools were all still evolving.

The application we were validating was Fox Sports XR. The product itself is not the main subject of this article. What matters is the engineering problem behind it: how do you automate an end-to-end test when part of the interface exists in three-dimensional space and the established mobile testing tools do not understand that space yet?

Starting without a reliable automation path

The first problem was the lack of mature tooling.

At that time, Appium was a natural option to consider for mobile device automation, but it was not capable of understanding the spatial environment in which Android XR applications ran. Its traditional interaction model assumes that elements have usable two-dimensional coordinates on a conventional screen. That assumption does not hold reliably when an application is rendered through spatial panels and XR surfaces.

Espresso was also not a practical solution for this project. The framework was designed around Android UI components and application-level front-end testing, but it did not provide a reliable way to automate the spatial interactions we needed on Android XR. This was not necessarily a permanent limitation of Espresso. Android XR was still in beta, and the ecosystem had not yet reached the point where its testing requirements were supported by established tools.

Google did not yet provide a complete, mature solution for end-to-end automation of spatial applications either. As a result, QA could not remain completely outside the development project. We had to integrate the automation infrastructure directly into the application codebase while keeping the production impact as small as possible.

Choosing UIAutomator2 as the foundation

We chose UIAutomator2 because it provided access to the device-level accessibility tree. That gave us something essential: visibility into what the system believed was being rendered.

UIAutomator could find Compose elements, read their text and content descriptions, inspect their state, and verify that elements existed. This made it useful for validating whether a screen had reached the expected state.

In practice, the accessibility tree became our observation layer:

val playPause = device.findObject(By.desc("Play/Pause Icon"))
val controlsAreVisible = playPause != null

That small distinction was fundamental. We could trust the discovery of an element more than we could trust a coordinate-based interaction with it.

However, UIAutomator was not designed to provide the complete QA workflow we needed. It could execute interactions and produce basic test results, but we needed a maintainable structure, evidence, reporting, and traceability to Xray. Those capabilities had to be added around it.

More importantly, discovering an element was not the same as interacting with it successfully.

The silent interaction failure in spatial environments

The most difficult problem appeared when entering spaces that were not completely flat.

UIAutomator was able to identify elements in the accessibility tree. It could find a button and report that the button was visible. It could also attempt to click it and behave as if the click had succeeded.

But in the XR environment, the coordinates reported by the accessibility layer did not necessarily correspond to the element’s actual position in three-dimensional space. The click could therefore miss the rendered control. In many cases, the failure was silent: the test framework found the element, issued the click, and continued without an explicit error, even though the application had not performed the expected action.

This made the problem particularly dangerous. A conventional test could appear to pass the interaction step while the application remained in the previous state.

The issue was not that UIAutomator could not see the interface. It was that its two-dimensional interaction model did not map correctly to the XR presentation layer.

TestBridge: keeping the UI visible while bypassing unreliable coordinates

Once we confirmed this behavior, we introduced TestBridge.

The bridge was designed as a small connection between the instrumentation tests and the running application’s internal objects. Instead of asking UIAutomator to click a spatial button, a test could first verify that the button existed in the accessibility tree and then invoke the same ViewModel method that the button was expected to call.

For example, a play/pause interaction followed this general pattern:

1. Find the play/pause element through its content description.
2. Confirm that it is actually present in the rendered accessibility tree.
3. Dispatch the action to the playback ViewModel on the application’s main thread.
4. Validate the resulting playback state.

The action was executed through the instrumentation main-thread mechanism rather than through a screen coordinate. This allowed us to work with spatial video controls on both a real XR device and an XR emulator with deterministic behavior.

The bridge exposed references to objects such as the playback controls ViewModel, the video player ViewModel, the navigation controller, and selected Compose state. It also provided helper operations for reading playback position, duration, seekbar percentage, pause state, and player readiness.

The core of the bridge was deliberately simple: expose volatile references that could be populated only for an instrumented debug run.

object TestBridge {
    @Volatile
    var playbackControlsViewModel: YBVRPlaybackControlsViewModel? = null

    @Volatile
    var videoPlayerViewModel: VideoPlayerViewModel? = null

    fun clear() {
        playbackControlsViewModel = null
        videoPlayerViewModel = null
    }
}

The real implementation also exposed navigation and selected Compose state. clear() was called during teardown so that ViewModels from one test could not leak into another test execution.

We tried to keep this integration as non-invasive as possible. The bridge existed in the debug path, while the release implementation was a no-op stub. Instrumentation detection was performed without making the production application depend directly on the test layer.

There was an important trade-off.

With this approach, we were no longer testing the physical click-to-handler connection itself. We were testing that the UI element was rendered and that the underlying action method behaved correctly. We had to assume that the button was correctly wired to that method, or cover that connection through other forms of UI validation.

That limitation was explicit and acceptable. In an XR environment where coordinate clicks could fail silently, testing the method deterministically was more valuable than pretending that an unreliable click represented a valid end-to-end interaction.

The interaction helper preserved the relationship between UI validation and internal execution:

private fun clickPlayPause(): Boolean {
    if (!device.hasObject(By.desc("Play/Pause Icon"))) return false

    val viewModel = TestBridge.playbackControlsViewModel
    if (viewModel == null) {
        val element = device.findObject(By.desc("Play/Pause Icon"))
            ?: return false
        element.click()
        return true
    }

    InstrumentationRegistry.getInstrumentation().runOnMainSync {
        viewModel.playPauseButtonAction()
    }
    return true
}

The production page object was more defensive and supported a UIAutomator fallback, but the principle remained the same: first prove that the control exists, then bypass the unreliable XR coordinate mapping.

From a working bridge to a usable QA system

Once TestBridge proved that the core interaction strategy worked, we moved to the next problem: making the solution usable at scale.

The first requirement was readability. Tests needed to be understandable by QA and development teams, so we adopted a Page Object structure. Page objects centralized selectors, content descriptions, waits, navigation helpers, and XR-specific interaction behavior.

The second requirement was evidence. A failed test needed to explain what happened without requiring someone to reproduce the failure manually on an XR device. The test base therefore supported:

• Screenshots on failure.
• Optional screenshots for every recorded step.
• Per-test structured logs.
• Optional screen recordings.
• Validation inside named test steps.

The step() helper became particularly important. A step could execute a validation and fail immediately when the expected state was not reached. When that happened, the infrastructure captured a failure screenshot before the assertion was raised.

The important part of the test base was that a named step could also be a real assertion:

protected fun step(name: String, validation: (() -> Boolean)? = null) {
    val passed = validation?.invoke() ?: true
    if (!passed) {
        takeScreenshot("STEP_FAILED_$name")
    }

    assertWithMessage("Step validation failed: $name")
        .that(passed)
        .isTrue()

    xrayListener.recordStep(name)
}

This prevented the report from containing steps that had merely been logged without actually reaching the expected state.

Building uiautomator-reporter

UIAutomator’s native output was not enough for this workflow, so we built uiautomator-reporter as a separate pure JVM Kotlin module.

Keeping the reporter independent from the Android SDK gave us two advantages. It could process artifacts outside the device, and it could run in a CI environment without becoming part of the production application. The reporter could be used as a Gradle module, a standalone fat JAR, a programmatic API, or a command-line tool.

Its processing pipeline was:

JUnit XML results
        +
Screenshots
        +
Per-test logs
        |
        v
uiautomator-reporter
        |
        v
Interactive HTML report

The reporter parses JUnit XML into a common test result model with passed, failed, skipped, and error states. It then associates screenshots with test classes and methods using a naming convention such as:

ClassName_methodName_STEP_01_description_timestamp.png

The same principle is used for log files. This allows the final report to display the execution timeline, failure evidence, structured logs, error details, and summary information for each test.

The reporter’s public entry point keeps the post-processing flow small and explicit:

fun generateReport(
    artifactsDir: File,
    outputFile: File,
    testResultsSubdir: String = "results",
    screenshotsSubdir: String = "screenshots",
    testSourcesDir: File? = null
): Boolean {
    var results = JUnitXmlParser.parseTestResults(
        File(artifactsDir, testResultsSubdir)
    )

    val screenshots = ScreenshotManager.findScreenshots(
        File(artifactsDir, screenshotsSubdir)
    )
    val screenshotsByTest = ScreenshotManager.organizeScreenshotsByTest(
        screenshots,
        results
    )
    val logsByTest = LogsManager.readLogs(
        File(artifactsDir, "logs"),
        results
    )

    results = results.map { result ->
        result.copy(
            systemOut = logsByTest[
                "${result.className}#${result.name}"
            ].orEmpty()
        )
    }

    val html = HtmlReportGenerator().generateReport(
        results,
        screenshotsByTest,
        emptyMap(),
        File(artifactsDir, testResultsSubdir)
    )
    outputFile.writeText(html)
    return true
}

The implementation also enriches results with summaries extracted from the test source and expands class-level skipped tests when the XML does not contain individual test cases.

The HTML report was intentionally generated as a self-contained artifact with embedded styling and behavior. It included result filters, text search, test summaries, expandable details, screenshots, and logs. This made it practical to inspect a test run without direct access to the device.

Xray integration

Traceability to QA management was another requirement. Each test could declare its Xray case with an annotation:

@XrayTestCase(
    testKey = "TEST-123",
    summary = "Validate spatial video playback"
)

The reporter extracted these annotations from the Kotlin test sources and mapped them to the actual JUnit XML results. It then generated Xray-compatible JSON, created execution metadata, authenticated against Xray Cloud, and uploaded the results through the multipart execution API.

The mapping was intentionally based on the stable Xray key rather than on the display name of the test:

val result = testResults.firstOrNull { xmlResult ->
    xmlResult["methodName"] == methodName &&
        xmlResult["className"]?.endsWith(className) == true
}

val xrayResult = mapOf(
    "testKey" to testKey,
    "status" to (result?.get("status") ?: "PASSED")
)

The final upload used Xray Cloud’s multipart execution endpoint:

val token = XrayUploader.authenticate(config)
XrayUploader.uploadResults(
    config = config,
    token = token,
    resultsFile = resultsFile,
    infoFile = infoFile,
    testCount = testsList.size
)

This separated three concerns:

• The test code remained readable.
• The JUnit output represented what actually happened during execution.
• The reporter connected that execution to the corresponding Xray cases.

The test-side listener also recorded test duration, status, errors, and named steps. In practice, this resulted in two related reporting paths: detailed results captured during instrumentation and a post-processing path that reconstructed Xray results from source annotations and JUnit XML.

The Gradle pipeline

The complete workflow was integrated into Gradle rather than requiring a collection of manual commands.

The main task cleaned old artifacts, ran the instrumented tests, pulled screenshots, videos, and logs from the device with ADB, copied the JUnit XML results, built the reporter fat JAR, and generated the HTML report.

This turned the automation system into a repeatable pipeline:

Run tests
    -> Collect device artifacts
    -> Parse JUnit results
    -> Match screenshots and logs
    -> Generate HTML
    -> Optionally import to Xray

The Gradle task was the glue between the Android test process and the JVM reporter:

tasks.register("runUiAutomatorAndCollect") {
    dependsOn("cleanUiAutomatorArtifacts")
    dependsOn("connectedWhitelabelForcedEnglishNoCustomEnvironmentDebugAndroidTest")
    dependsOn(":uiautomator-reporter:fatJar")

    doLast {
        adbPull("/sdcard/Pictures/uiautomator_screenshots", "screenshots")
        adbPull("/sdcard/Documents/uiautomator_logs", "logs")
        adbPull("/sdcard/Movies/uiautomator_videos", "videos")
        generateHtmlReport(uiautomatorArtifactsDir)
    }
}

The real task also copied the JUnit XML and Android test reports and loaded the reporter fat JAR through URLClassLoader. The shortened example shows the architectural boundary: Android produces the evidence, while the JVM module interprets and presents it.

The same reporter could also regenerate the HTML from existing artifacts without rerunning the tests. That was useful when the test execution was expensive or when the report presentation needed to be refined after the device run.

What we learned

The main lesson was that XR automation could not be treated as ordinary mobile automation with a different screen size.

The testing model had to separate three questions:

1. Is the element actually present in the accessibility tree?
2. Can the intended behavior be executed reliably in the spatial environment?
3. Can the result be explained and traced after the test has finished?

UIAutomator2 helped answer the first question. TestBridge solved the most critical part of the second. uiautomator-reporter, structured logs, screenshots, and Xray integration solved the third.

The solution was created while Android XR was still in beta, so some of the limitations were specific to the platform and tooling available at that time. It should not be interpreted as proof that Espresso, Appium, or other frameworks can never support XR. It was the practical response to the state of the ecosystem during development.

We did not wait for a complete testing ecosystem to appear. We built the smallest bridge needed to make spatial interactions deterministic, then built the reporting and traceability infrastructure required to turn that experiment into a usable QA process.