Starting March 2nd we started getting numerous reports of Meta Quest 2 devices lagging and crashing. After initial testing, it became clear that this problem was happening at the device operating system level and that we'd need support from the Meta team to find a solution. This issue has had our highest attention since it was reported and we're doing everything we can to find short-term and long-term solutions.
UPDATES
March 3rd at 5pm PST: Meta has confirmed that they were able to reproduce the issue and has informed us that their next firmware version (v50) will fix the issue. This firmware version is slated to be released next week. We are actively working to get early access to this firmware version to confirm the fix, and will also be working to expedite the fix to our customers. Please contact us at support@managexr.com if you'd like to get expedited access to this update.
March 5th at 3pm PST: We have submitted a large number of device serial numbers of devices affected by the bug to Meta for expedited access to the new firmware. We are still waiting for confirmation.
March 6th at 5pm PST: Meta has informed us that the v50 firmware will be rolled out to all ManageXR customers by March 8th. We are working to get this firmware out to you even sooner.
March 7th at 6pm PST: We have confirmed that firmware v50 solves the issue! The fix is currently being rolled out to all Meta Quest devices and should be available to you by March 8th. The update will be installed automatically if the device is kept on and connected to the internet, or you can manually check for updates following these instructions. (Note: To manually check for updates you may need to temporarily disable Kiosk Mode/Home Screen)
March 9th at 7:30am PST: Meta has informed us that the full v50 firmware rollout has been delayed by a day. They now plan for the update to be available to all devices by March 9th. We know how frustrating it is to know that a fix is available and to not yet be able to access it. Thank you for your patience. The update will be installed automatically if the device is kept on and connected to the internet, or you can manually check for updates following these instructions. (Note: To manually check for updates you may need to temporarily disable Kiosk Mode/Home Screen)
March 10th at 8am PST: Meta has informed us that 100% rollout of the v50 firmware has been delayed again. We are now working with Meta to expedite the update to our customer’s devices. We know how frustrating it is to know that a fix is available and to not yet be able to access it. Thank you for your patience.
March 13th, 11am PST: The v50 firmware continues to be delayed, with only about 30% of devices able to access the update. We continue to work with Meta to unblock the update for our customers, but they are running into infrastructure issues as they try to create a special case for devices on ManageXR. They are still working on pushing this update through for us, but it is taking longer than expected. Thank you for your patience. We'll continue to provide updates as soon as we get them.
March 13th, 7:45pm PST: Meta has informed us that they intend to complete the firmware rollout across all devices by March 14th. We will provide an update when we receive confirmation that the update has been rolled out to 100% of devices.
March 15th, 10:20am PST: Meta confirmed that they've rolled out firmware v50 to 100% of devices. Online devices should update to firmware v50 automatically. If your devices are not updating to firmware v50, you can follow these manual updating instructions.
Read on to learn more about the issue and how to reproduce it.
Symptoms
Some Quest 2 devices are crashing during standard device usage. It starts with the device frame rate getting choppy, and then a popup appearing saying "com.oculus.guardian keeps stopping". At this point the device's RAM usage is unusually high (we can see this with ManageXR). Closing the popup solves it for a bit, but eventually, the lag comes back and at some point the whole system crashes and boots the user out of whatever app they were in.
It's clear the device is experiencing a CPU/memory overload to the point of failure.
Reproducing the Issue
We were able to track down the bug to being related to the sideloaded MDM device enrollment process - specifically, the bug is caused by removing the Android Accounts from the device (which is a necessary step to getting any MDM app onto a Quest).
To reproduce the bug:
With a freshly factory reset device, launch the Android Settings menu using adb:
adb shell am start -n com.android.settings/.SettingsHit "Accounts"
Remove all accounts from the device
Reboot the device
The device is now primed for failure. You don't need to install an MDM like ManageXR to experience the issues, it's entirely related to deleting the Accounts themselves. At this point, the device's logcat logs will be going wild and any typical usage may result in the Guardian App crash popup, and then eventually the current foreground app will crash. You can find example log output here.
The logs that we think are important are:
MobileConfigServiceImpl exception while trying to subscribe java.lang.IllegalArgumentException: account is null
Which is then followed by a wall of text printing at a rate of hundreds of log lines per second, primarily repeating the following 2 log lines:
VrGuardianUserService: DeviceConfig - onFailure - Error:
exceptionDeviceConfigTelemetryLogger: onFailure(exception)
We're reproducing this internally on several devices with the following version information:
System Version: 49882890235800150
Version: 49.0.0.207.357.445159226
Runtime Version: 49.0.0.207.358.445158841
OS Version: user-49882890235800150