A developer found out a intelligent strategy to entry Quest 3’s passthrough cameras to run customized pc imaginative and prescient fashions.
Meta doesn’t enable third-party builders to entry the cameras of Quest headsets, however Michael Gschwandtner, the co-founder and lead developer of the hand monitoring health app XRWorkout, posted a video to social media platforms exhibiting the outcomes of an fascinating workaround to this restriction.
The demo video exhibits the open-source MobileNet V2 object detection mannequin working natively on Quest 3 at 0.75 Hz. This was chosen for practicality, somewhat than for efficiency causes.
Gschwandtner despatched UploadVR a check construct and we independently verified that that is actual and works. He additionally completely revealed to UploadVR how precisely he pulled it off.
How It Works
Gschwandtner’s app construct creates a webview, a Chromium internet browser occasion embedded inside the app, which routinely masses the oculus.com/casting URL which Quest headsets can forged their view to.
You then must undergo the total technique of logging into your Meta account on this webview. When you’re executed, you utilize the Quest system menu to provoke internet casting. Uniquely, you are casting from the headset to an app inside the identical headset, to not a special machine.
Gschwandtner’s app then samples the pixel content material, the rendertexture, of the webview receiving the casted view, and from there can do with it no matter he likes. Within the demo instance, he is utilizing Google’s MediaPipe on-device SDK.
A key limitation right here is that the view will embrace digital objects and interfaces too, not simply the digital camera view, that means it’s going to solely work when a lot of the picture is passthrough. However Gschwandtner claims he has a “plan” for a workaround for this.
What Occurs Subsequent?
It is unclear how Meta will reply to Gschwandtner’s discovery. It is doable the corporate might make this method inoperable in a future model of Quest’s Horizon OS. Gschwandtner’s social media posts embrace a public request to “hop on a name” with Meta CEO Mark Zuckerberg or CTO Andrew Bosworth to “present them” the approach.
Final month Bosworth gave his ideas on the thought of deliberately giving Quest builders entry to the passthrough cameras throughout an interview, saying Meta will “preserve taking a look at” the difficulty.
Meta “Wanting At” Giving Quest Builders Uncooked Digicam Entry
Meta CTO Andrew Bosworth stated the corporate will “preserve taking a look at” giving Quest builders uncooked digital camera entry.
Compared, for its Imaginative and prescient Professional headset Apple solely provides uncooked entry to enterprise corporations, and just for personal inner apps.
