XR World Weekly

Menhera

Siqi

XR World Weekly 031

XRealityZone — Mon, 21 Oct 2024 14:09:19 GMT

Cover of This Issue

For this issue's cover, we're sharing NEXBOT by Max.

Opening Words

Hey, have you noticed that XRealityZone's official website (https://xreality.zone/) has quietly undergone some changes recently?

That's right, over the past two weeks, we've made some renovations to XRealityZone's official website, making the new site look more vibrant! This renovation also includes many small detail updates, such as more natural language switching without directly returning to the homepage, and article recommendations at the bottom of the page, etc.

Most importantly, we have officially launched the email subscription feature. If you want to see updates of XR World Weekly as soon as possible, you're welcome to subscribe to XR World Weekly via email in your preferred language.

Note that if you subscribe on the Chinese interface, you'll receive emails in Chinese, and if you subscribe on the English interface, you'll receive emails in English. Don't mix them up!

Perhaps the XR world isn't evolving as fast as we imagined, unable to meet our eager expectations, but it should be like the gears of fate, slowly and steadily turning, and XR World Weekly is the same💪. We might be moving slowly, and sometimes we might even stop to catch our breath, but we will still continue to move forward steadily. If you ask why, we would say, friend, do you believe in light.

No matter which issue you started reading this Newsletter from, we are very grateful for your continued support🙏. Sometimes we think that the Newsletter is like a small boat on the vast sea, carrying us to explore and move forward bit by bit, and you are that gentle sea breeze, our motivation.

It's also because of you that we have more motivation to continue bringing you some surprises. So here, we also give you a little surprise, please see the image below😉:

That's right, in 2025, we can still meet at the Let's visionOS conference, to go crazy together, play together, learn together, and — be happy together.

Finally, if you like our content, please don't forget to like👍 or share📤 it with your friends. We also hope that XR World Weekly can continue to be a bond for everyone in the future, and witness together the day when the fire of XR🔥 burns brightly.

Recap

What sparks will be ignited when PICO 4 Ultra meets spatial video?

News

Apple TV has released two new Immersive Videos: Submerged and NBA All Star Weekend
Unity 6 official version is released, and PolySpatial has also ended its Pre version, with the official 2.04 version
Meta has updated its developer documentation

Tool

ShaderVision

Article

Drawing Graphics on Apple Vision with the Metal Rendering API
Configuring iOS Development Environment in Cursor/VSCode

Video

Hands-On: Meta Orion Augmented Reality Glasses!
Submerged: Technical Breakdown

Code

Meta Horizon UI Set
Depth Pro: Apple's Official Open-Source Depth Detection Model

Recap

What sparks will be ignited when PICO 4 Ultra meets spatial video?

In this article, we introduced the latest support for spatial video on PICO 4 Ultra and how to use related features. So far, we've written quite a bit about spatial video. If you haven't read it yet, you can review our previous articles:

News

Apple TV has released two new Immersive Videos: Submerged and NBA All Star Weekend

Apple TV has released two new immersive videos, Submerged and NBA All Star Weekend. With this, all the immersive videos "to be released this fall" have been fully delivered😄.

If you've forgotten what immersive videos are available in the second half of this year, you can review our XR World Weekly Issue 022.

Among them, NBA All Star Weekend is still a documentary-style short film like other immersive videos. If you haven't watched this film yet, remember to be careful not to bump your head when you dodge backwards around 1:20～

In comparison, Submerged is more special because this short film of less than 20 minutes is the first immersive video with a plot. The overall story is simple, telling the story of a group of submariners escaping after their submarine was sunk during World War II. Obviously, in this short film, the plot is not the focus, but the immersion is. The cramped environment of the submarine and the dim light underwater will make you fully experience what "immersive" means.

It's worth mentioning that the director of this film is Edward Berger. He won an Oscar for directing the new version of "All Quiet on the Western Front", and he has always been praised for his delicate cinematography. This immersive video is also a new challenge to his cinematography skills. You can pay more attention to the cinematography of this film when watching.

In addition, to complement this short film, Apple has also released official trailer and behind-the-scenes on its YouTube channel to promote Submerged. In the Video recommendations below, we also recommend a video about in-depth analysis of the behind-the-scenes production.

Unity 6 official version is released, and PolySpatial has also ended its Pre version, with the official 2.04 version

After six months of testing, the first LTS version of Unity 6 (6000.0.23f1) was officially released on October 16, 2024. Unity official also wrote a blog post to introduce all the new features.

In short, the main new features of Unity 6 include:

Performance improvement: GPU Resident Drawer brings 2x performance improvement in some scenarios. Developers can experience it through Fantasy Kingdom in the Asset Store.
Simpler multiplayer game development: Multiplayer game development will be simpler on Unity 6. Official provides a demo called Megacity Metro as an example.
Better support for Web environment: The Unity Web platform features based on WebAssembly will be more complete. This time, Unity's project officially supports mobile platform web browsers, and can even use local storage like PWA.
Better multi-platform workflow support: Mainly includes Build Profiles and Platform Browser. The former can set different build configurations for each platform, and the latter can better preview the configuration content of each platform in the Unity editor.
Improvement of visual tools: The improvement of lighting, VFX and other tools allows developers to achieve more beautiful visual effects on Unity 6.

In addition, PolySpatial 2.0 has also ended its Pre version and has the official 2.04 version.

However, currently the template project of PolySpatial 2 has not been updated and is still using the previous 6000.0.23f1 version. Please pay attention⚠️ when initializing projects using the template.

Meta has updated its developer documentation

Perhaps feeling the pressure of Apple's excellent performance in developer relations, Meta recently not only released Spatial SDK, but also updated the style of the developer homepage for MR development. Similar to the visionOS Developer homepage, it categorizes and organizes existing technologies, making the overall structure clearer for developers who are new to the field:

Tool

ShaderVision

Hey, my old buddy, are you still troubled by writing Shaders on visionOS? Well, you need this tool called ShaderVision. With this tool, you can directly write Shader code in Apple Vision Pro and preview the effect in real-time.

It's worth mentioning that the author of ShaderVision also created an application called Pulsargeist at VisionHack, which we've promoted before, and won first place in the Games and Entertainment category.

Article

Drawing Graphics on Apple Vision with the Metal Rendering API

In this article, the author introduces how to use the Metal Rendering API to draw graphics on Apple Vision Pro. If you want to learn how to use Metal to draw graphics on visionOS, this article is a good start.

Configuring iOS Development Environment in Cursor/VSCode

If you're a developer new to visionOS development and really need to use AI to help with development, then using VSCode or Cursor is an ideal start.

However, due to some toolchain limitations, you need some configuration to make these tools perfectly integrate with the iOS development environment.

In this article, the author introduces how to configure the iOS development environment for Cursor/VSCode. The third step is particularly important, allowing Swift code to be recognized by the compiler without too many red error messages.

Video

Hands-On: Meta Orion Augmented Reality Glasses!

In this video and related article, the author delves into some technical details of Meta Orion, including battery life limit (about three hours), resolution (12 PPD, about half of Quest 3), and so on.

Submerged: Technical Breakdown

As the saying goes, laymen watch the fun, experts watch the technique. If you're curious about the story behind the filming of Submerged, but find the official behind-the-scenes a bit vague, you can check out this in-depth analysis of Submerged.

The author is also a VR industry film practitioner, so he can break down for us from a very professional perspective the details that are easily overlooked in the fleeting shots of the director's behind-the-scenes.

Code

Meta Horizon UI Set

Unlike Apple, which has a large and detailed HIG and design requirements, Meta seems to have never had a so-called "UI specification" for a long time.

However, recently, Meta launched a UI component library called Meta Horizon OS UI Set. This component library contains some common UI components, such as buttons, input boxes, sliders, etc. Based on this, developers can complete the UI design of applications very quickly.

Depth Pro: Apple's Official Open-Source Depth Detection Model

Depth Pro is Apple's official open-source depth detection model. This model can obtain depth information from a single photo, which can be used to make some applications based on depth information, such as background blur, portrait segmentation, etc. From the given examples, the accuracy is quite high. (I wonder if the function of converting ordinary photos to spatial photos on visionOS also uses this model? 🤔)

Contributors of This Issue

XR World Weekly 030

XRealityZone — Thu, 17 Oct 2024 03:00:22 GMT

Cover

For this issue's cover, we're sharing the Spline scene Lighting Bulb created by Max.

Editorial

The most significant news this week should be the debut of the Orion prototype. Although it's more expensive than the Apple Vision Pro and people have had longer and less expected hands-on time (yes, Meta hasn't given a clear expectation of when Orion will be officially available to consumers), the release of Orion has still caused quite a stir both within and outside the industry. Looking at the WeChat index data, this release of Orion has received more attention than the Apple Vision Pro's launch in February.

Many articles and videos might say that the future is already here, blah blah blah, but personally, I think the greatest significance of Orion's release at this stage is to give people hope. It shows people that AR glasses are not an impossible product, and existing technology can still create a product that barely meets hardware specifications. At the very least, the day when we expect everyone to be able to use AR glasses should not be more than ten years away (🤣). Just like the light bulb on this issue's cover, we can now see some faint but steady light.

News

Meta Connect Conference
Meta Released Meta Spatial SDK
Apple Provided New visionOS Sample Code

Tools

PICO Emulator(Beta)

App

Story: Spatial Storytelling

Code

SGMExamples: Shader Graph Material Examples in visionOS
RealityMixerVisionPro: Mixed Reality Capture app for the Vision Pro

News

Meta Connect Conference

On September 25th, local time in the US, the Meta Connect conference officially opened. At this conference, Meta unsurprisingly released the Meta Quest 3s that everyone was already very familiar with in advance.

There's not much to say about the Meta Quest 3s. You can think of it as a cheaper version of the Quest 3. The CPU and lenses haven't changed much (the lenses have even slightly decreased, returning to the Quest 2's resolution level). The 128GB version is priced at $299, and the 256GB version at $399, which is about 2000 RMB. It's available for pre-order on the day of release and will go on sale on October 15th. Buyers will receive a Batman: Arkham game.

By the way, the name Quest 3s is indeed confusing. It might have been better to call it Quest 3 lite.

In this release, Meta's description of Quest 3s, as well as the operating system Horizon OS, all describe it as a mixed reality product.

The two bigger points of this conference were Orion—a product that Meta claims to have been developing for 10 years. This is an AR glasses that is not yet available to consumers. It uses silicon carbide lenses, has a 70° field of view, a magnesium frame, a self-developed chip, supports eye tracking and gesture recognition, has about two hours of battery life, and weighs only 90g. It also comes with a portable computing unit called the Wireless Compute Puck (which seems to have a camera on it, but it's not yet known what this camera is specifically used for), and a wristband that can sense myoelectric signals, which can be used when Orion can't detect the hand.

Currently, in addition to the pinching of the thumb and index finger, this wristband can also sense a sliding gesture, which seems perfect for up and down scrolling scenarios:

From the information released at the conference, the cost of an Orion AR glasses is about $10,000 (based on 2022 mass production data). Due to the high material cost, Meta had to shelve the official launch plan for this product and only manufactured 1,000 pairs for internal company demonstrations and partner trials.

Personally, I think the greatest significance of Orion's appearance at this stage is to let everyone realize that existing hardware technology can actually achieve a lightweight, 6DOF AR glasses that supports eye and hand tracking. Although the current technology level can only achieve a 70° FOV (which is already a very large FOV in AR glasses), this is already a very good start. (If you look carefully at the reviews from foreign media, you'll find that Orion's current 70° FOV is still clearly visible with a truncated edge when worn, which will at least give people a very obvious feeling of "breaking immersion".)

Many excellent articles have already provided detailed introductions to more details about Orion. If you're interested, you can click to read:

Meta Released Meta Spatial SDK

Developers familiar with the love-hate history of Android and iOS will be no stranger to this scene: Meta has released a brand new SDK for development on Quest—Meta Spatial SDK. With this SDK, you can use Android Studio and Kotlin to develop spatial applications on Quest (these two things are very familiar to Android developers).

In Android Studio, you can edit the UI of spatial applications based on XML:

You can also edit 3D scenes in the Meta Spatial Editor, similar to Reality Composer Pro used in visionOS development:

The concept is truly consistent, and the official has also released 10 Sample Codes. Interested friends can go directly to Github and quickly get started according to the documentation.

Apple Provided New visionOS Sample Code

Recently, Apple provided new visionOS Code Samples. Compared to the previous relatively complete large project Sample Code, this time's Samples are all relatively simple small projects, suitable for beginners to get started.

Tools

PICO Emulator(Beta)

PICO recently released a new debug tool on their official website: PICO Emulator. Its main function is to simulate running applications that originally needed to run on PICO real machines on PC and Mac. You can install it via adb or by directly dragging and dropping the apk. At the same time, the emulator itself also supports keyboard and handle mapping.

App

Story: Spatial Storytelling

Story is an App that can bring an extremely immersive video experience to Apple Vision Pro users. Here, you can not only watch 360° immersive videos composed of pictures, models, and text like this:

You can also immersively view landscape videos with 16K clarity:

Currently, there are five videos in the App, and more videos will be released in the future.

As an immersive video playback application, Story has two surprising points. The first point is the video clarity of up to 16K (this is a perfect match with the resolution of Apple Vision Pro). When viewing landscapes, the extreme clarity will really make your eyes light up.

The second point is the vision behind Story: to enable more people to create such spatial videos. Therefore, we can say that Story is more of a content platform. Let's take a look at the editor that can be used in Vision Pro:

Code

SGMExamples: Shader Graph Material Examples in visionOS

To create cool and beautiful Apps, Shader Graph is definitely a must-have skill for visionOS developers. But unfortunately, the official only has documentation and lacks some more advanced examples, which brings some difficulties for beginners.

But now, with SGMExamples, you can better learn how to use Shader Graph through this project. This project includes many advanced Shader Graph usage scenarios, and this water ripple is one of them:

Many thanks to Yasuhito for open-sourcing such a great project

RealityMixerVisionPro: Mixed Reality Capture app for the Vision Pro

All friends using Apple Vision Pro may have encountered a problem: people around you can't see the content in your glasses. Although we can solve this problem through screen casting, this solution is not optimal.

Many thanks to Fabio Dela Antonio for his attempt on this issue. Through RealityMixerVisionPro, you can project what Vision Pro sees onto your phone in a three-dimensional form, so your friends can see the content in your glasses.

Contributors to this issue

xChester

XR World Weekly #029

XRealityZone — Sat, 28 Sep 2024 01:57:08 GMT

Cover of this issue

In this issue, we are sharing Roman Klčo 's Food Truck

Publication preface

As the Mid-Autumn Festival approaches and Apple releases various new systems, it's a good time to upgrade all the main devices at home. Although the long-awaited feature of the fish screen has not been released in visionOS 2, fortunately, the new features of other systems can still be played with. Happy Mid-Autumn Festival to everyone in advance~

Recap

visionOS 2 PortalComponent: A more anticipated teleportation artifact

News

Meta Quest v69 released
Unity no longer charges runtime fees
Unreal now supports perspective mode on visionOS

Article

Using visionOS to revolutionize surgery and medical treatment

App

AquaZen: Immerse yourself in the tranquility of water

Recap

visionOS 2 PortalComponent: A more anticipated teleportation artifact

Keywords: Portal, visionOS

In this article , Puffinwalker introduced us to the use of the new features of PortalComponent in visionOS 2, and made a very cute super dream with PortalComponent, worth reading.

News

Meta Quest v69 released

Keywords: Quest, Meta

In the past week, Meta officially released Quest v69 , in this update, the more interesting are these features:

Seamless multitasking: Simply put, when you were using Quest before, if you were playing a game and wanted to open an Android-like application floating window, you had to exit the game and come back. The updated Seamless multitasking allows you to use the Home button to display the multitasking panel that existed before, and can hold up to three applications, including the browser. This feature is greatly beneficial for games that need to check the strategy to play (such as various lost black myths, Wukong 🐒), or some slow-paced games. Watch some videos at the same time (like in the 👆 picture)
Meta AI adds Hey Meta wake-up word: a more common update, before this Meta AI can only be awakened by double-clicking the Meta button on the handle or clicking on the system menu, but Meta AI This function itself is only available in the US and Canada.
Window supports Spatial Audio : Simply put, the audio played in the flat window application in Quest also supports Spatial Audio, and the sound you hear will be more spatial

Unity no longer charges runtime fees

Keywords: Unity, Runtime Fee

The Unity runtime fee controversy, which began in September 2023, finally settled in September 2024, one year later: the new CEO Matt Bromberg announced that Unity will cancel runtime fees .

Tips
About the runtime cost of Unity, we have mentioned it in Newsletter 009 and Newsletter 010 .
If you are not familiar with this matter, it can be simply understood that Unity required all users of Unity to charge additional fees based on the number of installed users of the game in addition to the original fees in September last year. This move has caused strong dissatisfaction among industry developers. Former Unity CEO John Riccitiello resigned as CEO because of this matter, and Matt Bromberg took office in May 2024.

Along with the cancellation of runtime fees, Unity Personal has raised its fee threshold (from $100,000 to $200,000), and Unity Pro and Unity Enterprise have increased their subscription prices (by 8% and 25% respectively). Overall, Unity's strategy is friendly to small and medium-sized developers and raises prices for large customers to reduce its losses (yes, Unity has not yet achieved profitability since its launch).

Unreal now supports perspective mode on visionOS

Keywords: Unreal, visionOS, Passthrough

Although there are still some rough edges and imperfections, according to this X , Unreal can finally show the real world on visionOS 2!

Article

Using visionOS to revolutionize surgery and medical treatment

Keywords: visionOS, Surgery, Medical

In this article , the author proposes the following viewpoint: before spatial computing, 2D plane CT and MRI scans have brought better tools for surgeons' surgeries, allowing them to more accurately understand the internal structure of patients. Vision Pro only requires hand interaction and fine rendering of 3D objects, undoubtedly giving surgeons a more capable assistant on the operating table.

In addition, the author also lists and discusses a series of medical-oriented apps currently available on Apple Vision Pro, such as:

INSIGHT HEART : an immersive view of the structure of the human heart
Fundamental Surgery : Immersive surgical training simulation

App

AquaZen: Immerse yourself in the tranquility of water

Keywords: Apple Vision Pro, Zen, Zen

AquaZen is an app that can provide an immersive aquarium experience. It provides three modes internally: 360 ° Aquarium, Space Aquarium, and Shared Space .

In the 360 ° aquarium, you can watch various sea fish swimming around you in a panoramic underwater world, and the app also provides configuration options for you to choose which fish to display in the current scene.

In the space aquarium, these fish no longer swim in the vast and boundless water, but in the room next to you. Obviously, the scene perception ability of visionOS is used here. If you are in a very small room, you will find these fish swimming very "crowded" in the small room.

In the shared space, you can place the aquarium in the form of an aquarium, like a decoration, next to you. Similarly, you can also configure all the displayed fish here.

Next, let's talk about this application from the perspectives of user requests, product solutions, and Technology Implementation.

From the user request, AquaZen intends to solve the user with Apple Vision Pro, find a different environment and then change the mood of the needs, such applications on the App Store is not few, the official, meditation applications, Healium , Endel are intended to solve this kind of problem.

From the product plan, AquaZen uses a solution surrounded by swimming fish. Unlike the previously mentioned applications, the fish in AquaZen swim under certain rules and are not completely preset scenes. Therefore, compared to seeing the same video or audio every time, the threshold for visual fatigue for users will be slightly higher.

From Technology Implementation, the most interesting technology applied in the entire application should be how to make these fish swim. In this regard, Apple has a Demo: Building an Immersive Experience with RealityKit on the iOS.

Contributors of this issue

XR Word Weekly Newsletter #028

XRealityZone — Sat, 28 Sep 2024 01:57:00 GMT

Cover of this issue

In this issue, we are sharing about visionOS 2

Publication preface

I thought the Apple Special Event 2024 would not have any information about Apple Vision Pro, but the result is that there is really no information about Apple Vision Pro.

However, it is surprising that visionOS 2 will be released in 2024.9.16, this update rhythm is faster than iOS18, iPadOS 18, watchOS 11, macOS Sequoia 2024.9.17 update rhythm (although only fast one day), but from the current official website description , everyone is looking forward to the 4K widescreen projection function should not be launched in this visionOS 2 (marked Later this year)

News

New immersive videos "Elevated" and "4 Minutes Inside Super Bowl LVIII" are available on Apple TV
PICO System Update & Pre-orders for PICO 4 Ultra in Europe are now open

Idea

Is the FPS game of Apple Vision Pro just a pointing gun?

Article

Vision Pro Online Usage Survey Results: A survey on Apple Vision Pro usage

News

New immersive videos "Elevated" and "4 Minutes Inside Super Bowl LVIII" are available on Apple TV

Keywords: Apple Vision Pro, Immersive Video

In the 022 issue of the Newsletter , we mentioned that there will be a lot of immersive videos on Apple TV in the second half of the year. Now, we see two new immersive videos:

Elevated : Fly over volcanoes, climb over waterfalls, and discover the unknown natural wonders of Hawaii, the world's longest island chain
4 Minutes Inside Super Bowl LVIII : This is a video content that has not been previewed before. Here you can watch the behind-the-scenes story of the Kansas City Chiefs and San Francisco 49ers competing for the Super Bowl championship in the NFL (US Professional Football League). To be honest, in the immersive video, when you see these tall and strong football players rushing past you, you will still instinctively want to dodge.

Thanks to Mike Swanson for the Apple TV screenshot

PICO System Update & Pre-orders for PICO 4 Ultra in Europe are now open

Keywords: PICO

With the launch of PIOC 4 Ultra, PICO OS has also provided a new version for PICO 4 Ultra. Among the new features worth mentioning are:

Support Bluetooth keyboard and mouse connection
Mobile Pentium 4 supports watching videos in transparency mode

In addition to this, PICO also opened pre-orders in Europe , priced at 529 pounds sterling/599 Euros.

Idea

Is the FPS game of Apple Vision Pro just a pointing gun?

Keywords: Apple Vision Pro, FPS

Every boy may have played the simplest version of FPS with his fingers when he was a child, and such scenes may not be difficult to see in Apple Vision Pro.

Fabio Dela Antonio recently made a demo like this. When your finger bends, a ray that serves as a aiming line disappears, which also means you have "fired":

Article

Vision Pro Online Usage Survey Results: A survey on Apple Vision Pro usage

Keywords: Apple Vision Pro, Suvery

Recently, Tom Krikorian conducted a research report on Vision Pro users. In this report, Tom conducted a survey on multiple usage scenarios of Apple Vision Pro users, including:

How often to use Apple Vision Pro
Main usage scenarios
Feelings about the device
Dislike points about the device

In this survey report, there are many interesting points. Firstly, among the 163 respondents interviewed, most of them have been using it for more than 3 months.

Among these users, nearly half of them use it every day

Most people use Apple Vision Pro to watch movies and work.

Regarding the underutilized features of Apple Vision Pro, Spatial Persona, spatial video, productivity features, etc. are prominently listed.

The most interesting thing may be this. In the questionnaire, when asked about the unusual ways of using Apple Vision Pro, everyone's usage was varied. The ten most interesting ones are as follows:

Use First-Person Perspective sharing to get medical help for splinting a broken finger.
Host a virtual cooking club where hosts live-stream the food preparation process in real time.
While playing the piano, use the Mac virtual display to create music while overlooking the virtual Yosemite environment.
Locate the location of water damage in the house by placing the music player window in the suspected water damage area and looking through the wall/floor. ( Note: This means that the window in Apple Vision Pro is fixed in the real world and can be seen through to be able to see the location of the leak and its possible impact from a farther distance )
In virtual piano lessons, students provide their own perspectives to the teacher while demonstrating the sheet music application and hand movements.
Create custom immersive environments and view Gaussian splash renders for relaxation or entertainment.
Use the Eye of Da Vinci to track images that are only visible in Vision Pro for artistic creation. (The Eye of Da Vinci is an application that maps images onto a real-world plane, allowing for copying and painting)
Conduct a silent writing session on Zoom, where other participants can see the user's avatar.
Work at a campground in a state park, use multiple virtual displays, and run Windows through Parallels on your MacBook.
Record space videos of special events such as solar eclipses

Contributors of this issue

XR World Weekly #027

XRealityZone — Sat, 28 Sep 2024 01:56:54 GMT

Cover of this issue

In this issue's cover, we share the hottest AI Code Editor in the past few weeks - Cursor

Publication preface

Just in the past week, Apple officially announced that it will hold a special event at 1 am Beijing time on September 10, 2024, with the theme of "It's Glowtime" and the following very psychedelic video:

Anyone with discerning eyes knows the meaning of this video - it's too similar to the effect of the new Siri released by Apple at this year's WWDC. It seems that Apple's new hardware product (although we all know what it is, let's pretend not to know 😄) will have a deep relationship with AI - Apple Intelligence.

We have always believed that XR and AI must be a good pair of partners, so this year's WWDC saw Apple Intelligence and visionOS was a bit disappointed (although soon after, unable to use the Swift Assistant sad 😭 was cured by Cursor 😆, test Cursor wrote a lot of new visionOS RealityKit code is also very good)

But after disappointment, think carefully, this is also in line with Apple's usual style - new technologies are first tested on mature platforms and then expanded to other platforms. ARKit and RealityKit, which are now the backbone of visionOS, have also been polished for many years on iOS platforms before finally shining on Apple Vision Pro (you can take a look back at the story behind this Before developing visionOS, you need to know the panoramic view of Apple AR technology ). Let's be patient and look forward to next year's WWDC together~

Recap

The gesture matching framework HandVector on Vision Pro has been updated to 2.0, with the addition of FingerShape function

News

Meta decides to shut down Spark AR effects platform
Surreal Touch is now available on Kickstarter!

App

Mixed Snow Worlds: Turn your room into an ice and snow world in an instant

Tool

Swift-translate: A CLI tool to help you translate Internationalization files in your app

Code

Quest Display Access Demo: A New Way to Get Quest Camera Data

Recap

The gesture matching framework HandVector on Vision Pro has been updated to 2.0, with the addition of FingerShape function

Keywords: visionOS, HandVector, Hand Tracking, FingerShape

In this article , we introduce you to the major updates of HandVector 2.0, including:

Updated the calculation method of cosine similarity algorithm, changing from hand Anchor space to matrix space based on parent joint points, improving the similarity calculation of fingertip vectors.
Added FingerShape function, simplifying finger shape into 5 parameters, including curl and separation, for easy understanding and adjustment.
Updated the emulator debugging function SimHand to solve the problem that Google MediaPipes 3D gesture tracking only has joint position without matrix information.
Adjusted project structure and 3D joint style, code structure is clearer, joint visualization is clearer.

News

Meta decides to shut down Spark AR effects platform

Keywords: Meta, AR, Effect, Spark

Recently, Meta's "difficult decisions" have come one after another. On August 27th of the past week, Meta announced that it will shut down its AR effect platform Spark .

Specifically, starting from January 14, 2025, third-party creators will no longer be able to create new AR effects on the Spark platform (including Meta Spark Studio, Meta Spark Hub, and Meta Spark Player), and AR effects created by third-party creators will also be removed from the Meta platform (including Facebook, Instagram, and Messenger).

We have explained Lens Studio and Meta Spark as competing products in the 002 issue . Like the netizens on X , we are also sighing at the end of an era.

Surreal Touch is now available on Kickstarter!

Keywords: Controller, Apple Vision Pro

The world's first gesture controller, Surreal Touch, tailored specifically for the Apple Vision Pro, has been officially launched on Kickstarter and has reached its minimum fundraising amount.

With Surreal Touch, you can use Apple Vision Pro to play VR games on SteamVR at 4K 90Hz, including classics like Alyx. If you are interested in playing VR games with Apple Vision Pro, you can purchase it now for as little as 249 dollars, with estimated delivery time in December 2024.

App

Mixed Snow Worlds: Turn your room into an ice and snow world in an instant

Keywords: Quest, MR

We have always believed that simply allowing players to see the real world cannot be called Mixed Reality - at least not excellent Mixed Reality.

In Mixed Snow Worlds , when players play the game, they will see the game scan the player's surrounding environment and transform the corresponding environment into an ice and snow world before starting the next step of the game, which undoubtedly greatly enhances the Mixed Reality. The sense of on-site experience.

The game is currently available for free download on the Quest app store. Interested players can download and give it a try.

Tool

Swift-translate: A CLI tool to help you translate Internationalization files in your app

Keywords: visionOS, i18n

If you want to do internationalization when you are making a visionOS-related app, you can try using the tool swift-translate to try to simplify the workflow. This is a command line program that can help you send multilingual strings from Xcode to the ChatGPT backend and replace the characters.

Code

Quest Display Access Demo: A New Way to Get Quest Camera Data

Keywords: Quest, Android MediaProjector API

As we all know, the camera screen on Quest cannot be obtained by developers normally, but it is also well known that developers still have a way to obtain it. Previously, some developers shared their trick - to obtain it by opening a Quest Mirror page in the background.

This time, trev3d came up with a good solution - directly using the Android MediaProjector API to obtain the camera image on Quest. Although this method is feasible, there are also some small limitations:

Quest v68 and above system
Invalid when using QuestLink
Cannot record properly, need to use a tool like scrcpy for screen recording
Because it is not normal camera data acquisition, all virtual objects will obscure the entities on the image

The complete code can be viewed on GitHub

Contributors of this issue

XR World Weekly #026

XRealityZone — Sat, 28 Sep 2024 01:56:46 GMT

Cover of this issue

The cover of this issue is about Black Myth · Wukong from Game Science .

Publication preface

This week, whether you are a gamer or not, I believe your various timelines will be occupied by "Black Myth: Wukong". Moreover, the producer Hideo Kojima, who holds a high position in the gaming industry, posted a thread when Black Myth sold 10 million copies:

The road to the West has always been a difficult road, and Meta has many trade-offs on this road. For example, this week it is said that Meta has cancelled Quest Pro 2 (compared to last week's cancellation of the GTA VR version, this news seems to have a slightly smaller impact on players).

News

PICO releases PICO 4 Ultra

Idea

Use water droplets for transitions

Tool

3D Icons: An Open Source Collection of 3D Icons
WWDC Notes

Video

Blender Tutorial 4.0
Meta Immersive Debugger

Code

HLS 360 Video Material for visionOS

News

PICO releases PICO 4 Ultra

Keywords: PICO, MR

This Tuesday, PICO officially released the PICO 4 Ultra, which is PICO's first device defined by MR (previous PICO series products were all VR All in One).

In terms of hardware, in conjunction with the positioning of "MR devices", PICO has also equipped Ultra with four environmental tracking cameras, two color perspective cameras, and depth perception cameras.

In terms of price, the PICO 4 Ultra is priced at 4299 yuan (12GB + 256GB), with pre-orders starting on August 20th and official sales starting at 10:00 on September 2nd. During the pre-sale period, PICO can provide a subsidy of 300 yuan for old users (referring to those who have activated PICO machines), and provide a subsidy for exchanging old machines for new ones.

However, for creators in the XR industry, it is obvious that updates to the system and application ecology have a greater impact on us. Therefore, we will focus most of our attention on the system and application ecology.

In terms of software functionality, PIOC 4 Ultra, like Meta, is increasingly approaching mature system designs like Apple Vision Pro.

PICO Connect allows you to cast your Mac, Windows, iOS/Android phone to PICO 4 Ultra:

In addition, for computer screen casting, you can customize the window size, and even set it to the size of the fish screen. (Well, the functions of visionOS 2 have also caught up one by one), and when we can use PICO 4 Ultra to cast the phone to the device, a small pain point that has been unable to play with the phone for a long time has also been solved (there are similar third-party apps doing this on visionOS, see Bezel of XR World Weekly 019 ), the system implementation of PICO is not only screen casting, but also includes operation.

In addition, this time PICO 4 Ultra supports running multiple windows of native Android applications simultaneously (including the projected window seen earlier), which was called the Panoramic Screen Workbench at the PICO launch event. It supports up to 20 applications on the same screen and surrounds the user in a 360 ° manner.

These windows can be interacted with in the form of MultiModal Machine Learning (including keyboard, mouse, controller, gestures).

In terms of spatial video (hey, if you don't know what spatial video is, don't forget to check out iPhone What is spatial video on the 15 Pro/Apple Vision Pro? ), the PICO 4 Ultra supports 60FPS spatial video shooting, and you can also view the spatial video shot by the iPhone/Apple Vision Pro, and then use the editor on the PICO 4 Ultra to edit, score, and publish (no need to think, it must be published to Douyin--TikTok's Chinese version)

But, the functions of mobile screen projection, Spatial Audio recording of spatial videos, 3D special effects, and keyboard and mouse interaction for panoramic screens have not been officially released yet, and we need to wait for the OTA upgrade of the subsequent system.

Idea

Use water droplets for transitions

Keywords: Portal, Transition

Switching from one scene to another very immersive scene, instead of directly using a portal like MoonPortal , maybe we can also try to use something like Jason Cooper designed here, a transition of a water droplet from small to large :

Tool

3D Icons: An Open Source Collection of 3D Icons

Keywords: 3D, Icon, Blender

3D Icons is an open source collection of 3D Icons, providing a large number of free (commercial) 3D Icon resources, we can directly use these 3D Icons in a variety of formats (including Blender-based secondary modification).

WWDC Notes

Key words: WWDC, Sessions, Swift-Doc

As we all know, one of the best ways to learn about Apple Vision Pro and Apple-related framework development is to directly watch the Session videos of WWDC over the years. However, with so many videos, it is always a bit tiring to watch them one by one. Sometimes, for efficiency, you can directly come to WWDC Notes This website first to see if there are friends in the community who have helped you watch it and made some notes and summaries of the sessions you want to watch.

The entire WWDC Notes website is based on Apple's Swift-Doc , and the entire note content is maintained by the open source community. If you are interested in this project, you can also click here to contribute to this project.

Video

Blender Tutorial 4.0

Keywords: Blender, Tutorial

If you want to learn Blender, then Blender Guru 's classic Blender 4.0 donut tutorial is a set of tutorials you must not miss. This tutorial has been widely praised since the Blender 2.x era.

Meta Immersive Debugger

Keywords: Meta, Unity, Debugger

All developers who have debugged immersive VR games may have a common pain: when debugging immersive game scenes, they cannot see their own monitor, and then see the console output on the IDE, or turn on and off some options through the debugging tool.

Fortunately, Meta's recent Meta Immersive Debugger solves this problem to some extent. This Debugger provides some built-in debugging options, and developers can also set the switches themselves. Dilmer Valecillos also explains this tool in detail in his new video .

Code

HLS 360 Video Material for visionOS

Keywords: Apple Vision Pro, Material, HLS

If you are an Apple fan, I believe you must have seen the term HLS in some of Apple's official LIVE events. This is a video LIVE protocol launched by Apple.

In the HLS 360 Video Material for visionOS project, Yasuhito Nagatomo implemented an HLS version of Material based on RealityKit, enabling it to play HLS LIVE videos in the sky sphere.

Contributors of this issue

XR World Weekly #025

XRealityZone — Sat, 28 Sep 2024 01:56:37 GMT

Cover of this issue

On the cover of this issue, we share the Vision Pro Music Player produced by Ilknur Citak .

Preface: Bad News and Good News for Quest Players

The past week may have been a mixed bag for Quest players.

On the one hand, IGN has confirmed that Meta will indefinitely delay the release of the GTA San Andreas VR (Grand Theft Auto: San Andreas) project, mainly because Meta needs to focus on other more important projects. After all:

With the indefinite postponement of GTA San Andreas, Meta is losing billions of dollars in the VR gaming market. Yahoo Finance stated in a new report released earlier this month that the Reality Labs department is burning money, and in 2023 alone, the VR-centric department will accumulate a loss of $16 billion.

On the other hand, at this time, Meta has launched the August promotion, and before the end date of the August promotion (August 25th Pacific Time), a large wave of games will be discounted by up to 74%.

News

Apple Vision Pro updates with new immersive video
Spline Summer Event: Real Time API & Webhooks、Android Support、Code API
Meta has launched a new app: Meta Quest HDMI Link.

Activity

VisionHack: The world's first online visionOS online hackathon

Video

Intro to Bezi 101: Bezi's series of introductory videos
Use RealityKit and Reality Composer Pro to create a portal for a box
Unboxing Surreal Touch in advance: the world's first VR controller designed for Apple Vision Pro

Code

SharePlayMock: Helps you test the communication logic of SharePlay

News

Apple Vision Pro updates with new immersive video

Keywords: Apple Vision Pro, Immersive Video

In issue 022 , we mentioned that we would see more immersive videos on Apple TV with Apple Vision Pro in the second half of the year. This week, WildLife's second episode: Elephants is here. In this episode, we will follow the director's lens to the elephant orphanage in Kenya and learn how people here coexist friendly with elephants.

Spline Summer Event: Real Time API & Webhooks、Android Support、Code API

Keywords: Spline Summer Event

Since a series of major feature releases at the end of 2023 (corresponding to our issue 015 ), the Spline team has entered a busy development cycle. However, last week, at the Spline Summer Event , the Spline team announced some exciting data: currently 2 million people use Spline every day to design and share their 3D designs.

Meanwhile, at the Summer Event, Spline also released three very interesting new features: RealTime API & Webhooks, Android Support, and Code API. Let's take a look at them one by one.

First, let's talk about the RealTime API & Webhooks . In short, Spline can now accept data from network APIs as input sources for 3D models, and various interactions in the model can also send requests to external API servers. This way, we can make our 3D models display based on real-world data. For example, using this capability, we can directly generate beautiful 3D charts based on real-world conditions, and even make a real-time weather application if we want.

Just imagine, if we use this API to connect 3D scenes with various AI APIs, wouldn't it be possible to make a very interesting interactive game in an instant? Take a look at this interesting little duck below, we can control its actions and dialog box through AI. Even for the convenience of development, Spline specifically provides the ability of OpenAI API to help you quickly integrate various capabilities of OpenAI.

These two features can be used immediately in Spline, so you can start experiencing them immediately.

The second major feature is the support for the Android platform . Like the previous support for the iOS platform, Spline is also directly connected to the underlying rendering engine layer. However, on the Android platform, it is connected to Vulkan. From Spline, it can now seamlessly run on various Android devices, such as foldable screen phones, Android TV, etc. This feature is currently in beta and is expected to be officially released in Q4 2024. So far, Spline has basically completed the native support work for the three major platforms of Web, iOS, and Android.

The third update is the Code API. After we integrate the Spline SDK on the Web , iOS , and Android platforms, we can perform more refined operations on the integrated Spline view through variables, properties, and event listeners.

This feature currently only provides a direct API on the web platform, iOS and Android platforms need to wait a little longer (expected to start beta support in Q4 2024).

Meta has launched a new app: Meta Quest HDMI Link.

Keywords: Meta Quest HDMI Link

Meta recently launched a software called Meta Quest HDMI Link , which allows you to directly project images from devices that support HDMI or DP output to Quest in 1080P resolution. However, to achieve this function , you need to purchase an additional hardware capture card compatible with UVC and UAC.

Activity

VisionHack: The world's first online visionOS online hackathon

Keywords: Apple Vision Pro, Hackathon, OnLine, Huge Bonus

In less than a month, the grand global visionOS Hackathon VisionHack will begin (9.13-9). After intensive preparation, this Hackathon will finally provide a total of $25K (total RMB 180,000) in cash and prizes to the outstanding winners.

In addition, not only will there be wired matches, but there will also be offline matches. If you happen to live near New York, San Francisco, Paris, Montreal, and St. Louis , you can participate in the offline matches of this competition through this link .

This competition has also invited many outstanding developers from the international visionOS and XR industries to share or participate in awards.

If you are interested in this event, want to participate and learn more details, welcome to contact Onee ~

Video

Intro to Bezi 101: Bezi's series of introductory videos

Keywords: Bezi, 101, Introduction

As a very useful web-based XR prototyping tool, we introduced Bezi to you in 003 and 011 respectively. Recently, in order to help you better understand Bezi, Bezi official also launched the Bezi 101 series of videos , aiming to help you understand how to use Bezi to help XR development from the aspects of concept, prototype, collaboration, design, etc.

Use RealityKit and Reality Composer Pro to create a portal for a box

Keywords: RealityKit, Reality Composer Pro, Box Portals

If you've played with Moon Portal , you might be curious about how such portals are implemented in visionOS.

In this " How to make a Box of Portals in your Apple Vision Pro app using RealityKit and Reality Composer Pro " video by Sarang Borude , your doubts will be answered. Here, Sarang has created a PortalBox with four walls for us, and in the tutorial, he explains step by step the precautions needed for each step.

In addition, if you are interested in the effect of Portal, after watching this video, you can also go to see the update of PortalComponent in visionOS 2 on WWDC 24, and learn about the function of allowing crossing added to ProtalComponent in visionOS 2.

Unboxing Surreal Touch in advance: the world's first VR controller designed for Apple Vision Pro

Keywords: Apple Vision Pro, SteamVR, Controllers

With powerful gesture recognition, Apple Vision Pro has successfully become the first spatial computing device that does not require a controller. However, some traditional VR players may not be convinced of this. However, where there is demand, there is a solution. In Himels Tech 's video , he opened the box for us in advance Surreal Touch - the world's first VR controller designed specifically for Apple Vision Pro.

With these two Type-C controllers, we can directly use the Surreal Link app to stream various games on our SteamVR directly on Apple Vision Pro.

Currently, the official price of Surreal Touch has not been announced, but if you register an email through this link of Himels Tech, you can get a minimum price of 249 dollars. If you are interested, please register first~

Code

SharePlayMock: Helps you test the communication logic of SharePlay

Keywords: SharePlay, Apple Vision Pro

SharePlayMock is a framework that can help you use emulator and real machine to directly communicate and debug locally when developing SharePlay-related development, without the need for joint debugging with another developer who has Apple Vision Pro real machine.

His basic principle is to start a server on Mac to pretend to be the communication channel provided by SharePlay, and then directly replace the GroupActivity-related API in the code to achieve local debugging of SharePlay-related information transmission functions.

// before
var sharePlaySession: GroupSession?
// after
var sharePlaySession: GroupSessionMock?

In order to explain the specific use of this framework, the author also wrote an article to explain.

Contributors of this issue

XR World Weekly #024

XRealityZone — Sat, 28 Sep 2024 01:56:29 GMT

Cover of this issue

In this issue's cover, we share the halo mecha generated using Meshy.ai.

Corresponding prompt word:
Full tactical scifi body armor from Halo crysis infinite with a bit of iron man. black and blue colors for the armor and black for the joints, helmet with visor glow panel, halo helmet, scifi synthetic muscles from the Crysis nanosuit, #fashion show#
Reverse prompt word:
weapon, sticking fingers, 3 fingers, 4 fingers, bad proportions, bright colors, missing limbs, joined together legs

Publication preface

This week, PICO officially announced that it will hold a new product launch event on August 20, 2024. It has been nearly a year and a half since the last PICO 4 Pro launch event (April 27, 2023).

And the last PICO SDK update, has been half a year ago, do not know the new product release, will not bring the new PICO SDK update? Let's wait and see 👀

App

[Free] Camo Studio - Help you share your insights in Apple Vision Pro

Video

FASTER Iteration With Meta XR Simulator - macOS Support Available!

Article

How to Use Material Nodes in Reality Composer Pro’s Shader Graph

Code

Computer Graphics Tools-Photos Example

App

[Free] Camo Studio - Help you share your insights in Apple Vision Pro

Keywords: Apple Vision Pro, Spatial Persona, Stream, Record

If you are a loyal player of Apple Vision Pro and want to share your gaming experience in Apple Vision Pro with your fans through LIVE, then you can definitely try CamoStudio.

CamoStudio is a cross-platform LIVE and recording application. In addition to using Spatial Persona directly in Apple Vision Pro for LIVE and recording with the current field of view, users can also use the iPhone camera to record what the real world looks like.

In addition, the visionOS version of CamoStudio can also directly display LIVE comments next to the LIVE screen, and use a brush tool similar to the borderless note to prompt the content in the picture.

Video

FASTER Iteration With Meta XR Simulator - macOS Support Available!

Keywords: macOS, Unity, Meta XR Simulator

For some VR games that require frequent physical simulations and tactile adjustments, the process of constantly repeating, writing code - > packaging programs - > putting on devices - > verifying tactile feedback, is quite cumbersome. Fortunately, this process can be greatly simplified through the Meta XR Simulator provided by Meta. In the Meta XR Simulator, in addition to some visual content of the scene, other functions such as gamepad operations can be directly completed based on the Meta XR Simulator. This video by Dilmer Valecillos explains how to use Meta XR Simulator to simplify your development process and provides corresponding demo projects for demonstration.

By the way, the most important thing is that this Meta XR Simulator also supports macOS!

Article

How to Use Material Nodes in Reality Composer Pro’s Shader Graph

Keywords: ShaderGraph, Xcode, Tutorial

For most iOS developers, the programming concept of Shader and the Shader Graph in Reality Composer Pro are very unfamiliar (although it may be very easy for technical artists who are proficient in Unity to get started). If you want to learn how to use ShaderGraph, you can refer to this article How to Use Material Nodes in Reality Composer Pro's Shader Graph

Code

Computer Graphics Tools-Photos Example

Keywords: Shader, Metal

In the world of 3D graphics, shaders can do a lot of things, as long as you are skilled, you don't have to worry about being busy. In the following code, Lucky Iyinbor uses Metal Compute Shader , ARKit, and RealityKit to create a very interesting special effect of photo fragmentation into particle balls:

The complete project code is here , in addition to the current Photos Example, in this Computer Graphics Tools Organisation, there are other frameworks to help developers write graphics computing related code in Swift:

By the way, previously very 🔥 on social media in the Apple Vision Pro to imitate the fabric effect of the project , also from Lucky Iyinbor.

Contributors of this period

xChester

XR World Weekly 023

XRealityZone — Sat, 28 Sep 2024 01:56:22 GMT

Cover of this issue

For this issue, we recommend Roman Klčo 's Mixed Reality Pokemon .

OpenUSD: The increasingly influential OpenUSD

This week, AOUSD has two new "heavyweight" members joining : Microsoft and Sony . Along with these two heavyweight members, there are three new working groups of AOUSD and the v24.08 version of OpenUSD.

When writing last year's issue 006 and issue 014 , I did not expect that this Apple-led organization would gain more and more industry recognition. In addition to new members, NVDIA, which is riding the wave of AI and has unlimited glory, is also contributing to the development of OpenUSD with actions: at the recent SIGGRAPH conference, NVDIA announced that they will provide services based on a series of large models of OpenUSD . According to the press release, these services are mainly used to help users generate OpenUSD-related processing code and materials for 3D models using natural language, and are mainly used in the industrial and engineering fields to help Apple Vision Pro be utilized in industrial scenarios.

In addition, according to the WWDC 2024 Session , Finder on macOS will also support simple editing of usd files on the new macOS. According to this trend, it is very likely that in a few years, usd will become the jpg format in the field of 3D models.

Recap

What is OpenXR: How Qualcomm promotes OpenXR technology

News

Use "SUPERHOT VR" to get a half-price discount on Meta Quest +
Starting this week (2024-08-05) apps on Meta AppLab will be migrated to the Meta Horizon Store

Activity

VisionHack: The world's first online visionOS online Hackathon

Idea

3D Interactive Icons

Tool

Helm for App Store Connect

Article

PDF：Create virtual and mixed reality experiences in Unity
Apple Document——ShaderGraph

Code

RealityKit Terrain Shader

App

Spatial Physics Playground（Free）

Job

ILLUSION Tech: Front-end development, content cybertribe operations, operations development, 3D content designer

Recap

What is OpenXR: How Qualcomm promotes OpenXR technology

Keywords: OpenXR, Snapdragon Spaces, Enlight XR

In this article , we delve into Qualcomm's initiative to launch Snapdragon Spaces XR Development Platform in the XR field, as well as the OpenXR-related content derived from it, and the work done by Shenmu Technology in Localization landing.

Tips
In the recent Meta quest v68 , Meta also introduced a new Algorithm for OpenXR that reduces Latency and lag in OpenXR applications.

News

Use "SUPERHOT VR" to get a half-price discount on Meta Quest +

Keywords: Meta Quest + Subscription

Meta Quest + is a membership subscription service similar to Sony's PS + or Xbox launched by Meta. After subscribing to this service, users only need to pay 7.99 dollars per month or 59.99 dollars per year to play all the games in the game catalog given by Meta., and you can add the two games given each month to your game library (as long as you keep subscribing, you can play all the time). At the same time, you can enjoy limited-time discounts when purchasing some games.

From August 3rd until August 31st, if you want to subscribe to Meta Quest +, you can use " SUPERHOT VR " to get a half price discount.

If you plan to subscribe to Meta Quest + recently, it is recommended that you do not miss this discount.

Starting this week (2024-08-05) apps on Meta AppLab will be migrated to the Meta Horizon Store

Keywords: Meta, AppLab

In April of this year , Meta announced that it would rename Meta Quest OS to Meta Horizon OS and rename the app store to Meta Horizon Store to show its determination to turn to Mixed Reality.

Starting this week , AppLab, which was originally used as an internal testing channel, will also be merged into the Meta Horizon Store. The original internal testing AppLab will exist in the Meta Horizon Store in the form of Early Access. The overall process is very similar to Apple's AppStore and TestFlight.

However, for most AppLab developers, there is nothing special to do. After AppLab is merged into the Meta Horizon Store, developers who have published apps on AppLab will receive a confirmation email.

Activity

VisionHack: The world's first online visionOS online Hackathon

Keywords: Hackthon, Apple Vision Pro

From September 13th to 15th, visionOS Dev Partners will hold an online visionOS App Hacthon - VisionHack . If you are confident in your creativity or want to communicate with international developers through Discrod and Spatial Persona, you can come and participate in this Hackthon.

It should be noted that participants need to pay a fee of 25 dollars to ensure participation.

Idea

3D Interactive Icons

Keywords: 3D, Interactive

How to make the interaction of an App very vivid? When the user views the icon, there are some corresponding animations to play, which will definitely make the interaction inside the App more dynamic. bigloong also shows us similar interactions in his design :

If you want to see the actual effect of this interaction, you can directly through the Spline preview address to see the actual effect.

Happily, visionOS 2 also provides a new CustomHoverEffect API to allow developers to achieve similar effects (a more complete development approach can be found in the Create custom hover effects in visionOS Session).

Tool

Helm for App Store Connect

Keywords: App Store Connect, visionOS

Helm for App Store Connect is a native macOS app that helps developers submit to the App Store.

By using this application, we can obtain many practical functions when submitting applications, such as automatically translating submitted descriptions into multiple languages, using templates to restore application reviews, automatically checking for duplicate application descriptions or descriptions that do not use application search.

This is indeed a very useful productivity tool for independent developers who frequently need to submit applications.

Article

PDF：Create virtual and mixed reality experiences in Unity

Keywords: Unity, VR, MR

Recently, Unity has released a new VR/MR experience guide in its official community . In this guide, Unity mainly explains the following content:

The basic elements of building a world in XR, such as design purposes, implementing audio and visual cues, adding interactive elements, etc
Tips to follow when creating assets in XR to save time later in the development cycle
Decision factors for each development stage in the new VR project and actual steps in Unity
Practical tips for MR development on Apple Vision Pro

Apple Document——ShaderGraph

Keywords: ShaderGraph, RealityComposerPro

Since the release of Reality Composer Pro at WWDC 2023, the documentation for the ShaderGraph section in Reality Composer Pro has been missing. Fortunately, with the passage of time, this part of the documentation has gradually been supplemented. If you want to know the details of ShaderGraph, it is highly recommended that you take a look at the details of this document.

Code

RealityKit Terrain Shader

Keywords: RealityKit, Github, Shader

In the terminology of game engines, Terrain generally refers to technology specifically used for scene layout. Although this technology is currently missing in visionOS, the RealityKit Terrain Shader project shows us how to use Fractal Brownian Motion (fBM) technology to create a complex terrain through a series of parameters.

Unfortunately, the RealityKit CustomMaterial API can only be used on iOS and macOS, so this project cannot be used for visionOS yet. However, we can learn the technology now and migrate directly when visionOS supports CustomMaterial in the future.

If you are interested in the detailed technical content, you can read the accompanying article written by the author: RealityKit Terrain Shader .

App

Spatial Physics Playground（Free）

Keywords: Physics Simulator

Spatial Physics Playground is a simple and fun physics simulation app. With the spatial perception ability of Apple Vision Pro, this app can give you an intuitive understanding of the physical simulation and virtual-real combination ability of visionOS.

Job

ILLUSION Tech: Front-end development , content cybertribe operations, operations development, 3D content designer

ILLUSION Tech is a technology company focusing on the XR field. " Seasons " is the first Apple Vision Pro application under ILLUSION, which has been recommended by Apple several times since its launch.

The company is developing a space content creation tool to promote the emergence of more high-quality space content.

Currently, the company is recruiting positions.

Job titleJob requirementsBaseRecruitment typeFront-end developmentSwiftUI expert, familiar with native AR frameworks such as ARKit and RealityKit, and familiar with third-party AR frameworks such as Unity ARFoundationShenzhenSocial recruitment, campus recruitmentContent cybertribe operationsWork experience in content sharing products, experience in operating overseas media such as Twitter and Reddit, and able to use English for daily conversationsShenzhenSocial recruitmentOperations development1. Responsible for the overall cloud service structure of the product to ensure high availability, high security, and low cost

2. Responsible for building DevOps R & D operation and maintenance pipeline

3. Responsible for building product Big Data Analysis capability

Bonus points: Experience with AWS and CloudflareShenzhenSocial recruitment, campus recruitment3D content designerExperience in using tools such as Composer, Unity, and Unreal, as well as Shader design experience. Follow the latest developments in the AR industry and have a personal understanding of experience design.ShenzhenSocial recruitment

If you are interested in the above positions, please send your resume to aaron@illusion.tech.

Contributors of this issue

XR World Weekly 022

XRealityZone — Sat, 28 Sep 2024 01:56:14 GMT

Cover of this issue

In this issue, we share the Airbnb - AR/VR Animation Concept built by Aurélien Salomon UX .

Publication preface

Hi, long time no see! I am Siqi, the editor-in-chief of XReality.Zone!

After organizing the Let's visionOS 2024 Global Creator Conference this spring, we fully felt the enthusiasm of the entire ecosystem for spatial computing. Let's revisit the grand scene of the event through the following short video.

It is not difficult to notice that after this event, the XReality.Zone entered a short period of rest, and our XR World Weekly and visionOS teaching content have stopped updating. However, now I can happily announce to you - We are back!

Yes, the XReality.Zone is back!

During this seemingly "rest" period, we communicated with many domestic and foreign developers and enterprises (the number may far exceed your imagination). You may see many people on the Internet saying that this device is too bulky, the interaction is too poor, and the sales of this thing are too low. These information always give you a feeling that spatial computing is about to fail. However, through our actual observation, we believe that the exploration of developers and enterprises in this area is far beyond everyone's imagination, and these changes have practical significance and imagination.

Just like the one-month period after the official release of Apple Vision Pro in China, although the media criticized the market in various ways, Chinese internet giants have come forward one after another to show their own work. This is like saying no with words, but still being honest with the body.

At the same time, the exploration of AVP in the community is also showing an increasingly diverse trend. For example, two community projects that I have recently paid attention to:

To be honest, when I saw these two projects, the first thing that came to my mind was the touching boxing match between Zeus and Adam in Real Steel , as well as some of the world settings in this movie, such as the new electronic sports project of "robot boxing match".

These seemingly unattainable things may already be slowly developing and evolving around you and me.

I want to say, the future has arrived, would you like to join us on the road?

Activity

Qualcomm Snapdragon Spaces XR Technology National Tour Salon · Beijing Station

News

In the second half of the year, you can see more immersive videos on Apple Vision Pro
Apple Vision Pro now has a new virtual environment to view
You can now replace Persona's background

Idea

Playing games with gestures, maybe we can be more creative

Code

visionOS 2 30 Days

Video

How to Make a Bubble in VisionOS: Reality Composer Pro Shader Graph Tutorial

App

Meshy.ai

Job

ILLUSION Tech: front-end development, Content cybertribe operations, operations development, 3D content designer

Activity

Qualcomm Snapdragon Spaces XR Technology National Tour Salon · Beijing Station

Keywords: Qualcomm, Snapdragon Spaces XR

On August 3, 2024 , the first stop of Qualcomm Snapdragon Spaces Technology National Tour Salon will be opened in Beijing . This XR technology sharing meeting is jointly organized by Shenmu Technology and China Electronics Cloud, and has received strong support from Beijing Zhongguancun Entrepreneurship Street Technology Service Co., Ltd.

This technology salon will bring together many XR industry leaders and innovators, focusing on sharing XR innovative application cases, bringing the latest XR technology sharing and solutions to developers. The event is divided into the morning XR application sharing session and the afternoon technology practice session and electronic cloud metaverse closed-door exchange meeting . You will have the opportunity to learn about the most cutting-edge XR technology applications and gain valuable industry insights.

💡 Tips:
Those who fill in the questionnaire from the registration entrance, remember to select "XReality.Zone" in the channel , send the screenshot of the selected channel and completed questionnaire to the "XReality.Zone" official account background, and you can get a small gift - Minority gift three-piece set 😉 .

News

In the second half of the year, you can see more immersive videos on Apple Vision Pro

Key words: Apple Vision Pro, immersive video

Finally, the Apple Vision Pro we bought can have more "exclusive" content! 😆

In this official Apple article , Apple announced that more immersive videos will exclusively land on Apple Vision Pro in the second half of the year . These videos include:

Release timeEpisode nameHighlightPreview imagePublishedBoundless Episode 1: Hot Air BalloonsTake a hot air balloon ride and overlook the strange landforms of Cappadocia, Turkey

This fallBoundless Episode 2: Arctic SurfingGo to the area inside the Arctic Circle and search for the giant waves🈚️This fallWild Life Episode 2: ElephantsElephant orphanages in Kenya

This fallElevatedFly over volcanoes, climb over waterfalls, and discover the unknown natural wonders of Hawaii, the world's longest island chain

This fallSubmergedAgainst the backdrop of World War II, the audience will board a submarine and witness how the crew survived after being hit by a torpedo

This fall2024 NBA All-Star WeekendWatch the Rookie Game, Slam Dunk Contest, the first-ever NBA vs. WNBA 3-Point Challenge Stephen V.S. Sabrina, and the All-Star Game

Later this yearThe Weeknd Immersive PerformanceImmersive experience of Apple Music Award winner and artist with 7 diamond sales certifications

Later this yearRed Bull: Big-Wave SurfingWitness surfing masters trying to conquer towering waves on the Tiahup coast in Tahiti

It can be seen that all these immersive videos are "big scenes" from the setting, which is very consistent with the characteristics of immersive videos.

Apple Vision Pro now has a new virtual environment to view

Keywords: Apple Vision Pro, Environment

In the latest visionOS 2 beta 4, Apple has finally put together the final piece of the puzzle for the virtual environment of the system: Lake Vrangla. With this, all the pieces of the virtual environment puzzle are finally completed.

Compared to the previous Mount Hood, this Lake Vrangla has a misty effect and is very suitable for watching horror movies: P

You can now replace Persona's background

Keywords: Persona, Apple Vision Pro

Outside of FaceTime, if we obtain the front-facing camera of Apple Vision Pro, what we get is a Persona video stream with a background. However, we cannot replace the background of this video stream.

In the recent visionOS 2 beta 4, we can see that in the settings, we can now modify the background of Persona in scenes other than FaceTime. In addition to the background of the virtual environment, users can also set it with photos themselves (more and more like Zoom and other meeting software).

Idea

Playing games with gestures, maybe we can be more creative

Keywords: games, gestures

If we use gestures to operate RPG games, how should we design gestures? Perhaps, Gabriele Romagnoli 's idea of using fingers to operate small people can give us some ideas:

A similar scene also appeared in a movie called Her :

Code

visionOS 2 30 Days

Keywords: visionOS, 30 Days

He's here, he's here. After receiving 2k Stars for visionOS 30 Days last year, our old friend Satoshi This year, he has started updating his 2.0 version - visionOS 2 30 Days . The entire project has reached Day 2. If you have also borrowed Satoshi's code in the development of visionOS last year, then continue to give this project a Star 🤩

Video

How to Make a Bubble in VisionOS: Reality Composer Pro Shader Graph Tutorial

Keywords: visionOS, RealityComposerPro

If you've played Blackbox , the famous game on the Apple Vision Pro, you'll be fascinated by the variety of real bubbles.

In this video from Sarang Borude , he shows us how to use Reality Composer Pro's Shader Graph to generate realistic bubbles.

App

Meshy.ai

Keywords: Generated Model,

Meshy.ai is an AI tool that can quickly generate models through text and images. Recently, Meshy also launched their Apple Vision Pro version . In the current version, we can quickly browse the models accumulated by the community on Meshy and zoom and move them in the form of volumes in Apple Vision Pro:

Job

ILLUSION Tech: front-end development, Content cybertribe operations, operations development, 3D content designer

The company is developing a space content creation tool to promote the emergence of more high-quality space content.

Currently, the company is recruiting positions.

If you are interested in the above positions, please send your resume to aaron@illusion.tech.

Contributors of this issue

XR World Weekly 021

XRealityZone — Sat, 28 Sep 2024 01:56:06 GMT

This Issue's Cover

Dazzling particle effects always feel very interesting. In this issue's cover, we share a creation by zhantemir31 using Spline, titled a star like our own.

BigNews

Apple launches the web version of the visionOS App Store

Idea

AmazedVR: Watch high-definition panoramic 180° videos here

Tool

Ensemble: Open source software for projecting Mac screens to Vision Pro

Article

What you should know about visionOS volumes before using them in an app

Code

Vortex: A handy open-source particle system framework for SwiftUI
Vision Panes: Vision Pro multi-pane expansion animation toolkit
ImmersiveMoveAndRotate: How to move and rotate objects in immersive scenes in visionOS
SpatialMediaKit: Another free Mac command-line tool for converting between MV-HEVC and regular formats
HandVector: Gesture matching toolkit for Vision Pro

QuickNews x 12

Work x 1

BigNews

Apple launches the web version of the visionOS App Store

Keywords: Apple Vision Pro, App Store

Apple recently launched a web page version of the App Store, which includes a number of Apple Vision Pro apps recommended by Apple. If you're still hesitating about whether to buy the Apple Vision Pro, why not come here first to see if there are any fun apps to persuade you😉

Idea

AmazedVR: Watch high-definition panoramic 180° videos here

Keywords: Apple Vision Pro, concert
Thanks to netizen In the Clouds for the recommendation

Besides the 2D and 3D videos on Apple TV and Disney+, if you want a new video-watching experience on Apple Vision Pro, we recommend an app — AmazedVR. It's an app for watching 180° angle concerts with built-in videos at 8K resolution, which, combined with the high-quality visuals of Apple Vision Pro, offer a very immersive experience (although it does take up a lot of disk space, a 16min video requires 19GB of disk space).

In addition to regular video playback, AmazedVR also includes some interactions related to gestures, such as sending specific heart-shaped barrages by making a heart gesture with your hands.

Tool

Ensemble: Open source software for projecting Mac screens to Vision Pro

Keywords: Vision Pro, Mac

Ensemble is a free open source screen-casting software that can project multiple Mac screens to Vision Pro for viewing and use. It consists of two parts: the Mac version and the Vision Pro version, and you can download the code and run it on both devices, or join TestFlight to use the provided version. The usage effect is as follows:

The current version requires three steps to use on a Mac:

Allow screen recording permission to share the Mac screen

Allow accessibility permissions to synchronize Vision Pro operations back to Mac

Copy ensemble_helper_v1.sh to the library: /Users/yourusername/Library/Application Scripts/com.saagarjha.MacCast

Then open the corresponding software on Vision Pro, and it should automatically connect and allow you to choose the window to display.

Article

What you should know about visionOS volumes before using them in an app

Keywords: visionOS, Volume

Volume in visionOS is a very novel presence. Before this, developers in the Apple ecosystem rarely manipulated 3D controls in 3D space as they would on a flat surface.

This article mainly explains some things to be aware of when using Volume in visionOS, including:

Unlike Window, the boundaries of Volume cannot be scaled
The maximum size of Volume is a 2-meter long Box
Similar to Window, we can't set the initial appearance position of Volume, only allowing it to appear directly in front of the user (sometimes this setting is indeed annoying)

Additionally, if you encounter strange situations where the boundaries of Volume are clipped during development, you can also view the author's another article to understand the current settings of visionOS in this regard.

Code

Vortex: A handy open-source particle system framework for SwiftUI

Keywords: Particle system, SwiftUI

Vortex is a powerful particle system framework designed for SwiftUI. With this framework, you can create dazzling particle effects, such as flames and fireworks, in SwiftUI with simple code.

The overall usage is also very simple, just like adding a Modifier to a View in a normal SwiftUI project:

VortexView(.fireworks) {
    Circle()
        .fill(.white)
        .blendMode(.plusLighter)
        .frame(width: 32)
        .tag("circle")
}

Thanks to SwiftUI's cross-platform feature, Vortex also supports multiple Apple platforms, including visionOS:

The project's author Paul Hudson also introduces this open-source framework in his video. If you prefer watching videos, you can learn about Vortex through this video.

Additionally, Paul Hudson also maintains the highly regarded tutorial website Hacking With Swift. If you are interested in Swift, you can also find many high-quality tutorials on Hacking With Swift.

Vision Panes: Vision Pro multi-pane expansion animation toolkit

Keywords: Vision Pro, SwiftUI

VisionPanes provides multiple animations for expanding and closing panes, allowing you to add window size change functionality to apps in Vision Pro, fully utilizing the unlimited screen feature of Vision Pro. It is already supported by Swift Package Manager (SPM) and can be directly used in your projects.

It supports not only left and right pane expansion and closure but also upper and lower panes, offering flexible and diverse forms and convenient use.

ImmersiveMoveAndRotate: How to move and rotate objects in immersive scenes in visionOS

Keywords: Vision Pro, SwiftUI, DragGesture, RotateGesture3D

ImmersiveMoveAndRotate demonstrates how to present ModelEntity in immersive scenes on visionOS using RealityView and how to use DragGesture to drag the entity and RotateGesture3D to rotate it.

This example project currently only allows movement along the X and Z axes to prevent users from lifting objects off the ground. However, moving along the Y-axis is easily achievable with a simple modification of the code in EntityMovementViewModifier.swift.

SpatialMediaKit: Another free Mac command-line tool for converting between MV-HEVC and regular formats

Keywords: Spatial Video, MV-HEVC

In Issue 19, we recommended the spatial tool, a command-line tool for converting between MV-HEVC and regular videos on Mac. This time's SpatialMediaKit does something similar, handling the conversion between the two formats, including two sub-commands:

split: Obtains an MV-HEVC file and outputs the left and right eyes as separate files in the current directory. The output format is ProRes 422 HQ, video only. Users should be familiar with tools like ffmpeg for all other needs, including adding audio back in.
merge: Takes two video files (left and right) and re-encodes them into an MV-HEVC file. Again, output is video only. Users should be familiar with tools like MP4Box for all other needs, including adding audio back in.

HandVector: Gesture matching toolkit for Vision Pro

Keywords: Vision Pro, SPM

HandVector is a gesture matching toolkit for Vision Pro, supporting import using Swift Package Manager (SPM). It uses the cosine similarity algorithm to calculate the similarity between a single hand gesture and pre-recorded gestures.

The project also integrates the VisionOS SimHands functionality for simulator debugging, allowing users without a real Vision Pro to test gesture functionality on a simulator. More details can be found here.

Quick News

Tips
If you don't want to read the text, you can also go to the podcast " XR Product Talk" - "XR News Early Bird" to listen.
Search for "XR Product Talk" on Apple Podcasts or go directly here
Search for "XR Product Talk" on XiaoYuZhou or click here

Work

【Hangzhou-Taotian Group-Internship】Vision Pro Development Engineer
- Job Summary: If you want to experience the Vision Pro, a space computing device filled with "tomorrow's technology," if you want to be involved in the latest and trendiest visionOS development, if you want to experience the development of a cross-era app from scratch, we welcome you to apply for this position.

Contributors to This Issue

XR World Weekly 020

XRealityZone — Sat, 28 Sep 2024 01:55:51 GMT

This Issue's Cover

This issue's cover is from Nikita Belousov's design of the visionOS ARC browser concept, where the essence of visionOS in a three-dimensional space is fully considered, giving the Arc browser's Space and Pined Site a sense of depth.

BigNews

visionOS 1.1 and Xcode 15.3 released!
PICO updated to 5.9 system, bringing PICO Connect and Avatar SDK

Weekly App

MagicRoom: Add some magic to your room
Share your space videos on Spatial Station, ImmersiShare, and Share Spatial!

Tool

Quickly search for visionOS apps on Mac

Video

GazeTrak: Eye tracking with sound?

Article

The first 1000 apps on visionOS
Apple Vision Pro has great potential in the healthcare field
Create With Swift-Make it Spatial: High-quality Swift and visionOS development tutorial articles

SmallNews

Perhaps, the lifelike environment in Apple Vision Pro is not just panoramic video

Quick News x 9️⃣

BigNews

visionOS 1.1 and Xcode 15.3 Released!

Keywords: visionOS, Xcode

After over a month of beta testing, visionOS finally released the official version 1.1 this week. Let's highlight some interesting points of this update~

First, a very useful update not mentioned in any update notes: visionOS 1.1 solved the issue of screen recording in low light conditions. Previously, if your room was dark, you couldn't screen record (unless you turned on travel mode to bypass this), as complained by Reddit users.

Now in visionOS 1.1, we can still record the screen even in almost pure black rooms.

In addition to solving the screen recording problem, in the new version of the system, if you want to record Persona, you can choose not to hold the Apple Vision Pro yourself.

Instead, you can choose the Hand-Free method in Accessibility, place the Apple Vision Pro somewhere, or have someone else hold it for recording.

One might ask, what's the point? Likely, this reflects Apple's consistent positioning of its products: making them accessible to a diverse range of people. As shown in the Equal Accessibility video, even a user without hands can effectively use the Apple Vision Pro. Knowing this, it makes sense why Apple Vision Pro specifically provides a Hands-Free Capture option.

Next is the improvement of Persona itself, which is subjective. Have a look at the comparison by NewAppleTimes, do you think Persona has improved?

The official update notes indicate that this update mainly targets the appearance of hair, makeup, neck, and mouth. By the way, this update also brought improvements to the look and stability of EyeSight, Mac Virtual Display (the function of projecting Mac to Apple Vision Pro), and the reliability of the virtual keyboard.

Moreover, after this update, all official apps except for Settings, Photos, Safari, App Store, Messages can now be deleted. (This logic is more like the iOS logic that wasn't implemented before.)

Additionally, following the visionOS update, Xcode 15.3 was also released, which didn't involve much content related to visionOS development. Perhaps the only relevant feature is the hidden "Export Preview Screenshot" in Editor > Canvas, allowing direct export of app screenshots from Xcode's Preview.

Overall, visionOS 1.1 and Xcode 15.3, as a minor version update, didn't bring very rich content, more like some basic feature optimizations.

PICO Updated to 5.9 System, Bringing PICO Connect and Avatar SDK

Keywords: PICO, Unity, Avatar SDK

After nearly half a year, PICO finally welcomed a new system upgrade – 5.9.0 (the last upgrade to 5.8.0 was in October '23). On the user side, PICO brought a major feature: PICO Connect. This is an upgraded product of the original streaming assistant.

With PICO Connect, Windows and Mac users can mirror computer screen content to PICO (similar to Mac Virtual Display on Apple Vision Pro), and on Mac, PICO Connect also supports switching between multiple connected screens:

Currently, PICO Connect seems usable in terms of stability and clarity (supporting up to 4K resolution (3840*2160), screen ratio of 16:9), but now due to the lack of a mature multitasking mechanism in the PICO system, other apps can't be opened for simultaneous operation when PICO Connect is active. Hopefully, this will improve in future versions.

However, thanks to the "Auto Connect to This Computer" setting in PICO Connect, your computer will automatically project to PICO when you put it on, which feels very convenient.

In addition, PICO has updated video playback and gesture recognition to some extent.

In terms of video playback, PICO mainly improved the recognition accuracy of its built-in video player. When you open various 3D, 180°, 360° videos in PICO's file manager, it will automatically help you identify the video format and play it correctly. Also, the new "Premium Image Mode" enhances the sharpness of 4K videos during playback.

For gesture recognition, PICO mainly optimized for some edge cases:

On the developer side, the newly released PICO Unity Integration SDK 2.5.0 now supports the XRI framework and Hands package.

Editor's Note
As early as June '23 in 002-Unity XR Interaction Toolkit released version 2.3, we introduced XRI and Hands, saying that this feature requires platform support. Nearly a year later, PICO finally supported it (official documentation here), feels like a wish come true 🫣

The PDC tool also now supports hand gesture preview:

On another note, the newly launched Avatar SDK fills a gap for developers on PICO's virtual human aspect. Previously, developers on the PICO platform could only create or use third-party SDKs like Ready Player Me to create avatars for users. PICO previously provided users with an avatar editing feature (click "My" -> "Edit Virtual Image" to enter this interface):

However, developers previously had no access to these avatars. This Avatar SDK now provides these avatars (including full-body skeletal models) to developers. With the Avatar SDK, developers can directly access these avatars like other PICO platform services (such as accounts), and can pop up similar UIs in the app for players to add or edit avatars:

If you want to know how to use the Avatar SDK, you can check out the sample project provided by the official on Github.

Idea

MagicRoom: Add Some Magic to Your Room

Keywords: The Matrix, visionOS

MagicRoom is a visionOS app that adds a layer of "spatial filter" to the real world. Using the Scene Reconstruction capability provided by visionOS, MagicRoom can make your surroundings look like they are covered with a colored grid:

Or like the digital rain in The Matrix:

Although currently there are only these two "filters", it is believed that more similar filters will be added to MagicRoom in the future (finally, our imagination is the only limit to the final effect😆)

Share Your Space Videos on Spatial Station, ImmersiShare, and Share Spatial!

Keywords: Space video, Sharing
Thanks to the user In the Clouds for the recommendation

Watching space videos on Apple Vision Pro truly feels like going back to those particular moments in the past. Now, by watching space videos shared by friends around the world, we can experience interesting activities and beautiful scenery from all over the world without leaving home. If you have captured wonderful moments in life, consider sharing your space videos! Currently, there are 3 ways to share space videos:

Spatial Station is an already launched app with a fairly complete user system and many user-uploaded space videos. The homepage has sections like Latest Uploads, Current Hot, Hot Trends, etc. However, the main issue is the varying quality of user-uploaded videos (such as some with severe shaking causing viewer dizziness, and some with less prominent parallax and lacking a strong 3D feel).

Another app, ImmersiShare, also allows for the sharing of space videos. This app is special in that the videos shared are actually uploaded to the developer's iCloud for sharing, so the loading speed of the videos is subject to network influence.

Apart from app-based sharing, Share Spatial is a website where you can find space videos, space photos, and even panoramic images. There are even some artists sharing their panoramic images, which are quite stunning. Share Spatial is also testing their own app, and soon everyone will be able to share and view more conveniently on Apple Vision Pro.

Tool

Quickly Search for visionOS Apps on Mac

Keywords: Raycast, vision.directory

The vision.directory we recommended in Issue 019 recently added a plugin on Raycast, a highly praised Mac launcher, allowing you to search for visionOS apps directly through Raycast.

Video

GazeTrak: Eye Tracking with Sound?

Keywords: Sonar technology, Eye tracking

Eye tracking is a key technology in XR, used for focus rendering in the user's gaze area to reduce system rendering pressure, and also for UX design, like the overall system interaction in Apple Vision Pro.

Although relying entirely on eye tracking for system interaction can cause problems, such as rapid fatigue for interactions requiring frequent, fast switching, and a difficult balance between providing APIs for developers to unlock more possibilities and user privacy.

However, achieving precise eye tracking using cameras is a significant challenge. Recently, researchers from Cornell University in New York publicized a technology that uses sonar-like technology instead of cameras for eye tracking: GazeTrak. This technology utilizes 8 miniature speakers, each capable of emitting sound at frequencies above 18 kHz.

As these signals are beyond most people's hearing range, they are imperceptible to humans and also immune to noise interference.

The research team suggests that sonar technology should have several advantages: compared to camera-based systems, GazeTrak can operate at low power and, since it doesn't require continuous data recording, offers more privacy for users. Additionally, it could reduce the manufacturing cost and weight of VR headsets. However, a main drawback of GazeTrak is that enough data needs to be collected for model training due to the different eye shapes of individuals.

Article

The First 1000 Apps on visionOS

Keywords: visionOS, AppStore

In The First 1000 Apps on visionOS, the author analyzes the 1217 apps currently on the AppStore, offering simple insights into their types and pricing.

It's evident that compared to other XR devices, apps on visionOS are still mostly productivity and tools focused, highlighting the difference between Apple Vision Pro and other devices—it's not a gaming console, but a spatial computing device.

Apple Vision Pro Has Great Potential in the Healthcare Field

Keywords: Apple Vision Pro, Healthcare

Utilizing XR to assist doctors and nurses in surgery has always been an exciting concept, but due to various reasons, XR technology has never been fully applied in real surgical environments, until recently. According to a report by Daily Mail Online, in a real surgery in London, nurses working with surgeons used Apple Vision Pro to help prepare, track the surgery, and assist in selecting the right surgical tools.

It's clear that Apple Vision Pro is driving development across various fields. This week, Apple also shared some visionOS-exclusive apps developed by the healthcare community. Using the new platform of Apple Vision Pro to simplify complex medical processes or 3D visualization of medical imaging is no longer out of reach.

For instance, myMako allows surgeons to easily share the details and insights of complex knee surgery plans in a 3D, intuitive, and dynamic way.

CyranoHealth simulates the use of medical equipment, helping frontline workers like nurses familiarize themselves with new and complex medical devices (such as medical infusion pumps), thereby reducing work-related anxiety.

Epic Systems allows doctors to perform tasks like gesture drawing, patient communication, and condition analysis in Apple Vision Pro, opening new collaborative and productivity workflows for future medical applications.

Apart from assisting doctors in their tasks, Vision Pro can also help us pay attention to our physical and mental health. For example, Cedars-Sinai's Xaia app provides intelligent conversational mental health support to users, who can also engage in deep breathing exercises and meditation.

There are many more healthcare apps, such as Visage Ease VP for improving patient care and aiding in medical data review for inefficient tumor committees; Falcon Vue aims to empower doctors with the ability to view all medical imaging data; Insight Heart allows us to understand our heart in 3D and explore various heart conditions. Cinematic Reality lets surgeons, medical students, and patients view medical images obtained through medical scans in the real environment.

Nature Magazine also notes the great potential of Apple Vision Pro in the healthcare field. For example, the advanced eye-tracking technology of Apple Vision Pro might detect early signs of motion sickness, stroke, or dementia. Surgeons can enhance their awareness of patient information by wearing Apple Vision Pro, such as seeing the patient's vascular structure during surgery for more precise operations.

Create With Swift-Make it Spatial: High-Quality Swift and visionOS Development Tutorial Articles

Keywords: Swift, visionOS, Spatial, Accessibility

Create With Swift is a website maintained by developers and designers at imaginary.institute, sharing many development and design insights on Swift app development. The website is mainly divided into two sections:

Make it Accessible: Mainly sharing design and development articles related to accessibility features
Make it Spatial: Mainly sharing design and development articles related to visionOS

Thanks to the collaboration between design and development, the content on the website is concise and useful, often accompanied by attractive illustrations. For example, this article discussing the design of icons on visionOS uses an animated GIF to represent the feel of Spatial Icons, which is both interesting and visually appealing:

Small News

Realistic Environments in Apple Vision Pro Are Not Just Panoramic Videos

Keywords: Apple Vision Pro, Environment

Environment is one of the few official Fully Immersive experiences on Apple Vision Pro. Everyone who tries Apple Vision Pro for the first time is amazed by the detailed quality of the scenes and the surrounding spatial audio in the environment. Many think these are panoramic images or videos, but these might actually be meticulously modeled scenes by Apple! Don't believe it? Check this out:

Apple Vision Pro's Mount Hood Environment

Through M1's roaming perspective, we can see that the entire scene is indeed modeled, and to reduce unnecessary computational load, modeling and texturing are omitted in areas not visible to the viewer. Once again, we are in awe of the immersive experiences Apple Vision Pro offers!

Apple Vision Pro's Joshua Tree Environment

Apple Vision Pro's Yosemite Environment

Quick News

Tips
If you prefer not to read text, you can also listen to the podcast 「XR Product Talk」 — 「XR News at a Glance」.
Search for 「 XR Product Talk」 on Apple podcast or go here
Search for 「 XR Product Talk」 on Xiaoyuzhou or click here

Contributors to This Issue

Zion

XR World Weekly 019

XRealityZone — Sat, 28 Sep 2024 01:55:02 GMT

Cover of this Issue

In this issue's cover, we share Mike Stang's creation Instagram 4 VisionOS. Currently, many applications on Apple Vision Pro still use a single-block design similar to the iPad, not fully utilizing the "space" as a larger "screen". We hope to see more multi-block design apps like Instagram 4 VisionOS in the future.

Foreword

Hi everyone, surprised to see the XR World Guide appear this Thursday, right?😉

Yes, in order to meet with you every week, we decided to switch from bi-weekly to weekly updates starting from issue 019. Unless there are unforeseen circumstances (like typhoons or volcanic eruptions), the XR World Guide will meet you every Thursday.

Of course, maintaining weekly updates is impossible with just one person's effort. We would like to thank all the evangelists at XReality.Zone. If you are passionate about the XR world and want to share the fun of XR world through XR World Guide, feel free to join our evangelist team at XReality.Zone.

How to join? You can contact us via WeChat (add CallMeOnee) or email (renyujie@xreality.zone)~

Without further ado, let's see what this issue of XR World Guide brings you~

BigNews

Let's visionOS — A playground for visionOS creators created by us at the end of March

Idea

Bezel: Show Your iPhone in Apple Vision Pro
Dino: Helping you type Chinese on Apple Vision Pro

Tool

Unity provides a new visionOS Project Template for developers
SplineMirror: Now you can view 3D models created in Spline on visionOS
Spatial: A free command-line tool for Mac to convert MV-HEVC to regular format

Code

HandsRuler: An app for measuring object lengths with both hands is now open source
ShaderGraphCoder: Writing ShaderGraph with code
ALVR for Apple visionOS: SteamVR streaming software now supports Vision Pro!

QuickNews x 🔟6️⃣

BigNews

Let's visionOS — A playground for visionOS creators created by us at the end of March

Keywords: Apple Vision Pro, visionOS Creators

Here it comes! 🌟 The first Let's visionOS Creator Conference is about to open grandly in Beijing at the end of March. 🌆 From March 30 to 31, friends at XReality.Zone invite you to focus on Apple Vision Pro and visionOS, exploring new trends in spatial computing. This is not just a gathering, but a great opportunity to deeply interact with global innovators and pioneers who love tinkering. There's also a chance to directly experience Apple Vision Pro, immersing in the future's appearance📍

At the conference, XReality.Zone's team has invited many domestic and international developers🌍, who will bring their extraordinary insights on Apple Vision Pro, such as:

Jordi Bruin — Creative independent developer, his app Navi was nominated for the Apple Design Award in 2022, revolutionizing communication with real-time subtitle translation for video calls.
Devin Davies — Outstanding iOS developer, his visionOS recipe app Crouton was highly acclaimed by the Wall Street Journal for its unique AR timer function, along with his creative ‘AR Plate Smashing Stress Relief App’ Plate Smasher, showing his exceptional talent in app innovation.
Hidde van der Ploeg — Product designer with ten years of experience, his visionOS app NowPlaying provides an upgraded music playback experience for Apple Vision Pro users, becoming a star product recommended by the App Store.
Adam Watters — Pioneer in game development, his visionOS app developed with the Godot game engine demonstrates more diverse development possibilities to the developer community, pioneering a new path for developing high-quality applications with non-mainstream engines.
Hai Xin and A Wen — The first AI space video creators to perform on the Spring Festival Gala stage, leading AI visual artists in China.

At this two-day grand conference, we will explore the various possibilities that visionOS brings us. If you are:

Engineer: You will explore cutting-edge technology, challenges, and solutions under the new platform, expanding your technical horizons.
Designer: You will explore new design concepts in spatial computing, get inspiration from peers from five continents, and boost your creativity.
Entrepreneur: You will build connections with emerging partners during the inception period of the new platform, seeking inspiration and opportunities for entrepreneurship.
Tech enthusiast: You will keep up with technological forefront, expanding your cognitive boundaries, igniting your passion and innovative drive for the future.

🌟 Embrace international perspectives, seize overseas opportunities: March 30 to 31, Beijing, join us to welcome the new chapter of technological innovation. 🚀 Click the link to secure your spot and enter the new era of spatial computing.

Idea

Bezel: Show Your iPhone in Apple Vision Pro

Keywords: Apple Vision Pro, iPhone, Mirror

Bezel is an app that helps you project your iPhone screen on your Mac, and it has recently launched an app for Apple Vision Pro to project your iPhone as an independent window.

If you want to use some apps on Apple Vision Pro in multiple windows, which are currently only available on mobile phones, like the one shown in the picture above, you can try this app.

Additionally, like the Mac app, Bezel also offers very exquisite iPhone shells, only this time they are fully 3D!

Dino: Helping you type Chinese on Apple Vision Pro

Keywords: Apple Vision Pro, Chinese Input

Even though the national version has not been released yet, Chinese developers and consumers' enthusiasm for Apple Vision Pro has not diminished at all🤏.

However, when everyone gets their Apple Vision Pro, they find that visionOS does not currently provide Chinese input method and voice recognition, making it very difficult to type Chinese on Apple Vision Pro.

Dino was launched to solve this problem. With Dino, you can input Chinese offline via voice. After the input is finished, Dino will automatically add the text content to the clipboard, and you can also drag and drop it into any text box.

The author's idea and initiative, Ciyou, deserve praise👍.

Tool

New visionOS Project Template from Unity for Developers

Keywords: Unity, visionOS

Unity now offers a visionOS Template Project for developers looking to start developing apps on visionOS. Dilmer showcased some demos running on Apple Vision Pro in his video.

This Template includes a Shared Space scene and a Full Space scene.

In native visionOS development, we typically use a Volumetric Window to contain the App's 3D content in Shared Space mode. In PolySpatial, we achieve the same effect through the VolumeCamera component. Just like the Volumetric Window in native development, we can also set the physical size of the 3D Volume in MR using the Dimensions parameter in VolumeCamera. When we set the mode of VolumeCamera to Bounded, as the name suggests, our 3D content is displayed within a fixed-size Volume. When we change the mode of VolumeCamera to Unbounded, the App switches from Shared Space mode to Full Space mode, and the 3D content in the App is no longer bounded, entering a fully immersive mode.

In the Bounded scenario of the case study, we can move floating blocks and other 3D objects in the scene through remote hand-eye interaction and direct interaction. This scenario also allows us to learn how to replicate the interaction modes native to the visionOS system in Unity. The two scenes in the Template can be switched in real-time. By dynamically loading scenes or switching the mode of VolumeCamera, we can enable the App to switch between Shared Space and Full Space modes. The Unbounded scenario of the case demonstrates how to use plane detection and gesture recognition.

In addition, this Template project also provides an interactive Tutorial to help you step by step understand the development components of PolySpatial.

If you want to quickly get started with developing visionOS Apps on Unity, this template project is recommended for you not to miss~

SplineMirror: View 3D Models from Spline on visionOS

Keywords: Spline, 3D Editor, visionOS

As a 3D artist, when you've finished your proud creation in the editor, it would be so thrilling to view it in 3D form.

Now, SplineMirror + Apple Vision Pro can satisfy this need. After you have finished editing a scene in the Spline Editor, just click Export -> visionOS -> Update, and you can export this scene file in a format suitable for visionOS preview.

Next, just open Spline Mirror in Apple Vision Pro, and you can see this scene in Recent:

Then, by clicking on the scene, you can view this model file in three different forms: Window, Volume, Immersive Space:

Of course, at this stage, Spline Mirror does not support all features of Spline files, as can be seen from the official documentation. There are still many features such as those involving interactive events, and overall scene post-processing that are not yet supported:

However, for Spline, previewing 3D models on Apple Vision Pro is definitely a scenario where the experience is greatly enhanced. Looking forward to further updates and upgrades of Spline~

Spatial: Free Command-Line Tool for Converting MV-HEVC on Mac

Keywords: Spatial Video, MV-HEVC

In previous articles, we recommended many tools for Apple spatial video, such as in 018 where we recommended HaloDepth, Owl3D, and Depthify.ai.

This time we recommend spatial, a command-line tool that makes it easy to convert between spatial videos and regular SBS format videos on a Mac (if you're unfamiliar with SBS, please read What is Spatial Video on iPhone 15 Pro / Apple Vision Pro? for a brief review).

The usage is very simple. If you have Homebrew installed, just execute the following command in the terminal:

brew install spatial

Then, you can use these four commands to perform the operations you want:

info: Get information about an MV-HEVC file
export: Convert an MV-HEVC into a regular SBS video
make: Convert a regular SBS video into an MV-HEVC format
metadata: Get and modify the metadata in an MV-HEVC file

For example, with the following command, you can convert a spatial video in MV-HEVC format into a regular SBS video:

spatial export -i spatial_test.mov -f ou -o over_under.mov

And with the following command, you can convert an SBS video into a spatial video in MV-HEVC format:

spatial make -i over_under.mov -f ou -o new_spatial.mov --cdist 19.24 \
--hfov 63.4 --hadjust 0.02 --primary right --projection rect

Although Quest has already supported direct viewing of MV-HEVC format spatial videos in v62, this command-line tool is still very practical in scenarios where a large number of video files need to be modified. Many thanks to the author Mike Swanson for bringing us such a convenient tool!

Code

HandsRuler: Open-Source App for Measuring Object Lengths with Hands

Keywords: Ruler, visionOS

HandsRuler is a free app that uses gesture tracking to measure lengths, now available on the App Store. Additionally, the author has open-sourced the project code for everyone to learn from, available at HandsRuler. If you are interested in developing visionOS applications, you can refer to and learn from this project's code to understand the basics of developing and launching with visionOS.

ShaderGraphCoder: Write ShaderGraph with Code

Keywords: visionOS, ShaderGraph, USD

visionOS natively supports the import and use of ShaderGraph, but Xcode itself does not currently offer a code writing feature. That means, to use ShaderGraph, one has to use external tools (such as Reality Composer Pro or Blender, MaYa) to pre-write by dragging components, and then import them via USDZ files.

The ShaderGraphCoder project provides the ability to write ShaderGraph code in Xcode. The specific implementation logic is:

Write Shader Graph code using the preset SGValue types, which make up a graph
Write the graph into a USDA file in a predetermined format (USDA is a plain-style USD file, similar to a json file)
Load the USDA file to obtain a usable ShaderGraphMaterial type

Most common shader operations and input types are supported. Example usage is as follows:

// Create a solid red material for visionOS
func solidRed() async throws -> ShaderGraphMaterial {
    let color: SGColor = .color3f([1, 0, 0])
    let surface = SGPBRSurface(baseColor: color)
    return try await ShaderGraphMaterial(surface: surface, geometryModifier: nil)
}

// Create a pulsing blue material for visionOS
func pulsingBlue() async throws -> ShaderGraphMaterial {
    let frequency = SGValue.floatParameter(name: "Frequency", defaultValue: 2)
    let color: SGColor = .color3f([0, 0, 1]) * sin(SGValue.time * frequency * (2*Float.pi))
    let surface = SGPBRSurface(baseColor: color)
    return try await ShaderGraphMaterial(surface: surface, geometryModifier: nil)
}

ALVR for Apple visionOS: SteamVR Streaming Software Now Supports Vision Pro!

Keywords: SteamVR, ALVR

The well-known streaming software ALVR recently announced its official launch on the Apple Vision Pro headset. Players can use this application to remotely play SteamVR games on the Vision Pro headset. However, the related application is not yet available on the App Store, and users need to download the project from GitHub for manual compilation and deployment. You can visit the project address to view the code, and follow the compilation steps to install it on Apple Vision Pro.

The main issue this project currently faces is the lack of adapted controllers, as Apple has not equipped Vision Pro with such devices. Therefore, players need to use PS5's DualShock controllers or purchase additional VR controllers to play games.

Quick News

Welcome to our new section - "Quick News", where we'll quickly bring you various industry news from the past week

Tips
If you don't want to read the text, you can also listen to the podcast "XR Product Talk" - "Early Knowhow of XR Dynamics".
Search for "XR Product Talk" on Apple podcast or go directly here
Search for "XR Product Talk" on Xiaoyuzhou or click here

Contributors to This Issue

XR World Weekly 018

XRealityZone — Sat, 28 Sep 2024 01:54:54 GMT

This Issue's Cover

This issue's cover features a design from Ramotion Future Labs, showcasing Movie trailers on IMDB. Pay attention to the light effects perfectly blending the floating screen with the real-world wall next to it. This embodies the core experience enhancement of watching movies in Apple Vision Pro, as envisioned by the designers.

BigNews

Is the $299 Developer Strap Worth It?
30 Best XR Blogs and Websites

Idea

A New Way to Play? Creating Spatial Videos with Sora
Enhance Your Apple Vision Pro Experience with 3D Printed Accessories!

Tool

Lifelike 3D Biological Models
Where to Find Fun Apps After Getting Apple Vision Pro?
SixD - Understanding Best Design Practices on visionOS
Splitscreen - Dual Screen Dreams on Apple Vision Pro Become Reality
Rodin Gen-1 - Experience Converting Images into 3D Models in Apple Vision Pro

Article

The Assistive Features of Apple Vision Pro
Tips for Cleaning Apple Vision Pro and Its Accessories

Code

GodotVision- Developing on Goddot for visionOS!

SmallNews

Is the Return Rate of Apple Vision Pro Really That High?

BigNews

Is the $299 Developer Strap Worth It?

Keywords: visionOS, Apple Vision Pro, Developer Strap

Weeks after its release, if you're a developer with an Apple Vision Pro, you can choose to debug projects directly through WIFI, as detailed in this article: Points to Note if You Are a Developer of Apple Vision Pro.

However, if your visionOS project is large, debugging via WIFI each time can be time-consuming. Now you have another option: wired debugging with the Apple Vision Pro Developer Strap.

Tips
Note that to purchase the Developer Strap, you must have a U.S. Apple Developer Program account, and one account can purchase a maximum of 5 Developer Straps.

Despite complaints, a rational analysis of its high price reveals that the Developer Strap is not just a cable but also equipped with a speaker on the right side. Currently, for large projects, spending $299 on this accessory is still considered worthwhile. Dilmer compared his project's installation time on Apple Vision Pro with and without the Developer Strap, dropping from 19s to 6s, almost a threefold difference.

According to Jeff Benjamin, the Developer Strap currently offers only a 100Mbps USB 2.0 connection when connected to a Mac. However, it supports Thunderbolt, indicating it's capable of connecting to external monitors and any USB-supported device.

Stephen Robles found that these external devices currently do not work, but this is merely a software limitation and might change in the future.

Apart from faster project transfer speeds, the Developer Strap also provides more stable Mac screen mirroring. If Apple Vision Pro running a Beta system bricks, system downgrade and recovery without a Developer Strap would require a visit to an Apple retail store in the U.S.

The specific downgrade steps are as follows:
Power off, install Developer Strap on Apple Vision Pro;
Connect Apple Vision Pro to a Mac using the USB-C port of the Developer Strap;
Connect the power cord, then immediately press and hold the top button on the Apple Vision Pro until the “Connect to Mac” display appears;
Open Apple Configurator, double-click the Apple Vision Pro device icon, and ensure it's recognized in recovery mode;
Click "Action" in the menu bar, hold the Option key on the Mac keyboard, then click "Restore";
Locate the visionOS IPSW restore image, then click "Restore".

30 Best XR Blogs and Websites

Keywords: Blog, XR

FeedSpot, an online service for aggregating, reading, monitoring, and managing content from various online sources like blogs, news websites, RSS feeds, and YouTube channels, recently listed the Top 30 XR Blogs and Websites. We are thrilled that XReality.Zone is among them. We thank everyone for their recognition. Also, check out other excellent XR blogs on this list.

Idea

A New Way to Play? Creating Spatial Videos with Sora

Keywords: Sora, Spatial Video

Recently, I visited Santorini Island in Greece and captured some aerial spatial video footage. How does it look?

If you've been following Sora, I confess, this video was generated using Sora. But how did it become a spatial video?

Andrew Mayne used a video generated by Sora, copied it for both eyes, and played one eye's video a few frames late, depending on the camera's movement direction, to create a disparity for spatial perception. More examples are available in his Github repository for experiencing on Apple Vision Pro.

However, this method requires the generated video to have horizontal camera movement, as later frames can't form correct parallax with earlier ones otherwise.

In addition to this method, YouTuber [Hugh Hou](https://www

.youtube.com/watch?v=vwL0-TdoIyg) introduced several other methods using neural network depth estimation to create spatial videos.

For a fee (about $0.75/min for 1080P videos and $10/min for 4K videos), you can use HaloDepth for online conversion of regular videos into 3D videos playable on Vision Pro. For those with powerful Windows PCs, Owl3D offers a free option to obtain 3D video formats, combined with ffmpeg to convert videos into Vision Pro-compatible spatial videos.

If you only have an Apple Silicon Mac, you can use Spatial Media Converter to create spatial photos/videos from regular 2D material, though conversion will be slower (as it runs the Depth Anything model locally to estimate frame-by-frame image depth). Note that these methods use neural network models whose estimated image depths are not physically accurate, often requiring adjustments to Depth Multiplier to control content's depth perception. Depth values may fluctuate over time, so final effects can differ significantly from actual footage.

Your raw markdown translation is:

Enhance Your Apple Vision Pro Experience with 3D Printed Accessories!

Keywords: Apple Vision Pro, Accessories, 3D Printing

Apple Vision Pro offers a truly magical experience, especially when its weight is evenly distributed on your head. There are already many small accessories available to redistribute the weight of 600g!

Pairing with a Hat

The Dual Band headband certainly allows us to use the Apple Vision Pro for longer periods, but it can ruin our hairstyle! Therefore, some have thought of using a hat combined with two hooks to redistribute the weight while also protecting the hairstyle, killing two birds with one stone!

However, if the wearer has a larger head, the hooks may cause the straps to deform, so use with caution...

Multiple Straps in Action

Apart from using a hat, could we use multiple straps simultaneously? The Solo Top designed by Nick McClure allows the combination of multiple headbands for weight redistribution. According to Big Ears's test, two Solo Knit Bands do provide an almost perfect experience! The downside is the need to purchase an additional $99 Solo Knit Band.

Other Accessories

In addition to weight redistribution accessories, there are also stands for placing the Apple Vision Pro, preventing dust from affecting the internal screen and turning the Apple Vision Pro into a decorative piece.

If you're interested, we could also try 3D printing...

Tool

Lifelike 3D Biological Models

Keywords: 3D Model, SketchFab

ffish.asia and florazia.com are two websites focused on natural science research. Recently, they uploaded over 2700 realistic models of flora and fauna on SketchFab, many of which are downloadable.

If you're looking to use real-life animal and plant models in your project, these websites are worth exploring.

Where to Find Fun Apps After Getting Apple Vision Pro?

Keywords: Apple Vision Pro, visionOS, vision.directory

After getting Apple Vision Pro, if you're unsure which apps to download, aside from exploring the official App Store, we recommend three websites to help you find fun apps for Apple Vision Pro.

First is vision.directory, which groups apps based on App Store categories and offers email subscriptions for daily new app updates.

The second is Apps For Apple Vision Pro, similar to visiono.directory, with apps grouped according to App Store categories.

This site uniquely features additional developer information on each app's detail page, like the official website, Twitter, and email, allowing users to connect with developers.

Developers can submit their apps to both websites through a submission form.

Additionally, Chinese developers can check out Side Vision Pro by V2XR on V2XR, where V2XR shares interesting Apple Vision Pro apps.

SixD - Understanding Best Design Practices on visionOS

Keywords: Apple Vision Pro, SwiftUI

SixD, developed by Yang Haolun, a student at the Pasadena Art Center College of Design and WWDC 2023 Scholarship recipient, is a SwiftUI design and development tool that helps quickly grasp Apple's design guidelines. The app was recently adapted for visionOS.

It offers interactive tutorials to learn best design practices by modifying common design errors.

In SixD's SwiftUI Lab, you can design SwiftUI components using a WYSIWYG interface and get the corresponding SwiftUI code. The new visionOS version allows adding continuous rotation and drag interactions to 3D models with one click.

The visionOS-adapted version adds sections about three-dimensional object interactions, offering a more vivid understanding of interactions in visionOS, like object auto-rotation and user-operated movement.

If unsure how to implement a design in SwiftUI, whether it's fonts, animations, buttons, navigation, or even MapKit and PencilKit, SixD offers the necessary SwiftUI components. Bridging a significant gap in the visionOS development experience, SixD is a guide to human-machine interface design and a WYSIWYG SwiftUI design tool.

Importantly, in a time when Apple Vision Pro apps often cost upwards of $19.9, SixD is free!

Splitscreen - Dual Screen Dreams on Apple Vision Pro Become Reality

Keywords: Mirror, Splitscreen

Mirroring a Mac screen to Apple Vision Pro is a fantastic feature, but the limitation of mirroring only one screen has been a common complaint. Now, Splitscreen, developed by Jordi Bruin and others, makes dual-screening on Vision Pro a reality.

After enabling Mac mirroring to Apple Vision Pro, open Splitscreen on Vision Pro and connect via the Mac app for a dual-screen work experience. Currently, Splitscreen supports only 1920 x 1080 mirroring. Splitscreen is in TestFlight, and interested users can join the waitlist for testing through Jordi's survey.

Rodin Gen-1 - Experience Converting Images into 3D Models in Apple Vision Pro

Keywords: AI 3D Generation

Deemos Technology from ShanghaiTech University recently demonstrated converting web images into immersive 3D models using Rodin Gen-1 on Apple Vision Pro. In a video, they transformed an image from X's live stream into a downloadable GLB model for free manipulation and browsing on Apple Vision OS. Deemos revealed that Rodin Gen-1 also utilizes a model architecture similar to OpenAI's Sora, known as Diffusion Transformer, enhancing the model's scalability.

Article

The Assistive Features of Apple Vision Pro

Keywords: Accessibility, Apple Vision Pro

Despite being a first-generation product, Apple's focus on human care is evident in the comprehensive assistive features provided in Apple Vision Pro. These features aim to make the spatial computing device accessible to a diverse range of users. The article Vision Accessibility on Apple Vision Pro details the experiences of an author with nystagmus, explaining the various assistive functionalities on Apple Vision Pro, including Voice Over, Voice Control, Bigger Text Size, Window Zoom, and more.

Tips
As someone with nystagmus, Onee agrees with the author's perspective. Although 90% of scenarios are manageable, precision focusing on minute components remains a challenge due to involuntary eye movements. Fortunately, Apple Vision Pro offers solutions like "head tracking" and "wrist tracking" to address this issue.

Similarly, another YouTuber, James Rath, who is nearly blind and has nystagmus, shared how Apple Vision Pro's features have aided him, including clearer vision due to the proximity of the screen to the eyes and Voice Over assisting in scene recognition. For further insights into Apple's capabilities in assisting visually impaired users, Kristy Viers's video offers a look into how the Magnifier app enhances reading capabilities for blind users.

Tips for Cleaning Apple Vision Pro and Its Accessories

Keywords: Apple Vision Pro, Cleaning

Like all electronic devices, Apple Vision Pro requires regular cleaning. Combining the official guidelines with our experience, we've compiled key points to consider when cleaning Apple Vision Pro and its accessories.

Cleaning the front panel glass.
How to clean the display.
Cleaning the Curtain around the display.
Cleaning the Light Seal pad, headband, and Apple Vision Pro protective case.
Special considerations for cleaning the Light Seal.
Tips for cleaning the audio strap.
How to clean the polishing cloth.
General advice for cleaning other accessories.

Each point includes specific instructions and precautions to ensure effective and safe cleaning of the device and its components.

Code

GodotVision- Developing on Godot for visionOS!

Keywords: Godot, visionOS

Remember Godot, as mentioned in our 001 issue? Recently, kev has made some initial attempts on Godot, using the GodotVision project to run a simple HelloWorld project on visionOS.

While the basic path has been proven successful, this project is not yet officially supported and currently has limitations, including lack of documentation, absence of skeletal models, shaders, and particle systems.

Fortunately, kev has outlined a clear roadmap for the project's future development. If you're interested in Godot and visionOS development, it's highly recommended to clone this project from GitHub and explore its possibilities with the author.

Small News

Is the Return Rate of Apple Vision Pro Really That High?

Keywords: Apple Vision Pro, Return Policy

You might have seen news about a high number of returns for the Apple Vision Pro, but according to Mike Wuerthele in his article, the return rate isn't significantly higher than other new products. He gathered information from Apple Store employees, revealing that returns before 2.16 (the last day for first-batch buyers to return) were mainly from two types of users:

Youtubers, who often treat new product return policies as a free rental opportunity.
Users who experienced discomfort while wearing the device.

Contributors to This Issue

XR World Weekly 017

XRealityZone — Sat, 28 Sep 2024 01:48:12 GMT

This Issue's Cover

This issue's cover features a scene designed by Attiq R. using Spline - ZenTree.

BigNews

Apple Vision Pro pre-orders start
2024, March 30-31, Beijing, Let's visionOS!

Idea

Navi: Simultaneous translation on Apple Vision Pro
The "aging" memory keyboard
Lungy: Smooth enjoyment in Apple Vision Pro

Tool

SF Symbols 5 lets you happily use Vision Pro icons in visionOS

Article

Remodeling enterprise apps through Apple Vision Pro
Chinese tutorial series for Vision Pro development

Design

Spatial IKEA on Vision Pro

Code

Open visionOS: Animation and UI design
Climate Spiral in visionOS

SmallNews

Feature to modify the ceiling might be removed by Apple from Apple Vision Pro
If you get an Apple Vision Pro, don't forget to buy this accessory (but where to buy it? 🤔)
VRChat announces automatic generation of mobile avatars for PC players

BigNews

Apple Vision Pro pre-orders start

Keywords: Apple Vision Pro

Finally, the pre-orders for Apple Vision Pro have officially started on January 19. A lot has happened during these days, which we will detail in the article What happened when Apple Vision Pro pre-orders started?.

2024, March 30-31, Beijing, Let's visionOS!

Keywords: visionOS, Developers, Apple Vision Pro, SwiftGG, XReality.Zone

After seeing so much news about Apple Vision Pro, if you still have doubts about its future possibilities, or if you are very optimistic about its future but don't know how to start, either way, you can come to Beijing on March 30-31, 2024, to attend the Let's visionOS Conference co-planned by SwiftGG and XR Base.

The conference will share content including programming technology, product design, human-computer interaction, and commercialization, aimed at helping entrepreneurs quickly stand out in the visionOS track!

In addition, we are actively contacting many developer communities in Asia, such as South Korea's @Async Swift community, Australia's @Cocoaheads Austria, Singapore's @iOS Conf community, and Japan's XR developer community representative @AR_Ojisan. We are inviting them to China for exchanges with local developers, all to help Chinese developers expand their horizons and strengthen international exchanges.

Therefore, we look forward to everyone's participation, whether as attendees, sponsors, speakers, or volunteers. We sincerely welcome you to join us in welcoming and exploring this new field!

Idea

Navi: Simultaneous translation on Apple Vision Pro

Keywords: Apple Vision Pro, Translation

X netizen Apple Vision Commentary recently shared a video of a simultaneous translation software called Navi on Apple Vision Pro, but did not provide the specific source. However, considering Apple Vision Pro's design standards and the speed of language recognition, this does not seem like a "concept video" but rather a recording of actual application use.

The "Aging" Memory Keyboard

Keywords: Apple Vision Pro, Keyboard

This is a share from Dribbble designer Oleg Frolov (we previously shared his dice project on visionOS). Imagine, as mentioned in the official video, paying attention to using other sensory effects to compensate for the lack of some sensory experiences on the device (for example, due to the lack of a physical keyboard, designing typing to compensate for the lack of tactile sensation through sound effects).

Oleg's mention of "aging buttons" could be a new solution to "visual tactile": as shown in the animation below, the virtual keyboard keys add a "fingerprint" texture as the user's selection frequency increases, just like a real physical keyboard being pressed many times (although personally, I think there might be some data security risks in using it on keyboards, imagine the next person using the device to input, seeing your just-typed bank card password...), but it's not a bad idea as a "tactile compensation" method?

Project address and Repository address here, interested friends can try it (the animation display is relatively inconspicuous, it is recommended to check the project page).

Lungy: Smooth Enjoyment in Apple Vision Pro

Keywords: Apple Vision Pro, Lungy, Interactive Breathing

Lungy is an iOS breathing training app that helps you practice breathing exercises on your iPhone, combined with various immersive light effects. The app's various realistic particle effects also give a very decompressing feel.

What would such an app look like on Apple Vision Pro? The answer is, the entire space will help you enter the atmosphere of breathing exercises. You can not only slap the bubbles in the space:

But also strike the keys in the space:

And the particle effects and bubbles that could only be felt on the screen can now be experienced throughout the space:

Tool

SF Symbols 5 lets you happily use Vision Pro icons in visionOS

Keywords: SF Symbols, visionOS, iOS, Apple Vision Pro

As a company that values design, Apple's SF Symbols, launched in 2019, is definitely one of the important tools for creating tactile apps. With SF Symbols, developers can easily use up to 5000 icons.

In the latest SF Symbols 5, Apple has also updated many icons related to Apple Vision Pro to help with identification in Apple Vision Pro-related feature development:

For example, the spatial video features of the iPhone 15 Pro series use the visionpro.slash icon here.

Article

Remodeling enterprise apps through Apple Vision Pro

Keywords: Apple, Apple Vision Pro

Remodeling enterprise apps through Apple Vision Pro is a series launched by Apple about building and testing apps in visionOS, as well as the languages, tools, and frameworks needed for the work. Explore videos and resources that show efficiency and collaboration, simulation and training, and guidance in the workplace. It can serve as a simple roadmap for learning.

Vision Pro development series Chinese tutorial

Keywords: Apple Vision Pro, Development Tutorial

Author xChester recently updated a series of tutorials for Vision Pro development, including complete code examples.

Each article comes with illustrations and key code explanations, some with videos, overall progressing from easy to advanced, very suitable for developers to get started.

The current tutorial directory is as follows:

Windows, Spatial Containers, and Spaces
Window Position and Size
Tab Bar & Side Bar
Ornaments
Basic Video Playback
Object Movement
How to make objects always face the camera?
Shaders and Code Control
Controlling Car Movement with UI and Handles

Design

Spatial IKEA on Vision Pro

Keywords: IKEA, visionOS

Also from Dribbble designer Yanosh Govoshi, imagine IKEA on visionOS and how spatial computing devices might better help us beautify our living environment (more information awaits your unboxing and downloading of the official IKEA app). Project address here, welcome to Dribble for a look.

Code

Open visionOS: Animation and UI Design

Keywords: Animation, visionOS

Amos Gyamfi is proficient in SwiftUI animation and UI design. His previously open-sourced Open SwiftUI Animations project has gained over 3.2K stars on GitHub. He now brings similar designs to visionOS and combines them with 3D models. OpenvisionOS is such a project for animation and UI design suitable for visionOS. If you encounter UI and animation issues in visionOS development, consider referring to his designs.

Climate Spiral in visionOS

Keywords: visinoOS, ECS, RealityKit

VOClimateSpiral is a demo example using a climate spiral to display global warming on visionOS. It uses NASA GISS data to show climate change.

Technical points of RealityKit used:

Runtime geometry and material generation: Making a spiral 3D model from temperature data.
ECS-based time series animation: Playing annual and monthly temperature changes.
Particle system: Adding visual effects and presets for easy use.

SmallNews

Feature to modify the ceiling might be removed by Apple from Apple Vision Pro

Keywords: Apple Vision Pro

According to M1, Apple may remove the feature of adjusting the Digital Crown to replace the ceiling with the sky from the official version. In Apple's latest promotional video, the above scene has been removed, and the promotional copy has been modified:

If you get an Apple Vision Pro, don't forget to buy this accessory (but where to buy it? 🤔)

Keywords: Developers, visionOS, Xcode

After the release of Apple Vision Pro, as developers, seeing its packing list, we instantly had a big question:

This question is about the lack of a special USB cable needed by developers. In various developer videos at WWDC, if you want to debug Apple Vision Pro on a real device, you need such a cable:

Developers might be getting a bit nervous now — what? My Apple Vision Pro can't be developed? Don't worry, according to Apple's official documentation, the Apple Vision Pro might be able to connect and authenticate directly through a wireless network. Of course, how to specifically develop during the development stage, and what pitfalls might be encountered, we will also introduce in a separate article later.

VRChat announces automatic generation of mobile avatars for PC players

Keywords: VRChat, PC VR

According to reports, VRChat, a cross-terminal social software (including PC, Android mobile Google Play Store, VR headsets like Quest, Pico 4, etc.), has always faced a challenge: previously, if PC players wanted users on VR headsets and mobile terminals to see their content or avatar, they had to upload their optimized mobile terminal avatar separately. Otherwise, other players could only see a system-generated default avatar representing the PC player (officially called "fallback avatar").

Now VRChat officially announces the launch of an Imposter system, which can generate a corresponding, slightly lower quality mobile model for PC players who have not uploaded their own optimized mobile avatar (although users are still supported/encouraged to upload their own optimized mobile avatar model files).

The quality of the system-generated mobile model compared to the original model is shown below.

Contributors to This Issue

XR World Weekly 016

XRealityZone — Tue, 16 Jan 2024 01:47:00 GMT

This Issue's Cover

This issue's cover features the TVA Timedoor, designed by Max using Spline. Indeed, for Spline, creating such photorealistic, interactive 3D model scenes is no longer a challenge.

BigNews

Release date for Apple Vision Pro confirmed!

Idea

Seasons—Immersive Weather Experience
Doing chores with Quest, making chores no longer boring!

Tool

UEVR—Bringing VR support to Unreal Engine games in your Steam library
Rokid—Free export of spatial videos to SBS format
Genie—Luma.ai's Text-Generated 3D Tool

Video

Polyspatial Quick Start / Play to Device
Making a Cel / Toon Shader in Reality Composer Pro's Shader Graph

SmallNews

At CES, Sony launches new XR headset

BigNews

Release Date for Apple Vision Pro Confirmed!

Keywords: Apple Vision Pro

Long-awaited, Apple officially announced in this press release that the Apple Vision Pro will be officially on sale in the US on February 2, 2024, with pre-orders starting on Friday, January 19, at 5 AM Pacific Time (corresponding to 9 PM on Friday, January 19, Beijing time).

Not just the release date, from the official announcement (combined with some of our exclusive news), we can also glean previously undisclosed information, including:

The starting price of Apple Vision Pro is $3,499, equipped with 16 GB of memory and 256GB of storage space (versions with higher capacity will be available)
In terms of productivity, apps like Fantastical, Freeform, JigSpace, Microsoft 365, and Slack have been adapted specifically (as Apple's announcement wouldn't mention an app for no reason)
In gaming, Blackbox, Game Room, What the Golf?, and Super Fruit Ninja have been adapted or newly developed for spatial computing scenarios (Apple's official term for these games is Spatial Games)
In entertainment, content providers like Apple TV+, Disney+, and Max are offering HDR-supported video content, and in Apple TV, 150 3D movies are already prepared (as we mentioned in 012, there were 19 movies supporting 3D format on Apple TV as of November 2023); furthermore, Apple revealed a new format for 180-degree panoramic videos—Apple Immersive Video, a 3D format with a clarity of 8K covering 180 degrees (the official website even showcased the scenes of Apple shooting these Immersive Videos)

There are some more details in the announcement, such as this fleeting video from the visionOS App Store in the announcement, which gives us a sneak peek at some of the native visionOS apps that will soon be on the Vision Pro App Store homepage.

For example, the most prominent on the homepage is Disney+. You can watch Star Wars on a Star Destroyer, watch the NBA from a God's eye view, and maybe even become the Watcher Uatu, experiencing What if...

Another app that made an early appearance on the App Store homepage in the official introduction is Sky Guide — this AR software used for astrophotography and constellation observation will be a whole new experience on Vision Pro:

Other first-launch native visionOS apps include: the Apple Design Award-winning Blackbox (Blackbox has been entirely redesigned for the visionOS platform!), JigSpace showcased as part of the ARKit feature, Microsoft Office software (including Microsoft Word and Microsoft Excel), and a new Apple Arcade game, Game Rooms.

JigSpace and another app PTC, not featured in the App store showcase video, were chosen by Apple as exemplars of enterprise-level app design and development.

The video's finale showcases native apps like Zoom, newly launched Onshape and Super Fruit Ninja, and of course, djay and Complete HeartX, which were showcased on the first day of Vision Pro's announcement at Keynote.

djay, in particular, frequently appears in the official Developer News, making it a highly anticipated app!

The latest report on how djay achieved their vision was published on January 11, 2024 here.

Also of note, though briefly glimpsed and easily overlooked in the Keynote, is the anticipation for what will be presented in visionOS's Keynote. Previous media reports also covered what Keynote will look like on visionOS, including features such as simulating presentations in spatial environments.

Based on our summary of various sources, the current Apple Vision Pro package contents are (prescription lenses are an additional purchase):

The specific contents of the package and all Chinese translations of terms are subject to the final release by Apple

In addition, in anticipation of the Apple Vision Pro, a revolutionary product, Apple officially released an advertisement: Get Ready. In this ad, Apple featured scenes from various classic movies where different characters put on blindfolds, as if to say, wearing the Apple Vision Pro, you too can become "Some One". From a user's perspective, this indeed might change the minds of many who think "wearing a head cover is silly."

It's worth noting that 17 years ago, when the iPhone was first released, Apple used a similar filming technique for an advertisement. In this ad, scenes from various classic movies were featured, with different characters picking up the phone and saying "Hello":

It seems Apple is fully comparing the Apple Vision Pro with the iPhone. Additionally, if you're a developer in the Apple ecosystem, you've likely received an email from Apple pushing the latest guide on how to submit apps to the visionOS app store here. With the Apple Vision Pro's release date set, Xcode has officially added support for visionOS in version 15.2 (yes, one less barrier to starting to learn visionOS).

One last thing to remember, if you plan to purchase the Apple Vision Pro, make sure to download the Apple Store app on a device equipped with Face ID in advance, as facial scanning is required before purchase to determine the size of the headband. And according to Mark Gurman's report, we can reasonably guess that in the future, the iPhone might also be needed to calibrate some Vision Pro accessories.

Tips
If you haven't yet experienced the magic of visionOS apps, you're welcome to try them out in a simulator using the Let's xrOS project.
Let's xrOS is a platform for exchanging visionOS apps, allowing developers to distribute their visionOS apps to enthusiasts around the world. Rest assured, the distribution mechanism is as secure as Apple's TestFlight, only requiring the app's binary, no source code required.

Idea

Seasons—Immersive Weather Experience

Keywords: visionOS, Apple Vision Pro, WeatherKit

If you're lucky enough to get an Apple Vision Pro after its release, be sure to visit the visionOS AppStore to try this app called Seasons.

This app is a prime example of Apple's convenient developer tools. By leveraging Apple's WeatherKit combined with SwiftUI + Reality Composer Pro, Seasons successfully displays 34 types of weather through immersive scenes. In Seasons, we can see:

Cloudy days with drifting leaves:

Drizzly rainy days:

And of course, snowy days with falling snowflakes:

It's noteworthy that this app was developed by a Chinese developer, Cosmo, and his team from Shenzhen. We're very excited to see more great works from Chinese developers in the world of spatial computing.

Doing Chores with Quest, Making Chores No Longer Boring!

Keywords: Quest Hackathon Winner, MR, Quest

What if doing chores felt like playing a game? Would you start enjoying household tasks?

The following product concept comes from a four-person team at the Meta Quest Hackathon. By utilizing Meta's Presence Platform, they transformed the mundane act of vacuuming into an exciting "weed-clearing" experience.

When spatial applications move beyond simply displaying 3D models to integrating more with real-world scenarios, they create wonderful chemical reactions.

Tool

UEVR—Bringing VR Support to Unreal Engine Games in Your Steam Library

Keywords: Steam, VR, UEVR, Unreal

UEVR, developed by praydog, is a mod for Unreal Engine games. By installing UEVR, you can convert games in your Steam library, developed with Unreal Engine (versions 4.8-5.3), into VR games playable with various VR headsets.

As a mod rather than a framework requiring developer integration, UEVR is incredibly user-friendly. The basic usage involves:

Opening UEVR on your PC.
Launching the game you want to inject.
Configuring the injection parameters, including the target game process and selecting the runtime framework (OpenVR/OpenXR).
Starting the game!

The actual gameplay experience varies with the game. For example, in the game illWill, you can play in first-person view, and use VR headset controllers to operate weapons realistically:

In other games, you might only be able to play in third-person view with a traditional game controller:

Currently, the list of games supported by UEVR is continually expanding. If you want to experience your favorite games in VR, you can check the official spreadsheet to see if they're included. Additionally, you can learn more about getting started with UEVR through these two videos: UEVR - Getting Started Basics and UEVR Mod: An Idiots Guide.

Rokid—Free Export of Spatial Videos to SBS Format

Keywords: Rokid, Spatial Video

In our previous post How to Play Spatial Video on iOS 17.2, we recommended Halocline and Spatialify for converting spatial videos into SBS format for viewing on devices like PICO/Quest. Considering the dedication behind the Spatialify app, its 18 Yuan price tag is quite reasonable, especially with the latest version supporting additional features like spatial photo capture!

However, if you just want to simply export an SBS format video, the Rokid AR App is also a great option.

In the latest 2.0.8 version, Rokid has updated the app to include a feature for exporting spatial videos, with a basic operation logic similar to Spatialify. Just select the spatial video and export it:

Genie—Luma.ai's Text-Generated 3D Tool

Keywords: AI, Luma.ai, Text to 3D

Luma.ai, frequently mentioned in our previous issues, has recently launched their Text to 3D tool—Genie. Our trials found that the tool generally generates models quite quickly, taking about half a minute to create a basic usable model. For example, we entered “A cute cat”, and in about half a minute, a (somewhat rough) cat model was generated.

It's worth mentioning that besides Luma.ai's Genie, another startup company, tripo3d.ai, has also launched a similar rapid 3D model generation tool, providing an API for enthusiasts to experiment with. For instance, a netizen created an online PK game, where the winner of the PK takes the loser's model keyword, merges prompts using GPT, and regenerates (evolves) into a new model using tripo3d.ai. Here, a cat successfully PKed a little girl and evolved into a cool Catwoman.

Those who follow related technology may notice that compared to CSM.ai introduced in Issue 012, the field has evolved from minute or hour-long generations to generating four sets in just tens of seconds over just a few months.

Further analysis shows that early methods, like zero123 and related technologies, used a combination of 2D diffusion models and NeRF reconstruction, mainly limited by the reconstruction speed of NeRF itself. In contrast, Luma.ai and tripo3d.ai trained a large Transformer-based reconstruction model (like LRM or TriplaneGaussian shown above), directly inferring the final NeRF or 3D Gaussians reconstruction result from 2D images.

Video

Polyspatial Quick Start / Play to Device

Keywords: Unity, visionOS, Polyspatial

If you're eager to quickly learn how to develop visionOS apps using Unity with Polyspatial, these two videos by Dilmer Valecillos are highly recommended for beginners:

The first video primarily introduces how to incorporate the Polyspatial SDK into a project and debug in visionOS. The second video mainly explains how to use Play to Device, a method that allows for real-time preview of effects in the Unity Editor upon saving.

Making a Cel / Toon Shader in Reality Composer Pro's Shader Graph

Keywords: visionOS, RealityKit, Reality Composer Pro, Shader

In this video, Mark Robinson explains how to use Reality Composer Pro to build a Cel/Toon Shader. The end result is the effect seen on the spheres, skull, and airplane in the image above.

It's important to note that this tutorial isn't aimed at complete beginners. Therefore, before watching this video, you should have some understanding of Reality Composer Pro's Shader Graph (or a basic understanding of Unity's Shader Graph).

SmallNews

CES - Sony Launches New XR Headset

Keywords: CES, Sony

At the recent CES, Siemens CEO Roland Busch announced that they will be launching a new XR headset later in 2024, in collaboration with Sony. This headset will feature a 4K OLED screen and a Qualcomm XR2+ Gen 2 chip. It's primarily aimed at professional engineers, meaning it's designed for those who can use CAD software, rather than for general consumers interested in movies and gaming.

The headset features a flip-up design for easy on and off.

It also comes with a ring and a controller to help engineers position 3D models in the real world.

And make fine adjustments to the models:

Siemens has also released a promotional video for this device, which can be viewed here.

Contributors to This Issue

Zion

ybbbbt

XR World Weekly 015

XRealityZone — Mon, 08 Jan 2024 01:45:00 GMT

This Issue's Cover

For this issue's cover, we share a concept design from Ben Artis titled Apple Maps for visionOS. It reminds us that the Map app in the official version of visionOS is still just a Window-style compatible version, making Ben's design concept quite avant-garde and trendy.

In addition to the cover, Ben has also designed a series of product promotion pages, very much in the Apple style. Interested friends can click the link for more details.

BigNews

Spline released multiple major updates related to the Apple platform
Meta Quest SDK welcomes a wave of updates

Idea

MR smart home control app "Immersive Home(BETA)" arrives on App Lab and Side Quest
An aquarium on the wall!
Neko Neko at home, petting cats!

Video

Unity6 New Features Introduction Series

Article

In-depth analysis of the spatial computing industry: related devices, key elements, industry chain, and relevant companies

Code

Custom rendering with Metal on visionOS
Spatial Metal: Custom rendering on visionOS using Metal and Compositing Services
Third-party keyboards on visionOS: Vision Keyboard Kit
Hand tracking debugging on the visionOS simulator

SmallNews

Ray-Ban Meta smart glasses + multimodal AI, a true smart life assistant

BigNews

Spline releases multiple major updates related to the Apple platform

Keywords: Spline, visionOS, SwiftUI

Hey, friends, do you remember Spline, which we started recommending to you from Issue 001? Just recently, the Spline team released an update video revealing some fun stuff they're about to launch during the Spline Online Event held on December 19th:

Particles
3D embed for Apple Platforms
Generate App
UI Scenes

Let's take a closer look at these fun features.

First up, the particle system. Now, in Spline, we can finally use particle systems to create more "dynamic" scenes:

The particle system is just the beginning. The next three updates are quite revolutionary.

First, 3D embed for Apple Platforms. Simply put, the Spline team has used Metal — Apple's flagship graphics framework — to achieve nearly seamless 3D scene embedding on Apple platforms. With Spline's SplineRuntime, developers can embed 3D scenes in their apps with simple code like this, using Spline's exported splineswift files:

import SplineRuntime
import SwiftUI

struct ContentView: View {
        var body: some View {
                // // fetching from local
                // let url = Bundle.main.url(forResource: "scene", withExtension: "splineswift")!

                // fetching from cloud
                let url = URL(string: "https://build.spline.design/8DX7ysrAJ3oDSok9hNgs/scene.splineswift")!

                try? SplineView(sceneFileURL: url).ignoresSafeArea(.all)
        }
}

If your app has a need for a 3D scene like the one below, Spline's new feature is undoubtedly a great help:

Moreover, leveraging the event system Spline has been updating, you can even create a game that runs on iPhone/iPad directly using Spline:

Currently, this feature is still in Beta, and support for both the Apple platform and Spline's own functions is not yet complete, but the potential for the future is definitely exciting!

Besides, if you're not familiar with Apple platform development but know Spline's features well, you can use Spline's newly launched Generate App feature to quickly generate an Xcode project. Then, knowing the basic process of submitting an app, you can launch an app on Apple's App Store.

Tips
Similarly, this feature is currently in Beta, and you also need to subscribe to Spline Super to use it.

The first two features are aimed at helping modelers collaborate better with Apple platform developers. The next feature is intended to assist spatial computing scene interaction designers in prototyping more effectively. Now, in Spline, you can use UI Scenes to design 2D views within Spline:

And embed these flat UI Scenes into 3D scenes in a manner similar to Windows in visionOS:

Thanks to the event system Spline has built, button interactions in UI Scenes can seamlessly interact with the entire 3D scene. For example, you can create effects like the entire scene's light and shadow transformation by using "Get Started" on the UI Scene:

Tips
Like the previous two features, UI Scenes is also in an early stage (still Early Preview), and if you encounter any difficulties using it, the Spline Team is very welcoming of all kinds of feedback.

Meta Quest SDK welcomes a wave of updates

Keywords: Meta, Quest, Avatar SDK, MR Utility Kit, Haptics SDK, Passthrough Relighting (PTRL)

Recently, with the release of the Quest v60.0 version, Meta updated the Meta XR Haptics SDK, Meta Avatars SDK, and officially launched the MR Utility Kit (which we mentioned in Detailing Meta Quest 3's MR Technology Upgrades, and Meta promised to release at Meta Connect 2023).

Tips
For these three SDK updates, Meta also published three blog posts for introduction:
v24 of the Meta Avatars SDK is now publicly available
Introducing MR Utility Kit: Build Spatially-Aware Apps Faster
Haptics Public Release - Enhance Your Immersive Experiences on Meta Quest

In the Meta Avatars SDK v24 version, Meta mainly uploaded it to the Unity Asset Store and made some performance optimizations, making the default appearance of Avatars look a bit more realistic. For example, in the comparison below, you can see that Avatars' hair, skin, and clothing will have more natural light reflection processing.

The release of the new MR Utility Kit can help developers build an MR App more quickly. As the name of this SDK (Utility Kit) suggests, it is not a core SDK providing complex functions, but a collection of resources, utility functions, and debugging tools. Many of these tools are built on the Scene API. Currently, the basic features included in the MR Utility Kit are:

Scene queries
Graphical Helpers
Development Tools
Passthrough Relighting (PTRL)

Let's discuss these features one by one.

Scene queries allow for easy checking of a scene's location and its relationship to the real world. For instance, we can quickly check whether a position is inside the room using the IsPositionInSceneVolume API.

Graphical Helpers are aids for graphical imaging. For example, using the Effect Mesh component provided by the SDK, we can easily add textures to real-world planes like walls, ceilings, and floors.

Development Tools are pretty self-explanatory, providing various developer tools for easier debugging. The SDK mainly provides Scene Debugger and Room Prefabs. The former helps developers visually inspect the previously mentioned Scene Query, while the latter offers a room prefab, facilitating rapid development in a room setting:

Regarding Passthrough Relighting (PTRL), we're keeping some details under wraps. Remember the realistic light reflection effect video player we mentioned in Issue 011, shared by a Redditor?

At that time, we noted that the effect was developer-implemented, lacking a low-cost solution. Now, the PTRL feature provided by MR Utility Kit is designed to help developers quickly add shadows:

And lighting to objects:

Finally, the Haptics SDK has moved out of its Beta phase, which began in March, to officially enter version 1.0. Paired with the Mac, Windows, and Quest Meta Haptics Studio apps, developers can use audio files to tweak the vibration waveform of the controllers, preview it on the Quest app, and export it in the .haptic format recognized by the Unity/Unreal SDKs.

Additionally, the official version of the Studio now includes several tutorials for beginners, helping new developers get up to speed quickly.

Idea

MR Smart Home Control App "Immersive Home(BETA)" Arrives on App Lab and Side Quest

Keywords: MR, smart home

We recently came across several interesting apps on Side Quest and would like to share them with you.

Let's start with a practical one. The concept of smart homes has been touted by major platforms for many years, and there have been some demos combining smart homes with XR. For instance, in our last issue, we introduced a smart home control interface designed by a designer for Vision Pro. Numerous other concept videos also abound. Now, we can see a mature MR application, "Immersive Home(BETA)," on Side Quest and Quest App Lab.

This smart home app by Nils Twelker utilizes Quest 3's MR capabilities for a fascinating application: users wearing Quest 3 can view power consumption by gazing at or pointing to smart home devices and toggle lights with a wave of the hand. Alternatively, a virtual control panel can be opened to operate dozens of smart home devices. The developer notes that while it runs better on Quest 3, Quest 2 is also compatible.

Looking at the official website's plans, in addition to the current support for gesture-based power viewing and light switching, the developer hopes to add features like automation control, voice interaction, whole-house power consumption, and weather display in the future.

It's worth noting that the connected home products need to be compatible with Home Assistant before pairing with the app. More information is available on the app's official website, and the project repository address is also provided here.

An Aquarium on the Wall!

Keywords: MR, aquarium

The next app is from one of the earliest modern VR experiences — Ocean Rift.

Their recent update supports the latest MR mixed reality mode for the Quest series. For example, Quest 2 users can use their fingers to draw a square area on their room wall to display their favorite fish species, as if turning their home into an aquarium. Different lighting effects can also be added to the environment. On Quest 3, the app can intelligently recognize a user's previously recorded room environment and walls, turning the selected wall edge into an aquarium display glass. If set on the ceiling, it can even simulate the lighting effect seen from underwater. Head over to Side Quest / Quest App Store to bring the aquarium home.

Neko Neko at Home, Petting Cats!

Keywords: MR, cats, Neko Atsume

The next app comes from developer Hit-Point, who previously developed a mobile game called Neko Atsume: Kitty Collector. This game allows you to collect different breeds of cartoon cats and interact with them. The iOS version is available here, and the Android version can be downloaded here.

Now, this adorable cat petting and collecting game is available on Meta Quest! Combined with Quest 3's see-through capabilities, you can easily bring cats into your home, collect different breeds, and play games with them using items like catnip. More importantly, you can pick up the cats with your hands (especially with the recently released Haptic SDK, the feeling must be great). The interaction is expected to be much stronger than the mobile version. Interested friends can get it here on Quest Store.

Video

Unity6 New Features Introduction Series

The Unity6 New Features Series introduces a series of new features in Unity6. Regarding support for the USD format, it is no longer recommended to use the old version of USD Packages, but instead the New OpenUSD Packages, supporting USD version v23.02. It provides import and export of scenes and animations, meeting the needs of collaborative creation of large scenes and offers a powerful compositing engine capable of assembling new scenes losslessly from numerous resources.

The new toolkit includes USD Importer, USD Exporter, and USD core:

USD Importer: A tool for importing USD files into Unity;
USD Exporter: A tool for exporting from Unity to USD files;
USD core: A C# wrapper based on the open-source USD engine.

Article

In-depth Analysis of the Spatial Computing Industry: Related Devices, Key Elements, Industry Chain, and Relevant Companies

Keywords: Spatial computing, Apple Vision Pro, Meta

The In-depth Report on the Spatial Computing Industry is a compilation by Hui Bo, referencing multiple industry reports. It covers a wide range of content concerning the spatial computing and XR industry's current landscape and future development, offering high value for understanding these fields.

The report starts with What is Spatial Computing and What Role Does Spatial Computing Play, suggesting that computational dimensionality is a disruptive trend of the era. It then analyzes key technology points, device industry chains, and relevant companies, finally discussing current factions and main tracks and making future predictions: Large models × spatial computing platforms, expecting frequent hits in 24 and 25.

The full article is lengthy, so here we've summarized its content, covering several aspects:

Overview of Spatial Computing
The Disruptive and Inevitable Nature of Spatial Computing
Key Elements of Spatial Computing Platforms
Analysis of the Spatial Computing Device Industry Chain
Related Companies
Analysis of Factions and Main Tracks
Future Outlook

Code

Custom Rendering with Metal on visionOS

Keywords: Metal, visionOS, Compositor Service

Metal Spatial Rendering is a sample program that uses Metal, ARKit, and visionOS Compositing Services for custom rendering. At WWDC2023's session 10089, Apple unveiled the Compositor Services framework and demonstrated basic usage. It allows developers to implement fully custom 3D rendering on visionOS using Metal cpp, enabling third-party rendering engines to integrate with visionOS.

Fully immersive scene: A fully immersive scenario, akin to VR's fully virtual mode, can present scene objects like hands, tables, and floors, but lacks video passthrough capabilities.

However, Apple only provided partial key code and explanations, not a complete sample. This project, by metal-by-example, is a complete sample code based on Apple's public information. It implements basic lighting rendering and model loading, serving as a valuable reference for developers needing to integrate custom rendering into visionOS.

Spatial Metal: Custom Rendering on visionOS Using Metal and Compositor Services

Keywords: Metal, Compositor Services, visionOS

Similar to the Metal Spatial Rendering project, SpatialMetal and SpatialMetal2 are also sample programs for custom rendering on visionOS using Metal and Compositor Services. They render Earth models and cosmic starry skies, representing foreground and background, respectively. The difference between SpatialMetal and SpatialMetal2 lies in SpatialMetal2 implementing multi-uniform buffers for binocular vision.

Third-Party Keyboards on visionOS: Vision Keyboard Kit

Keywords: visionOS, Third Party Keyboard

VisionKeyboardKit is a keyboard component for visionOS, supporting full-size 104 keys and 60% mode without a numeric keypad.

However, since the visionOS simulator currently does not support installing other input method keyboards, it cannot provide system-level keyboard support and can only be embedded in one's app as a regular window. The usage is similar to the encrypted keyboards found in banking apps.

Hand Tracking Debugging on the visionOS Simulator

Keywords: visionOS, Hand Tracking, Simulator

VisionOS-SimHands is a project for debugging hand tracking features on the visionOS simulator, utilizing WebCam, Bonjour, and Google MediaPipes. This project is divided into two parts:

A macOS app that uses the Google MediaPipe framework and a Mac camera to perform three-dimensional hand tracking, starting a Bonjour sharing service to transmit gesture information.
A visionOS project running in the simulator, which starts a Bonjour service to receive gesture information and display it in visionOS.

It's important to note that the number of hand joint points recognized by Google MediaPipe differs from those in visionOS. Not only are the names different, but there are also fewer palm joints, one less forearm joint, and one wrist root joint.

This innovative project can significantly facilitate hand tracking code debugging on the visionOS simulator, effectively reducing development costs.

SmallNews

Ray-Ban Meta Smart Glasses + Multimodal AI, a True Smart Life Assistant

Keywords: Meta, Ray-Ban, Smart Glasses, Multimodal AI

In September this year, Meta collaborated with Ray-Ban to launch smart glasses at the Connect Conference. According to reports, Meta is currently testing more advanced artificial intelligence features on these glasses and randomly pushing them to some users in the United States to participate in early testing of Meta AI's image recognition function on the glasses.

Meta's CTO Andrew Bosworth revealed that this new AI feature is expected to be available to all users of the smart glasses next year. This feature allows the glasses to capture first-person perspective photos and use Meta's multimodal AI technology to interact with the photos in a Q&A format, providing users with more fun experiences or assistance. For instance, you could ask the glasses how much longer to grill for medium-rare steak, or translate a restaurant's sign or menu in real-time to help you order food while traveling abroad.

In the following scene, for example, you could use the Ray-Ban glasses to attempt translating the poster's photo:

Contributors to This Issue

XR World Weekly 014

XRealityZone — Sun, 24 Dec 2023 01:44:00 GMT

This Issue's Cover

This issue's cover features Yehor Haiduk 🇺🇦's creation Smart Home Vision Pro OS UI. Compare it with the compact list style of smart home applications commonly found on mobile phones. Doesn't it feel like in spatial computing, not only developers but also designers are given more freedom due to the "unlimited canvas"?

BigNews

AOUSD announced its roadmap for the next two years along with new partnership members.
Official release of iOS 17.2: Start recording your own three-dimensional memories from now on.
MLX — Another big move by Apple AI.
Meta introduces Relightable Gaussian Codec Avatars to present real-time relit head models in VR.

Ideas

AI + XR = A Door to Anywhere?
Unmissable apps on Quest.
Using Passthrough AR to meet from afar.

Tools

Evolution of the Gaussian Splatting Web app: Special effects & WebXR.
Generating 3D models from 2D sketches.
Cognitive3D — A user analytics tool in XR.

Video & Code

SwiftUI + Metal - Create special effects by building your own shaders.

SmallNews

Wow! These gesture operations have been realized!
Xbox Cloud Gaming (beta) lands on Quest 3.

BigNews

AOUSD Announces Its Two-Year Roadmap and New Members

Keywords: AOUSD, Unity, Meta, Khronos Group

In Issue 006, we introduced the establishment of AOUSD and its potential impact.

Four months have flown by, and AOUSD has released two major pieces of news: a two-year roadmap and new members including Cesium, Chaos, Epic Games, Foundry, Hexagon, IKEA, Lowe's, Meta, OTOY, SideFX, Spatial, and Unity.

These new members, who are leaders in their respective fields such as film, gaming, retail, and research, all have a demand for 3D models.

AOUSD also announced its collaboration with Khronos Group, the organization behind glTF, to ensure unnecessary compatibility issues between glTF and USD do not arise.

The roadmap details AOUSD's plan to divide its standards into three layers: Computed Data, Composed Data, and Foundation layer, with the outer layers stably utilizing the capabilities of the inner layers:

Over the next two years, AOUSD will focus on the Foundation layer, planning to release its first draft in Q1 of 2024, revise it in Q4 of 2024, and finally publish the final draft in Q3 of 2025.

If all goes well, by Q4 of 2025, our Newsletter should reach 60+ issues, and we'll review AOUSD's accomplishments then (Hey! Is it really good to dig such a big hole for ourselves so early? 🥴)

Official Release of iOS 17.2: Start Recording Your Own Three-Dimensional Memories

Keywords: iOS 17.2, Spatial Video, Apple Vision Pro

With the official release of iOS 17.2, users who previously purchased the iPhone 15 Pro series can finally try out this exclusive feature. To help everyone get started with spatial video, we detailed in How to Play Spatial Video on iOS 17.2 the current ways to enjoy spatial video.

MLX — Another Big Move by Apple AI

Keywords: Deep Learning Framework, Apple Silicon, GPU

Deep learning enthusiasts are surely familiar with Pytorch, but Apple has recently launched its own deep learning framework: MLX! It's not just another framework for developers to choose

from, but a game-changing tool for the growing GPU performance of Macs!

Left image above: Training Graph Conv Nets, right image: Real-time performance comparison with Whisper

However, one might wonder why Apple introduced MLX instead of implementing it in Pytorch.

Apple engineers explained that first, it fully utilizes Mac features (like unified memory, which eliminates the need to consider whether arrays are created on CPU or GPU); retains the best parts of other frameworks; and the overall framework is simple, flexible, and hackable.

We agree with their statement, especially in specific domains where GPU memory is often a bottleneck for many algorithms. Mac's unified memory design can easily surpass NVIDIA H100's 80 GB. (Although Mac's memory is as expensive as gold, it's still more reasonable compared to NVIDIA.) Moreover, official model examples have been provided.

LLM models: Meta's LLaMA with 7B to 70B parameters, Mistral with 7B parameters, Microsoft's Phi-2 with 2.7B parameters.
Voice recognition models: OpenAI's Whisper.
Fine-tune models: LoRA.

If you wish to run models on mobile devices (like iPad and Apple Vision Pro equipped with Apple Silicon), you can refer to:

Additionally, asitop is recommended for real-time monitoring of CPU, GPU, and memory performance on Apple Silicon. However, there's a small bug regarding monitoring on M2 and M3 series chips, so stay updated with the latest progress.

Meta Introduces Relightable Gaussian Codec Avatars for Real-Time VR Head Models

Keywords: Head Avatar, Relighting, Gaussian Splatting, Codec Avatar

To enhance user immersion when wearing MR headsets, whether it's Apple Vision Pro's Persona or Meta's Codec Avatar, they both aim to capture users' expressions in real-time through device cameras to drive a realistic head model. Persona has achieved real-time driving and display on the headset, while Codec Avatar, though more detailed in reconstruction, requires substantial computational power.

A few days ago, Meta announced a new work, using Gaussian Splatting (GSplats) to accelerate the most time-consuming body rendering step in Codec Avatar, ultimately achieving real-time presentation of high-precision head models in VR.

Unlike general GSplats, the human head model can be standardized (facial features are similarly positioned). Thus, Meta's work parameterizes GSplats and radiative transfer functions on a standardized head UV map (the flattened head parameterization UV map shown above), combined with GSplats and precomputed acceleration rendering processes, to achieve relightable realistic head rendering.

However, despite the acceleration with Gaussian Splatting, there are still many challenges to overcome. For instance, head reconstruction data requires capture with hundreds of synchronized cameras and controllable LED environments, and real-time presentation in VR usually needs PC-level GPU streaming.

Ideas

AI + XR = A Door to Anywhere?

Keywords: Quest, MR, MoonVR, Skybox

Anyone who has seen Doraemon will have a lasting impression of the "Anywhere Door" from the manga - a door that leads to various different worlds— step through it, and you can arrive in a multitude of different places.

When writing the topic of SkyboxLab in Newsletter 003, we were thinking, can we combine this technology with the XR world?

Now, MoonVR brings us such a practical product — AI Skybox Generator.

It's an app in Quest that uses AI prompts to generate Skyboxes in real-time. In addition to creating and inputting your own prompts, you can choose your favorite sky walls from the rich creations of other users.

Thanks to the Quest v54 update's Custom Skybox feature, you can also download these beautiful Skyboxes to your Quest and use them as the background for your control room.

It's worth mentioning that the MoonVR team is actively working on porting this app to Apple Vision Pro. You can see their current progress on Reddit.

Unmissable Apps on Quest

Keywords: Quest, Official Demos, Free Apps

With the Black Friday sales and online user numbers, more and more people are buying Quest devices. However, you might find that some interesting apps and games are either expensive (and you have to buy them again even if you've bought them on another platform) or hard to find. To prevent your Quest from collecting dust too quickly, here are a few official apps we recommend. Although they have relatively simple processes, they are well-made and worth experiencing as each app showcases a new feature. Don't miss out!

Oculus First Contact

Description: An official creation filled with toy boxes and a robot guide in a small house, where you can make a new friend and play together. It doesn't support hand gestures, but the interaction design of the items is creative.

Compatible Devices: All Quest devices, no internet connection required during use.

Interaction SDK Sample

Description: Mainly for developers to reference the Interaction SDK sample effects. The SDK allows developers to build high-quality interactions in VR or MR more simply, quickly, and easily. This app provides a quick overview of the SDK's support for interactions such as poking, hand grabbing, touch grabbing, ray, distance grabbing, and post recognition.

Compatible Devices: All Quest devices, supports hand gestures, no internet connection required during use.

First Hand

Description: The first official hand gesture interaction demo after the release of the Quest Interaction SDK. There are four tasks, from making a robotic glove and playing a shooting game to interactions, teleportation, and decryption. In this demo, you can truly feel the charm of hand gesture interaction.

Compatible Devices: Quest 2 and later devices, supports hand gestures, no internet connection required during use.

The World Beyond

Description: Interact with Oppy using voice and gestures. Make Oppy jump, come closer, or just say hello. Use a magic flashlight to find energy orbs in your room and feed them to Oppy so she can go home. Although it lasts only 5 minutes, it's a magical MR experience!

Compatible Devices: All Quest devices, supports hand gestures, supports Passthrough, requires an internet connection.

Maestro: The Masterclass

Description: Don't miss out if you love rhythm games! Conduct an orchestra to play epic symphonies in an opera house. (Holding a pen or a chopstick can unlock the advanced conductor experience, haha.)

Compatible Devices: All Quest devices, supports hand gestures, no internet connection needed.

First Encounters

Description: Finally, a MR game that's been talked about a lot. The main content is to help a group of cute creatures that have crash-landed in your room to get home. Shoot to help them get into the lifeboat within a limited time. Missed bullets will even break the walls of your room, truly an interesting blend of virtual and real gameplay. If you visit Meta's offline stores in Japan, you'll see that this game is one of the MR games recommended by the staff.

Compatible Devices: Quest 3, supports Passthrough, no internet connection needed.

Using Passthrough AR to Meet from Afar

Keywords: Passthrough AR, VRChat

Twitter user ShodahVR used Quest3's Passthrough AR feature to create an interesting display, bringing a distant friend into their world. In implementation, ShodahVR first activated Virtual Desktop's passthrough feature, then entered a world named greenscreen in VRChat, achieving the effect of overlaying the virtual world onto the real one.

Tools

Evolution of the Gaussian Splatting Web App: Special Effects & WebXR

Keywords: Gaussian Splatting, WebGL, WebXR

LumaAI recently released the Luma WebGL Library, a framework that integrates the Gaussian Splatting rendering framework with three.js rendering pipeline, making it convenient to share various Luma splats reconstruction results on the web and add interesting effects.

For example, adding a fog effect to a reconstructed Hollywood scene.

Or rendering Luma splats scenes into cubemaps and using them as the HDRI ambient light of a three.js scene (the high-reflective spaceship in the scene can reflect the outline of the snow-capped mountains).

We can even use custom Shaders to hook the vertex and fragment shaders of Luma splats, creating unique motion effects (like making the entire mountain range roll like a sine wave).

It's worth noting that Luma splats also support rendering on WebXR, meaning we can experience three-dimensional Gaussian Splatting on Quest, PICO, Vision Pro, and even the Leia holographic display.

For those who don't have these devices, we can immerse ourselves in Luma splats previews in Safari on the Vision Pro simulator.

Note: You need to enable WebXR-related features in Settings > Apps > Safari, under Advanced at the bottom of Feature Flags.

Generating 3D Models from 2D Sketches

Keywords: Sketch-to-3D, AI, 3D Assets Generation

Do you remember in Newsletters Issue 12 when we introduced CSM.ai's explosion of single-image 3D model generation and real-time AI painting tools? Now, CSM.ai has combined these two functions, allowing users to directly create objects with clean backgrounds from 2D sketches and text descriptions on the web, and then generate 3D models.

Cognitive3D—A User Analytics Tool in XR

Keywords: 3D Analytics, Unity, Cognitive3D

In mobile applications, many products use tools like Google Mobile App Analytics to analyze user behavior, aiding in the better iteration of their apps.

So, how should the same be done in VR/AR?

Here, we introduce the star of this segment — Cognitive3D, a 3D analytics platform that measures user behavior in VR/AR. It's compatible for integration in Unity/Unreal/C++ platforms.

Integrating and using Cognitive3D is very straightforward. Taking a Unity project as an example, after importing the necessary Unity Package and entering the required Developer Key, you just need to drag and drop the Camera and Left/Right Controllers from the Hierarchy Window into Cognitive3D's Setup Window, then upload the project's Geometry, completing the basic setup.

After that, when users play in the VR device, all their gameplay will be recorded and uploaded to Cognitive3D's backend, and can be displayed as a replay of all user actions (seeing such a complete scene, you can understand why we needed to upload Geometry earlier😉):

Besides visualization, Cognitive3D also possesses capabilities like event reporting and other common analytics platform features. You can even set up custom combinations of user behaviors (referred to as an Objective in Cognitive3D) and directly view on the web whether users have achieved the set goals.

If you want a quick and intuitive understanding of Cognitive3D, you can quickly check out Valem Tutorials' this video. For more detailed usage, you can refer to Cognitive3D's official documentation.

Video & Code

SwiftUI + Metal - Creating Special Effects with Your Own Shaders

Keywords: SwiftUI, Metal, Shader
This topic is a community contribution, thanks to XR Chestnut for submitting this topic

Shaders, the backbone behind all kinds of magical effects in the gaming world, might seem unfamiliar to many client-side developers.

However, this might start to change with the emergence of SwiftUI and Metal. In Paul's this video, he introduces us step-by-step on how to create basic shaders in Metal and use them in SwiftUI. At the end of the video, he also recommends The Book Of Shaders and ShaderToy, with the former explaining Shaders in a more systematic and in-depth manner, and the latter providing beautiful Shader effects and source codes.

Besides this video, Paul also kindly launched the Inferno project, which gathers various Shaders designed for SwiftUI. It's recommended to use in conjunction with this video.

Small News

Wow! These Gesture Operations Have Been Realized!

Keywords: Gesture Operations, Gesture Screenshot, Gesture Pause/Play

As early as in Newsletter Issue 8, we envisioned implementing a gesture screenshot operation in Vision Pro. According to user Luna's sharing, this operation is now realized in Horizon!

At the same time, the air gesture operation, previously seen in Huawei, is now supported by Readdle! In the app, while playing a video, you can use gestures for operations like pausing or continuing to play the video by moving your hand from far to close, or fast-forwarding the video by moving your hand left or right. All these gesture recognitions are done on the device. We tried it out, and the gestures for pausing or continuing to play the video are highly sensitive, while the gesture for fast-forwarding the video has some sensitivity issues.

Xbox Cloud Gaming (beta) Lands on Quest

Keywords: Quest, Xbox, Microsoft, Cloud Gaming

Finally, you can now play various AAA titles on Quest using Xbox Cloud Gaming. You might want to start by trying Xbox Game Pass Ultimate.

According to Meta's official statement, this feature is currently supported on Quest 2, Quest Pro, and Quest 3. As for controllers, it currently supports Xbox controllers, PS4 controllers, and Nintendo Switch Pro controllers, with plans to support PS5 controllers in the future.

If you're a die-hard Xbox fan, then hurry to the Meta App Store to download Xbox Cloud Gaming (Beta)!

This Issue's Contributors

Zion

ybbbbt

XR World Weekly 013

XRealityZone — Sun, 24 Dec 2023 01:43:00 GMT

This Issue's Cover

This issue's cover features a concept design for visionOS sports live broadcast from Vektora, shared at concept design.

BigNews

The next version of Unity will be known as Unity 6
Spline now also supports Gaussian Splatting

Idea

Tracking Craft: Remote control racing cars, be careful
Mobile version of AR Cubism?

Tool

sudo.ai: Create a 3D model in 60 seconds with a photo
Swift Angel: GPTs that help you develop visionOS Apps with Swift

Article

Apple visionOS and iOS SwiftUI development (Augmented Reality AR)

Code

Vision Pro Agora Sample App: Agora SDK's Vision Pro integration demo
Grape: A Swift library for force simulation and graphics visualization
GoncharKit: A toolkit developed specifically for RealityKit on visionOS

SmallNews

Apple releases spatial video examples
Apple's new patent application: Cross-terminal multiplayer games in Facetime, but with real objects (I really have printed playing cards in my hand!)
TikTok adds AR filter editing to its mobile version
Microsoft: Industrial AR AI, two-pronged approach, more efficient!
Apple's new patent: From the Genius Bar to your living room, using AR to promote new product chopping

BigNews

The Next Version of Unity will be Known as Unity 6

Keywords: Unity, Developer Community

"A new broom sweeps clean", this time, Unity's version number might be the first thing to clean for new CEO Jim Whitehurst.

Since 2017, Unity has changed its version number to a yearly-based major version mode, such as Unity 2017.1, 2017.2, 2017.3, etc. This naming method has continued to the present, such as 2020.3, 2021.3, etc. Before this, whether it was Unity 1.0 in 2005 or version 5.4 in 2016, Unity used a more common pure digital version number.

However, at Unite 2023, Unity announced its next Unity version will be Unity 6, to be released in 2024.

The new CEO also said he didn't like the original yearly naming model:

"Frankly I never liked the yearly naming model, because it's always this weird thing where the LTS came out in a different year than the one it was named for," he said. He said the new naming system should be more "consistent and clean."

Spline now also supports Gaussian Splatting

Keywords: Spline, Gaussian Splatting

In Newsletter 012, we mentioned Gaussian Splatting technology, where each data point is represented as a shape with a Gaussian distribution (or "splat"), rather than a single pixel point or geometric shape. This allows point clouds or data sets to be presented in a smoother, more continuous way, improving visual effects and data interpretation quality. Some 3D scanning tools, such as Polycam and Luma, also support exporting scene files in Gaussian Splatting format.

And the 3D online design tool Spline that we recommended in Newsletter 001 now also supports Gaussian Splatting format, like this work.

The usage is also very simple. Just drag the .ply file from Polycam or the Gaussian Splat result scanned by Luma directly into Spline (note, do not export .ply files directly on the mobile end), and you can directly use Gaussian Splatting in the Spline scene and make some edge trimming according to your own needs:

However, due to the different rendering technologies behind Gaussian Splatting, there are still some limitations in Spline's support for it. For example, although Spline can crop and transform the imported .ply files (displacement, scaling, rotation), it cannot support shadows, physical simulation, lighting, and other features.

But in the end, the Spline team also said that technology is developing very fast, and these incompatible features may be resolved one day, so please don't be discouraged and keep an eye on Spline.

Idea

Tracking Craft: Remote Control Racing Cars, Be Careful

Keywords: MR, Cars, Quest 3

Hey, kids, do you want to play remote control racing cars in your living room?

In the past, you might have to get your parents to buy you lots of little cars or tracks to realize this dream, but with this MR App Tracking Craft, you can achieve this dream just by wearing Quest 3.

Tracking Craft is an MR game where you drive a remote control racing car through various interesting levels. The player's task is to predict the physical settings of the levels and let the remote control car go from the starting gate to the end gate within a limited time.

Of course, thanks to Quest 3's spatial awareness, you can also let the remote control car run wild in your living room, just like the official demonstration video shows:

In actual gameplay, making good use of different remote control car characteristics and observing the unique physical settings of the levels is the key to successfully passing them. For example, in the level below, as long as the car has speed, it will not fall off the track when it flips over, so you can pass the flipped section at a slower speed.

Currently, Tracking Craft is available for free download on both META APP LAB and SIDE QUEST, and the official said it will be launched on PICO in early 2024. If you have Quest 3 at hand, this game is highly recommended for you to try.

Mobile Version of AR Cubism?

Keywords: AR, Cubism, Cubism, Building Blocks

A few days ago, I found a casual game supporting AR mode called Projekt while browsing Artifact, which is similar to building blocks according to prompts from different directions to make the projection satisfy both left and right views (reminds me of the days dominated by three views/engineering drawings in my freshman year, and the style is also very similar to the schematic diagrams I used to draw at school). Similar to 《Cubism》 on Quest and PICO. Made with Unity, it also supports AR mode on mobile phones (just like Cubism supports MR mode), which is quite relaxing. Interested friends can try it.

What, you say this does not reflect the superiority and uniqueness of MR/AR? That's because you don't understand the fun of Lego and Jenga~

Tool

sudo.ai: Createa 3D Model in 60 Seconds with a Photo

Keywords: AI, 3D Assets Generation

sudo.ai is an online platform for generating 3D models, currently available for free trial. Its main feature is to create 3D models based on a photo or text description. Its advantages include:

Just one photo is needed to identify the main subject and automatically guess the shape of the back, generating a corresponding 3D model.
Fast speed, capable of quickly generating usable models within 60 seconds, and also creating a 360-degree rotation video for sharing on social networks.
Automatically recommends similar models based on shape and color, eliminating the need for manual keyword searches.

The official promotional video also highlights the speed of sudo.ai:

It also emphasizes the fidelity of sudo.ai-generated models to the original images:

We also conducted practical tests with sudo.ai. On the left side of sudo.ai, we uploaded a photo of a flower taken with a mobile phone. It automatically identified the main subject as the flower, discarded the background leaves, and generated a model of the flower. After completion, the right side had recommended similar models.

If you remember the CSM.ai we recommended in issue 012, they essentially use the same principle, first generating multi-angle images based on the uploaded picture, then using technologies like NeRF for 3D model generation. sudo.ai has now open-sourced the code for multi-angle image generation: Zero123++, with an online demo. The 3D model generation part is currently only publicly available in the form of a paper, with no plans to open-source the code in the short term.

Using the same flower photo for testing, the left side with sudo.ai is faster and the 3D model is smoother. Currently, export supports only .glb format; while the right side with CSM.ai is a bit slower but has stronger detail and supports more 3D formats.

Swift Angel: GPTs that Help Develop visionOS Apps with Swift

Keywords: AI, Swift, VisionOS, GPTs

Since the launch of GPTs, multiple Swift GPT assistants created by GPT users can help accelerate the development of iOS and visionOS apps using Swift and SwiftUI. We introduced some in the previous issue.

This time, the only novice in the editorial team personally tried and tested the writing and communication efficiency of some of these GPT assistants. Due to time constraints, we did not test all the GPT assistants that claim to help write Swift apps. The following is a brief review of the usage process with a GPT assistant that has been tested and found workable.

Before ChatGPT supported image recognition and online query functions simultaneously, the novice author had a pure text conversation with the GPT4 native body to implement a simple visionOS app front-end interface (very simple, almost window-based, with some volume display in a few places). Since GPT4's knowledge base had not been updated at that time, still stuck in 2020, much of the visionOS documentation information was learned and updated in the knowledge base by the author submitting to GPT during the debug process. The overall experience was that GPT4 "learns" and "understands" quickly, but the author's English written language expression ability also improved greatly after not writing papers.

After the GPTs assistant appeared, the author took the simple interface implemented by the GPT4 body and simplified it again to test individual assistants. Some assistants, when tested, the author uploaded hand-drawn interface sketches and stated that they "could not provide code language," so the author gave up. Meanwhile, to ensure smooth testing and fearing that the sketch was too "rough," the author also quickly drew a set of wireframes for submission testing.

It turns out that a GPT assistant named Swift Angel was able to understand the submitted wireframes (with simple text explanations) and provide Swift code, as well as guide you on how to create project files (truly an Angel). For subsequent misunderstandings or errors, you can also communicate with Angel for modifications (the interface implementation after the first description did not fully meet my expectations; the video and pre/post-modification images here are only for demonstrating GPTs capabilities). Mainly, the efficiency of communication with GPTs/GPT is greatly improved after being able to upload images, no longer relying solely on our "verbal" communication. This example is relatively simple; whether this GPT assistant can implement more complex visionOS app interfaces is something for everyone to explore and discover.

Or do you have any insights or recommendations from your recent interactions with GPTs/GPT in programming? Feel free to leave a comment and interact with us (and pray for server stability)~

Some communication processes with Swift Angel are shown below:

The final implementation effect after the first description is as follows:

Article

Apple visionOS and iOS SwiftUI Development (Augmented Reality AR)

Keywords: visionOS, tutorial, SwiftUI

Apple visionOS and iOS SwiftUI Development (Augmented Reality AR) is a Chinese introductory tutorial on visionOS and SwiftUI development. The author refers to the Japanese developer's visionOS_30Days open-source project, combining their own ideas, creating demos with explanatory thoughts and API usage guides.

Each article briefly completes a small function and comes with pictures and explanations, very suitable for beginners who are not very good at English.

Code

Vision Pro Agora Sample App: Agora SDK's Vision Pro Integration Demo

Keywords: visionOS, Sample Code

visionOS-Quickstart is a visionOS version sample program that uses Agora RTC SDK to implement video calling functions.

If you need to integrate video calling or voice conferencing features in visionOS, you can refer to this official demo from Agora.

Grape: A Swift Library for Force Simulation and Graphics Visualization

Keywords: visionOS, force simulation

Grape is a Swift library for force simulation and graphics visualization, capable of performing force-directed graph and lattice simulation calculations, supporting 2D and 3D simulations, with a visionOS version demo for learning.

GoncharKit: A Toolkit Developed Specifically for RealityKit on visionOS

Keywords: RealityKit, visionOS

GoncharKit is a toolkit developed specifically for RealityKit development on visionOS. It conveniently adds visual effects to 3D models, currently offering three functions: Outline Mesh Generation, DoubleSided materials, Skeleton Visualization.

Outline Mesh Generation: Since visionOS's RealityKit currently neither supports custom multi-channel rendering nor screen post-processing for edge processing, to add outlines (external contour lines) to models, it's necessary to generate new meshes by outwardly extending normals.

Although this method is not perfect for objects with discontinuous normals, such as cubes, resulting in breaks at discontinuities, it is currently the only feasible way to add outlines on visionOS.

DoubleSided materials: Since RealityKit and USDZ themselves do not support double-sided materials, flat objects like leaves may lead to the backside not being displayed. To display the backside of the model, it is necessary to reverse the vertex indices to generate new faces, ultimately achieving a stable display effect.

Skeleton Visualization: Adds markers to the joints of skeletal animations, facilitating debugging of skeletal animations.

SmallNews

Apple Releases Spatial Video Examples

Keywords: visionOS, Spatial Video

On the Apple Developer Streaming Examples page, Apple provided an example based on the 3D movie Prehistoric Earth. However, the video link provided can only be played in Safari on a real Apple Vision Pro device; it cannot be played in simulators or browsers on a Mac. (Possibly the current browser version does not support this HLS format playback)

Apple's new patent application: Cross-terminal multiplayer games in Facetime, but with real objects (I really have printed playing cards in my hand!)

Keywords: Patent, Facetime, Cross-terminal, XR Multiplayer Games

At this year's Apple WWDC and subsequent videos, Apple mentioned creating a SharePlay experience in Facetime using visionOS in Facetime. In a patent application revealed recently by PatentlyApple (Application No. 20230293998), this experience will be upgraded.

It's not just about playing the same virtual game across terminals and systems (iOS, iPadOS, visionOS) in Facetime. The patent application aims to use Fiducial marker technology to bring real-world multiplayer gaming experiences to Facetime. For example, you play poker in Facetime with a friend using an iPhone while wearing Apple Vision Pro (both holding playing cards). Unlike straightforward video presentation (which may not be clear), this patent hopes to use holographic projection or similar forms to bring the cards in your hand to your friend's presence. Or when your friend hands you a "real" item, this technology might generate a "real" projection in your hand, simulating the effect of truly passing it to you, blurring the boundary between reality and virtual, thus realizing the concept of extended reality and bringing people closer together.

TikTok Adds AR Filter Editing to Mobile Version

Keywords: AR, TikTok, EffectHouse

Following the launch of the AR creation software EffectHouse （PC） that emulates AR camera program Snapchat, mobile version TikTok recently announced a new feature that allows users to create AR filters directly in the mobile version of TikTok, eliminating the need to download EffectHouse separately (of course, the PC version of EffectHouse can create richer effects). Users can combine more than 2000 materials from the mobile AR filter library. Previously, TikTok also provided a creator fund worth $6 million to encourage the production of AR filters. Currently, TikTok is adjusting its latest creator program.

Note: For some reason, TikTok later took down the Tutorial video

As shown in the screenshot of the video tutorial below, users can select the corresponding FilterBase, Trigger on the mobile end, similar to a simplified version of the PC EffectHouse and Unity modules; users can combine different 3D assets, Triggers, and tracking Anchors to complete the creation of filters. AR filters created on the mobile end also enjoy the benefits of the creator program.

Microsoft: Industrial AR AI, Two-Pronged Approach, More Efficient!

Keywords: AR, AI, Copilot, Microsoft

Earlier this year, Microsoft introduced Copilot for white-collar workers. Now, they are integrating this AI assistant into their mixed reality applications Dynamics 365 Guides, targeting technicians in factories, with the goal of helping them work better and spend less time managing and maintaining equipment. In this way, on-site workers no longer need to do everything themselves; they can communicate with Copilot using natural language and gestures to complete daily tasks. Copilot provides assistance by displaying relevant technical documentation, service records, and other data sources on the device. This enables easier conversational interactions and the creation of three-dimensional models, among other things. Moreover, as some older employees retire, adding this AI assistant helps companies retain their knowledge base. Currently, Copilot is in preview and will first be released on Microsoft's HoloLens 2 head-mounted device, and later on mobile devices.

Apple's New Patent: From Genius Bar to Your Living Room, Using AR to Promote New Product Purchases

Keywords: Patent, AR, Sales, Apple

Apple has undoubtedly been driving industry integration and technological advancement while marketing remains a crucial aspect for them. A recent patent disclosed by PatentlyApple (link) that Apple has recently been granted (Application No. 11816800, filed in 2021) suggests they may incorporate AR technology in future sales of their products. Even if there's no Apple store within 500 miles, remote sales personnel with devices can come to customers' living rooms in Avatar form, showing and interacting with products in real-time.

Apple also plans to combine AR-guided shopping with online chat systems. This means customers can consult and get help while seeing products come to life through AR technology. It's like having a personal shopping advisor who can help you make the best choices based on your needs.

It appears that Apple is truly pushing the boundaries of AR technology. They are not only making it easier for customers to access this technology but also enhancing the overall shopping experience. I can't wait to see how these features develop and improve in the future!

Writing at the End

Whether it's the quality information you see or the premium content you write, you can contribute through Github's Issue 😉.

Contributors to This Issue