Portfolio Review Day

Today was Portfolio Review. I managed to get good feedback on my portfolio.

The overall impression is that portfolio is strong. I would later write what feedback was given.

Internship: XSens Trackers Issue (and Choosing Different Trackers)

For context, I have been trying to solve the positioning with XSens trackers.

I have encountered an issue where the trackers said they are moving while being completely still.

I have messaged XSens Support to understand what is the problem, and they replied that this is expected and intended behavior. They mentioned that XSens DOT trackers were not intended for positioning and that my solution inherently would not work due to accumulating free acceleration errors over time.

You know what that means?

This means that I would look for other trackers to use!

I already have an eye for Haritora because I saw that the fans of it managed to reverse-engineer the Bluetooth UUIDs to get data from the trackers directly.

I would do a comparison of options and compile a proper offer document for my company

Internship: Successfully Convincing My Employers to Buy Haritora Trackers

In this post, I managed to convince my employers to move away from XSens trackers and to try out the HaritoraX 2 trackers.

The main 2 reasons to use HaritoraX 2 trackers:

•⁠ ⁠Price: HaritoraX 2 trackers cost at 300 euros, almost 3 times cheaper than XSens trackers.
•⁠ ⁠Technical Viability: HaritoraX 2 provides a proven, ready-to-use solution for our tracking needs.

Haritora has a VR manager app which processes tracker data and outputs a complete skeleton position and rotation data through OSC which can then be caught by our Unity application.

This means that from our side, we need to program an OSC bridge similar to that of VR-Chat

Click the comments to read the deep dive analysis!
╰➤https://t.me/JonGearBlog/692

Internship: The HaritoraX2 is finally here!

My plan is to:
1. Test the trackers in VR chat with the app installed on the headset
2. Check out the VR chat packages I found on the internet
3. Implement the trackers into the existing tennis project

Internship: Trying out HaritoraX 2 Trackers Full-Body

In this video, I test out the HaritoraX 2 trackers in full-body.

Good news! It seems to be working as intended.

I have tried out the "install the VR manager on the headset" route, it DOES work, but it seems to be slow. I presume its because VR chat is a resource-intensive game and the Meta Quest is spread thin between them. I presume that our baseball and tennis application could be less power intensive than VR chat.

In this video, I have used the "install the VR manager on the phone" route, and the performance and responsiveness is better. I have managed to set up everything correctly, though for note, the last step of the calibration in VR chat was the most confusing to me, so the character MIGHT not be 100% perfect, but that is because of that one last part.

But overall, the tracking works! The next step is to check out the Unity packages from VR chat which enables OSC communication with the trackers.

For some reason, VR chat crashed at the end of the recording, but it seems to be more related to the Meta Quest 2's limited processing power.

https://youtu.be/Rv3s95KBmvw

Internship: Functioning Full-Body Skeleton Rig Demo in VR Chat (HaritoraX2)

In this video I demonstrate that HaritoraX2 trackers are working as intended and track very well! Watch as I show how responsive the legs are tracked in VR with me doing funny moves.

https://www.youtube.com/watch?v=GZE1Uex3iqs

Internship Tangent: How to Connect Your Meta Quest to Eduroam

A helpful link for anyone who would want to try to connect your headset to eduroam.

It does provide the necessary Wi-Fi, but the drawback is that due to my university's firewall, you aren't able to send data from one device to another (this step is necessary if I would be sending data from the phone to the headset)

https://www.tiborudvari.com/blog/connect-meta-quest-to-eduroam/

Internship Tangent: About Eduroam and Iotroam

If you have been in a university, you would be aware of Eduroam. It allows you to connect to the internet on campus.

One issue with Eduroam is the firewall: a device cannot connect to another device on the same network.

This was a problem for me because the Quest and the phone must be connected together (and having the VR manager on the Quest itself is not an option due to bad performance)

To resolve this, I contacted the university which suggested me to connect to Iotroam. Iotroam is a network which has less restrictions. To connect to it, you must give your university the MAC address.

I managed to get Iotroam connection and I was able to successfully connect the phone and the headset in VR chat.

Internship: Attempting to Bridge OSC

Today, I was working on connecting the OSC in Unity.

I downloaded the OSC library which is used by VR Chat.

So far, the results is that nothing has happened yet. The headset is supposed to give out the IP and the port, and then just a blob of information it receives (our tracker data).

At first, I thought the reason was that the network issues. That's why I spent some time trying to resolve the Eduroam/Iotroam issue today. Fortunately, I did resolve it, but it seems that no change has happened.

I will try to look more into why it just doesn't work.

Internship: Trying to Understand Why It Doesn't Work

From yesterday, I understood that I have been using the wrong OSC-Core library. Today, I downloaded the one which VR-Chat uses.

From the uses, I got no results, but I now I got more info, this warning specifically:

SocketException: Access denied

I then switched to OSC-Jack, and I got almost the exact same error.

Something is blocking the sockets. This could explain why nothing was happening, as in there was literally no messages at all. I do know for certain that this is NOT a Wi-Fi network issue because I have Iotroam on my phone and headset which I have tested with VR-Chat.

So this must be something on the Unity/Android/Meta Quest side.

UPDATE:

I figured out what was wrong.

The Unity project did NOT have permission to use the internet. And you would think that something like this would be as easy as a small checkbox (it is, but with a few exceptions).

First, AndroidManifest.xml

My project didn't have one, I needed to make a custom one:

1. Open Project Settings > Player.
2. Click the Android tab (the robot icon).
3. Expand Publishing Settings.
4. Find Custom Main Manifest and check the box next to it.
5. Unity will now generate a file at: Assets/Plugins/Android/AndroidManifest.xml.

Then in there, I had to solved it by

I solved by adding these two lines outside the application tag in the AndroidManifest.xml
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

I got this from here:
https://communityforums.atmeta.com/discussions/dev-quest/how-to-receive-osc-data-via-udp-on-the-quest/742856

Then, I had to go and uncheck the setting which still overrides the internet access (even though there is a different setting checkbox which says to turn on the internet and even though you can have your own AndroidManifext.xml which explicitly states to turn on internet access)

If you are using the Oculus XR Plugin or Meta XR Core SDK, they sometimes override your manifest during the build.
1. Go to Project Settings > XR Plug-in Management > OpenXR.
2. In the Android Tab, look for Meta Quest Support.
3. Click the Gear icon next to it.
4. Crucial: Ensure "Force Remove Internet Permission" is UNCHECKED. If this is checked, it doesn't matter what your manifest says; the Meta build tools will delete the internet permission as a "security measure."

This resolves that SocketException Access Denied error. Now, when I launch OSC-Jack, I don't get such error. I would need to try it with OSC-Core later.

This is so many hoops you have to jump through to just get that to work. And they are all hidden behind these unintuitive settings.

Continuation

For OSC-Core, I managed to get the actual IP address instead of local-host. This is good progress!

Internship: Successful Applicaiton of OSC-Core from VR-Chat to My Own Unity Application

In this video, I demonstrate how I use OSC-Core from VR-Chat in my very own Unity application.

So what did I do?

First, I installed some proper OSC debug tools recommended by VR-Chat: "TouchOSC" and "Protokol" by Hexler. This will help with properly debugging with sending and receiving OSC information.

Second, I have set up the TouchOSC on my phone to mimic the Shiftall VR Manager. This way, when I slide the slider on my phone, it will send OSC data over the network.

Third, I have set up the OSC in my Unity project. As it turns out, it ACTUALLY IS that simple (like "plug-and-play"). It just works.

I would still need to investigate what I did wrong. I'm not sure what exactly different I did, but I'm very happy with the result.

https://youtu.be/KCrT6g94gVs

Internship: Full-Body Tracking in Unity Just Like VR-Chat With HaritoraX2 Trackers

In this video, I demonstrate a collection of cubes which represent me in the virtual world. When I move my legs in real life, the legs in the game move too.

The implementation is similar to what VR-Chat uses.

What is essentially left is to apply all of this data to a character model. And that's it.

https://youtu.be/QdWAARc9eOo

Internship: Dancing Robot (Full-Body Tracking in Unity Demonstration)

In this video, I demonstrate and build upon the dancing cube man to make a dancing robot.

The implementation is very simple. We are using Unity's native animator IK solver system. We are using the chest, hip, and 2 feet and knees as IK targets. This applies all of the data to the legs.

The head camera is a bit wonky, but my main job was to get the legs to work.

https://youtu.be/OAOXHxKc9dk