Theme options

Thank you for trusting this site to use Javascript. It is used in a way that degrades gracefully; for persistence of the below values. This uses local session storage, just so you know.

VR projects: FBT, mouth and eye tracking

Posted on
2026-07-29T23:18:00Z
Tags
Content warnings
None
This page is a draft! Content may not yet be finalized.
Table of Contents

§ Remaining before publishing (draft: false)

  • Take video(s) of working tracking; both VRChat-side and application(s) side.
  • Show libsnout + oscavmgr + VrcAdvert working in screenshots.
  • Get proof-readings.

§ My background with VR

I've been playing Beat Saber for years and years (since ~2020...? on the Rift CV1?). You can see some of my playing here and my BeatLeader. This was most of what I have done with my headsets (I eventually upgraded to the Quest 3 because I enjoyed Beat Saber so much), other than some time spent playing SUPERHOT VR and Half-Life Alyx.

§ VRChat

During the pandemic, although briefly, a partner had me come onto VRChat. I didn't really get it, nor spend any time with it past that initial meeting in some apartment-like setting. Fast-forward several years, a friend reintroduced me to it and this time, it stuck a lot better. I spent a good period of time world-hopping initially, eventually made friends and found places to be! I must really thank Cozy Calm (by Sam) for this.

A snow leopard girl looks out a window at several people floating in front of her. A catgirl sits on a mattress next to a Saya (from Saya no Uta) fumo, a dango daikazoku (from Clannad), paintings referencing Umineko, ... and a laptop. A snow leopard girl looking forwards sitting down on the grass in Cozy Calm. A catgirl stares at the camera! She has a nix flake pin, a toki pona pin and her own logo within her hair. She is wearing a nametag that reads "soweli Ka" (using the glyph for plant; kasi) in toki pona in the style of a Japanese kindergarten badge.

These experiences made me a lot more interested in avatar creation and VR technology.

To my VRChat friends: hi! you're cute, you're based.

toki a jan pona lon ma [wile,, jan,, sama,,] o. sina suwi mute. sina wawa mute.

§ Full body tracking

§ Pre-VRChat attempts at FBT

A long while back (must have been 2021) I had obtained an XBOX 360 Kinect for using for body tracking with Beat Saber via K2VR with owoTrack. This was not a good experience (no fault of the K2VR people); I did not have enough space for the required distance and I can't say the Kinect is a great tracking solution period. Around this time I had heard of SlimeVR for the first time as a result of the crowdfunding campaign, but I was not willing to part with my money for it at the time.

§ Earlier attempts with joycons

§ SlimeVR joycon wrangler

Due to circumstances, I had two sets of joycons (these circumstances being having a prior Switch 1 to my current, that I had modchipped but later broke in an attempt to replace a bezel) and I noticed that slimevr-wrangler existed. I noticed a Linux-specific issue with MAC address duplication and created a PR to fix it. Despite these changes and also having tested the WIP quaternions PR later, the accuracy of tracking with this was absolutely attrocious; drift was instant and rotation would end up going the wrong way in comparison to reality. I also used owoTrack during this for the waist tracker.

§ everything-imu

To see what it would look like in the "best case" I had decided to try everything-imu, a project I absolutely cannot recommend given its either heavily AI-assisted or vibecoded status. This slightly improved things in comparison to the aformentioned, but it was still woefully inadequate. While trying it, I noticed that on my system the UI could not even correctly connect to the server itself, so that was fun.

§ SlimeVR Smol Slimes

§ Prints

  • Ibis Trackers - I don't like Bris particularly much because of their heavy use of AI, but Ibis Trackers as prints seem to predate this.
  • Raye's Smolslime case edits - Had some tracker trays I needed for a particular width strap!

§ Results

After repeatedly doing cost calculations and staring at my 3D printer longingly, I realised I could probably make my own Slime Smols, regardless of the fact that Butterfly trackers are now pre-orderable. I decided to build them around the SlimeVR Mumo Breakout Modules (ICM-45686) because they were the best IMUs according to the IMU comparison. I used the typical knock-off nice!nanos (superminis, whatever) that are NRF52840s in a pro micro format. The pinecil, as always, came in clutch. I used the aliexpress straps that were not recommended, but hey, they were cheap!

I used the soldering fixture for the Ibis Trackers, it went relatively well other than pin mistakes that I made that required reworking what I had done. The first tracker took 11 hours, the remaining each took 30 minutes (and it could've been less than 30 mins for the remainder had I not had to do some rework from doing all of them a certain way initially).

With ~120mAh cells, I get around 30hrs of battery life on these trackers. They weigh very, very little.

Test fitting straps on programming socks Tracker mounting belts and straps. An individual smol slime with no housing or battery with additional wire antenna Many trackers next to their housing, no battery. Many trackers attached to the dock.

Finished trackers with battery and housings attached to the dock

§ Mouth & eye tracking

§ Project Babble & EyeTrackVR (with Baballonia + VRCFaceTracking.Avalonia)

Do not confuse Project Babble with Project Babel!

For this project, I opted for the Seeed Studio XIAO Sense ESP32-S3; it's tiny and I think the Project Babble DIY trackers are far smaller than the official boards for them. I bought 3 initially and later bought a fourth to replace one that I immediately broke; the camera hat board connector sheared off backwards when I was initially removing one of them to put on an antenna.

§ Part 1 - mouth!

§ Prints
§ Results

Initially, I printed in PLA (because I didn't want to open a new reel of PETG and I don't have a drybox yet) for testing. My first attempt with the mouth tracker was over WiFi with Baballonia and VRCFaceTracking.Avalonia. It did, in fact work, though initially it was off-center and I had to figure out how to do the calibrations.

Around this time, I also removed the IR filters on all of the 120/130° FOV cameras I had obtained (with two pairs of pliers to twist the lenses off and a pair of tweezers to shatter the filter).

The first attempt for the mouth tracker; rear shot, heatsink visible. The first attempt for the mouth tracker, front shot with a camera. The first attempt for the mouth tracker, third attempt with LEDs. An Epli model with an open mouth showing working mouth tracking

§ Part 2 - eye tracker!

The print for this I got from someone as a 3MF + blend set of files on the EyeTrackVR discord as I use VROptician prescription lenses with my Quest 3. I would otherwise link them here. My apologies!

A few days into this, Baballonia got the expression beta release! How very exciting!

A single eye tracker work-in-progress

§ Part 3 - reprints in PETG for the mouth tracker, slime smol strap trays and Quest 3 ESP and IR power board housings

At this point I was ready to finalize things, given that the mouth tracker was a little janky by virtue of PLA issues. I also think, had I committed early enough to using UVC for the cameras I would've had less issue with antenna wire routing causing issues with putting together the mouth tracker.

A picture of the reprinted mouth tracker, now PETG A picture of the reprinted mouth tracker, now PETG

§ All done!

A blurry picture of the Quest 3 with installed eye trackers A picture with mouth tracker and eye trackers in frame on the Quest 3, with the IR LEDs functional A wire mess attached to a Quest 3

By the end, I moved to using the cameras via UVC with the USB hub instead of using WiFi and treating them as IP cameras. I already use the Quest 3 via ADB-forwarded WiVRn over a Syntech Link Cable, 16ft with Charging, but having to have a second long cable is pretty rough. I accept it for now but I would like to get the whole setup down to one and a lot cleaner in future.

A picture of Babbalonia showing eyes and mouth.

§ libsnout + oscavmgr + VrcAdvert

Thank goodness for Rust developers! I have been able to avoid the C# Avalonia programs due to the work of those mentioned below. libsnout even allows for proper GPU accelerated training on Linux, something that Baballonia natively does not (outside of Windows). I was also able to make an adjusted version of the Nix flake libsnout has to allow for NixOS libtorch training!

I can't express enough how much I appreciate and respect Darksecond for making libsnout. (It also seems libsnout was moved to codeberg! Chad shit.)

The same can be said for galister and oscavmgr, VrcAdvert and WayVR.

Over time, libsnout has gotten better and better for my use case, including being able to use eye models trained by Baballonia including expression beta things (because with their own training, it seemed a little rough for me on certain parameters, mostly blink did not seem to work!). I can't complain.

§ Conclusion

Although a little janky, I've been able to make a Linux-only VRChat experience that works well. I have all the tracking I could want, although it requires adjustment from time to time and could do with cleanup efforts. I'm relatively happy with the results!

Something I would like to see is modded Beat Saber having SlimeVR that works because right now it is dependent upon SteamVR (which I do not use) set to a compatibility mode to expose it as fake Vive trackers since that is what the Custom Avatars mod requires. I like being able to dance in my Beat Saber and it'd be nice to be able to stream that working without having to use Windows.

§ Remaining things to do

  • Clean up the wire mess! Shorter cables, better routing.
  • Figure out how to get the setup to only require one cable, two long cables is rough!