finSCHIZODIO::: Mouth Pipeline & UV Shapekey Standards

Eyes were the warmup. Mouth is where SAUCE learns to speak. If the eyes are the window to the soul, the mouth is the door — and we need that door to open, close, purse, pout, and sync to audio without exploding the vertex count.

Why Mouth Is Harder Than Eyes

Eyes blink. That’s one state transition: open ↔ closed. The UV slides horizontally, shader does the rest, done.

Mouths speak. That’s not one transition — that’s a continuous deformation problem. Every phoneme in every language reshapes the lips, jaw, and cheeks differently. And because SAUCE isn’t a static portrait — it’s a 3D character that tilts, turns, and animates — those deformations need to work from every angle.

The good news: we already solved the hard part. The eye pipeline (Gemini → Hunyuan3D → 3D Studio → Blender → Paint.NET) proved the AI-assisted workflow works. Now we’re adapting it for mouths with one critical addition: shapekeys.

The Pipeline: Same Players, New Rules

The only new stage is shapekey creation — everything else is the same pipeline with mouth-specific conventions.

The Trait Inventory: 15 Mouths, One Base

From the collection tree (devlog1), we have 15 mouth traits to support:

All 15 are currently flat UV textures. The goal: each becomes a 3D mouth with the same shapekey set, so any SAUCE head can speak regardless of which mouth trait is equipped.

The Base Mesh: Mouth-Specific Topology

The SAUCE mouth base mesh follows rules learned from the eye radial topology:

• Concentric lip loops: 3 edge loops minimum (outer lip, mid-lip, inner lip)
• Radial corner fans: Edge flow radiates from mouth corners for natural creasing
• Jaw adjacency: Lower lip loops connect to chin geometry, not floating
• Quad purity: No triangles in the deform zone — quads deform predictably
• Symmetry plane: Mirror across X, break only for asymmetric traits (Smirk)

Target stats:

• Mouth mesh: ~120–180 tris
• Lip deformation zone: ~40 vertices
• Full head with mouth: consistent with existing SAUCE base

UV Shapekey Standards: The Core Spec

This is the heart of the document. Every mouth trait — flat or 3D — needs these shapekeys to support speech animation.

Shapekey Naming Convention

Mouth_<Category>_<Name>

Examples:

• Mouth_Phoneme_AA — wide open “ah” sound
• Mouth_State_Closed — rest/neutral
• Mouth_State_Wide — grin width (non-speech)
• Mouth_Emotion_Frown — sad downturn

Required Shapekey Set (16 Total)

1. Rest State (1 shapekey)

2. Viseme Shapekeys (11 shapekeys)

Based on the standard OVRLipSync viseme set (industry standard for VR/chat avatars):

3. Emotional/Expression Shapekeys (4 shapekeys)

Shapekey Value Rules

• Range: 0.0 to 1.0 (Blender standard)
• Default rest: All shapekeys at 0.0, base mesh = Mouth_State_Closed
• Combinability: Visemes are mutually exclusive — only one viseme active at a time
• Emotion overlay: Emotional shapekeys CAN combine with visemes (e.g., smiling while saying “oh”)
• Combination method: Additive blending — final = base + (viseme * weight) + (emotion * weight)
• Max total weight: Clamp at 1.5 to prevent over-deformation

UV Atlas Layout: Expression States

For flat-texture mouth traits (the 15 existing variants), the UV atlas follows the eye pattern but with more columns:

+--------+--------+--------+--------+
| Closed | Open   | Wide   | Pursed |
+--------+--------+--------+--------+
| Smile  | Frown  | Ooh    | Aah    |
+--------+--------+--------+--------+

Atlas spec:

• Dimensions: 1024×512 minimum (4×2 grid of 256×256 expression tiles)
• Grid: 4 columns × 2 rows = 8 expression states per trait
• States per tile: Closed, Open (surprise), Wide (grin), Pursed, Smile, Frown, Ooh (rounded), Aah (wide)
• Shader logic: UV offset driven by animation state, same as eye blink
• Alpha channel: Teeth and inner mouth masked for layering over face texture

For 3D mouth traits (future mechanical/expression variants), the atlas becomes a single diffuse + normal + roughness — the shapekeys handle all deformation.

Lip-Sync Compatibility: The Translation Layer

SAUCE needs to talk. Whether it’s a user voice chat, pre-recorded audio, or AI TTS, the mouth must sync. Here’s the compatibility matrix:

OVRLipSync (Meta / VRChat standard)

OVRLipSync outputs 6 visemes + laughter/blink. Our 11-shapekey set is a superset — we map 6-to-11 and fill gaps with nearest-neighbor.

Rhubarb Lip-Sync (offline, free)

Rhubarb outputs 9 mouth shapes (A, B, C, D, E, F, G, H, X). Map:

• A → Mouth_Viseme_AA
• B → Mouth_Viseme_PP
• C → Mouth_Viseme_E
• D → Mouth_Viseme_TH
• E → Mouth_Viseme_O
• F → Mouth_Viseme_U
• G → Mouth_Viseme_FF
• H → Mouth_Viseme_I
• X → Mouth_State_Closed

Custom JSON Animation Format

For web-based SAUCE implementations (three.js and friends):

{
  "track": "mouth_viseme",
  "keyframes": [
    {"time": 0.0, "value": "sil"},
    {"time": 0.12, "value": "AA"},
    {"time": 0.28, "value": "PP"},
    {"time": 0.45, "value": "sil"}
  ]
}

The runtime maps string viseme IDs to shapekey indices. 16 shapekeys = index 0–15.

Blender Implementation: Shapekey Creation Workflow

For the artist (me, in Blender at 2 AM):

1. Duplicate base mesh → create shapekey from mix
2. Enter edit mode on the new shapekey
3. Deform only lip vertices — jaw vertices follow if jaw is separate mesh
4. Use proportional editing (O key) with smooth falloff for natural deformation
5. Test in object mode — slider from 0 to 1, check silhouette from front and side
6. Name strictly — Mouth_Viseme_AA, not ah_mouth or wide_open
7. Set slider range — 0 to 1, never negative
8. Pin for preview — pin the shapekey at 1.0 to sculpt the next variant

Proportional Editing Settings

Paint.NET: The Expression Atlas

For flat-texture mouth traits, Paint.NET stays the texture source. The workflow:

1. Base mouth shape — neutral closed, lips together
2. Duplicate layer — one per expression state
3. Transform each layer — warp lips to match shapekey target
4. Export grid — 4×2 tiles, 1024×512 total
5. Name convention: mouth_<traitname>_<YYYY-MM-DD>_<HHMM>.webp

Critical: Each tile must align pixel-perfect with the UV shell. One pixel offset = lip seam in engine. Zoom to 800%, use grid snap, check alignment against UV reference overlay.

Integration: From Blender to SAUCE

The export pipeline:

1. Blender → Shapekeys baked, base mesh frozen
2. Export format: GLB (glTF 2.0) with morph targets
3. Morph target naming: Preserved from shapekey names
4. Engine import: Three.js family → mesh.morphTargetInfluences
5. Runtime animation: AnimationMixer or custom driver updates influence array

Three.js Morph Target Example

// Access morph targets by name
const mouthMesh = scene.getObjectByName('mouth')
const morphDict = mouthMesh.morphTargetDictionary

// Set viseme influence (0.0 to 1.0)
mouthMesh.morphTargetInfluences[morphDict['Mouth_Viseme_AA']] = 0.85

// Blend with emotion
mouthMesh.morphTargetInfluences[morphDict['Mouth_Emotion_Smile']] = 0.3

Vertex Count Budget

• Each shapekey = full vertex position array for deformable verts
• 120 mouth verts × 16 shapekeys = 1,920 extra vertex positions
• In GLB: stored as morph target accessors (~23KB uncompressed)
• Acceptable overhead for the expressiveness gain

Quality Gates: Definition of Done

Before a mouth trait graduates from “in progress” to “collection ready”:

• All 16 shapekeys created and named correctly
• No vertex pinching at 1.0 influence (check from 45° angle)
• Viseme set produces readable lip-sync on test audio
• Texture atlas exported (flat traits) or material assigned (3D traits)
• Weight painted to head rig (if using bones + shapekeys)
• Exported to GLB and reimported cleanly
• Tested in a live web build with morphTargetInfluences driver
• Screenshots: front view, 3/4 view, profile, all shapekeys at 1.0

The Mechanical Mouth Future

Just like Ekubo and Kano’s bionic eyes, some mouth traits deserve 3D geometry:

• Vampire — fangs protrude in Mouth_Viseme_PP
• Buck Teeth — incisors extend in Mouth_Viseme_AA
• Clown Smile — painted grin widens asymmetrically
• Tongue Out — tongue mesh with independent bone

These follow the same shapekey spec but add geometry-specific deformation. The jaw opens, the fangs track with it, the tongue extends on its own bone layer.

Summary: The Spec At a Glance

Next

This document is the spec. The next task is the execution: take one trait (probably Smile — simplest geometry, most recognizable) and run it through the full pipeline with shapekeys. If Smile works, the other 14 are variations on the same rig.

The mouth is where SAUCE stops being a picture and starts being a person. Let’s make it count. 🫙

Days until June 30 deadline: 32 | Pipeline: locked | Shapekeys: specified | Status: ready to sculpt