Innovation 9: Vocabulary Mining & Injection

Overview

The Generic Speech Problem

Without vocabulary mining, LLMs use standard expressions:

"That's interesting"
"I understand"
"How fascinating"
# Generic, lacks personality

With Vocabulary Injection

Character uses authentic expressions from source:

"omg that's lit 🔥"
"literally same!!"
"no cap that's wild"
# Authentic voice, distinctive personality

Implementation

Vocabulary Mining Prompt

# You are a linguistic expert and your objective is to identify all of the 
# characteristic vocabulary and emojis from the following text
# output the characteristic vocabulary in about 12 expressions

text="""
{source_text}
"""

# result =

Extraction Process

async def mine_vocabulary(source_text):
    """Extract characteristic expressions"""
    
    prompt = VOCABULARY_MINING_PROMPT.format(text=source_text)
    
    result = await call_llm(
        prompt=prompt,
        model="gpt-4",
        temperature=0.0  # Deterministic extraction
    )
    
    # Parse output
    expressions = [expr.strip() for expr in result.split(',')]
    
    return expressions[:12]  # Top 12 expressions

Injection into Character DNA

def create_role_string(profile, name):
    """Include vocabulary in role string"""
    
    # ... other profile elements ...
    
    # Add vocabulary
    vocab = ', '.join(profile['vocabulary'][:10])
    role_string += f"; vocabulary: {vocab}"
    
    return role_string

# Example output:
# "Sarah, Female, 28yo; [backstory]; vocabulary: omg, literally, vibes, fire 🔥"

Prompts Reference Vocabulary

# Note: {character_name} chats with characteristic voice, limited world-view, catchphrases
# Note: {character_name} speaks with distinctive mannerisms, colloquialisms, regionalisms
# Note: {character_name} constantly learns to mimic voice, slang, idioms to connect

Real Examples

Gen Z Marketing Professional

Mined Vocabulary:
"omg, literally, vibes, lowkey, highkey, no cap, bet, it'll be lit, fire 🔥, 
heart eyes 😍, 100 💯, for real"

Generated Messages:
"omg literally just saw the best sunset ever 🌅 the vibes were immaculate!"
"no cap that place was fire 🔥 we HAVE to go back"
"lowkey thinking about it all day 😅"

Software Engineer

Mined Vocabulary:
"tbh, ngl, figured it out, race condition, async handler, debugging, 
refactoring, edge case, lol, honestly"

Generated Messages:
"tbh debugging this was a nightmare but figured it out eventually"
"ngl that's a pretty interesting edge case you found"
"honestly might need to refactor this entire module lol"

Southern US Regional

Mined Vocabulary:
"y'all, bless your heart, fixin' to, might could, reckon, ain't, 
down yonder, pretty near, supper time"

Generated Messages:
"y'all coming over for supper tonight?"
"I reckon we might could head down yonder around 6"
"bless your heart, that's mighty kind of you!"

Replication Guide

Step 1: Create Mining Prompt

VOCABULARY_PROMPT = """
You are a linguistic expert identifying characteristic expressions.

Analyze this text and extract:
- Catchphrases and expressions
- Emojis used frequently
- Slang and colloquialisms
- Regional expressions
- Distinctive vocabulary

Output 10-12 most characteristic expressions, comma-separated.

Text:
{text}

Expressions:"""

Step 2: Mine from Source

vocabulary = await mine_vocabulary(whatsapp_export)
# Returns: ["omg", "literally", "vibes", "fire 🔥", ...]

Step 3: Include in Character Profile

character_profile = {
    "name": "Sarah",
    "cognitive": {...},
    "backstory": "...",
    "vocabulary": vocabulary  # ← Mined expressions
}

# Synthesize into role string
role = create_role_string(character_profile)

Step 4: Reference in Prompts

prompt = f"""
You are acting like {name}, {role}

# Note: {name} uses characteristic expressions: {', '.join(vocabulary[:5])}
# Note: {name} speaks with distinctive voice and mannerisms

[{host}]: {message}
[{name}]:"""