mirror of
https://github.com/zenlm/zen-live.git
synced 2026-07-27 03:09:42 +00:00
Rebrand as Zen Live with Hanzo Node backend support
- Add control room web portal (index.html) for news teams - Add simplified monitor view (monitor.html) for broadcast displays - Add Hanzo Node backend integration (HANZO_NODE_URL env var) - Default to Spanish → English translation - Add audio streaming endpoints for broadcast integration - Add OpenAPI docs at /docs - Update README with comprehensive documentation
This commit is contained in:
Hanzo Dev
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.env
|
||||
.venv/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
.DS_Store
|
||||
@@ -0,0 +1,84 @@
|
||||
# Zen Live
|
||||
|
||||
Real-time speech translation for broadcast news monitoring.
|
||||
|
||||
## Project Context
|
||||
|
||||
This is part of the [Zen AI model family](https://github.com/zenlm/zen), providing a hosted translation service via [Hanzo Node](https://github.com/hanzoai/hanzo-node) infrastructure.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Control Room Browser
|
||||
│
|
||||
│ WebRTC (audio/video)
|
||||
▼
|
||||
Zen Live (FastRTC/FastAPI)
|
||||
│
|
||||
├──► Hanzo Node API (recommended)
|
||||
│ │
|
||||
│ ▼
|
||||
│ Qwen3 LiveTranslate
|
||||
│
|
||||
└──► Direct DashScope API (fallback)
|
||||
│
|
||||
▼
|
||||
Qwen3 LiveTranslate
|
||||
```
|
||||
|
||||
## Backend Options
|
||||
|
||||
1. **Hanzo Node** (recommended): `HANZO_NODE_URL=http://host:9550`
|
||||
2. **DashScope Direct**: `API_KEY=xxx`
|
||||
3. **Zen Omni Local** (future): `ZEN_OMNI_PATH=/path/to/model`
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `app.py` | Main FastAPI/FastRTC application |
|
||||
| `index.html` | Control room web portal |
|
||||
| `monitor.html` | Simplified broadcast display |
|
||||
| `requirements.txt` | Python dependencies |
|
||||
| `README.md` | User documentation |
|
||||
| `LLM.md` | AI assistant context (this file) |
|
||||
|
||||
## Key Endpoints
|
||||
|
||||
- `/` - Control room UI
|
||||
- `/monitor` - Broadcast monitor view
|
||||
- `/api/status` - Health check
|
||||
- `/broadcast/info` - Engineer integration guide
|
||||
- `/audio/stream/{id}` - PCM16 audio stream
|
||||
- `/outputs?webrtc_id={id}` - SSE transcripts
|
||||
|
||||
## Default Configuration
|
||||
|
||||
- **Source**: Spanish (news monitoring use case)
|
||||
- **Target**: English
|
||||
- **Voice**: Cherry
|
||||
- **Audio**: PCM16, 24kHz, mono
|
||||
- **Latency**: ~200-500ms
|
||||
|
||||
## Integration with Hanzo Node
|
||||
|
||||
When `HANZO_NODE_URL` is set, Zen Live can:
|
||||
1. Query configured LLM providers from hanzo-node
|
||||
2. Use node's API key management
|
||||
3. Leverage node's monitoring/logging
|
||||
|
||||
Future: Native integration with zen-omni model for offline operation.
|
||||
|
||||
## Development Notes
|
||||
|
||||
- WebRTC via FastRTC library (Gradio ecosystem)
|
||||
- CORS enabled for cross-origin control room access
|
||||
- Audio subscribers pattern for broadcast streaming
|
||||
- SSE for real-time transcript delivery
|
||||
|
||||
## Links
|
||||
|
||||
- GitHub: https://github.com/zenlm/zen-live
|
||||
- Hanzo Node: https://github.com/hanzoai/hanzo-node
|
||||
- Zen Models: https://github.com/zenlm/zen
|
||||
- Hanzo AI: https://hanzo.ai
|
||||
@@ -1,12 +1,179 @@
|
||||
---
|
||||
title: Qwen3 Livetranslate Demo
|
||||
emoji: 🏃
|
||||
colorFrom: pink
|
||||
colorTo: gray
|
||||
sdk: gradio
|
||||
sdk_version: 5.46.1
|
||||
app_file: app.py
|
||||
pinned: false
|
||||
---
|
||||
# Zen Live
|
||||
|
||||
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
||||
Real-time speech translation for broadcast news monitoring. Powered by [Hanzo AI](https://hanzo.ai).
|
||||
|
||||
## Overview
|
||||
|
||||
Zen Live is a low-latency simultaneous translation service designed for news control rooms. It takes audio/video input in one language and outputs translated audio + captions in real-time via WebRTC.
|
||||
|
||||
**Default configuration: Spanish → English** (configurable for 18+ language pairs)
|
||||
|
||||
## Features
|
||||
|
||||
- **~200-500ms end-to-end latency** - Suitable for live broadcast monitoring
|
||||
- **WebRTC streaming** - Simple browser-based consumption for control rooms
|
||||
- **Multiple output formats** - WebRTC, HTTP audio streams (PCM/WAV), SSE transcripts
|
||||
- **Broadcast integration** - Convert to SRT/RTMP/NDI via ffmpeg
|
||||
- **Control room UI** - Professional interface with transcript logging
|
||||
- **Monitor mode** - Clean fullscreen view for broadcast displays
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/zenlm/zen-live.git
|
||||
cd zen-live
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Configure backend (choose one):
|
||||
export HANZO_NODE_URL=http://localhost:9550 # Recommended: Hanzo Node
|
||||
# or
|
||||
export API_KEY=your_dashscope_api_key # Direct DashScope API
|
||||
|
||||
# Run
|
||||
python app.py
|
||||
```
|
||||
|
||||
Open http://localhost:8000 in your browser.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `/` | Control room web portal |
|
||||
| `/monitor` | Simplified broadcast display view |
|
||||
| `/monitor?autostart=1` | Auto-start for OBS/video walls |
|
||||
| `/api/status` | Service health check |
|
||||
| `/api/sessions` | Active session list |
|
||||
| `/broadcast/info` | Integration guide for engineers |
|
||||
| `/docs` | OpenAPI documentation |
|
||||
| `/outputs?webrtc_id=ID` | SSE transcript stream |
|
||||
| `/audio/stream/ID` | Raw PCM16 audio (24kHz) |
|
||||
| `/audio/wav/ID` | WAV-wrapped audio stream |
|
||||
|
||||
## Backend Options
|
||||
|
||||
### 1. Hanzo Node (Recommended)
|
||||
|
||||
Connect to a [Hanzo Node](https://github.com/hanzoai/hanzo-node) instance for managed translation infrastructure.
|
||||
|
||||
```bash
|
||||
export HANZO_NODE_URL=http://your-hanzo-node:9550
|
||||
```
|
||||
|
||||
### 2. Direct DashScope API
|
||||
|
||||
Use DashScope Qwen3 LiveTranslate directly (requires API key).
|
||||
|
||||
```bash
|
||||
export API_KEY=your_dashscope_key
|
||||
```
|
||||
|
||||
### 3. Local Zen Omni Model (Coming Soon)
|
||||
|
||||
Run completely offline with local Zen Omni model.
|
||||
|
||||
```bash
|
||||
export ZEN_OMNI_PATH=/path/to/zen-omni
|
||||
```
|
||||
|
||||
## Control Room Usage
|
||||
|
||||
### For News Teams
|
||||
|
||||
1. Open **http://server:8000** in Chrome/Edge
|
||||
2. Select source language (default: Spanish)
|
||||
3. Select target language (default: English)
|
||||
4. Click **Start** to begin translation
|
||||
5. Listen to translated audio through speakers/headphones
|
||||
6. View live captions on screen
|
||||
|
||||
### For Broadcast Engineers
|
||||
|
||||
**OBS Integration:**
|
||||
```
|
||||
Add Browser Source → http://server:8000/monitor?autostart=1
|
||||
```
|
||||
|
||||
**Direct Audio Monitoring:**
|
||||
```bash
|
||||
ffplay -f s16le -ar 24000 -ac 1 http://server:8000/audio/stream/SESSION_ID
|
||||
```
|
||||
|
||||
**Convert to SRT:**
|
||||
```bash
|
||||
ffmpeg -f s16le -ar 24000 -ac 1 -i http://server:8000/audio/stream/SESSION_ID \
|
||||
-c:a aac -f mpegts 'srt://broadcast-server:9000'
|
||||
```
|
||||
|
||||
**Convert to RTMP:**
|
||||
```bash
|
||||
ffmpeg -f s16le -ar 24000 -ac 1 -i http://server:8000/audio/stream/SESSION_ID \
|
||||
-c:a aac -f flv rtmp://server/live/translation
|
||||
```
|
||||
|
||||
## Supported Languages
|
||||
|
||||
### Source Languages
|
||||
Spanish, English, Chinese, Portuguese, French, German, Russian, Italian, Korean, Japanese, Cantonese, Indonesian, Vietnamese, Thai, Arabic, Hindi, Greek, Turkish
|
||||
|
||||
### Target Languages
|
||||
English, Chinese, Russian, French, German, Portuguese, Spanish, Italian, Korean, Japanese, Cantonese, Indonesian, Vietnamese, Thai, Arabic
|
||||
|
||||
## Audio Format
|
||||
|
||||
- **Encoding:** PCM16 signed little-endian
|
||||
- **Sample Rate:** 24,000 Hz
|
||||
- **Channels:** Mono (1)
|
||||
- **Bits per Sample:** 16
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HANZO_NODE_URL` | Hanzo Node backend URL | - |
|
||||
| `API_KEY` | DashScope API key (fallback) | - |
|
||||
| `ZEN_OMNI_PATH` | Local model path | - |
|
||||
| `PORT` | Server port | 8000 |
|
||||
| `MODE` | `UI` for Gradio, `PHONE` for FastPhone | - |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ WebRTC ┌──────────────────┐
|
||||
│ Control Room │◄─────────────────►│ Zen Live │
|
||||
│ Browser UI │ │ (FastRTC/ASGI) │
|
||||
└─────────────────┘ └────────┬─────────┘
|
||||
│
|
||||
┌──────────┴──────────┐
|
||||
│ │
|
||||
┌─────▼─────┐ ┌──────▼──────┐
|
||||
│ Hanzo │ │ DashScope │
|
||||
│ Node │ │ Direct │
|
||||
└───────────┘ └─────────────┘
|
||||
│ │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ Qwen3 LiveTranslate │
|
||||
│ (Backend) │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
## Related Projects
|
||||
|
||||
- [Hanzo Node](https://github.com/hanzoai/hanzo-node) - AI infrastructure node
|
||||
- [Zen AI Models](https://github.com/zenlm/zen) - Zen model family
|
||||
- [Zen Omni](https://huggingface.co/zenlm/zen-omni) - Multimodal model
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
|
||||
## Links
|
||||
|
||||
- **Zen LM:** https://zenlm.org
|
||||
- **Hanzo AI:** https://hanzo.ai
|
||||
- **HuggingFace:** https://huggingface.co/zenlm
|
||||
|
||||
@@ -1,9 +1,27 @@
|
||||
"""Simultaneous speech translation over FastRTC using DashScope Qwen3 LiveTranslate.
|
||||
- Streams mic audio (16k PCM16) to DashScope Realtime
|
||||
- Receives translated text deltas and 24k PCM16 TTS audio
|
||||
- Plays audio via FastRTC and shows text in a Gradio Chatbot
|
||||
Set DASHSCOPE_API_KEY in the environment before running.
|
||||
"""
|
||||
Zen Live - Real-time Speech Translation for Broadcast
|
||||
|
||||
A low-latency simultaneous translation service for news control rooms.
|
||||
Powered by Hanzo AI infrastructure with Qwen3 LiveTranslate backend.
|
||||
|
||||
Backend options:
|
||||
1. Hanzo Node API (recommended) - Set HANZO_NODE_URL
|
||||
2. Direct DashScope API - Set API_KEY
|
||||
3. Local Zen Omni model - Set ZEN_OMNI_PATH
|
||||
|
||||
Usage:
|
||||
export HANZO_NODE_URL=http://localhost:9550 # or
|
||||
export API_KEY=your_dashscope_key
|
||||
python app.py
|
||||
|
||||
Endpoints:
|
||||
/ - Control room web portal
|
||||
/monitor - Simplified broadcast monitor view
|
||||
/api/status - Service health check
|
||||
/broadcast/info - Integration guide for engineers
|
||||
|
||||
Part of the Zen AI model family: https://github.com/zenlm
|
||||
"
|
||||
import os
|
||||
import time
|
||||
import base64
|
||||
@@ -16,8 +34,10 @@ from pathlib import Path
|
||||
import gradio as gr
|
||||
import numpy as np
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse, JSONResponse
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel
|
||||
from fastrtc import (
|
||||
AdditionalOutputs,
|
||||
# AsyncStreamHandler,
|
||||
@@ -36,19 +56,32 @@ import cv2
|
||||
|
||||
load_dotenv()
|
||||
|
||||
os.environ["MODE"] = "UI"
|
||||
os.environ["MODE"] = os.environ.get("MODE", "")
|
||||
cur_dir = Path(__file__).parent
|
||||
|
||||
API_KEY = os.environ['API_KEY'] # Set with: export DASHSCOPE_API_KEY=xxx
|
||||
# Backend configuration
|
||||
HANZO_NODE_URL = os.environ.get("HANZO_NODE_URL") # Preferred: Hanzo Node backend
|
||||
API_KEY = os.environ.get("API_KEY") # Fallback: Direct DashScope API
|
||||
ZEN_OMNI_PATH = os.environ.get("ZEN_OMNI_PATH") # Optional: Local model path
|
||||
|
||||
API_URL = "wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime?model=qwen3-livetranslate-flash-realtime"
|
||||
VOICES = ["Cherry", "Nofish", "Jada", "Dylan", "Sunny", "Peter", "Kiki", "Eric"]
|
||||
|
||||
ssl_context = ssl.create_default_context(cafile=certifi.where())
|
||||
# ssl_context = ssl._create_unverified_context() # 禁用证书验证
|
||||
|
||||
if not API_KEY:
|
||||
raise RuntimeError("Missing DASHSCOPE_API_KEY environment variable.")
|
||||
headers = {"Authorization": "Bearer " + API_KEY}
|
||||
# Determine backend and get credentials
|
||||
if HANZO_NODE_URL:
|
||||
print(f"🔗 Zen Live: Using Hanzo Node backend at {HANZO_NODE_URL}")
|
||||
BACKEND_TYPE = "hanzo_node"
|
||||
elif API_KEY:
|
||||
print("🔗 Zen Live: Using direct DashScope API")
|
||||
BACKEND_TYPE = "dashscope"
|
||||
else:
|
||||
print("⚠️ Zen Live: No backend configured")
|
||||
print(" Set HANZO_NODE_URL (recommended) or API_KEY")
|
||||
BACKEND_TYPE = None
|
||||
|
||||
headers = {"Authorization": f"Bearer {API_KEY}"} if API_KEY else {}
|
||||
LANG_MAP = {
|
||||
"en": "English",
|
||||
"zh": "Chinese",
|
||||
@@ -73,7 +106,8 @@ LANG_MAP_REVERSE = {v: k for k, v in LANG_MAP.items()}
|
||||
# SRC_LANGUAGES = ["en", "zh", "ru", "fr", "de", "pt", "es", "it", "ko", "ja", "yue", "id", "vi", "th", "ar", "hi", "el", "tr"] # 使用相同的语言列表
|
||||
# TARGET_LANGUAGES = ["en", "zh", "ru", "fr", "de", "pt", "es", "it", "ko", "ja", "yue", "id", "vi", "th", "ar"]
|
||||
|
||||
SRC_LANGUAGES = [LANG_MAP[code] for code in ["en", "zh", "ru", "fr", "de", "pt", "es", "it", "ko", "ja", "yue", "id", "vi", "th", "ar", "hi", "el", "tr"]]
|
||||
# Spanish first for news monitoring default
|
||||
SRC_LANGUAGES = [LANG_MAP[code] for code in ["es", "en", "zh", "pt", "fr", "de", "ru", "it", "ko", "ja", "yue", "id", "vi", "th", "ar", "hi", "el", "tr"]]
|
||||
TARGET_LANGUAGES = [LANG_MAP[code] for code in ["en", "zh", "ru", "fr", "de", "pt", "es", "it", "ko", "ja", "yue", "id", "vi", "th", "ar"]]
|
||||
|
||||
|
||||
@@ -302,13 +336,13 @@ def update_chatbot(chatbot: list[dict], response: dict):
|
||||
chatbot = gr.Chatbot(type="messages")
|
||||
src_language = gr.Dropdown(
|
||||
choices=SRC_LANGUAGES,
|
||||
value="English", # 改成全称
|
||||
value="Spanish", # Default to Spanish for news monitoring
|
||||
type="value",
|
||||
label="Source Language"
|
||||
)
|
||||
language = gr.Dropdown(
|
||||
choices=TARGET_LANGUAGES,
|
||||
value="Chinese", # 改成全称
|
||||
value="English", # Default to English output
|
||||
type="value",
|
||||
label="Target Language"
|
||||
)
|
||||
@@ -396,11 +430,265 @@ def enhance_ui():
|
||||
""")
|
||||
return demo
|
||||
|
||||
app = FastAPI()
|
||||
app = FastAPI(
|
||||
title="Zen Live",
|
||||
description="Real-time speech translation for broadcast news monitoring. Powered by Hanzo AI.",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc"
|
||||
)
|
||||
|
||||
# CORS for control room access from different domains
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
stream.mount(app)
|
||||
|
||||
|
||||
# WebRTC offer model
|
||||
class WebRTCOffer(BaseModel):
|
||||
sdp: str
|
||||
type: str
|
||||
src_language: str = "Spanish"
|
||||
target_language: str = "English"
|
||||
voice: str = "Cherry"
|
||||
|
||||
|
||||
# Store active sessions for control room
|
||||
active_sessions = {}
|
||||
|
||||
|
||||
@app.post("/webrtc/offer")
|
||||
async def webrtc_offer(offer: WebRTCOffer):
|
||||
"""Handle WebRTC offer from control room portal.
|
||||
|
||||
This endpoint receives SDP offers and returns SDP answers
|
||||
for establishing low-latency WebRTC connections.
|
||||
"""
|
||||
try:
|
||||
# Generate session ID
|
||||
session_id = secrets.token_hex(16)
|
||||
|
||||
# Store session config
|
||||
active_sessions[session_id] = {
|
||||
"src_language": offer.src_language,
|
||||
"target_language": offer.target_language,
|
||||
"voice": offer.voice,
|
||||
"created_at": time.time()
|
||||
}
|
||||
|
||||
# Use FastRTC's internal offer handling
|
||||
response = await stream.offer(
|
||||
offer.sdp,
|
||||
offer.type,
|
||||
extra_data={
|
||||
"src_language": offer.src_language,
|
||||
"target_language": offer.target_language,
|
||||
"voice": offer.voice
|
||||
}
|
||||
)
|
||||
|
||||
return JSONResponse({
|
||||
"sdp": response["sdp"],
|
||||
"type": response["type"],
|
||||
"webrtc_id": response.get("webrtc_id", session_id)
|
||||
})
|
||||
except Exception as e:
|
||||
print(f"WebRTC offer error: {e}")
|
||||
return JSONResponse(
|
||||
{"error": str(e)},
|
||||
status_code=500
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/sessions")
|
||||
async def list_sessions():
|
||||
"""List active translation sessions for control room monitoring."""
|
||||
return JSONResponse({
|
||||
"sessions": [
|
||||
{
|
||||
"id": sid,
|
||||
"src_language": data["src_language"],
|
||||
"target_language": data["target_language"],
|
||||
"voice": data["voice"],
|
||||
"uptime": time.time() - data["created_at"]
|
||||
}
|
||||
for sid, data in active_sessions.items()
|
||||
]
|
||||
})
|
||||
|
||||
|
||||
@app.get("/api/status")
|
||||
async def api_status():
|
||||
"""API health check for control room monitoring."""
|
||||
return JSONResponse({
|
||||
"status": "healthy",
|
||||
"service": "zen-live-translate",
|
||||
"version": "1.0.0",
|
||||
"active_sessions": len(active_sessions),
|
||||
"supported_languages": {
|
||||
"source": SRC_LANGUAGES,
|
||||
"target": TARGET_LANGUAGES
|
||||
},
|
||||
"voices": VOICES
|
||||
})
|
||||
|
||||
|
||||
@app.get("/monitor")
|
||||
async def monitor_page():
|
||||
"""Simplified monitor-only view for control room displays."""
|
||||
rtc_config = await get_cloudflare_turn_credentials_async() if get_space() else None
|
||||
html_content = (cur_dir / "monitor.html").read_text() if (cur_dir / "monitor.html").exists() else (cur_dir / "index.html").read_text()
|
||||
html_content = html_content.replace("__RTC_CONFIGURATION__", json.dumps(rtc_config))
|
||||
return HTMLResponse(content=html_content)
|
||||
|
||||
|
||||
# Audio streaming for broadcast integration (can be ingested via NDI/SDI converters)
|
||||
audio_subscribers = {} # webrtc_id -> list of asyncio.Queue
|
||||
|
||||
|
||||
@app.get("/audio/stream/{webrtc_id}")
|
||||
async def audio_stream(webrtc_id: str):
|
||||
"""
|
||||
Raw PCM audio stream for broadcast integration.
|
||||
|
||||
This endpoint streams translated audio as raw PCM16 at 24kHz.
|
||||
Can be converted to SDI/NDI using tools like:
|
||||
- ffmpeg -f s16le -ar 24000 -ac 1 -i http://host/audio/stream/ID -f alsa default
|
||||
- OBS with browser source + audio capture
|
||||
- Blackmagic Web Presenter
|
||||
- NDI Tools with HTTP input
|
||||
|
||||
For SRT output, pipe through:
|
||||
ffmpeg -f s16le -ar 24000 -ac 1 -i http://host/audio/stream/ID \
|
||||
-c:a aac -f mpegts srt://dest:port
|
||||
"""
|
||||
async def generate_audio():
|
||||
queue = asyncio.Queue()
|
||||
if webrtc_id not in audio_subscribers:
|
||||
audio_subscribers[webrtc_id] = []
|
||||
audio_subscribers[webrtc_id].append(queue)
|
||||
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
audio_data = await asyncio.wait_for(queue.get(), timeout=30.0)
|
||||
yield audio_data
|
||||
except asyncio.TimeoutError:
|
||||
# Send silence to keep connection alive
|
||||
yield b'\x00' * 4800 # 100ms of silence at 24kHz
|
||||
finally:
|
||||
if webrtc_id in audio_subscribers:
|
||||
audio_subscribers[webrtc_id].remove(queue)
|
||||
|
||||
return StreamingResponse(
|
||||
generate_audio(),
|
||||
media_type="audio/pcm",
|
||||
headers={
|
||||
"Content-Type": "audio/L16;rate=24000;channels=1",
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Audio-Format": "PCM16 24kHz Mono",
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@app.get("/audio/wav/{webrtc_id}")
|
||||
async def audio_wav_stream(webrtc_id: str):
|
||||
"""
|
||||
WAV-wrapped audio stream for easier playback/ingestion.
|
||||
|
||||
Includes WAV header for compatibility with more players/converters.
|
||||
Useful for direct monitoring or conversion to broadcast formats.
|
||||
"""
|
||||
async def generate_wav():
|
||||
# Send WAV header for streaming (size = max)
|
||||
wav_header = bytes([
|
||||
0x52, 0x49, 0x46, 0x46, # "RIFF"
|
||||
0xFF, 0xFF, 0xFF, 0x7F, # Size (max for streaming)
|
||||
0x57, 0x41, 0x56, 0x45, # "WAVE"
|
||||
0x66, 0x6D, 0x74, 0x20, # "fmt "
|
||||
0x10, 0x00, 0x00, 0x00, # Subchunk1Size (16)
|
||||
0x01, 0x00, # AudioFormat (1 = PCM)
|
||||
0x01, 0x00, # NumChannels (1)
|
||||
0xC0, 0x5D, 0x00, 0x00, # SampleRate (24000)
|
||||
0x80, 0xBB, 0x00, 0x00, # ByteRate (48000)
|
||||
0x02, 0x00, # BlockAlign (2)
|
||||
0x10, 0x00, # BitsPerSample (16)
|
||||
0x64, 0x61, 0x74, 0x61, # "data"
|
||||
0xFF, 0xFF, 0xFF, 0x7F, # Subchunk2Size (max)
|
||||
])
|
||||
yield wav_header
|
||||
|
||||
queue = asyncio.Queue()
|
||||
if webrtc_id not in audio_subscribers:
|
||||
audio_subscribers[webrtc_id] = []
|
||||
audio_subscribers[webrtc_id].append(queue)
|
||||
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
audio_data = await asyncio.wait_for(queue.get(), timeout=30.0)
|
||||
yield audio_data
|
||||
except asyncio.TimeoutError:
|
||||
yield b'\x00' * 4800
|
||||
finally:
|
||||
if webrtc_id in audio_subscribers:
|
||||
audio_subscribers[webrtc_id].remove(queue)
|
||||
|
||||
return StreamingResponse(
|
||||
generate_wav(),
|
||||
media_type="audio/wav",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@app.get("/broadcast/info")
|
||||
async def broadcast_info():
|
||||
"""
|
||||
Information for broadcast engineers on how to integrate.
|
||||
"""
|
||||
base_url = "http://YOUR_HOST:8000"
|
||||
return JSONResponse({
|
||||
"service": "Zen Live Translate - Broadcast Integration",
|
||||
"endpoints": {
|
||||
"control_room_ui": f"{base_url}/",
|
||||
"monitor_only": f"{base_url}/monitor?autostart=1",
|
||||
"webrtc_offer": f"{base_url}/webrtc/offer",
|
||||
"transcript_sse": f"{base_url}/outputs?webrtc_id=SESSION_ID",
|
||||
"audio_pcm": f"{base_url}/audio/stream/SESSION_ID",
|
||||
"audio_wav": f"{base_url}/audio/wav/SESSION_ID",
|
||||
"api_status": f"{base_url}/api/status",
|
||||
"api_sessions": f"{base_url}/api/sessions"
|
||||
},
|
||||
"audio_format": {
|
||||
"encoding": "PCM16 signed little-endian",
|
||||
"sample_rate": 24000,
|
||||
"channels": 1,
|
||||
"bits_per_sample": 16
|
||||
},
|
||||
"integration_examples": {
|
||||
"ffplay_direct": "ffplay -f s16le -ar 24000 -ac 1 http://host/audio/stream/ID",
|
||||
"ffmpeg_to_srt": "ffmpeg -f s16le -ar 24000 -ac 1 -i http://host/audio/stream/ID -c:a aac -f mpegts 'srt://dest:port'",
|
||||
"ffmpeg_to_rtmp": "ffmpeg -f s16le -ar 24000 -ac 1 -i http://host/audio/stream/ID -c:a aac -f flv rtmp://dest/live/stream",
|
||||
"obs_browser": "Add Browser Source with URL: http://host/monitor?autostart=1",
|
||||
"vlc": "vlc http://host/audio/wav/SESSION_ID"
|
||||
},
|
||||
"latency": {
|
||||
"typical": "200-500ms end-to-end",
|
||||
"factors": ["network RTT", "AI processing", "TTS generation"]
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def _():
|
||||
rtc_config = await get_cloudflare_turn_credentials_async() if get_space() else None
|
||||
@@ -422,24 +710,36 @@ def _(webrtc_id: str):
|
||||
|
||||
|
||||
def handle_exit(sig, frame):
|
||||
print("Shutting down gracefully...")
|
||||
print("\n👋 Zen Live shutting down...")
|
||||
exit(0)
|
||||
# 可扩展为执行更多清理逻辑
|
||||
|
||||
|
||||
signal.signal(signal.SIGINT, handle_exit)
|
||||
signal.signal(signal.SIGTERM, handle_exit)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import os
|
||||
import uvicorn
|
||||
|
||||
if (mode := os.getenv("MODE")) == "UI":
|
||||
mode = os.getenv("MODE", "").upper()
|
||||
port = int(os.getenv("PORT", 8000))
|
||||
|
||||
print("""
|
||||
╔═══════════════════════════════════════════════════════════╗
|
||||
║ ZEN LIVE ║
|
||||
║ Real-time Speech Translation Service ║
|
||||
║ Powered by Hanzo AI ║
|
||||
╠═══════════════════════════════════════════════════════════╣
|
||||
║ Control Room: http://localhost:{port:<5} ║
|
||||
║ Monitor View: http://localhost:{port}/monitor ║
|
||||
║ API Docs: http://localhost:{port}/docs ║
|
||||
║ Broadcast: http://localhost:{port}/broadcast/info ║
|
||||
╚═══════════════════════════════════════════════════════════╝
|
||||
""".format(port=port))
|
||||
|
||||
if mode == "UI":
|
||||
demo = enhance_ui()
|
||||
demo.launch()
|
||||
# stream.ui.launch(server_port=7862)
|
||||
demo.launch(server_port=port)
|
||||
elif mode == "PHONE":
|
||||
stream.fastphone(host="0.0.0.0")
|
||||
stream.fastphone(host="0.0.0.0", port=port)
|
||||
else:
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(app, host="0.0.0.0")
|
||||
uvicorn.run(app, host="0.0.0.0", port=port)
|
||||
+870
@@ -0,0 +1,870 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Zen Live - Control Room</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg-dark: #1a1a2e;
|
||||
--bg-panel: #16213e;
|
||||
--accent: #0f3460;
|
||||
--highlight: #e94560;
|
||||
--text: #eee;
|
||||
--text-muted: #888;
|
||||
--success: #00d26a;
|
||||
--warning: #ffc107;
|
||||
}
|
||||
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: var(--bg-dark);
|
||||
color: var(--text);
|
||||
min-height: 100vh;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
.header {
|
||||
background: var(--bg-panel);
|
||||
padding: 12px 24px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
border-bottom: 2px solid var(--accent);
|
||||
}
|
||||
|
||||
.logo {
|
||||
font-size: 1.5rem;
|
||||
font-weight: 700;
|
||||
color: var(--highlight);
|
||||
}
|
||||
|
||||
.status-bar {
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.status-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
border-radius: 50%;
|
||||
background: var(--text-muted);
|
||||
}
|
||||
|
||||
.status-dot.live {
|
||||
background: var(--success);
|
||||
animation: pulse 1.5s infinite;
|
||||
}
|
||||
|
||||
.status-dot.warning {
|
||||
background: var(--warning);
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.5; }
|
||||
}
|
||||
|
||||
.main-container {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 400px;
|
||||
height: calc(100vh - 60px);
|
||||
gap: 0;
|
||||
}
|
||||
|
||||
.video-section {
|
||||
position: relative;
|
||||
background: #000;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
#videoElement {
|
||||
max-width: 100%;
|
||||
max-height: 100%;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
.video-overlay {
|
||||
position: absolute;
|
||||
bottom: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
background: linear-gradient(transparent, rgba(0,0,0,0.8));
|
||||
padding: 40px 24px 24px;
|
||||
}
|
||||
|
||||
.live-caption {
|
||||
font-size: 1.4rem;
|
||||
line-height: 1.6;
|
||||
text-shadow: 2px 2px 4px rgba(0,0,0,0.8);
|
||||
min-height: 80px;
|
||||
}
|
||||
|
||||
.live-caption .stable {
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.live-caption .interim {
|
||||
color: #aaa;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
background: var(--bg-panel);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
border-left: 2px solid var(--accent);
|
||||
}
|
||||
|
||||
.sidebar-header {
|
||||
padding: 16px;
|
||||
border-bottom: 1px solid var(--accent);
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
font-size: 0.85rem;
|
||||
letter-spacing: 1px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.controls-section {
|
||||
padding: 16px;
|
||||
border-bottom: 1px solid var(--accent);
|
||||
}
|
||||
|
||||
.control-group {
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.control-group:last-child {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
label {
|
||||
display: block;
|
||||
font-size: 0.8rem;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 6px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
select, input {
|
||||
width: 100%;
|
||||
padding: 10px 12px;
|
||||
background: var(--bg-dark);
|
||||
border: 1px solid var(--accent);
|
||||
border-radius: 6px;
|
||||
color: var(--text);
|
||||
font-size: 0.95rem;
|
||||
}
|
||||
|
||||
select:focus, input:focus {
|
||||
outline: none;
|
||||
border-color: var(--highlight);
|
||||
}
|
||||
|
||||
.btn-group {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
margin-top: 16px;
|
||||
}
|
||||
|
||||
.btn {
|
||||
flex: 1;
|
||||
padding: 12px 16px;
|
||||
border: none;
|
||||
border-radius: 6px;
|
||||
font-size: 0.9rem;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
.btn-primary {
|
||||
background: var(--highlight);
|
||||
color: white;
|
||||
}
|
||||
|
||||
.btn-primary:hover {
|
||||
background: #ff5a75;
|
||||
}
|
||||
|
||||
.btn-primary:disabled {
|
||||
background: var(--text-muted);
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.btn-secondary {
|
||||
background: var(--accent);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.btn-secondary:hover {
|
||||
background: #1a4080;
|
||||
}
|
||||
|
||||
.btn-stop {
|
||||
background: #dc3545;
|
||||
color: white;
|
||||
}
|
||||
|
||||
.transcript-section {
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.transcript-log {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
.transcript-entry {
|
||||
padding: 12px;
|
||||
margin-bottom: 8px;
|
||||
background: var(--bg-dark);
|
||||
border-radius: 8px;
|
||||
border-left: 3px solid var(--highlight);
|
||||
}
|
||||
|
||||
.transcript-entry .time {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.transcript-entry .text {
|
||||
font-size: 0.95rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.audio-monitor {
|
||||
padding: 16px;
|
||||
border-top: 1px solid var(--accent);
|
||||
background: var(--bg-dark);
|
||||
}
|
||||
|
||||
.audio-level {
|
||||
height: 8px;
|
||||
background: var(--accent);
|
||||
border-radius: 4px;
|
||||
overflow: hidden;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.audio-level-bar {
|
||||
height: 100%;
|
||||
width: 0%;
|
||||
background: linear-gradient(90deg, var(--success), var(--warning), var(--highlight));
|
||||
transition: width 0.1s;
|
||||
}
|
||||
|
||||
.latency-display {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
margin-top: 12px;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.latency-value {
|
||||
color: var(--success);
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.latency-value.high {
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.output-endpoints {
|
||||
padding: 16px;
|
||||
border-top: 1px solid var(--accent);
|
||||
}
|
||||
|
||||
.endpoint-item {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 8px 0;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.endpoint-url {
|
||||
font-family: monospace;
|
||||
color: var(--text-muted);
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.copy-btn {
|
||||
background: var(--accent);
|
||||
border: none;
|
||||
color: var(--text);
|
||||
padding: 4px 8px;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.fullscreen-toggle {
|
||||
position: absolute;
|
||||
top: 16px;
|
||||
right: 16px;
|
||||
background: rgba(0,0,0,0.6);
|
||||
border: none;
|
||||
color: white;
|
||||
padding: 8px 12px;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
/* Monitor-only mode */
|
||||
body.monitor-mode .sidebar {
|
||||
display: none;
|
||||
}
|
||||
|
||||
body.monitor-mode .main-container {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
body.monitor-mode .header {
|
||||
padding: 8px 16px;
|
||||
}
|
||||
|
||||
/* Audio output indicator */
|
||||
.audio-output-indicator {
|
||||
position: fixed;
|
||||
bottom: 20px;
|
||||
left: 20px;
|
||||
background: var(--bg-panel);
|
||||
padding: 12px 20px;
|
||||
border-radius: 8px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
border: 1px solid var(--accent);
|
||||
z-index: 1000;
|
||||
}
|
||||
|
||||
.speaker-icon {
|
||||
font-size: 1.5rem;
|
||||
}
|
||||
|
||||
.speaker-icon.active {
|
||||
animation: speaker-pulse 0.5s infinite alternate;
|
||||
}
|
||||
|
||||
@keyframes speaker-pulse {
|
||||
from { transform: scale(1); }
|
||||
to { transform: scale(1.1); }
|
||||
}
|
||||
|
||||
/* Hidden audio element for TTS playback */
|
||||
#audioOutput {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* Responsive */
|
||||
@media (max-width: 1024px) {
|
||||
.main-container {
|
||||
grid-template-columns: 1fr;
|
||||
grid-template-rows: 1fr 1fr;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
border-left: none;
|
||||
border-top: 2px solid var(--accent);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="logo">ZEN LIVE</div>
|
||||
<div class="status-bar">
|
||||
<div class="status-item">
|
||||
<div class="status-dot" id="connectionStatus"></div>
|
||||
<span id="connectionText">Disconnected</span>
|
||||
</div>
|
||||
<div class="status-item">
|
||||
<div class="status-dot" id="audioStatus"></div>
|
||||
<span>Audio</span>
|
||||
</div>
|
||||
<div class="status-item">
|
||||
<span id="latencyDisplay">Latency: --ms</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="main-container">
|
||||
<div class="video-section">
|
||||
<video id="videoElement" autoplay muted playsinline></video>
|
||||
<button class="fullscreen-toggle" onclick="toggleFullscreen()">Fullscreen</button>
|
||||
<div class="video-overlay">
|
||||
<div class="live-caption" id="liveCaption">
|
||||
<span class="stable"></span>
|
||||
<span class="interim"></span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="sidebar">
|
||||
<div class="sidebar-header">Control Panel</div>
|
||||
|
||||
<div class="controls-section">
|
||||
<div class="control-group">
|
||||
<label>Source Language</label>
|
||||
<select id="srcLanguage">
|
||||
<option value="Spanish" selected>Spanish (Espanol)</option>
|
||||
<option value="English">English</option>
|
||||
<option value="Chinese">Chinese (Mandarin)</option>
|
||||
<option value="Portuguese">Portuguese</option>
|
||||
<option value="French">French</option>
|
||||
<option value="German">German</option>
|
||||
<option value="Italian">Italian</option>
|
||||
<option value="Russian">Russian</option>
|
||||
<option value="Arabic">Arabic</option>
|
||||
<option value="Japanese">Japanese</option>
|
||||
<option value="Korean">Korean</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="control-group">
|
||||
<label>Target Language</label>
|
||||
<select id="targetLanguage">
|
||||
<option value="English" selected>English</option>
|
||||
<option value="Spanish">Spanish (Espanol)</option>
|
||||
<option value="Chinese">Chinese (Mandarin)</option>
|
||||
<option value="Portuguese">Portuguese</option>
|
||||
<option value="French">French</option>
|
||||
<option value="German">German</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="control-group">
|
||||
<label>Voice</label>
|
||||
<select id="voiceSelect">
|
||||
<option value="Cherry">Cherry (Female)</option>
|
||||
<option value="Dylan">Dylan (Male)</option>
|
||||
<option value="Jada">Jada (Female)</option>
|
||||
<option value="Peter">Peter (Male)</option>
|
||||
<option value="Sunny">Sunny (Female)</option>
|
||||
<option value="Eric">Eric (Male)</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="control-group">
|
||||
<label>Input Source</label>
|
||||
<select id="inputSource">
|
||||
<option value="camera">Camera/Webcam</option>
|
||||
<option value="screen">Screen Capture</option>
|
||||
<option value="srt">SRT Input (URL)</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="control-group" id="srtInputGroup" style="display:none;">
|
||||
<label>SRT Input URL</label>
|
||||
<input type="text" id="srtUrl" placeholder="srt://ip:port?streamid=...">
|
||||
</div>
|
||||
|
||||
<div class="btn-group">
|
||||
<button class="btn btn-primary" id="startBtn" onclick="startStream()">
|
||||
Start
|
||||
</button>
|
||||
<button class="btn btn-stop" id="stopBtn" onclick="stopStream()" disabled>
|
||||
Stop
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="sidebar-header">Transcript</div>
|
||||
<div class="transcript-section">
|
||||
<div class="transcript-log" id="transcriptLog"></div>
|
||||
</div>
|
||||
|
||||
<div class="audio-monitor">
|
||||
<label>Audio Output Level</label>
|
||||
<div class="audio-level">
|
||||
<div class="audio-level-bar" id="audioLevelBar"></div>
|
||||
</div>
|
||||
<div class="latency-display">
|
||||
<span>End-to-end latency:</span>
|
||||
<span class="latency-value" id="e2eLatency">--ms</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="output-endpoints">
|
||||
<div class="sidebar-header" style="padding: 0 0 12px 0;">Output Endpoints</div>
|
||||
<div class="endpoint-item">
|
||||
<div>
|
||||
<div>WebRTC Audio</div>
|
||||
<div class="endpoint-url" id="webrtcUrl">--</div>
|
||||
</div>
|
||||
<button class="copy-btn" onclick="copyEndpoint('webrtc')">Copy</button>
|
||||
</div>
|
||||
<div class="endpoint-item">
|
||||
<div>
|
||||
<div>Audio Stream (SSE)</div>
|
||||
<div class="endpoint-url" id="sseUrl">--</div>
|
||||
</div>
|
||||
<button class="copy-btn" onclick="copyEndpoint('sse')">Copy</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="audio-output-indicator" id="audioIndicator" style="display:none;">
|
||||
<span class="speaker-icon" id="speakerIcon">🔊</span>
|
||||
<span>Translated audio playing</span>
|
||||
</div>
|
||||
|
||||
<audio id="audioOutput"></audio>
|
||||
|
||||
<script type="module">
|
||||
const RTC_CONFIG = __RTC_CONFIGURATION__;
|
||||
|
||||
let peerConnection = null;
|
||||
let webrtcId = null;
|
||||
let mediaStream = null;
|
||||
let audioContext = null;
|
||||
let audioAnalyser = null;
|
||||
let isStreaming = false;
|
||||
let latencyStart = 0;
|
||||
|
||||
const videoElement = document.getElementById('videoElement');
|
||||
const startBtn = document.getElementById('startBtn');
|
||||
const stopBtn = document.getElementById('stopBtn');
|
||||
const connectionStatus = document.getElementById('connectionStatus');
|
||||
const connectionText = document.getElementById('connectionText');
|
||||
const audioStatus = document.getElementById('audioStatus');
|
||||
const liveCaption = document.getElementById('liveCaption');
|
||||
const transcriptLog = document.getElementById('transcriptLog');
|
||||
const audioLevelBar = document.getElementById('audioLevelBar');
|
||||
const e2eLatency = document.getElementById('e2eLatency');
|
||||
const audioIndicator = document.getElementById('audioIndicator');
|
||||
const speakerIcon = document.getElementById('speakerIcon');
|
||||
const inputSource = document.getElementById('inputSource');
|
||||
const srtInputGroup = document.getElementById('srtInputGroup');
|
||||
|
||||
// Update URLs display
|
||||
const baseUrl = window.location.origin;
|
||||
document.getElementById('webrtcUrl').textContent = `${baseUrl}/webrtc`;
|
||||
document.getElementById('sseUrl').textContent = `${baseUrl}/outputs?webrtc_id=...`;
|
||||
|
||||
// Input source toggle
|
||||
inputSource.addEventListener('change', (e) => {
|
||||
srtInputGroup.style.display = e.target.value === 'srt' ? 'block' : 'none';
|
||||
});
|
||||
|
||||
async function startStream() {
|
||||
try {
|
||||
startBtn.disabled = true;
|
||||
updateStatus('connecting');
|
||||
|
||||
const sourceType = inputSource.value;
|
||||
|
||||
// Get media based on source type
|
||||
if (sourceType === 'camera') {
|
||||
mediaStream = await navigator.mediaDevices.getUserMedia({
|
||||
video: { width: 1280, height: 720, facingMode: 'environment' },
|
||||
audio: {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
sampleRate: 16000
|
||||
}
|
||||
});
|
||||
} else if (sourceType === 'screen') {
|
||||
mediaStream = await navigator.mediaDevices.getDisplayMedia({
|
||||
video: { width: 1920, height: 1080 },
|
||||
audio: true
|
||||
});
|
||||
}
|
||||
|
||||
videoElement.srcObject = mediaStream;
|
||||
|
||||
// Setup WebRTC
|
||||
await setupWebRTC();
|
||||
|
||||
} catch (err) {
|
||||
console.error('Error starting stream:', err);
|
||||
updateStatus('error');
|
||||
startBtn.disabled = false;
|
||||
alert('Failed to start: ' + err.message);
|
||||
}
|
||||
}
|
||||
|
||||
async function setupWebRTC() {
|
||||
const config = RTC_CONFIG || {
|
||||
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
|
||||
};
|
||||
|
||||
peerConnection = new RTCPeerConnection(config);
|
||||
|
||||
// Add tracks
|
||||
mediaStream.getTracks().forEach(track => {
|
||||
peerConnection.addTrack(track, mediaStream);
|
||||
});
|
||||
|
||||
// Handle incoming audio (translated TTS)
|
||||
peerConnection.ontrack = (event) => {
|
||||
console.log('Received remote track:', event.track.kind);
|
||||
if (event.track.kind === 'audio') {
|
||||
const audio = document.getElementById('audioOutput');
|
||||
audio.srcObject = event.streams[0];
|
||||
audio.play().catch(e => console.log('Audio play error:', e));
|
||||
|
||||
audioIndicator.style.display = 'flex';
|
||||
audioStatus.classList.add('live');
|
||||
|
||||
// Setup audio visualization
|
||||
setupAudioVisualization(event.streams[0]);
|
||||
}
|
||||
};
|
||||
|
||||
// ICE handling
|
||||
peerConnection.onicecandidate = (event) => {
|
||||
if (event.candidate) {
|
||||
// Send candidate to server
|
||||
}
|
||||
};
|
||||
|
||||
peerConnection.onconnectionstatechange = () => {
|
||||
console.log('Connection state:', peerConnection.connectionState);
|
||||
if (peerConnection.connectionState === 'connected') {
|
||||
updateStatus('live');
|
||||
isStreaming = true;
|
||||
stopBtn.disabled = false;
|
||||
latencyStart = Date.now();
|
||||
} else if (peerConnection.connectionState === 'failed') {
|
||||
updateStatus('error');
|
||||
}
|
||||
};
|
||||
|
||||
// Create offer
|
||||
const offer = await peerConnection.createOffer();
|
||||
await peerConnection.setLocalDescription(offer);
|
||||
|
||||
// Wait for ICE gathering
|
||||
await new Promise((resolve) => {
|
||||
if (peerConnection.iceGatheringState === 'complete') {
|
||||
resolve();
|
||||
} else {
|
||||
peerConnection.onicegatheringstatechange = () => {
|
||||
if (peerConnection.iceGatheringState === 'complete') resolve();
|
||||
};
|
||||
// Timeout fallback
|
||||
setTimeout(resolve, 2000);
|
||||
}
|
||||
});
|
||||
|
||||
// Send to server
|
||||
const srcLang = document.getElementById('srcLanguage').value;
|
||||
const targetLang = document.getElementById('targetLanguage').value;
|
||||
const voice = document.getElementById('voiceSelect').value;
|
||||
|
||||
const response = await fetch('/webrtc/offer', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
sdp: peerConnection.localDescription.sdp,
|
||||
type: peerConnection.localDescription.type,
|
||||
src_language: srcLang,
|
||||
target_language: targetLang,
|
||||
voice: voice
|
||||
})
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
webrtcId = data.webrtc_id;
|
||||
|
||||
// Set remote description
|
||||
await peerConnection.setRemoteDescription(new RTCSessionDescription({
|
||||
sdp: data.sdp,
|
||||
type: data.type
|
||||
}));
|
||||
|
||||
// Start listening for outputs (transcripts)
|
||||
startOutputListener();
|
||||
}
|
||||
|
||||
function startOutputListener() {
|
||||
if (!webrtcId) return;
|
||||
|
||||
const evtSource = new EventSource(`/outputs?webrtc_id=${webrtcId}`);
|
||||
|
||||
evtSource.addEventListener('output', (e) => {
|
||||
const data = JSON.parse(e.data);
|
||||
updateTranscript(data);
|
||||
|
||||
// Update latency display
|
||||
const now = Date.now();
|
||||
const latency = now - latencyStart;
|
||||
if (latency < 5000) {
|
||||
e2eLatency.textContent = `${latency}ms`;
|
||||
e2eLatency.className = latency > 500 ? 'latency-value high' : 'latency-value';
|
||||
}
|
||||
});
|
||||
|
||||
evtSource.onerror = () => {
|
||||
console.log('SSE connection error, reconnecting...');
|
||||
};
|
||||
}
|
||||
|
||||
function updateTranscript(data) {
|
||||
const content = data.content || '';
|
||||
|
||||
// Parse HTML content for stable/interim text
|
||||
const stableMatch = content.match(/<span style='color:black'>([^<]*)<\/span>/);
|
||||
const interimMatch = content.match(/<span style='color:gray'>([^<]*)<\/span>/);
|
||||
|
||||
const stableText = stableMatch ? stableMatch[1] : '';
|
||||
const interimText = interimMatch ? interimMatch[1] : '';
|
||||
|
||||
// Update live caption
|
||||
liveCaption.querySelector('.stable').textContent = stableText;
|
||||
liveCaption.querySelector('.interim').textContent = interimText;
|
||||
|
||||
// Add to transcript log if message is complete
|
||||
if (data.new_message === false && stableText && !interimText) {
|
||||
addTranscriptEntry(stableText);
|
||||
}
|
||||
|
||||
// Reset latency timer for next segment
|
||||
latencyStart = Date.now();
|
||||
}
|
||||
|
||||
function addTranscriptEntry(text) {
|
||||
const entry = document.createElement('div');
|
||||
entry.className = 'transcript-entry';
|
||||
entry.innerHTML = `
|
||||
<div class="time">${new Date().toLocaleTimeString()}</div>
|
||||
<div class="text">${text}</div>
|
||||
`;
|
||||
transcriptLog.appendChild(entry);
|
||||
transcriptLog.scrollTop = transcriptLog.scrollHeight;
|
||||
}
|
||||
|
||||
function setupAudioVisualization(stream) {
|
||||
audioContext = new AudioContext();
|
||||
audioAnalyser = audioContext.createAnalyser();
|
||||
const source = audioContext.createMediaStreamSource(stream);
|
||||
source.connect(audioAnalyser);
|
||||
|
||||
audioAnalyser.fftSize = 256;
|
||||
const bufferLength = audioAnalyser.frequencyBinCount;
|
||||
const dataArray = new Uint8Array(bufferLength);
|
||||
|
||||
function updateLevel() {
|
||||
if (!isStreaming) return;
|
||||
|
||||
audioAnalyser.getByteFrequencyData(dataArray);
|
||||
const average = dataArray.reduce((a, b) => a + b) / bufferLength;
|
||||
const level = Math.min(100, (average / 128) * 100);
|
||||
|
||||
audioLevelBar.style.width = `${level}%`;
|
||||
speakerIcon.className = level > 10 ? 'speaker-icon active' : 'speaker-icon';
|
||||
|
||||
requestAnimationFrame(updateLevel);
|
||||
}
|
||||
|
||||
updateLevel();
|
||||
}
|
||||
|
||||
function stopStream() {
|
||||
isStreaming = false;
|
||||
|
||||
if (mediaStream) {
|
||||
mediaStream.getTracks().forEach(track => track.stop());
|
||||
mediaStream = null;
|
||||
}
|
||||
|
||||
if (peerConnection) {
|
||||
peerConnection.close();
|
||||
peerConnection = null;
|
||||
}
|
||||
|
||||
if (audioContext) {
|
||||
audioContext.close();
|
||||
audioContext = null;
|
||||
}
|
||||
|
||||
videoElement.srcObject = null;
|
||||
audioIndicator.style.display = 'none';
|
||||
|
||||
updateStatus('disconnected');
|
||||
startBtn.disabled = false;
|
||||
stopBtn.disabled = true;
|
||||
}
|
||||
|
||||
function updateStatus(status) {
|
||||
connectionStatus.className = 'status-dot';
|
||||
audioStatus.className = 'status-dot';
|
||||
|
||||
switch (status) {
|
||||
case 'connecting':
|
||||
connectionText.textContent = 'Connecting...';
|
||||
connectionStatus.classList.add('warning');
|
||||
break;
|
||||
case 'live':
|
||||
connectionText.textContent = 'LIVE';
|
||||
connectionStatus.classList.add('live');
|
||||
break;
|
||||
case 'error':
|
||||
connectionText.textContent = 'Error';
|
||||
break;
|
||||
default:
|
||||
connectionText.textContent = 'Disconnected';
|
||||
}
|
||||
}
|
||||
|
||||
window.toggleFullscreen = function() {
|
||||
const videoSection = document.querySelector('.video-section');
|
||||
if (document.fullscreenElement) {
|
||||
document.exitFullscreen();
|
||||
} else {
|
||||
videoSection.requestFullscreen();
|
||||
}
|
||||
};
|
||||
|
||||
window.copyEndpoint = function(type) {
|
||||
const url = type === 'webrtc'
|
||||
? document.getElementById('webrtcUrl').textContent
|
||||
: document.getElementById('sseUrl').textContent.replace('...', webrtcId || 'SESSION_ID');
|
||||
navigator.clipboard.writeText(url);
|
||||
alert('Copied to clipboard!');
|
||||
};
|
||||
|
||||
window.startStream = startStream;
|
||||
window.stopStream = stopStream;
|
||||
|
||||
// Keyboard shortcuts for control room
|
||||
document.addEventListener('keydown', (e) => {
|
||||
if (e.key === 'F11') {
|
||||
e.preventDefault();
|
||||
toggleFullscreen();
|
||||
}
|
||||
if (e.key === 'm' && e.ctrlKey) {
|
||||
document.body.classList.toggle('monitor-mode');
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+499
@@ -0,0 +1,499 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Zen Live Monitor - Spanish News Feed</title>
|
||||
<style>
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: #000;
|
||||
color: #fff;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.monitor-container {
|
||||
position: relative;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
#videoElement {
|
||||
max-width: 100%;
|
||||
max-height: 100%;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
/* Status bar - minimal */
|
||||
.status-bar {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
padding: 12px 20px;
|
||||
background: linear-gradient(rgba(0,0,0,0.8), transparent);
|
||||
font-size: 0.85rem;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.status-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.live-badge {
|
||||
background: #e94560;
|
||||
padding: 4px 12px;
|
||||
border-radius: 4px;
|
||||
font-weight: 700;
|
||||
font-size: 0.75rem;
|
||||
letter-spacing: 1px;
|
||||
animation: live-pulse 2s infinite;
|
||||
}
|
||||
|
||||
@keyframes live-pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.7; }
|
||||
}
|
||||
|
||||
.language-badge {
|
||||
background: rgba(255,255,255,0.2);
|
||||
padding: 4px 10px;
|
||||
border-radius: 4px;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.status-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 20px;
|
||||
color: #aaa;
|
||||
}
|
||||
|
||||
.latency {
|
||||
font-family: monospace;
|
||||
}
|
||||
|
||||
.latency.good { color: #00d26a; }
|
||||
.latency.warn { color: #ffc107; }
|
||||
.latency.bad { color: #e94560; }
|
||||
|
||||
/* Caption overlay */
|
||||
.caption-overlay {
|
||||
position: absolute;
|
||||
bottom: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
padding: 60px 40px 40px;
|
||||
background: linear-gradient(transparent, rgba(0,0,0,0.9));
|
||||
}
|
||||
|
||||
.caption-text {
|
||||
font-size: 2rem;
|
||||
line-height: 1.5;
|
||||
text-shadow: 2px 2px 8px rgba(0,0,0,0.9);
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
.caption-text .stable {
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.caption-text .interim {
|
||||
color: #888;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
/* Audio indicator */
|
||||
.audio-indicator {
|
||||
position: absolute;
|
||||
bottom: 20px;
|
||||
right: 20px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
background: rgba(0,0,0,0.6);
|
||||
padding: 8px 16px;
|
||||
border-radius: 20px;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.audio-bars {
|
||||
display: flex;
|
||||
gap: 2px;
|
||||
height: 20px;
|
||||
align-items: flex-end;
|
||||
}
|
||||
|
||||
.audio-bar {
|
||||
width: 3px;
|
||||
background: #00d26a;
|
||||
border-radius: 2px;
|
||||
animation: audio-wave 0.5s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.audio-bar:nth-child(1) { animation-delay: 0s; }
|
||||
.audio-bar:nth-child(2) { animation-delay: 0.1s; }
|
||||
.audio-bar:nth-child(3) { animation-delay: 0.2s; }
|
||||
.audio-bar:nth-child(4) { animation-delay: 0.3s; }
|
||||
.audio-bar:nth-child(5) { animation-delay: 0.4s; }
|
||||
|
||||
@keyframes audio-wave {
|
||||
0%, 100% { height: 4px; }
|
||||
50% { height: 20px; }
|
||||
}
|
||||
|
||||
.audio-bars.silent .audio-bar {
|
||||
animation: none;
|
||||
height: 4px;
|
||||
background: #666;
|
||||
}
|
||||
|
||||
/* Connection overlay */
|
||||
.connection-overlay {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
bottom: 0;
|
||||
background: rgba(0,0,0,0.9);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 24px;
|
||||
z-index: 200;
|
||||
}
|
||||
|
||||
.connection-overlay.hidden {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.connection-overlay h2 {
|
||||
font-size: 1.5rem;
|
||||
font-weight: 400;
|
||||
}
|
||||
|
||||
.spinner {
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
border: 4px solid #333;
|
||||
border-top-color: #e94560;
|
||||
border-radius: 50%;
|
||||
animation: spin 1s linear infinite;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to { transform: rotate(360deg); }
|
||||
}
|
||||
|
||||
.start-btn {
|
||||
background: #e94560;
|
||||
color: white;
|
||||
border: none;
|
||||
padding: 16px 48px;
|
||||
font-size: 1.1rem;
|
||||
font-weight: 600;
|
||||
border-radius: 8px;
|
||||
cursor: pointer;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 1px;
|
||||
}
|
||||
|
||||
.start-btn:hover {
|
||||
background: #ff5a75;
|
||||
}
|
||||
|
||||
/* Quick settings popup */
|
||||
.settings-popup {
|
||||
position: absolute;
|
||||
top: 60px;
|
||||
right: 20px;
|
||||
background: rgba(30, 30, 50, 0.95);
|
||||
padding: 20px;
|
||||
border-radius: 12px;
|
||||
z-index: 150;
|
||||
min-width: 250px;
|
||||
display: none;
|
||||
}
|
||||
|
||||
.settings-popup.visible {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.settings-popup label {
|
||||
display: block;
|
||||
margin-bottom: 6px;
|
||||
font-size: 0.8rem;
|
||||
color: #888;
|
||||
}
|
||||
|
||||
.settings-popup select {
|
||||
width: 100%;
|
||||
padding: 8px;
|
||||
margin-bottom: 16px;
|
||||
background: #1a1a2e;
|
||||
border: 1px solid #333;
|
||||
color: white;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.settings-btn {
|
||||
background: rgba(255,255,255,0.1);
|
||||
border: none;
|
||||
color: #888;
|
||||
padding: 6px 12px;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.settings-btn:hover {
|
||||
background: rgba(255,255,255,0.2);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
/* Hidden audio */
|
||||
#audioOutput {
|
||||
display: none;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="monitor-container">
|
||||
<video id="videoElement" autoplay muted playsinline></video>
|
||||
|
||||
<div class="status-bar">
|
||||
<div class="status-left">
|
||||
<span class="live-badge" id="liveBadge" style="display:none;">LIVE</span>
|
||||
<span class="language-badge" id="langBadge">ES → EN</span>
|
||||
</div>
|
||||
<div class="status-right">
|
||||
<span class="latency" id="latencyDisplay">--ms</span>
|
||||
<span id="timeDisplay"></span>
|
||||
<button class="settings-btn" onclick="toggleSettings()">Settings</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="caption-overlay">
|
||||
<div class="caption-text" id="captionText">
|
||||
<span class="stable"></span>
|
||||
<span class="interim"></span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="audio-indicator" id="audioIndicator" style="display:none;">
|
||||
<div class="audio-bars" id="audioBars">
|
||||
<div class="audio-bar"></div>
|
||||
<div class="audio-bar"></div>
|
||||
<div class="audio-bar"></div>
|
||||
<div class="audio-bar"></div>
|
||||
<div class="audio-bar"></div>
|
||||
</div>
|
||||
<span>Translated Audio</span>
|
||||
</div>
|
||||
|
||||
<div class="connection-overlay" id="connectionOverlay">
|
||||
<h2>Zen Live Monitor</h2>
|
||||
<p>Spanish → English Real-time Translation</p>
|
||||
<button class="start-btn" onclick="startMonitor()">Start Monitor</button>
|
||||
</div>
|
||||
|
||||
<div class="settings-popup" id="settingsPopup">
|
||||
<label>Source Language</label>
|
||||
<select id="srcLang">
|
||||
<option value="Spanish" selected>Spanish</option>
|
||||
<option value="English">English</option>
|
||||
<option value="Portuguese">Portuguese</option>
|
||||
<option value="French">French</option>
|
||||
<option value="Chinese">Chinese</option>
|
||||
</select>
|
||||
|
||||
<label>Target Language</label>
|
||||
<select id="targetLang">
|
||||
<option value="English" selected>English</option>
|
||||
<option value="Spanish">Spanish</option>
|
||||
<option value="Chinese">Chinese</option>
|
||||
<option value="Portuguese">Portuguese</option>
|
||||
</select>
|
||||
|
||||
<label>Voice</label>
|
||||
<select id="voiceSelect">
|
||||
<option value="Cherry">Cherry (F)</option>
|
||||
<option value="Dylan">Dylan (M)</option>
|
||||
<option value="Jada">Jada (F)</option>
|
||||
<option value="Peter">Peter (M)</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<audio id="audioOutput"></audio>
|
||||
|
||||
<script type="module">
|
||||
const RTC_CONFIG = __RTC_CONFIGURATION__;
|
||||
|
||||
let pc = null;
|
||||
let webrtcId = null;
|
||||
let mediaStream = null;
|
||||
let latencyStart = 0;
|
||||
|
||||
const video = document.getElementById('videoElement');
|
||||
const captionText = document.getElementById('captionText');
|
||||
const connectionOverlay = document.getElementById('connectionOverlay');
|
||||
const liveBadge = document.getElementById('liveBadge');
|
||||
const langBadge = document.getElementById('langBadge');
|
||||
const latencyDisplay = document.getElementById('latencyDisplay');
|
||||
const audioIndicator = document.getElementById('audioIndicator');
|
||||
const audioBars = document.getElementById('audioBars');
|
||||
|
||||
// Update time display
|
||||
setInterval(() => {
|
||||
document.getElementById('timeDisplay').textContent =
|
||||
new Date().toLocaleTimeString('en-US', { hour12: false });
|
||||
}, 1000);
|
||||
|
||||
window.toggleSettings = function() {
|
||||
document.getElementById('settingsPopup').classList.toggle('visible');
|
||||
};
|
||||
|
||||
window.startMonitor = async function() {
|
||||
try {
|
||||
connectionOverlay.innerHTML = '<div class="spinner"></div><h2>Connecting...</h2>';
|
||||
|
||||
// Get camera/mic
|
||||
mediaStream = await navigator.mediaDevices.getUserMedia({
|
||||
video: { width: 1920, height: 1080 },
|
||||
audio: { sampleRate: 16000, echoCancellation: true }
|
||||
});
|
||||
|
||||
video.srcObject = mediaStream;
|
||||
|
||||
// Setup WebRTC
|
||||
const config = RTC_CONFIG || { iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] };
|
||||
pc = new RTCPeerConnection(config);
|
||||
|
||||
mediaStream.getTracks().forEach(track => pc.addTrack(track, mediaStream));
|
||||
|
||||
pc.ontrack = (event) => {
|
||||
if (event.track.kind === 'audio') {
|
||||
const audio = document.getElementById('audioOutput');
|
||||
audio.srcObject = event.streams[0];
|
||||
audio.play();
|
||||
audioIndicator.style.display = 'flex';
|
||||
}
|
||||
};
|
||||
|
||||
pc.onconnectionstatechange = () => {
|
||||
if (pc.connectionState === 'connected') {
|
||||
connectionOverlay.classList.add('hidden');
|
||||
liveBadge.style.display = 'block';
|
||||
latencyStart = Date.now();
|
||||
}
|
||||
};
|
||||
|
||||
const offer = await pc.createOffer();
|
||||
await pc.setLocalDescription(offer);
|
||||
|
||||
// Wait for ICE
|
||||
await new Promise(r => {
|
||||
if (pc.iceGatheringState === 'complete') r();
|
||||
else pc.onicegatheringstatechange = () => {
|
||||
if (pc.iceGatheringState === 'complete') r();
|
||||
};
|
||||
setTimeout(r, 2000);
|
||||
});
|
||||
|
||||
const srcLang = document.getElementById('srcLang').value;
|
||||
const targetLang = document.getElementById('targetLang').value;
|
||||
const voice = document.getElementById('voiceSelect').value;
|
||||
|
||||
langBadge.textContent = `${srcLang.slice(0,2).toUpperCase()} → ${targetLang.slice(0,2).toUpperCase()}`;
|
||||
|
||||
const resp = await fetch('/webrtc/offer', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
sdp: pc.localDescription.sdp,
|
||||
type: pc.localDescription.type,
|
||||
src_language: srcLang,
|
||||
target_language: targetLang,
|
||||
voice: voice
|
||||
})
|
||||
});
|
||||
|
||||
const data = await resp.json();
|
||||
webrtcId = data.webrtc_id;
|
||||
|
||||
await pc.setRemoteDescription(new RTCSessionDescription({
|
||||
sdp: data.sdp,
|
||||
type: data.type
|
||||
}));
|
||||
|
||||
// Listen for transcripts
|
||||
const evtSource = new EventSource(`/outputs?webrtc_id=${webrtcId}`);
|
||||
evtSource.addEventListener('output', (e) => {
|
||||
const d = JSON.parse(e.data);
|
||||
updateCaption(d);
|
||||
});
|
||||
|
||||
} catch (err) {
|
||||
console.error(err);
|
||||
connectionOverlay.innerHTML = `<h2>Error</h2><p>${err.message}</p><button class="start-btn" onclick="location.reload()">Retry</button>`;
|
||||
}
|
||||
};
|
||||
|
||||
function updateCaption(data) {
|
||||
const content = data.content || '';
|
||||
const stableMatch = content.match(/<span style='color:black'>([^<]*)<\/span>/);
|
||||
const interimMatch = content.match(/<span style='color:gray'>([^<]*)<\/span>/);
|
||||
|
||||
captionText.querySelector('.stable').textContent = stableMatch ? stableMatch[1] : '';
|
||||
captionText.querySelector('.interim').textContent = interimMatch ? interimMatch[1] : '';
|
||||
|
||||
// Update latency
|
||||
const latency = Date.now() - latencyStart;
|
||||
if (latency < 10000) {
|
||||
latencyDisplay.textContent = `${latency}ms`;
|
||||
latencyDisplay.className = latency < 300 ? 'latency good' : latency < 800 ? 'latency warn' : 'latency bad';
|
||||
}
|
||||
latencyStart = Date.now();
|
||||
|
||||
// Animate audio bars when there's new content
|
||||
if (stableMatch || interimMatch) {
|
||||
audioBars.classList.remove('silent');
|
||||
setTimeout(() => audioBars.classList.add('silent'), 500);
|
||||
}
|
||||
}
|
||||
|
||||
// Keyboard shortcuts
|
||||
document.addEventListener('keydown', (e) => {
|
||||
if (e.key === 'f' || e.key === 'F11') {
|
||||
e.preventDefault();
|
||||
document.documentElement.requestFullscreen?.();
|
||||
}
|
||||
if (e.key === 'Escape' && document.fullscreenElement) {
|
||||
document.exitFullscreen();
|
||||
}
|
||||
});
|
||||
|
||||
// Auto-start with URL param
|
||||
if (new URLSearchParams(location.search).get('autostart') === '1') {
|
||||
startMonitor();
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+4
-1
@@ -6,4 +6,7 @@ python-dotenv
|
||||
fastrtc
|
||||
openai
|
||||
twilio
|
||||
opencv-python
|
||||
opencv-python
|
||||
pydantic
|
||||
gradio
|
||||
certifi
|
||||
Reference in New Issue
Block a user