zenlm/claude-collector
1
0
Fork 0
mirror of https://github.com/zenlm/claude-collector.git synced 2026-07-26 22:30:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Initial commit: Claude Collector v0.1.0

Browse Source 
- Extract agent conversations
- Auto-sanitize PII
- Token counting
- Training dataset export
- Works with uvx/uv tool
This commit is contained in:
Hanzo Dev
2025-11-13 13:00:50 -08:00
commit 1c726c4710
5 changed files with 595 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Hanzo AI
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+277
View File
@@ -0,0 +1,277 @@
# 🤖 Claude Collector
**One command to extract all your Claude Code conversations for training datasets.**
## Quick Start
### Install and run with uvx (recommended)
```bash
uvx claude-collector
```
That's it! The tool will:
- ✅ Auto-find your Claude Code data (`~/.claude/projects`)
- ✅ Extract all conversations
- ✅ Sanitize PII (emails, API keys, paths)
- ✅ Count total tokens
- ✅ Save as training-ready JSONL
### Or install globally
```bash
uv tool install claude-collector
claude-collector
```
## What It Does
Scans your Claude Code session files and:
1. **Finds** all conversation data in `~/.claude/projects`
2. **Extracts** user/assistant message pairs
3. **Sanitizes** sensitive information:
- Emails → `[EMAIL]`
- API keys → `[API_KEY]`
- File paths → `/Users/[USER]/...`
- IP addresses → `[IP]`
- OAuth tokens → `[REDACTED]`
4. **Counts** actual token usage
5. **Saves** as clean JSONL dataset
## Example Output
```
🤖 Claude Collector v0.1.0
Extract & sanitize Claude Code conversations
✓ Found Claude data: /Users/z/.claude/projects
📂 Processing 1394 files...
╭─────────────────────┬──────────────╮
│ Metric │ Value │
├─────────────────────┼──────────────┤
│ Files scanned │ 1,394 │
│ Files with data │ 1,273 │
│ Total messages │ 46,029 │
│ Training examples │ 3,653 │
│ Total tokens │ 4.04B │
╰─────────────────────┴──────────────╯
✅ Dataset saved!
File: claude_dataset_20251113.jsonl
Size: 19.13 MB
Examples: 3,653
🎉 Ready for training!
```
## Options
```bash
# Dry run (see stats without saving)
uvx claude-collector --dry-run
# Custom output location
uvx claude-collector --output ~/my-dataset.jsonl
# Specify input directory
uvx claude-collector --input ~/.config/claude/projects
# Filter by minimum tokens
uvx claude-collector --min-tokens 1000
# Skip sanitization (NOT recommended for sharing!)
uvx claude-collector --no-sanitize
```
## Use Cases
### 1. Create Training Dataset
```bash
uvx claude-collector --output training-data.jsonl
```
### 2. Audit Your Usage
```bash
uvx claude-collector --dry-run
```
Shows total tokens without saving.
### 3. Collect from Multiple Machines
On each computer:
```bash
# Machine 1
uvx claude-collector --output machine1-data.jsonl
# Machine 2
uvx claude-collector --output machine2-data.jsonl
# Combine later
cat machine1-data.jsonl machine2-data.jsonl > combined-dataset.jsonl
```
### 4. Add to Existing Dataset
```bash
uvx claude-collector --output new-sessions.jsonl
cat existing-dataset.jsonl new-sessions.jsonl > updated-dataset.jsonl
```
## Output Format
Each line is a JSON object:
```json
{
"messages": [
{"role": "user", "content": "How do I..."},
{"role": "assistant", "content": "You can..."}
],
"metadata": {
"timestamp": "2025-11-13T...",
"tokens": {
"input_tokens": 100,
"output_tokens": 200,
"cache_creation_input_tokens": 5000,
"cache_read_input_tokens": 1000
}
}
}
```
Perfect for:
- Fine-tuning LLMs
- Training coding assistants
- Building instruction datasets
- Analysis and research
## Finding Claude Data
### Default Locations
```bash
~/.claude/projects/ # Primary
~/.config/claude/projects/ # Alternative
```
### Check All Users
```bash
ls -la /Users/*/.claude/projects # macOS
ls -la /home/*/.claude/projects # Linux
```
### Find Anywhere
```bash
find ~ -name "*.jsonl" -path "*/.claude/*" 2>/dev/null
```
## Privacy & Security
⚠️ **Important**: Claude Code logs contain sensitive data!
**The tool sanitizes:**
- ✅ Email addresses
- ✅ API keys and tokens
- ✅ File paths (username removed)
- ✅ IP addresses
- ✅ OAuth credentials
- ✅ Passwords
**Still check before sharing:**
- Project names (if sensitive)
- Company-specific terminology
- Proprietary code patterns
For maximum privacy, review the output file before uploading anywhere.
## Requirements
- Python 3.8+
- Claude Code installed (for data to exist)
## Installation Methods
### 1. uvx (easiest, no install)
```bash
uvx claude-collector
```
### 2. uv tool (global install)
```bash
uv tool install claude-collector
claude-collector
```
### 3. pip
```bash
pip install claude-collector
claude-collector
```
### 4. From source
```bash
git clone https://github.com/hanzoai/claude-collector
cd claude-collector
uv pip install -e .
claude-collector
```
## Troubleshooting
**"No Claude Code data found"**
- Make sure Claude Code is installed
- Check you've had at least one session
- Try specifying path: `--input ~/.claude/projects`
**"Only found a few conversations"**
- This is normal if you're new to Claude Code
- Each session creates one file
- More usage = more data
**"Tokens show 0"**
- Some messages don't have usage tracking
- This is normal for system messages
- Real conversations will have token counts
## Advanced: Custom Processing
```python
import json
# Read dataset
with open('claude_dataset.jsonl', 'r') as f:
for line in f:
example = json.loads(line)
# Access messages
user_msg = example['messages'][0]['content']
assistant_msg = example['messages'][1]['content']
# Access metadata
tokens = example['metadata']['tokens']
timestamp = example['metadata']['timestamp']
# Your custom processing here
```
## License
MIT - Free to use for any purpose
## Credits
Built by [Hanzo AI](https://hanzo.ai) for the AI development community.
---
**Found a bug?** Open an issue: https://github.com/hanzoai/claude-collector/issues
**Want to contribute?** PRs welcome!
claude_collector/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Claude Collector - Extract Claude Code conversations for training datasets"""
__version__ = "0.1.0"
claude_collector/cli.py
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env python3
"""
Claude Collector CLI
Extract and sanitize Claude Code conversations
"""
import json
import re
from pathlib import Path
from datetime import datetime
import click
from rich.console import Console
from rich.progress import track
from rich.table import Table
from rich import print as rprint
console = Console()
def sanitize_text(text):
"""Remove PII from text"""
if not isinstance(text, str):
return str(text)
replacements = [
(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]'),
(r'sk-[a-zA-Z0-9]{48}', '[API_KEY]'),
(r'xox[baprs]-[a-zA-Z0-9-]+', '[SLACK_TOKEN]'),
(r'/Users/[^/\s]+', '/Users/[USER]'),
(r'/home/[^/\s]+', '/home/[USER]'),
(r'C:\\Users\\[^\\]+', 'C:\\Users\\[USER]'),
(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b', '[IP]'),
(r'oauth[_-]?token[_-]?secret?["\']?\s*[:=]\s*["\']?[a-f0-9:]+', 'oauth_token=[REDACTED]'),
(r'(password|passwd|pwd)["\']?\s*[:=]\s*["\'][^"\']+["\']', r'\1=[REDACTED]'),
]
for pattern, replacement in replacements:
text = re.sub(pattern, replacement, text, flags=re.IGNORECASE)
return text
def extract_text_content(content):
"""Extract text from content (string or array)"""
if isinstance(content, str):
return content
if isinstance(content, list):
texts = []
for item in content:
if isinstance(item, dict):
if item.get('type') == 'text':
texts.append(item.get('text', ''))
elif 'text' in item:
texts.append(item['text'])
elif isinstance(item, str):
texts.append(item)
return '\n'.join(filter(None, texts))
return str(content)
def process_jsonl_file(file_path):
"""Extract conversations from a JSONL file"""
messages = []
total_tokens = 0
try:
with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
for line in f:
if not line.strip():
continue
try:
data = json.loads(line)
# Only process actual user/assistant messages
if data.get('type') not in ['user', 'assistant']:
continue
msg = data.get('message', {})
if not msg:
continue
# Extract content
content = extract_text_content(msg.get('content', ''))
if not content or len(content) < 10:
continue
# Get usage data (in message.usage or top-level)
usage = msg.get('usage') or data.get('usage') or {}
token_count = (
usage.get('input_tokens', 0) +
usage.get('output_tokens', 0) +
usage.get('cache_creation_input_tokens', 0) +
usage.get('cache_read_input_tokens', 0)
)
total_tokens += token_count
messages.append({
'role': data['type'],
'content': sanitize_text(content),
'timestamp': data.get('timestamp'),
'tokens': token_count,
'usage': usage
})
except (json.JSONDecodeError, KeyError, TypeError):
continue
except Exception:
pass
return messages, total_tokens
def find_claude_directories():
"""Find all Claude Code data directories"""
locations = []
home = Path.home()
# Check common locations
candidates = [
home / '.claude' / 'projects',
home / '.config' / 'claude' / 'projects',
]
for loc in candidates:
if loc.exists() and loc.is_dir():
locations.append(loc)
return locations
@click.command()
@click.option('--input', '-i', 'input_dir', type=click.Path(exists=True),
help='Input directory (auto-detects ~/.claude/projects if not specified)')
@click.option('--output', '-o', 'output_file', type=click.Path(),
help='Output file (default: claude_dataset_YYYYMMDD.jsonl)')
@click.option('--dry-run', is_flag=True, help='Show stats without saving')
@click.option('--no-sanitize', is_flag=True, help='Skip PII sanitization (NOT RECOMMENDED)')
@click.option('--min-tokens', type=int, default=0, help='Minimum tokens per conversation')
def main(input_dir, output_file, dry_run, no_sanitize, min_tokens):
"""
🤖 Claude Collector - Extract Claude Code conversations
Extracts and sanitizes your Claude Code conversation history
into a training dataset.
\b
Quick start:
uvx claude-collector
\b
Custom output:
uvx claude-collector --output my-dataset.jsonl
\b
Check what you have:
uvx claude-collector --dry-run
"""
console.print("\n[bold cyan]🤖 Claude Collector v0.1.0[/bold cyan]")
console.print("[dim]Extract & sanitize Claude Code conversations[/dim]\n")
# Find Claude directories
if input_dir:
dirs = [Path(input_dir)]
else:
dirs = find_claude_directories()
if not dirs:
console.print("[red]❌ No Claude Code data found![/red]")
console.print("\nChecked:")
console.print(" • ~/.claude/projects")
console.print(" • ~/.config/claude/projects")
console.print("\nSpecify manually: --input /path/to/projects")
return
console.print(f"[green]✓[/green] Found Claude data: {dirs[0]}\n")
# Find all JSONL files
all_files = []
for directory in dirs:
all_files.extend(directory.rglob('*.jsonl'))
console.print(f"[cyan]📂 Processing {len(all_files)} files...[/cyan]\n")
# Process files
all_messages = []
grand_total_tokens = 0
files_with_data = 0
for file_path in track(all_files, description="Extracting"):
messages, tokens = process_jsonl_file(file_path)
if messages:
all_messages.extend(messages)
grand_total_tokens += tokens
files_with_data += 1
# Create training examples (user + assistant pairs)
examples = []
i = 0
while i < len(all_messages) - 1:
if all_messages[i]['role'] == 'user' and all_messages[i+1]['role'] == 'assistant':
example_tokens = all_messages[i+1]['tokens']
if example_tokens >= min_tokens:
examples.append({
'messages': [
{'role': 'user', 'content': all_messages[i]['content']},
{'role': 'assistant', 'content': all_messages[i+1]['content']}
],
'metadata': {
'timestamp': all_messages[i]['timestamp'],
'tokens': all_messages[i+1]['usage']
}
})
i += 2
else:
i += 1
# Display results
table = Table(title="Extraction Results")
table.add_column("Metric", style="cyan")
table.add_column("Value", style="green")
table.add_row("Files scanned", f"{len(all_files):,}")
table.add_row("Files with data", f"{files_with_data:,}")
table.add_row("Total messages", f"{len(all_messages):,}")
table.add_row("Training examples", f"{len(examples):,}")
table.add_row("Total tokens", f"{grand_total_tokens:,} ({grand_total_tokens/1e9:.2f}B)")
console.print()
console.print(table)
console.print()
if dry_run:
console.print("[yellow]🔍 DRY RUN - No files saved[/yellow]")
return
# Save dataset
if not output_file:
output_file = f"claude_dataset_{datetime.now().strftime('%Y%m%d_%H%M')}.jsonl"
output_path = Path(output_file)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
for example in examples:
f.write(json.dumps(example) + '\n')
size_mb = output_path.stat().st_size / 1024 / 1024
console.print(f"[green]✅ Dataset saved![/green]")
console.print(f" File: [cyan]{output_path.name}[/cyan]")
console.print(f" Size: [cyan]{size_mb:.2f} MB[/cyan]")
console.print(f" Examples: [cyan]{len(examples):,}[/cyan]")
console.print(f"\n[bold green]🎉 Ready for training![/bold green]\n")
if __name__ == '__main__':
main()
pyproject.toml
+38
View File
@@ -0,0 +1,38 @@
[project]
name = "claude-collector"
version = "0.1.0"
description = "Extract and sanitize Claude Code conversation data for training datasets"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
authors = [
{name = "Hanzo AI", email = "hello@hanzo.ai"}
]
keywords = ["claude", "ai", "dataset", "training", "llm"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dependencies = [
"rich>=13.0.0",
"click>=8.0.0",
]
[project.scripts]
claude-collector = "claude_collector.cli:main"
[project.urls]
Homepage = "https://github.com/hanzoai/claude-collector"
Repository = "https://github.com/hanzoai/claude-collector"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"