Add Troubleshooting

Troubleshooting.md

@@ -0,0 +1,603 @@
+# Troubleshooting
+
+Complete troubleshooting guide for common GTS-HolMirDas issues and their solutions.
+
+## 🚨 Common Issues
+
+### RSS Feed URL Control Characters Error
+
+**Error Message:**
+```
+Error fetching RSS feed https://example.org/tags/homelab.rss: URL can't contain control characters
+```
+
+**Cause:** 
+The RSS feed URLs in `rss_feeds.txt` contain invisible control characters, often introduced during copy/paste operations from web browsers or text editors.
+
+**Solutions:**
+
+#### Method 1: Clean the file automatically
+```bash
+# Stop the container first
+docker-compose down
+
+# Remove all control characters from rss_feeds.txt
+sed -i 's/[[:cntrl:]]//g' rss_feeds.txt
+
+# Alternative approach using tr
+cat rss_feeds.txt | tr -d '\000-\031' > rss_feeds_clean.txt
+mv rss_feeds_clean.txt rss_feeds.txt
+
+# Restart container
+docker-compose up -d
+```
+
+#### Method 2: Recreate the file manually
+```bash
+# Backup current file
+cp rss_feeds.txt rss_feeds.txt.backup
+
+# Create new clean file
+cat > rss_feeds.txt << 'EOF'
+https://mastodon.social/tags/homelab.rss
+https://fosstodon.org/tags/homelab.rss?limit=50
+https://mastodon.social/tags/selfhosting.rss?limit=100
+EOF
+
+# Restart container
+docker-compose restart gts-holmirdas
+```
+
+#### Method 3: Text editor approach
+1. Open `rss_feeds.txt` in a plain text editor (nano, vim, notepad++)
+2. Manually retype the URLs instead of copy/pasting
+3. Ensure each URL is on its own line with no trailing spaces
+4. Save and restart the container
+
+---
+
+### Permission Denied Error
+
+**Error Message:**
+```
+Could not save processed URLs: [Errno 13] Permission denied: "/app/data/processed_urls.json"
+```
+
+**Cause:** 
+The Docker container cannot write to the mounted data directory due to incorrect file permissions or user/group ownership.
+
+**Solutions:**
+
+#### Method 1: Fix host directory permissions
+```bash
+# Stop the container
+docker-compose down
+
+# Check current ownership
+ls -la ./data
+
+# Fix ownership (adjust user ID if needed)
+sudo chown -R 1000:1000 ./data
+sudo chmod -R 755 ./data
+
+# Restart container
+docker-compose up -d
+```
+
+#### Method 2: Configure container user
+Add user specification to your `docker-compose.yml`:
+
+```yaml
+services:
+  gts-holmirdas:
+    image: your-image:latest
+    user: "1000:1000"  # Match your host user ID
+    volumes:
+      - ./data:/app/data:rw
+    # ... other configurations
+```
+
+#### Method 3: Check your user ID
+```bash
+# Find your user ID and group ID
+id -u    # User ID
+id -g    # Group ID
+
+# Use these values in docker-compose.yml user field
+# Example: if output is 1001 and 1001, use user: "1001:1001"
+```
+
+#### Method 4: Create data directory with correct permissions
+```bash
+# Create data directory if it doesn't exist
+mkdir -p ./data
+
+# Set correct ownership and permissions
+sudo chown -R $(id -u):$(id -g) ./data
+chmod -R 755 ./data
+```
+
+---
+
+### Network Connection Issues
+
+**Error Messages:**
+```
+Failed to get instance count: HTTPSConnectionPool(host='gts.teqqy.social', port=443): Max retries exceeded
+Network is unreachable
+Connection timeout
+SSL: CERTIFICATE_VERIFY_FAILED
+```
+
+**Causes and Solutions:**
+
+#### Issue: DNS Resolution Problems
+```bash
+# Test DNS resolution
+nslookup your-gts-instance.tld
+dig your-gts-instance.tld
+
+# If DNS fails, check /etc/resolv.conf or network settings
+```
+
+#### Issue: HTTPS Connectivity Problems
+```bash
+# Test HTTPS connectivity
+curl -I https://your-gts-instance.tld/api/v1/instance
+
+# Test with verbose output for debugging
+curl -v https://your-gts-instance.tld/api/v1/instance
+```
+
+#### Issue: Container Network Problems
+```bash
+# Restart Docker daemon
+sudo systemctl restart docker
+
+# Recreate container with fresh networking
+docker-compose down
+docker-compose up -d
+
+# Check container network
+docker network ls
+docker network inspect gts-holmirdas_default
+```
+
+#### Issue: Firewall/Proxy Issues
+```bash
+# Check if running behind firewall/proxy
+# Ensure outbound HTTPS (port 443) is allowed
+
+# Test from host system
+curl -I https://mastodon.social/tags/test.rss
+
+# If host works but container doesn't, check Docker networking
+```
+
+#### Issue: SSL Certificate Problems
+```bash
+# If you see SSL certificate errors
+# Check if system clock is correct
+date
+
+# Update CA certificates in container
+docker-compose exec gts-holmirdas apt-get update && apt-get install -y ca-certificates
+```
+
+---
+
+### GoToSocial API Issues
+
+**Error Messages:**
+```
+Unauthorized: Invalid access token
+Forbidden: Insufficient permissions  
+Rate limit exceeded
+```
+
+#### Issue: Invalid Access Token
+```bash
+# Verify token in .env file
+cat .env | grep GTS_ACCESS_TOKEN
+
+# Test token manually
+curl -H "Authorization: Bearer YOUR_TOKEN" \
+     https://your-gts-instance.tld/api/v1/accounts/verify_credentials
+```
+
+**Solution:**
+1. Go to your GoToSocial Settings → Applications
+2. Delete the old application
+3. Create new application with correct scopes: `read`, `read:search`, `read:statuses`
+4. Update `.env` file with new token
+5. Restart container
+
+#### Issue: Insufficient Permissions
+**Solution:** Recreate application with all required scopes:
+- ✅ `read`
+- ✅ `read:search` 
+- ✅ `read:statuses`
+
+#### Issue: Rate Limiting
+```bash
+# Increase delay between requests
+DELAY_BETWEEN_REQUESTS=3  # Increase from 1-2
+
+# Reduce processing volume temporarily
+MAX_POSTS_PER_RUN=20     # Reduce from higher values
+```
+
+---
+
+### Container Won't Start
+
+**Error Messages:**
+```
+Error response from daemon: invalid reference format
+Environment variable not set
+Port already in use
+```
+
+#### Issue: Invalid Docker Compose Configuration
+```bash
+# Validate docker-compose.yml syntax
+docker-compose config
+
+# Check for common issues:
+# - Incorrect indentation (use spaces, not tabs)
+# - Missing quotes around special characters
+# - Invalid environment variable references
+```
+
+#### Issue: Environment Variables Not Set
+```bash
+# Check .env file exists and has correct format
+ls -la .env
+cat .env
+
+# Ensure no spaces around equals signs:
+# ✅ Correct: GTS_SERVER_URL=https://example.com
+# ❌ Wrong:   GTS_SERVER_URL = https://example.com
+
+# Check for required variables
+grep -E "(GTS_SERVER_URL|GTS_ACCESS_TOKEN)" .env
+```
+
+#### Issue: Port Conflicts
+```bash
+# Check if port is already in use
+sudo netstat -tlnp | grep :8080
+
+# Change port in docker-compose.yml if needed
+ports:
+  - "8081:8080"  # Use different external port
+```
+
+---
+
+### Performance Issues
+
+#### Issue: Processing Takes Too Long
+```
+Runtime: 0:25:30  # Over 25 minutes is too slow
+```
+
+**Solutions:**
+```bash
+# Reduce processing volume
+MAX_POSTS_PER_RUN=25     # Reduce from higher values
+DELAY_BETWEEN_REQUESTS=2 # Increase from 1
+
+# Reduce RSS feed count temporarily
+# Keep only highest-quality feeds
+
+# Check network latency
+ping mastodon.social
+ping your-gts-instance.tld
+```
+
+#### Issue: High Memory Usage
+```bash
+# Monitor memory usage
+docker stats gts-holmirdas
+docker stats gotosocial
+
+# If memory usage is too high:
+MAX_POSTS_PER_RUN=30     # Reduce processing volume
+# Reduce RSS feeds with high ?limit= parameters
+```
+
+#### Issue: Duplicate Detection Slow
+```bash
+# Clean processed URLs database (monthly maintenance)
+docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json
+
+# This forces fresh state tracking
+# Posts will be reprocessed once, then normal operation resumes
+```
+
+---
+
+## 🔧 Complete Reset Procedures
+
+### Soft Reset (Keep Configuration)
+```bash
+# Stop container
+docker-compose down
+
+# Clear runtime data only
+rm -rf ./data/*
+
+# Restart with clean state
+docker-compose up -d
+```
+
+### Hard Reset (Full Clean)
+```bash
+# Stop everything
+docker-compose down
+
+# Remove all containers and images
+docker-compose rm -f
+docker rmi $(docker images | grep gts-holmirdas | awk '{print $3}')
+
+# Clean up RSS feeds file
+cat rss_feeds.txt | tr -d '\000-\031' > rss_feeds_clean.txt
+mv rss_feeds_clean.txt rss_feeds.txt
+
+# Reset data directory
+sudo rm -rf ./data
+mkdir -p ./data
+sudo chown -R $(id -u):$(id -g) ./data
+chmod -R 755 ./data
+
+# Rebuild and restart
+docker-compose build --no-cache
+docker-compose up -d
+```
+
+### Nuclear Reset (Start From Scratch)
+```bash
+# Stop and remove everything
+docker-compose down
+docker-compose rm -f
+
+# Remove cloned repository
+cd ..
+rm -rf gts-holmirdas
+
+# Start fresh installation
+git clone https://git.klein.ruhr/matthias/gts-holmirdas
+cd gts-holmirdas
+cp .env.example .env
+cp rss_feeds.example.txt rss_feeds.txt
+
+# Configure and deploy
+nano .env
+nano rss_feeds.txt
+docker-compose up -d
+```
+
+---
+
+## 🛡️ Prevention Tips
+
+### 1. Configuration Best Practices
+```bash
+# Always use plain text editors for config files
+# Avoid copy/paste from web browsers
+# Use consistent line endings (Unix: LF)
+
+# Validate configuration before deployment
+docker-compose config
+```
+
+### 2. Regular Maintenance
+```bash
+# Weekly: Check logs and performance
+docker-compose logs --tail=100 gts-holmirdas
+
+# Monthly: Clean processed URLs
+docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json
+
+# Quarterly: Review and update RSS feeds
+nano rss_feeds.txt
+```
+
+### 3. Monitoring Setup
+```bash
+# Monitor resource usage
+docker stats --no-stream gts-holmirdas gotosocial
+
+# Set up log rotation in docker-compose.yml
+logging:
+  driver: "json-file"
+  options:
+    max-size: "10m"
+    max-file: "3"
+```
+
+### 4. Backup Strategy
+```bash
+# Regular backups of working configuration
+tar -czf gts-holmirdas-backup-$(date +%Y%m%d).tar.gz \
+    ./data ./rss_feeds.txt ./docker-compose.yml ./.env
+
+# Store backups securely
+# Test restore procedure periodically
+```
+
+### 5. URL Testing Before Adding
+```bash
+# Always test new RSS URLs manually
+curl -I "https://example.org/tags/homelab.rss"
+
+# Check for proper RSS format
+curl -s "https://example.org/tags/homelab.rss" | head -20
+
+# Validate RSS structure
+curl -s "https://example.org/tags/homelab.rss" | grep -E "(rss|feed|entry|item)"
+```
+
+---
+
+## 🔍 Advanced Debugging
+
+### Enable Debug Mode
+```bash
+# Add to .env file
+LOG_LEVEL=DEBUG
+
+# Restart container
+docker-compose restart gts-holmirdas
+
+# Watch detailed logs
+docker-compose logs -f gts-holmirdas
+```
+
+### Container Shell Access
+```bash
+# Access running container
+docker-compose exec gts-holmirdas /bin/bash
+
+# Check internal files
+ls -la /app/
+cat /app/rss_feeds.txt
+cat /app/data/processed_urls.json
+
+# Test RSS fetching manually
+python3 -c "
+import feedparser
+feed = feedparser.parse('https://mastodon.social/tags/test.rss')
+print(f'Feed title: {feed.feed.title}')
+print(f'Entry count: {len(feed.entries)}')
+"
+```
+
+### Network Debugging Inside Container
+```bash
+# Access container shell
+docker-compose exec gts-holmirdas /bin/bash
+
+# Test network connectivity
+ping -c 3 mastodon.social
+nslookup your-gts-instance.tld
+
+# Test RSS feeds
+curl -I https://mastodon.social/tags/test.rss
+curl -I https://your-gts-instance.tld/api/v1/instance
+
+# Check Python dependencies
+pip list | grep -E "(requests|feedparser)"
+```
+
+### Log Analysis
+```bash
+# Extract specific error patterns
+docker-compose logs gts-holmirdas 2>&1 | grep -i error
+
+# Show last run statistics
+docker-compose logs gts-holmirdas 2>&1 | grep -A 10 "Run Statistics"
+
+# Monitor real-time processing
+docker-compose logs -f gts-holmirdas | grep -E "(Processing|Error|Statistics)"
+
+# Save full logs for analysis
+docker-compose logs --no-log-prefix gts-holmirdas > debug.log
+```
+
+---
+
+## 📞 Getting Help
+
+### Before Reporting Issues
+
+1. **Try the troubleshooting steps** above
+2. **Check existing issues** at the repository
+3. **Gather complete information** (see below)
+
+### Information to Include
+
+When reporting issues, always include:
+
+```bash
+# System Information
+docker --version
+docker-compose --version
+uname -a
+df -h  # Disk space
+
+# Container Status
+docker-compose ps
+docker-compose logs --tail=50 gts-holmirdas
+
+# Configuration (remove sensitive data)
+cat docker-compose.yml
+cat .env | sed 's/GTS_ACCESS_TOKEN=.*/GTS_ACCESS_TOKEN=***HIDDEN***/'
+wc -l rss_feeds.txt  # Number of RSS feeds
+```
+
+### Issue Template
+
+When creating an issue, use this template:
+
+```markdown
+## Problem Description
+Brief description of what's not working
+
+## Error Messages
+```
+Paste complete error messages here
+```
+
+## System Information
+- OS: 
+- Docker version: 
+- Docker Compose version:
+- RAM available:
+- Disk space:
+
+## Configuration
+- Number of RSS feeds: 
+- MAX_POSTS_PER_RUN: 
+- DELAY_BETWEEN_REQUESTS:
+- GoToSocial version:
+
+## Steps to Reproduce
+1. 
+2. 
+3. 
+
+## Expected vs Actual Behavior
+Expected: 
+Actual: 
+
+## Additional Context
+Any other relevant information
+```
+
+### Community Support
+
+- **Repository Issues:** [Create Issue](../../issues)  
+- **Discussions:** [Repository Discussions](../../discussions)
+- **Matrix Chat:** [#gotosocial-space:superseriousbusiness.org](https://matrix.to/#/#gotosocial-space:superseriousbusiness.org)
+- **Email:** Available in repository profile
+
+### Emergency Recovery
+
+If your instance is completely broken:
+
+```bash
+# Stop everything immediately
+docker-compose down
+
+# Restore from backup (if available)
+tar -xzf gts-holmirdas-backup-YYYYMMDD.tar.gz
+
+# Or nuclear reset and reconfigure
+rm -rf gts-holmirdas
+git clone https://git.klein.ruhr/matthias/gts-holmirdas
+# Reconfigure from scratch following Installation Guide
+```
+
+Remember: Most issues are configuration-related and can be resolved by carefully following the troubleshooting steps above. Don't hesitate to ask for help if you're stuck!