matthias/gts-holmirdas
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions
gts-holmirdas/README.md

9.2 KiB
Raw Permalink Blame History

GTS-HolMirDas 🚀

RSS-based content discovery for GoToSocial instances.

Automatically discovers and federates content from RSS feeds across the Fediverse, helping small GoToSocial instances populate their federated timeline without relying on traditional relays.

Inspired by the original HolMirDas for Misskey by @aliceif (@aliceif@mkultra.x27.one), this GoToSocial adaptation extends the RSS-to-ActivityPub concept with enhanced Docker deployment and multi-instance processing.

Features

  • 📡 Multi-Instance RSS Discovery - Fetches content from configurable RSS feeds across Fediverse instances
  • ⚡ Efficient Processing - Configurable rate limiting and duplicate detection
  • 🔧 Production Ready - Environment-based config, Docker deployment, health monitoring
  • 📊 Comprehensive Statistics - Runtime metrics, content processing, and federation growth tracking
  • 🐳 Containerized - Simple Docker Compose deployment
  • 📁 File-based Configuration - Easy RSS feed management via text files

How it Works

GTS-HolMirDas reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach:

  • Maintains proper ActivityPub federation (posts remain interactive)
  • Respects rate limits and instance policies
  • Provides better content discovery for small instances
  • Works alongside tools like FediFetcher for comprehensive federation

Quick Start

# Clone the repository
git clone https://git.klein.ruhr/matthias/gts-holmirdas
cd gts-holmirdas

# Copy configuration templates
cp .env.example .env
cp rss_feeds.example.txt rss_feeds.txt

# Edit configuration
nano .env          # Add your GTS credentials
nano rss_feeds.txt # Customize RSS feeds

# Deploy
docker compose up -d

# Monitor
docker compose logs -f

Performance Scaling & Configuration

🚀 RSS Feed Optimization (v1.1.0+)

GTS-HolMirDas supports URL parameters to dramatically increase content discovery without additional API calls.

RSS Feed Limits

Most Mastodon-compatible instances support the ?limit=X parameter:

# Default behavior (20 posts per feed)
https://mastodon.social/tags/homelab.rss

# Increased limits (up to 100 posts per feed)
https://mastodon.social/tags/homelab.rss?limit=50
https://fosstodon.org/tags/docker.rss?limit=100

Supported limits: 20 (default), 50, 75, 100 (instance-dependent)

Performance Impact

Configuration Posts/Run API Calls Processing Time
Standard (limit=20) ~100 posts 30+ feeds 2-5 minutes
Optimized (limit=50) ~300 posts 30+ feeds 5-10 minutes
Maximum (limit=100) ~600 posts 30+ feeds 8-15 minutes

⚙️ Configuration Tuning

Environment Variables

# Processing Configuration
MAX_POSTS_PER_RUN=75          # Increase for higher limits
DELAY_BETWEEN_REQUESTS=1      # Balance speed vs. server load
RSS_URLS_FILE=/app/rss_feeds.txt

# Recommended combinations:
# Conservative: MAX_POSTS_PER_RUN=40, limit=50
# Balanced:     MAX_POSTS_PER_RUN=75, limit=100  
# Aggressive:   MAX_POSTS_PER_RUN=100, limit=100

RSS Feed Strategy

# Progressive scaling approach:
# 1. Start with mixed limits to test performance
# 2. Increase gradually based on server capacity
# 3. Monitor GoToSocial memory usage

# Example progression:
https://mastodon.social/tags/homelab.rss?limit=50
https://fosstodon.org/tags/selfhosting.rss?limit=75
https://chaos.social/tags/docker.rss?limit=100

📊 Monitoring & Optimization

Performance Metrics

The statistics output shows real-time performance:

📊 GTS-HolMirDas Run Statistics:
   ⏱️  Runtime: 0:08:42
   📄 Total posts processed: 487
   🌐 Current known instances: 3150
   ➕ New instances discovered: +45
   📡 RSS feeds processed: 102
   ⚡ Posts per minute: 56.0

Optimization Guidelines

Memory Usage:

  • Monitor GoToSocial memory consumption during runs
  • Each 100 additional posts ≈ ~2-5MB additional RAM
  • Recommended: 1GB+ RAM for aggressive configurations

Processing Time:

  • Scales linearly with MAX_POSTS_PER_RUN × number_of_feeds
  • Duplicate detection becomes more important at scale
  • Consider running frequency vs. content volume

Federation Growth:

  • Higher limits = more diverse instance discovery
  • Expect 20-50+ new instances per optimized run
  • Balance discovery rate with storage capacity

Troubleshooting High-Volume Setups

If processing takes too long:

MAX_POSTS_PER_RUN=50          # Reduce from 75/100
DELAY_BETWEEN_REQUESTS=2      # Increase from 1

If GoToSocial uses too much memory:

  • Reduce RSS feed count temporarily
  • Lower ?limit= parameters to 50 instead of 100
  • Increase run frequency instead of volume

If duplicate detection is slow:

  • Storage cleanup: docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json
  • This forces fresh state tracking (posts will be reprocessed once)

🎯 Best Practices

Scaling Strategy

  1. Start Conservative: limit=50, MAX_POSTS_PER_RUN=40
  2. Monitor Performance: Check RAM usage and processing time
  3. Scale Gradually: Increase to limit=75, then limit=100
  4. Optimize Mix: Use different limits per instance based on quality

Instance Selection

High-quality instances for aggressive limits:

# Tech-focused instances (good signal-to-noise ratio)
https://fosstodon.org/tags/homelab.rss?limit=100
https://infosec.exchange/tags/security.rss?limit=100

# General instances (moderate limits recommended)  
https://mastodon.social/tags/technology.rss?limit=50

Performance tip: Specialized instances often have higher content quality at scale than general-purpose instances.

Configuration

Environment Variables (.env)

# GTS Server Configuration
GTS_SERVER_URL=https://your-gts-instance.tld
GTS_ACCESS_TOKEN=your_gts_access_token

# Processing Configuration
MAX_POSTS_PER_RUN=25              # Posts per feed per run
DELAY_BETWEEN_REQUESTS=1          # Seconds between API calls
LOG_LEVEL=INFO                    # Logging verbosity

# RSS Configuration
RSS_URLS_FILE=/app/rss_feeds.txt  # Path to RSS feeds file

# Optional: Monitoring
HEALTHCHECK_URL=https://hc-ping.com/your-uuid

RSS Feeds (rss_feeds.txt)

# Example RSS feeds - customize for your interests
# homelab
https://mastodon.social/tags/homelab.rss
https://fosstodon.org/tags/homelab.rss

# selfhosting
https://mastodon.social/tags/selfhosting.rss
https://infosec.exchange/tags/selfhosting.rss

# Add your preferred instances and hashtags

Access Token Setup

  1. Login to your GoToSocial instance
  2. Go to Settings → Applications
  3. Create new application with scopes: read, read:search, read:statuses
  4. Copy the access token to your .env file

Statistics Output

📊 GTS-HolMirDas Run Statistics:
   ⏱️ Runtime: 0:04:14
   📄 Total posts processed: 45
   🌐 Current known instances: 2519
   ➕ New instances discovered: +3
   📡 RSS feeds processed: 25
   ⚡ Posts per minute: 10.6

Resource Requirements

  • Memory: ~200-500MB depending on feed count
  • CPU: Minimal (mostly I/O bound)
  • Storage: <100MB for application, plus log storage
  • Network: Depends on RSS feed count and frequency

Deployment Options

Docker Compose (Recommended)

docker compose up -d

Standalone Docker

docker build -t gts-holmirdas .
docker run -d --env-file .env \
  -v ./data:/app/data \
  -v ./gts_holmirdas.py:/app/gts_holmirdas.py:ro \
  -v ./rss_feeds.txt:/app/rss_feeds.txt:ro \
  gts-holmirdas

Monitoring

  • Logs: docker compose logs -f
  • Health: Optional Healthchecks.io integration
  • Statistics: Built-in runtime and performance metrics
  • Resource Usage: Docker stats or container monitoring tools

Troubleshooting

Common Issues

  • No posts processed: Check access token permissions and RSS feed URLs
  • Rate limiting errors: Increase DELAY_BETWEEN_REQUESTS or reduce feed count
  • High memory usage: Reduce MAX_POSTS_PER_RUN or feed frequency
  • Container won't start: Verify .env file format and required variables

Debug Mode

# Enable debug logging
echo "LOG_LEVEL=DEBUG" >> .env
docker compose restart gts-holmirdas

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Test thoroughly
  5. Submit a pull request

Related Projects

  • FediFetcher - Fetches missing replies and posts
  • GoToSocial - Lightweight ActivityPub server
  • slurp - Import posts from other instances

License

MIT License - see LICENSE file for details.

Acknowledgments

  • Inspired by HolMirDas by @aliceif (@aliceif@mkultra.x27.one) - the original RSS-to-ActivityPub concept
  • Built for the GoToSocial community
  • RSS-to-ActivityPub approach inspired by Fediverse discovery challenges