diff --git a/.env.example b/.env.example index 2593cf5..23d6c88 100644 --- a/.env.example +++ b/.env.example @@ -3,7 +3,7 @@ GTS_SERVER_URL=https://your-gts-instance.tld GTS_ACCESS_TOKEN=your_gts_access_token_here # Processing Configuration -MAX_POSTS_PER_RUN=75 +MAX_POSTS_PER_RUN=25 DELAY_BETWEEN_REQUESTS=1 LOG_LEVEL=INFO diff --git a/README.md b/README.md index 96916f9..a4fcd75 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,9 @@ # GTS-HolMirDas 🚀 -RSS-based content discovery for **[GoToSocial](https://codeberg.org/superseriousbusiness/gotosocial)** instances. +RSS-based content discovery for GoToSocial instances. Automatically discovers and federates content from RSS feeds across the Fediverse, helping small GoToSocial instances populate their federated timeline without relying on traditional relays. -*Inspired by the original [HolMirDas](https://github.com/aliceif/HolMirDas) for Misskey by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)), this GoToSocial adaptation extends the RSS-to-ActivityPub concept with enhanced Docker deployment and multi-instance processing.* - ## Features - 📡 **Multi-Instance RSS Discovery** - Fetches content from configurable RSS feeds across Fediverse instances @@ -17,7 +15,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers ## How it Works -**GTS-HolMirDas** reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach: +GTS-HolMirDas reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach: - Maintains proper ActivityPub federation (posts remain interactive) - Respects rate limits and instance policies @@ -28,7 +26,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers ```bash # Clone the repository -git clone https://git.klein.ruhr/matthias/gts-holmirdas +git clone https://your-forgejo-instance.tld/user/gts-holmirdas cd gts-holmirdas # Copy configuration templates @@ -40,142 +38,11 @@ nano .env # Add your GTS credentials nano rss_feeds.txt # Customize RSS feeds # Deploy -docker compose up -d +docker-compose up -d # Monitor -docker compose logs -f +docker-compose logs -f ``` -# Performance Scaling & Configuration - -## 🚀 RSS Feed Optimization (v1.1.0+) - -GTS-HolMirDas supports URL parameters to dramatically increase content discovery without additional API calls. - -### RSS Feed Limits - -Most Mastodon-compatible instances support the `?limit=X` parameter: - -``` -# Default behavior (20 posts per feed) -https://mastodon.social/tags/homelab.rss - -# Increased limits (up to 100 posts per feed) -https://mastodon.social/tags/homelab.rss?limit=50 -https://fosstodon.org/tags/docker.rss?limit=100 -``` - -**Supported limits:** 20 (default), 50, 75, 100 (instance-dependent) - -### Performance Impact - -| Configuration | Posts/Run | API Calls | Processing Time | -|---------------|-----------|-----------|-----------------| -| Standard (limit=20) | ~100 posts | 30+ feeds | 2-5 minutes | -| Optimized (limit=50) | ~300 posts | 30+ feeds | 5-10 minutes | -| Maximum (limit=100) | ~600 posts | 30+ feeds | 8-15 minutes | - -## ⚙️ Configuration Tuning - -### Environment Variables - -```env -# Processing Configuration -MAX_POSTS_PER_RUN=75 # Increase for higher limits -DELAY_BETWEEN_REQUESTS=1 # Balance speed vs. server load -RSS_URLS_FILE=/app/rss_feeds.txt - -# Recommended combinations: -# Conservative: MAX_POSTS_PER_RUN=40, limit=50 -# Balanced: MAX_POSTS_PER_RUN=75, limit=100 -# Aggressive: MAX_POSTS_PER_RUN=100, limit=100 -``` - -### RSS Feed Strategy - -``` -# Progressive scaling approach: -# 1. Start with mixed limits to test performance -# 2. Increase gradually based on server capacity -# 3. Monitor GoToSocial memory usage - -# Example progression: -https://mastodon.social/tags/homelab.rss?limit=50 -https://fosstodon.org/tags/selfhosting.rss?limit=75 -https://chaos.social/tags/docker.rss?limit=100 -``` - -## 📊 Monitoring & Optimization - -### Performance Metrics - -The statistics output shows real-time performance: - -``` -📊 GTS-HolMirDas Run Statistics: - ⏱️ Runtime: 0:08:42 - 📄 Total posts processed: 487 - 🌐 Current known instances: 3150 - ➕ New instances discovered: +45 - 📡 RSS feeds processed: 102 - ⚡ Posts per minute: 56.0 -``` - -### Optimization Guidelines - -**Memory Usage:** -- Monitor GoToSocial memory consumption during runs -- Each 100 additional posts ≈ ~2-5MB additional RAM -- Recommended: 1GB+ RAM for aggressive configurations - -**Processing Time:** -- Scales linearly with `MAX_POSTS_PER_RUN × number_of_feeds` -- Duplicate detection becomes more important at scale -- Consider running frequency vs. content volume - -**Federation Growth:** -- Higher limits = more diverse instance discovery -- Expect 20-50+ new instances per optimized run -- Balance discovery rate with storage capacity - -### Troubleshooting High-Volume Setups - -**If processing takes too long:** -```env -MAX_POSTS_PER_RUN=50 # Reduce from 75/100 -DELAY_BETWEEN_REQUESTS=2 # Increase from 1 -``` - -**If GoToSocial uses too much memory:** -- Reduce RSS feed count temporarily -- Lower `?limit=` parameters to 50 instead of 100 -- Increase run frequency instead of volume - -**If duplicate detection is slow:** -- Storage cleanup: `docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json` -- This forces fresh state tracking (posts will be reprocessed once) - -## 🎯 Best Practices - -### Scaling Strategy - -1. **Start Conservative:** `limit=50`, `MAX_POSTS_PER_RUN=40` -2. **Monitor Performance:** Check RAM usage and processing time -3. **Scale Gradually:** Increase to `limit=75`, then `limit=100` -4. **Optimize Mix:** Use different limits per instance based on quality - -### Instance Selection - -**High-quality instances for aggressive limits:** -``` -# Tech-focused instances (good signal-to-noise ratio) -https://fosstodon.org/tags/homelab.rss?limit=100 -https://infosec.exchange/tags/security.rss?limit=100 - -# General instances (moderate limits recommended) -https://mastodon.social/tags/technology.rss?limit=50 -``` - -**Performance tip:** Specialized instances often have higher content quality at scale than general-purpose instances. ## Configuration @@ -224,7 +91,7 @@ https://infosec.exchange/tags/selfhosting.rss ``` 📊 GTS-HolMirDas Run Statistics: - ⏱️ Runtime: 0:04:14 + ⏱️ Runtime: 0:04:14 📄 Total posts processed: 45 🌐 Current known instances: 2519 ➕ New instances discovered: +3 @@ -243,7 +110,7 @@ https://infosec.exchange/tags/selfhosting.rss ### Docker Compose (Recommended) ```bash -docker compose up -d +docker-compose up -d ``` ### Standalone Docker @@ -258,7 +125,7 @@ docker run -d --env-file .env \ ## Monitoring -- **Logs**: `docker compose logs -f` +- **Logs**: `docker-compose logs -f` - **Health**: Optional Healthchecks.io integration - **Statistics**: Built-in runtime and performance metrics - **Resource Usage**: Docker stats or container monitoring tools @@ -267,17 +134,20 @@ docker run -d --env-file .env \ ### Common Issues -- **No posts processed**: Check access token permissions and RSS feed URLs -- **Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count -- **High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency -- **Container won't start**: Verify `.env` file format and required variables +**No posts processed**: Check access token permissions and RSS feed URLs + +**Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count + +**High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency + +**Container won't start**: Verify `.env` file format and required variables ### Debug Mode ```bash # Enable debug logging echo "LOG_LEVEL=DEBUG" >> .env -docker compose restart gts-holmirdas +docker-compose restart gts-holmirdas ``` ## Contributing @@ -300,6 +170,6 @@ MIT License - see LICENSE file for details. ## Acknowledgments -- Inspired by [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)) - the original RSS-to-ActivityPub concept +- Inspired by the [HolMirDas](https://github.com/aliceif/HolMirDas) concept - Built for the GoToSocial community - RSS-to-ActivityPub approach inspired by Fediverse discovery challenges diff --git a/gts_holmirdas.py b/gts_holmirdas.py index 642695e..2137c32 100644 --- a/gts_holmirdas.py +++ b/gts_holmirdas.py @@ -1,13 +1,7 @@ #!/usr/bin/env python3 """ GTS-HolMirDas: RSS-based content discovery for GoToSocial - -Inspired by HolMirDas by @aliceif: -- GitHub: https://github.com/aliceif/HolMirDas -- Fediverse: @aliceif@mkultra.x27.one - -This GoToSocial adaptation extends the original RSS-to-ActivityPub concept -with Docker deployment, multi-instance processing, and comprehensive monitoring. +Fetches URLs from RSS feeds and uses GTS search API to federate content """ import os diff --git a/rss_feeds.example.txt b/rss_feeds.example.txt index d532f51..2284fc4 100644 --- a/rss_feeds.example.txt +++ b/rss_feeds.example.txt @@ -1,17 +1,15 @@ # Example RSS feeds - customize for your interests -# Add ?limit=X parameter to increase posts per feed (default: 20, max: 100) -# Higher limits = more content discovery, but longer processing time -# Performance tip: Start with limit=50, then increase to 100 if needed +# homelab +https://mastodon.social/tags/homelab.rss +https://fosstodon.org/tags/homelab.rss -# homelab (up to 100 posts per feed) -https://mastodon.social/tags/homelab.rss # 20 posts/feed (default) -https://fosstodon.org/tags/homelab.rss?limit=50 # 50 posts/feed +# selfhosting +https://mastodon.social/tags/selfhosting.rss +https://infosec.exchange/tags/selfhosting.rss -# selfhosting (up to 100 posts per feed) -https://mastodon.social/tags/selfhosting.rss?limit=100 # 100 posts/feed -https://infosec.exchange/tags/selfhosting.rss?limit=100 # 100 posts/feed +# docker +https://social.tchncs.de/tags/docker.rss +https://fosstodon.org/tags/docker.rss -# docker (up to 100 posts per feed) -https://social.tchncs.de/tags/docker.rss?limit=100 # 100 posts/feed -https://fosstodon.org/tags/docker.rss?limit=100 # 100 posts/feed +# Add your preferred instances and hashtags...