diff --git a/.env.example b/.env.example index 2593cf5..23d6c88 100644 --- a/.env.example +++ b/.env.example @@ -3,7 +3,7 @@ GTS_SERVER_URL=https://your-gts-instance.tld GTS_ACCESS_TOKEN=your_gts_access_token_here # Processing Configuration -MAX_POSTS_PER_RUN=75 +MAX_POSTS_PER_RUN=25 DELAY_BETWEEN_REQUESTS=1 LOG_LEVEL=INFO diff --git a/README.md b/README.md index b23f68e..ca316f2 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,34 @@ # GTS-HolMirDas 🚀 -RSS-based content discovery for [GoToSocial](https://codeberg.org/superseriousbusiness/gotosocial) instances. +RSS-based content discovery for **[GoToSocial](https://codeberg.org/superseriousbusiness/gotosocial)** instances. Automatically discovers and federates content from RSS feeds across the Fediverse, helping small GoToSocial instances populate their federated timeline without relying on traditional relays. -Inspired by the original [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://mkultra.x27.one/@aliceif), adapted for GoToSocial with enhanced Docker deployment and multi-instance processing. +*Inspired by the original [HolMirDas](https://github.com/aliceif/HolMirDas) for Misskey by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)), this GoToSocial adaptation extends the RSS-to-ActivityPub concept with enhanced Docker deployment and multi-instance processing.* -## ✨ Key Features +## Features -- **📡 Multi-Instance Discovery** - Fetches content from configurable RSS feeds across Fediverse instances -- **⚡ Performance Scaling** - 20-100 posts per feed with URL parameters (`?limit=100`) -- **🐳 Production Ready** - Docker deployment, environment-based config, health monitoring -- **📊 Comprehensive Stats** - Runtime metrics, federation growth, performance tracking -- **🔧 Zero Maintenance** - Runs automatically every hour with duplicate detection +- 📡 **Multi-Instance RSS Discovery** - Fetches content from configurable RSS feeds across Fediverse instances +- ⚡ **Efficient Processing** - Configurable rate limiting and duplicate detection +- 🔧 **Production Ready** - Environment-based config, Docker deployment, health monitoring +- 📊 **Comprehensive Statistics** - Runtime metrics, content processing, and federation growth tracking +- 🐳 **Containerized** - Simple Docker Compose deployment +- 📁 **File-based Configuration** - Easy RSS feed management via text files -## 🚀 Quick Start +## How it Works + +**GTS-HolMirDas** reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach: + +- Maintains proper ActivityPub federation (posts remain interactive) +- Respects rate limits and instance policies +- Provides better content discovery for small instances +- Works alongside tools like FediFetcher for comprehensive federation + +## Quick Start ```bash # Clone the repository -git clone https://git.klein.ruhr/matthias/gts-holmirdas +git clone https://git.klein.ruhr/user/gts-holmirdas cd gts-holmirdas # Copy configuration templates @@ -26,8 +36,8 @@ cp .env.example .env cp rss_feeds.example.txt rss_feeds.txt # Edit configuration -nano .env # Add your GTS credentials -nano rss_feeds.txt # Customize RSS feeds +nano .env # Add your GTS credentials +nano rss_feeds.txt # Customize RSS feeds # Deploy docker compose up -d @@ -36,77 +46,129 @@ docker compose up -d docker compose logs -f ``` -## 📈 Performance at Scale - -**Real Production Data:** -``` -📊 Runtime: 8:42 | 487 posts processed | 3,150+ instances discovered -⚡ 56 posts/minute | 102 RSS feeds | +45 new instances per run -💾 Resource usage: ~450MB RAM total (GoToSocial + tools) -``` - -**Scaling Options:** -- **Conservative:** 20 posts/feed (~100 posts/run) -- **Balanced:** 50 posts/feed (~300 posts/run) -- **Aggressive:** 100 posts/feed (~600 posts/run) - -## 🛠️ Configuration Essentials +## Configuration ### Environment Variables (.env) + ```bash -# Required +# GTS Server Configuration GTS_SERVER_URL=https://your-gts-instance.tld GTS_ACCESS_TOKEN=your_gts_access_token -# Performance Tuning +# Processing Configuration MAX_POSTS_PER_RUN=25 # Posts per feed per run DELAY_BETWEEN_REQUESTS=1 # Seconds between API calls -LOG_LEVEL=INFO # DEBUG for troubleshooting +LOG_LEVEL=INFO # Logging verbosity + +# RSS Configuration +RSS_URLS_FILE=/app/rss_feeds.txt # Path to RSS feeds file + +# Optional: Monitoring +HEALTHCHECK_URL=https://hc-ping.com/your-uuid ``` ### RSS Feeds (rss_feeds.txt) -```bash -# Use URL parameters to scale performance -https://mastodon.social/tags/homelab.rss?limit=50 -https://fosstodon.org/tags/selfhosting.rss?limit=100 -https://infosec.exchange/tags/security.rss?limit=75 + +``` +# Example RSS feeds - customize for your interests +# homelab +https://mastodon.social/tags/homelab.rss +https://fosstodon.org/tags/homelab.rss + +# selfhosting +https://mastodon.social/tags/selfhosting.rss +https://infosec.exchange/tags/selfhosting.rss + +# Add your preferred instances and hashtags ``` -### GoToSocial Access Token +## Access Token Setup + 1. Login to your GoToSocial instance -2. Settings → Applications → Create new application -3. Required scopes: `read`, `read:search`, `read:statuses` -4. Copy access token to `.env` file +2. Go to Settings → Applications +3. Create new application with scopes: `read`, `read:search`, `read:statuses` +4. Copy the access token to your `.env` file -## 📖 Complete Documentation +## Statistics Output -For detailed information, visit our **[Wiki](https://git.klein.ruhr/matthias/gts-holmirdas/wiki)**: +``` +📊 GTS-HolMirDas Run Statistics: + ⏱️ Runtime: 0:04:14 + 📄 Total posts processed: 45 + 🌐 Current known instances: 2519 + ➕ New instances discovered: +3 + 📡 RSS feeds processed: 25 + ⚡ Posts per minute: 10.6 +``` -- **[📋 Installation Guide](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/Installation-Guide.-)** - Detailed setup, Docker configuration, deployment options -- **[📈 Performance & Scaling](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/Performance-%26-Scaling)** - Optimization tables, scaling strategies, resource planning -- **[🛠️ Troubleshooting](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/Troubleshooting)** - Common issues, Docker problems, debugging guide -- **[⚙️ Advanced Configuration](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/Advanced-Configuration)** - Environment variables, RSS strategies, production tips -- **[📊 Monitoring & Stats](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/Monitoring-%26-Stats)** - Understanding output, health monitoring, metrics -- **[❓ FAQ](https://git.klein.ruhr/matthias/gts-holmirdas/wiki/FAQ+-+Frequently+Asked+Questions.-)** - Common questions and answers +## Resource Requirements -## 🤝 Community & Support +- **Memory**: ~200-500MB depending on feed count +- **CPU**: Minimal (mostly I/O bound) +- **Storage**: <100MB for application, plus log storage +- **Network**: Depends on RSS feed count and frequency -- **[Contributing Guide](Contributing)** - Development setup and contribution guidelines *(coming soon)* -- **Issues**: [Report bugs or request features](https://git.klein.ruhr/matthias/gts-holmirdas/issues) -- **Contact**: [@matthias@me.klein.ruhr](https://me.klein.ruhr/@matthias) on the Fediverse +## Deployment Options -## 🔗 Related Projects +### Docker Compose (Recommended) +```bash +docker compose up -d +``` -- **[FediFetcher](https://github.com/nanos/fedifetcher)** - Fetches missing replies and posts -- **[GoToSocial](https://github.com/superseriousbusiness/gotosocial)** - Lightweight ActivityPub server -- **[slurp](https://github.com/VyrCossont/slurp)** - Import posts from other instances +### Standalone Docker +```bash +docker build -t gts-holmirdas . +docker run -d --env-file .env \ + -v ./data:/app/data \ + -v ./gts_holmirdas.py:/app/gts_holmirdas.py:ro \ + -v ./rss_feeds.txt:/app/rss_feeds.txt:ro \ + gts-holmirdas +``` -## 📄 License +## Monitoring -MIT License - see [LICENSE](LICENSE) file for details. +- **Logs**: `docker compose logs -f` +- **Health**: Optional Healthchecks.io integration +- **Statistics**: Built-in runtime and performance metrics +- **Resource Usage**: Docker stats or container monitoring tools -## 🙏 Acknowledgments +## Troubleshooting -- Inspired by [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://mkultra.x27.one/@aliceif) +### Common Issues + +- **No posts processed**: Check access token permissions and RSS feed URLs +- **Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count +- **High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency +- **Container won't start**: Verify `.env` file format and required variables + +### Debug Mode + +```bash +# Enable debug logging +echo "LOG_LEVEL=DEBUG" >> .env +docker compose restart gts-holmirdas +``` + +## Contributing + +1. Fork the repository +2. Create a feature branch +3. Make your changes +4. Test thoroughly +5. Submit a pull request + +## Related Projects + +- [FediFetcher](https://github.com/nanos/fedifetcher) - Fetches missing replies and posts +- [GoToSocial](https://github.com/superseriousbusiness/gotosocial) - Lightweight ActivityPub server +- [slurp](https://github.com/VyrCossont/slurp) - Import posts from other instances + +## License + +MIT License - see LICENSE file for details. + +## Acknowledgments + +- Inspired by [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)) - the original RSS-to-ActivityPub concept - Built for the GoToSocial community -- RSS-to-ActivityPub federation approach \ No newline at end of file +- RSS-to-ActivityPub approach inspired by Fediverse discovery challenges diff --git a/gts_holmirdas.py b/gts_holmirdas.py index 77d1eaf..642695e 100644 --- a/gts_holmirdas.py +++ b/gts_holmirdas.py @@ -46,8 +46,8 @@ class GTSHolMirDas: try: with open(rss_urls_file, 'r') as f: self.config["rss_urls"] = [ - line.split('#', 1)[0].strip() for line in f - if line.strip() and not line.strip().startswith('#') + line.strip() for line in f + if line.strip() and not line.startswith('#') ] self.logger.info(f"Loaded {len(self.config['rss_urls'])} RSS URLs from file: {rss_urls_file}") except Exception as e: diff --git a/rss_feeds.example.txt b/rss_feeds.example.txt index d532f51..2284fc4 100644 --- a/rss_feeds.example.txt +++ b/rss_feeds.example.txt @@ -1,17 +1,15 @@ # Example RSS feeds - customize for your interests -# Add ?limit=X parameter to increase posts per feed (default: 20, max: 100) -# Higher limits = more content discovery, but longer processing time -# Performance tip: Start with limit=50, then increase to 100 if needed +# homelab +https://mastodon.social/tags/homelab.rss +https://fosstodon.org/tags/homelab.rss -# homelab (up to 100 posts per feed) -https://mastodon.social/tags/homelab.rss # 20 posts/feed (default) -https://fosstodon.org/tags/homelab.rss?limit=50 # 50 posts/feed +# selfhosting +https://mastodon.social/tags/selfhosting.rss +https://infosec.exchange/tags/selfhosting.rss -# selfhosting (up to 100 posts per feed) -https://mastodon.social/tags/selfhosting.rss?limit=100 # 100 posts/feed -https://infosec.exchange/tags/selfhosting.rss?limit=100 # 100 posts/feed +# docker +https://social.tchncs.de/tags/docker.rss +https://fosstodon.org/tags/docker.rss -# docker (up to 100 posts per feed) -https://social.tchncs.de/tags/docker.rss?limit=100 # 100 posts/feed -https://fosstodon.org/tags/docker.rss?limit=100 # 100 posts/feed +# Add your preferred instances and hashtags...