matthias/gts-holmirdas
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

Compare commits

base: matthias:master
Branches Tags
matthias:master
matthias:main
matthias:v1.1.0
matthias:v1.0.1
matthias:v1.0.0
..
compare: matthias:v1.0.0
Branches Tags
matthias:master
matthias:main
matthias:v1.1.0
matthias:v1.0.1
matthias:v1.0.0

No commits in common. "master" and "v1.0.0" have entirely different histories.
master ... v1.0.0

4 changed files with 29 additions and 167 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.env.example
View file

@ -3,7 +3,7 @@ GTS_SERVER_URL=https://your-gts-instance.tld
GTS_ACCESS_TOKEN=your_gts_access_token_here
# Processing Configuration
MAX_POSTS_PER_RUN=75
MAX_POSTS_PER_RUN=25
DELAY_BETWEEN_REQUESTS=1
LOG_LEVEL=INFO

164
README.md
View file

@ -1,11 +1,9 @@
# GTS-HolMirDas 🚀
RSS-based content discovery for **[GoToSocial](https://codeberg.org/superseriousbusiness/gotosocial)** instances.
RSS-based content discovery for GoToSocial instances.
Automatically discovers and federates content from RSS feeds across the Fediverse, helping small GoToSocial instances populate their federated timeline without relying on traditional relays.
*Inspired by the original [HolMirDas](https://github.com/aliceif/HolMirDas) for Misskey by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)), this GoToSocial adaptation extends the RSS-to-ActivityPub concept with enhanced Docker deployment and multi-instance processing.*
## Features
- 📡 **Multi-Instance RSS Discovery** - Fetches content from configurable RSS feeds across Fediverse instances
@ -17,7 +15,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers
## How it Works
**GTS-HolMirDas** reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach:
GTS-HolMirDas reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach:
- Maintains proper ActivityPub federation (posts remain interactive)
- Respects rate limits and instance policies
@ -28,7 +26,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers
```bash
# Clone the repository
git clone https://git.klein.ruhr/matthias/gts-holmirdas
git clone https://your-forgejo-instance.tld/user/gts-holmirdas
cd gts-holmirdas
# Copy configuration templates
@ -40,142 +38,11 @@ nano .env # Add your GTS credentials
nano rss_feeds.txt # Customize RSS feeds
# Deploy
docker compose up -d
docker-compose up -d
# Monitor
docker compose logs -f
docker-compose logs -f
```
# Performance Scaling & Configuration
## 🚀 RSS Feed Optimization (v1.1.0+)
GTS-HolMirDas supports URL parameters to dramatically increase content discovery without additional API calls.
### RSS Feed Limits
Most Mastodon-compatible instances support the `?limit=X` parameter:
```
# Default behavior (20 posts per feed)
https://mastodon.social/tags/homelab.rss
# Increased limits (up to 100 posts per feed)
https://mastodon.social/tags/homelab.rss?limit=50
https://fosstodon.org/tags/docker.rss?limit=100
```
**Supported limits:** 20 (default), 50, 75, 100 (instance-dependent)
### Performance Impact
| Configuration | Posts/Run | API Calls | Processing Time |
|---------------|-----------|-----------|-----------------|
| Standard (limit=20) | ~100 posts | 30+ feeds | 2-5 minutes |
| Optimized (limit=50) | ~300 posts | 30+ feeds | 5-10 minutes |
| Maximum (limit=100) | ~600 posts | 30+ feeds | 8-15 minutes |
## ⚙️ Configuration Tuning
### Environment Variables
```env
# Processing Configuration
MAX_POSTS_PER_RUN=75 # Increase for higher limits
DELAY_BETWEEN_REQUESTS=1 # Balance speed vs. server load
RSS_URLS_FILE=/app/rss_feeds.txt
# Recommended combinations:
# Conservative: MAX_POSTS_PER_RUN=40, limit=50
# Balanced: MAX_POSTS_PER_RUN=75, limit=100
# Aggressive: MAX_POSTS_PER_RUN=100, limit=100
```
### RSS Feed Strategy
```
# Progressive scaling approach:
# 1. Start with mixed limits to test performance
# 2. Increase gradually based on server capacity
# 3. Monitor GoToSocial memory usage
# Example progression:
https://mastodon.social/tags/homelab.rss?limit=50
https://fosstodon.org/tags/selfhosting.rss?limit=75
https://chaos.social/tags/docker.rss?limit=100
```
## 📊 Monitoring & Optimization
### Performance Metrics
The statistics output shows real-time performance:
```
📊 GTS-HolMirDas Run Statistics:
⏱️ Runtime: 0:08:42
📄 Total posts processed: 487
🌐 Current known instances: 3150
➕ New instances discovered: +45
📡 RSS feeds processed: 102
⚡ Posts per minute: 56.0
```
### Optimization Guidelines
**Memory Usage:**
- Monitor GoToSocial memory consumption during runs
- Each 100 additional posts ≈ ~2-5MB additional RAM
- Recommended: 1GB+ RAM for aggressive configurations
**Processing Time:**
- Scales linearly with `MAX_POSTS_PER_RUN × number_of_feeds`
- Duplicate detection becomes more important at scale
- Consider running frequency vs. content volume
**Federation Growth:**
- Higher limits = more diverse instance discovery
- Expect 20-50+ new instances per optimized run
- Balance discovery rate with storage capacity
### Troubleshooting High-Volume Setups
**If processing takes too long:**
```env
MAX_POSTS_PER_RUN=50 # Reduce from 75/100
DELAY_BETWEEN_REQUESTS=2 # Increase from 1
```
**If GoToSocial uses too much memory:**
- Reduce RSS feed count temporarily
- Lower `?limit=` parameters to 50 instead of 100
- Increase run frequency instead of volume
**If duplicate detection is slow:**
- Storage cleanup: `docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json`
- This forces fresh state tracking (posts will be reprocessed once)
## 🎯 Best Practices
### Scaling Strategy
1. **Start Conservative:** `limit=50`, `MAX_POSTS_PER_RUN=40`
2. **Monitor Performance:** Check RAM usage and processing time
3. **Scale Gradually:** Increase to `limit=75`, then `limit=100`
4. **Optimize Mix:** Use different limits per instance based on quality
### Instance Selection
**High-quality instances for aggressive limits:**
```
# Tech-focused instances (good signal-to-noise ratio)
https://fosstodon.org/tags/homelab.rss?limit=100
https://infosec.exchange/tags/security.rss?limit=100
# General instances (moderate limits recommended)
https://mastodon.social/tags/technology.rss?limit=50
```
**Performance tip:** Specialized instances often have higher content quality at scale than general-purpose instances.
## Configuration
@ -224,7 +91,7 @@ https://infosec.exchange/tags/selfhosting.rss
```
📊 GTS-HolMirDas Run Statistics:
⏱️ Runtime: 0:04:14
⏱️ Runtime: 0:04:14
📄 Total posts processed: 45
🌐 Current known instances: 2519
➕ New instances discovered: +3
@ -243,7 +110,7 @@ https://infosec.exchange/tags/selfhosting.rss
### Docker Compose (Recommended)
```bash
docker compose up -d
docker-compose up -d
```
### Standalone Docker
@ -258,7 +125,7 @@ docker run -d --env-file .env \
## Monitoring
- **Logs**: `docker compose logs -f`
- **Logs**: `docker-compose logs -f`
- **Health**: Optional Healthchecks.io integration
- **Statistics**: Built-in runtime and performance metrics
- **Resource Usage**: Docker stats or container monitoring tools
@ -267,17 +134,20 @@ docker run -d --env-file .env \
### Common Issues
- **No posts processed**: Check access token permissions and RSS feed URLs
- **Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count
- **High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency
- **Container won't start**: Verify `.env` file format and required variables
**No posts processed**: Check access token permissions and RSS feed URLs
**Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count
**High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency
**Container won't start**: Verify `.env` file format and required variables
### Debug Mode
```bash
# Enable debug logging
echo "LOG_LEVEL=DEBUG" >> .env
docker compose restart gts-holmirdas
docker-compose restart gts-holmirdas
```
## Contributing
@ -300,6 +170,6 @@ MIT License - see LICENSE file for details.
## Acknowledgments
- Inspired by [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)) - the original RSS-to-ActivityPub concept
- Inspired by the [HolMirDas](https://github.com/aliceif/HolMirDas) concept
- Built for the GoToSocial community
- RSS-to-ActivityPub approach inspired by Fediverse discovery challenges

8
gts_holmirdas.py
View file

@ -1,13 +1,7 @@
#!/usr/bin/env python3
"""
GTS-HolMirDas: RSS-based content discovery for GoToSocial
Inspired by HolMirDas by @aliceif:
- GitHub: https://github.com/aliceif/HolMirDas
- Fediverse: @aliceif@mkultra.x27.one
This GoToSocial adaptation extends the original RSS-to-ActivityPub concept
with Docker deployment, multi-instance processing, and comprehensive monitoring.
Fetches URLs from RSS feeds and uses GTS search API to federate content
"""
import os

22
rss_feeds.example.txt
View file

@ -1,17 +1,15 @@
# Example RSS feeds - customize for your interests
# Add ?limit=X parameter to increase posts per feed (default: 20, max: 100)
# Higher limits = more content discovery, but longer processing time
# Performance tip: Start with limit=50, then increase to 100 if needed
# homelab
https://mastodon.social/tags/homelab.rss
https://fosstodon.org/tags/homelab.rss
# homelab (up to 100 posts per feed)
https://mastodon.social/tags/homelab.rss # 20 posts/feed (default)
https://fosstodon.org/tags/homelab.rss?limit=50 # 50 posts/feed
# selfhosting
https://mastodon.social/tags/selfhosting.rss
https://infosec.exchange/tags/selfhosting.rss
# selfhosting (up to 100 posts per feed)
https://mastodon.social/tags/selfhosting.rss?limit=100 # 100 posts/feed
https://infosec.exchange/tags/selfhosting.rss?limit=100 # 100 posts/feed
# docker
https://social.tchncs.de/tags/docker.rss
https://fosstodon.org/tags/docker.rss
# docker (up to 100 posts per feed)
https://social.tchncs.de/tags/docker.rss?limit=100 # 100 posts/feed
https://fosstodon.org/tags/docker.rss?limit=100 # 100 posts/feed
# Add your preferred instances and hashtags...