Compare commits
No commits in common. "master" and "v1.0.1" have entirely different histories.
3 changed files with 12 additions and 145 deletions
|
@ -3,7 +3,7 @@ GTS_SERVER_URL=https://your-gts-instance.tld
|
||||||
GTS_ACCESS_TOKEN=your_gts_access_token_here
|
GTS_ACCESS_TOKEN=your_gts_access_token_here
|
||||||
|
|
||||||
# Processing Configuration
|
# Processing Configuration
|
||||||
MAX_POSTS_PER_RUN=75
|
MAX_POSTS_PER_RUN=25
|
||||||
DELAY_BETWEEN_REQUESTS=1
|
DELAY_BETWEEN_REQUESTS=1
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
|
|
||||||
|
|
133
README.md
133
README.md
|
@ -28,7 +28,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clone the repository
|
# Clone the repository
|
||||||
git clone https://git.klein.ruhr/matthias/gts-holmirdas
|
git clone https://git.klein.ruhr/user/gts-holmirdas
|
||||||
cd gts-holmirdas
|
cd gts-holmirdas
|
||||||
|
|
||||||
# Copy configuration templates
|
# Copy configuration templates
|
||||||
|
@ -45,137 +45,6 @@ docker compose up -d
|
||||||
# Monitor
|
# Monitor
|
||||||
docker compose logs -f
|
docker compose logs -f
|
||||||
```
|
```
|
||||||
# Performance Scaling & Configuration
|
|
||||||
|
|
||||||
## 🚀 RSS Feed Optimization (v1.1.0+)
|
|
||||||
|
|
||||||
GTS-HolMirDas supports URL parameters to dramatically increase content discovery without additional API calls.
|
|
||||||
|
|
||||||
### RSS Feed Limits
|
|
||||||
|
|
||||||
Most Mastodon-compatible instances support the `?limit=X` parameter:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Default behavior (20 posts per feed)
|
|
||||||
https://mastodon.social/tags/homelab.rss
|
|
||||||
|
|
||||||
# Increased limits (up to 100 posts per feed)
|
|
||||||
https://mastodon.social/tags/homelab.rss?limit=50
|
|
||||||
https://fosstodon.org/tags/docker.rss?limit=100
|
|
||||||
```
|
|
||||||
|
|
||||||
**Supported limits:** 20 (default), 50, 75, 100 (instance-dependent)
|
|
||||||
|
|
||||||
### Performance Impact
|
|
||||||
|
|
||||||
| Configuration | Posts/Run | API Calls | Processing Time |
|
|
||||||
|---------------|-----------|-----------|-----------------|
|
|
||||||
| Standard (limit=20) | ~100 posts | 30+ feeds | 2-5 minutes |
|
|
||||||
| Optimized (limit=50) | ~300 posts | 30+ feeds | 5-10 minutes |
|
|
||||||
| Maximum (limit=100) | ~600 posts | 30+ feeds | 8-15 minutes |
|
|
||||||
|
|
||||||
## ⚙️ Configuration Tuning
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
```env
|
|
||||||
# Processing Configuration
|
|
||||||
MAX_POSTS_PER_RUN=75 # Increase for higher limits
|
|
||||||
DELAY_BETWEEN_REQUESTS=1 # Balance speed vs. server load
|
|
||||||
RSS_URLS_FILE=/app/rss_feeds.txt
|
|
||||||
|
|
||||||
# Recommended combinations:
|
|
||||||
# Conservative: MAX_POSTS_PER_RUN=40, limit=50
|
|
||||||
# Balanced: MAX_POSTS_PER_RUN=75, limit=100
|
|
||||||
# Aggressive: MAX_POSTS_PER_RUN=100, limit=100
|
|
||||||
```
|
|
||||||
|
|
||||||
### RSS Feed Strategy
|
|
||||||
|
|
||||||
```
|
|
||||||
# Progressive scaling approach:
|
|
||||||
# 1. Start with mixed limits to test performance
|
|
||||||
# 2. Increase gradually based on server capacity
|
|
||||||
# 3. Monitor GoToSocial memory usage
|
|
||||||
|
|
||||||
# Example progression:
|
|
||||||
https://mastodon.social/tags/homelab.rss?limit=50
|
|
||||||
https://fosstodon.org/tags/selfhosting.rss?limit=75
|
|
||||||
https://chaos.social/tags/docker.rss?limit=100
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📊 Monitoring & Optimization
|
|
||||||
|
|
||||||
### Performance Metrics
|
|
||||||
|
|
||||||
The statistics output shows real-time performance:
|
|
||||||
|
|
||||||
```
|
|
||||||
📊 GTS-HolMirDas Run Statistics:
|
|
||||||
⏱️ Runtime: 0:08:42
|
|
||||||
📄 Total posts processed: 487
|
|
||||||
🌐 Current known instances: 3150
|
|
||||||
➕ New instances discovered: +45
|
|
||||||
📡 RSS feeds processed: 102
|
|
||||||
⚡ Posts per minute: 56.0
|
|
||||||
```
|
|
||||||
|
|
||||||
### Optimization Guidelines
|
|
||||||
|
|
||||||
**Memory Usage:**
|
|
||||||
- Monitor GoToSocial memory consumption during runs
|
|
||||||
- Each 100 additional posts ≈ ~2-5MB additional RAM
|
|
||||||
- Recommended: 1GB+ RAM for aggressive configurations
|
|
||||||
|
|
||||||
**Processing Time:**
|
|
||||||
- Scales linearly with `MAX_POSTS_PER_RUN × number_of_feeds`
|
|
||||||
- Duplicate detection becomes more important at scale
|
|
||||||
- Consider running frequency vs. content volume
|
|
||||||
|
|
||||||
**Federation Growth:**
|
|
||||||
- Higher limits = more diverse instance discovery
|
|
||||||
- Expect 20-50+ new instances per optimized run
|
|
||||||
- Balance discovery rate with storage capacity
|
|
||||||
|
|
||||||
### Troubleshooting High-Volume Setups
|
|
||||||
|
|
||||||
**If processing takes too long:**
|
|
||||||
```env
|
|
||||||
MAX_POSTS_PER_RUN=50 # Reduce from 75/100
|
|
||||||
DELAY_BETWEEN_REQUESTS=2 # Increase from 1
|
|
||||||
```
|
|
||||||
|
|
||||||
**If GoToSocial uses too much memory:**
|
|
||||||
- Reduce RSS feed count temporarily
|
|
||||||
- Lower `?limit=` parameters to 50 instead of 100
|
|
||||||
- Increase run frequency instead of volume
|
|
||||||
|
|
||||||
**If duplicate detection is slow:**
|
|
||||||
- Storage cleanup: `docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json`
|
|
||||||
- This forces fresh state tracking (posts will be reprocessed once)
|
|
||||||
|
|
||||||
## 🎯 Best Practices
|
|
||||||
|
|
||||||
### Scaling Strategy
|
|
||||||
|
|
||||||
1. **Start Conservative:** `limit=50`, `MAX_POSTS_PER_RUN=40`
|
|
||||||
2. **Monitor Performance:** Check RAM usage and processing time
|
|
||||||
3. **Scale Gradually:** Increase to `limit=75`, then `limit=100`
|
|
||||||
4. **Optimize Mix:** Use different limits per instance based on quality
|
|
||||||
|
|
||||||
### Instance Selection
|
|
||||||
|
|
||||||
**High-quality instances for aggressive limits:**
|
|
||||||
```
|
|
||||||
# Tech-focused instances (good signal-to-noise ratio)
|
|
||||||
https://fosstodon.org/tags/homelab.rss?limit=100
|
|
||||||
https://infosec.exchange/tags/security.rss?limit=100
|
|
||||||
|
|
||||||
# General instances (moderate limits recommended)
|
|
||||||
https://mastodon.social/tags/technology.rss?limit=50
|
|
||||||
```
|
|
||||||
|
|
||||||
**Performance tip:** Specialized instances often have higher content quality at scale than general-purpose instances.
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
|
|
|
@ -1,17 +1,15 @@
|
||||||
# Example RSS feeds - customize for your interests
|
# Example RSS feeds - customize for your interests
|
||||||
|
|
||||||
# Add ?limit=X parameter to increase posts per feed (default: 20, max: 100)
|
# homelab
|
||||||
# Higher limits = more content discovery, but longer processing time
|
https://mastodon.social/tags/homelab.rss
|
||||||
# Performance tip: Start with limit=50, then increase to 100 if needed
|
https://fosstodon.org/tags/homelab.rss
|
||||||
|
|
||||||
# homelab (up to 100 posts per feed)
|
# selfhosting
|
||||||
https://mastodon.social/tags/homelab.rss # 20 posts/feed (default)
|
https://mastodon.social/tags/selfhosting.rss
|
||||||
https://fosstodon.org/tags/homelab.rss?limit=50 # 50 posts/feed
|
https://infosec.exchange/tags/selfhosting.rss
|
||||||
|
|
||||||
# selfhosting (up to 100 posts per feed)
|
# docker
|
||||||
https://mastodon.social/tags/selfhosting.rss?limit=100 # 100 posts/feed
|
https://social.tchncs.de/tags/docker.rss
|
||||||
https://infosec.exchange/tags/selfhosting.rss?limit=100 # 100 posts/feed
|
https://fosstodon.org/tags/docker.rss
|
||||||
|
|
||||||
# docker (up to 100 posts per feed)
|
# Add your preferred instances and hashtags...
|
||||||
https://social.tchncs.de/tags/docker.rss?limit=100 # 100 posts/feed
|
|
||||||
https://fosstodon.org/tags/docker.rss?limit=100 # 100 posts/feed
|
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue