Merge branch 'master' of https://git.klein.ruhr/matthias/gts-holmirdas

- Add support for ?limit= URL parameters (up to 100 posts/feed)
- Update documentation with performance scaling guidelines
- Add progressive scaling examples in rss_feeds.example.txt
- Include optimization strategies and troubleshooting
- Maintain backward compatibility with standard RSS feeds

.env.example

@@ -3,7 +3,7 @@ GTS_SERVER_URL=https://your-gts-instance.tld
 GTS_ACCESS_TOKEN=your_gts_access_token_here
 
 # Processing Configuration
-MAX_POSTS_PER_RUN=25
+MAX_POSTS_PER_RUN=75
 DELAY_BETWEEN_REQUESTS=1
 LOG_LEVEL=INFO
 

README.md

@@ -1,9 +1,11 @@
 # GTS-HolMirDas 🚀
 
-RSS-based content discovery for GoToSocial instances.
+RSS-based content discovery for **[GoToSocial](https://codeberg.org/superseriousbusiness/gotosocial)** instances.
 
 Automatically discovers and federates content from RSS feeds across the Fediverse, helping small GoToSocial instances populate their federated timeline without relying on traditional relays.
 
+*Inspired by the original [HolMirDas](https://github.com/aliceif/HolMirDas) for Misskey by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)), this GoToSocial adaptation extends the RSS-to-ActivityPub concept with enhanced Docker deployment and multi-instance processing.*
+
 ## Features
 
 - 📡 **Multi-Instance RSS Discovery** - Fetches content from configurable RSS feeds across Fediverse instances
@@ -15,7 +17,7 @@ Automatically discovers and federates content from RSS feeds across the Fedivers
 
 ## How it Works
 
-GTS-HolMirDas reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach:
+**GTS-HolMirDas** reads RSS feeds from various Fediverse instances and uses GoToSocial's search API to federate the discovered content. This approach:
 
 - Maintains proper ActivityPub federation (posts remain interactive)
 - Respects rate limits and instance policies
@@ -26,7 +28,7 @@ GTS-HolMirDas reads RSS feeds from various Fediverse instances and uses GoToSoci
 
 ```bash
 # Clone the repository
-git clone https://your-forgejo-instance.tld/user/gts-holmirdas
+git clone https://git.klein.ruhr/matthias/gts-holmirdas
 cd gts-holmirdas
 
 # Copy configuration templates
@@ -38,11 +40,142 @@ nano .env          # Add your GTS credentials
 nano rss_feeds.txt # Customize RSS feeds
 
 # Deploy
-docker-compose up -d
+docker compose up -d
 
 # Monitor
-docker-compose logs -f
+docker compose logs -f
 ```
+# Performance Scaling & Configuration
+
+## 🚀 RSS Feed Optimization (v1.1.0+)
+
+GTS-HolMirDas supports URL parameters to dramatically increase content discovery without additional API calls.
+
+### RSS Feed Limits
+
+Most Mastodon-compatible instances support the `?limit=X` parameter:
+
+```
+# Default behavior (20 posts per feed)
+https://mastodon.social/tags/homelab.rss
+
+# Increased limits (up to 100 posts per feed)
+https://mastodon.social/tags/homelab.rss?limit=50
+https://fosstodon.org/tags/docker.rss?limit=100
+```
+
+**Supported limits:** 20 (default), 50, 75, 100 (instance-dependent)
+
+### Performance Impact
+
+| Configuration | Posts/Run | API Calls | Processing Time |
+|---------------|-----------|-----------|-----------------|
+| Standard (limit=20) | ~100 posts | 30+ feeds | 2-5 minutes |
+| Optimized (limit=50) | ~300 posts | 30+ feeds | 5-10 minutes |
+| Maximum (limit=100) | ~600 posts | 30+ feeds | 8-15 minutes |
+
+## ⚙️ Configuration Tuning
+
+### Environment Variables
+
+```env
+# Processing Configuration
+MAX_POSTS_PER_RUN=75          # Increase for higher limits
+DELAY_BETWEEN_REQUESTS=1      # Balance speed vs. server load
+RSS_URLS_FILE=/app/rss_feeds.txt
+
+# Recommended combinations:
+# Conservative: MAX_POSTS_PER_RUN=40, limit=50
+# Balanced:     MAX_POSTS_PER_RUN=75, limit=100  
+# Aggressive:   MAX_POSTS_PER_RUN=100, limit=100
+```
+
+### RSS Feed Strategy
+
+```
+# Progressive scaling approach:
+# 1. Start with mixed limits to test performance
+# 2. Increase gradually based on server capacity
+# 3. Monitor GoToSocial memory usage
+
+# Example progression:
+https://mastodon.social/tags/homelab.rss?limit=50
+https://fosstodon.org/tags/selfhosting.rss?limit=75
+https://chaos.social/tags/docker.rss?limit=100
+```
+
+## 📊 Monitoring & Optimization
+
+### Performance Metrics
+
+The statistics output shows real-time performance:
+
+```
+📊 GTS-HolMirDas Run Statistics:
+   ⏱️  Runtime: 0:08:42
+   📄 Total posts processed: 487
+   🌐 Current known instances: 3150
+   ➕ New instances discovered: +45
+   📡 RSS feeds processed: 102
+   ⚡ Posts per minute: 56.0
+```
+
+### Optimization Guidelines
+
+**Memory Usage:**
+- Monitor GoToSocial memory consumption during runs
+- Each 100 additional posts ≈ ~2-5MB additional RAM
+- Recommended: 1GB+ RAM for aggressive configurations
+
+**Processing Time:**
+- Scales linearly with `MAX_POSTS_PER_RUN × number_of_feeds`
+- Duplicate detection becomes more important at scale
+- Consider running frequency vs. content volume
+
+**Federation Growth:**
+- Higher limits = more diverse instance discovery
+- Expect 20-50+ new instances per optimized run
+- Balance discovery rate with storage capacity
+
+### Troubleshooting High-Volume Setups
+
+**If processing takes too long:**
+```env
+MAX_POSTS_PER_RUN=50          # Reduce from 75/100
+DELAY_BETWEEN_REQUESTS=2      # Increase from 1
+```
+
+**If GoToSocial uses too much memory:**
+- Reduce RSS feed count temporarily
+- Lower `?limit=` parameters to 50 instead of 100
+- Increase run frequency instead of volume
+
+**If duplicate detection is slow:**
+- Storage cleanup: `docker-compose exec gts-holmirdas rm -f /app/data/processed_urls.json`
+- This forces fresh state tracking (posts will be reprocessed once)
+
+## 🎯 Best Practices
+
+### Scaling Strategy
+
+1. **Start Conservative:** `limit=50`, `MAX_POSTS_PER_RUN=40`
+2. **Monitor Performance:** Check RAM usage and processing time
+3. **Scale Gradually:** Increase to `limit=75`, then `limit=100`
+4. **Optimize Mix:** Use different limits per instance based on quality
+
+### Instance Selection
+
+**High-quality instances for aggressive limits:**
+```
+# Tech-focused instances (good signal-to-noise ratio)
+https://fosstodon.org/tags/homelab.rss?limit=100
+https://infosec.exchange/tags/security.rss?limit=100
+
+# General instances (moderate limits recommended)  
+https://mastodon.social/tags/technology.rss?limit=50
+```
+
+**Performance tip:** Specialized instances often have higher content quality at scale than general-purpose instances.
 
 ## Configuration
 
@@ -91,7 +224,7 @@ https://infosec.exchange/tags/selfhosting.rss
 
 ```
 📊 GTS-HolMirDas Run Statistics:
-   ⏱️  Runtime: 0:04:14
+   ⏱️ Runtime: 0:04:14
    📄 Total posts processed: 45
    🌐 Current known instances: 2519
    ➕ New instances discovered: +3
@@ -110,7 +243,7 @@ https://infosec.exchange/tags/selfhosting.rss
 
 ### Docker Compose (Recommended)
 ```bash
-docker-compose up -d
+docker compose up -d
 ```
 
 ### Standalone Docker
@@ -125,7 +258,7 @@ docker run -d --env-file .env \
 
 ## Monitoring
 
-- **Logs**: `docker-compose logs -f`
+- **Logs**: `docker compose logs -f`
 - **Health**: Optional Healthchecks.io integration
 - **Statistics**: Built-in runtime and performance metrics
 - **Resource Usage**: Docker stats or container monitoring tools
@@ -134,20 +267,17 @@ docker run -d --env-file .env \
 
 ### Common Issues
 
-**No posts processed**: Check access token permissions and RSS feed URLs
+- **No posts processed**: Check access token permissions and RSS feed URLs
-
+- **Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count
-**Rate limiting errors**: Increase `DELAY_BETWEEN_REQUESTS` or reduce feed count
+- **High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency
-
+- **Container won't start**: Verify `.env` file format and required variables
-**High memory usage**: Reduce `MAX_POSTS_PER_RUN` or feed frequency
-
-**Container won't start**: Verify `.env` file format and required variables
 
 ### Debug Mode
 
 ```bash
 # Enable debug logging
 echo "LOG_LEVEL=DEBUG" >> .env
-docker-compose restart gts-holmirdas
+docker compose restart gts-holmirdas
 ```
 
 ## Contributing
@@ -170,6 +300,6 @@ MIT License - see LICENSE file for details.
 
 ## Acknowledgments
 
-- Inspired by the [HolMirDas](https://github.com/aliceif/HolMirDas) concept
+- Inspired by [HolMirDas](https://github.com/aliceif/HolMirDas) by [@aliceif](https://github.com/aliceif) ([@aliceif@mkultra.x27.one](https://mkultra.x27.one/@aliceif)) - the original RSS-to-ActivityPub concept
 - Built for the GoToSocial community
 - RSS-to-ActivityPub approach inspired by Fediverse discovery challenges

gts_holmirdas.py

@@ -1,7 +1,13 @@
 #!/usr/bin/env python3
 """
 GTS-HolMirDas: RSS-based content discovery for GoToSocial
-Fetches URLs from RSS feeds and uses GTS search API to federate content
+
+Inspired by HolMirDas by @aliceif:
+- GitHub: https://github.com/aliceif/HolMirDas
+- Fediverse: @aliceif@mkultra.x27.one
+
+This GoToSocial adaptation extends the original RSS-to-ActivityPub concept
+with Docker deployment, multi-instance processing, and comprehensive monitoring.
 """
 
 import os

rss_feeds.example.txt

@@ -1,15 +1,17 @@
 # Example RSS feeds - customize for your interests
 
-# homelab
+# Add ?limit=X parameter to increase posts per feed (default: 20, max: 100)
-https://mastodon.social/tags/homelab.rss
+# Higher limits = more content discovery, but longer processing time
-https://fosstodon.org/tags/homelab.rss
+# Performance tip: Start with limit=50, then increase to 100 if needed
 
-# selfhosting  
+# homelab (up to 100 posts per feed)
-https://mastodon.social/tags/selfhosting.rss
+https://mastodon.social/tags/homelab.rss                 #  20 posts/feed (default)
-https://infosec.exchange/tags/selfhosting.rss
+https://fosstodon.org/tags/homelab.rss?limit=50          #  50 posts/feed
 
-# docker
+# selfhosting (up to 100 posts per feed)  
-https://social.tchncs.de/tags/docker.rss
+https://mastodon.social/tags/selfhosting.rss?limit=100   # 100 posts/feed
-https://fosstodon.org/tags/docker.rss
+https://infosec.exchange/tags/selfhosting.rss?limit=100  # 100 posts/feed
 
-# Add your preferred instances and hashtags...
+# docker (up to 100 posts per feed)
+https://social.tchncs.de/tags/docker.rss?limit=100       # 100 posts/feed
+https://fosstodon.org/tags/docker.rss?limit=100          # 100 posts/feed