phil/Podcast-Downloader
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated README.md file for new version of the script

Browse Source
This commit is contained in:
Phil
2026-07-24 22:29:24 +01:00
parent 5bb7703918
commit 79f98c74c1
1 changed files with 181 additions and 534 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+181 -534
View File
@@ -1,539 +1,186 @@
# Podcast-Downloader
# Podcast Downloader
A **bash script** that automatically downloads videos from YouTube playlists using `yt-dlp`, maintains a local SQLite database of downloaded content, and prevents duplicate downloads. Supports multiple playlists with individual destination folders and **cron job integration** for automated scheduling.
---
## Features
| Feature | Description |
|---------|-------------|
| **Multi-Playlist Support** | Configure and download from multiple YouTube playlists simultaneously |
| **Duplicate Prevention** | SQLite database tracks all downloaded videos to prevent re-downloading |
| **Individual Destinations** | Assign a different destination folder for each playlist |
| **Cron-Compatible** | Run automatically on a schedule with proper logging |
| **Download Sessions** | Detailed statistics on each download run (downloaded, skipped, errors) |
| **Database Statistics** | View total videos downloaded, storage used, and per-playlist breakdowns |
| **Configurable Format** | Specify video format, download delays, and audio-only mode via config |
| **Automatic Cleanup** | Removes temporary download folders after successful transfers |
| **Colored Output** | Terminal-friendly logging (disabled in cron mode) |
---
## Prerequisites
Before using this script, ensure you have the following tools installed:
### Required
- **bash** (version 4.0 or higher)
- **yt-dlp** – YouTube video downloader ([install](https://github.com/yt-dlp/yt-dlp#installation))
- **sqlite3** – Database management
- **jq** – JSON parser for config file handling
### Installation
**Ubuntu/Debian:**
```bash
sudo apt-get update
sudo apt-get install yt-dlp sqlite3 jq
```
**macOS:**
```bash
brew install yt-dlp sqlite3 jq
```
**Fedora/RHEL:**
```bash
sudo dnf install yt-dlp sqlite3 jq
```
---
## Installation
### 1. Clone or Download the Script
```bash
mkdir -p ~/.youtube_downloader
cd ~/.youtube_downloader
# Download or clone the script here
chmod +x youtube_downloader.sh
```
### 2. Create Configuration File
Create the config file at `~/.config/youtube_downloader/config.json`:
```bash
mkdir -p ~/.config/youtube_downloader
```
Then create `~/.config/youtube_downloader/config.json` with the following structure:
```json
{
"general": {
"db_dir": "$HOME/.youtube_downloader",
"temp_download_dir": "/tmp/youtube_downloads",
"video_format": "best[ext=mp4]",
"output_template": "%(title)s.%(ext)s",
"audio_only": false,
"download_delay": 2,
"debug": false
},
"playlists": [
{
"name": "My Music Playlist",
"url": "https://www.youtube.com/playlist?list=PLAYLIST_ID_1",
"destination": "/home/user/Music/YouTubeDownloads",
"enabled": true
},
{
"name": "Tutorial Playlist",
"url": "https://www.youtube.com/playlist?list=PLAYLIST_ID_2",
"destination": "/home/user/Videos/Tutorials",
"enabled": true
},
{
"name": "Disabled Playlist",
"url": "https://www.youtube.com/playlist?list=PLAYLIST_ID_3",
"destination": "/home/user/Videos/Archive",
"enabled": false
}
]
}
```
### 3. Find Your Playlist IDs
To get a playlist ID:
1. Go to a YouTube playlist
2. Look at the URL: `https://www.youtube.com/playlist?list=PLAYLIST_ID_HERE`
3. Copy the part after `list=`
---
## Configuration Guide
### General Settings
| Setting | Description | Example |
|---------|-------------|---------|
| `db_dir` | Directory for database and logs | `$HOME/.youtube_downloader` |
| `temp_download_dir` | Temporary folder for downloads (cleaned up after) | `/tmp/youtube_downloads` |
| `video_format` | yt-dlp format specification | `best[ext=mp4]` or `worst` |
| `output_template` | Filename pattern (yt-dlp variables) | `%(title)s.%(ext)s` |
| `audio_only` | Download audio only (ignores video_format) | `true` or `false` |
| `download_delay` | Seconds between downloads (avoids rate-limiting) | `2` |
| `debug` | Enable debug logging | `true` or `false` |
### Video Format Examples
```json
"best[ext=mp4]" // Best quality MP4
"worst" // Lowest quality (smallest file)
"bestvideo+bestaudio" // Best video + best audio (merged)
"bestvideo[height<=720]" // Max 720p
```
### Playlist Settings
| Setting | Description | Example |
|---------|-------------|---------|
| `name` | Display name for the playlist | `"My Music Playlist"` |
| `url` | Full YouTube playlist URL | `https://www.youtube.com/playlist?list=...` |
| `destination` | Where videos are saved | `/home/user/Music/YouTubeDownloads` |
| `enabled` | Enable/disable this playlist | `true` or `false` |
---
## Usage
### Interactive Usage
```bash
# Download all enabled playlists
./youtube_downloader.sh run
# Download a specific playlist
./youtube_downloader.sh run "My Music Playlist"
# View statistics
./youtube_downloader.sh stats
# List last 20 downloaded videos
./youtube_downloader.sh list 20
# Show all configured playlists
./youtube_downloader.sh playlists
# View help
./youtube_downloader.sh help
# Reset database (WARNING: deletes all download history)
./youtube_downloader.sh reset
```
### Cron Job Usage
Run downloads automatically on a schedule. Use `--cron` flag to disable colors and log only to file.
#### Example Cron Jobs
**Download all playlists daily at 2 AM:**
```bash
0 2 * * * /home/user/.youtube_downloader/youtube_downloader.sh --cron run >> /home/user/.youtube_downloader/cron.log 2>&1
```
**Download specific playlist every 6 hours:**
```bash
0 */6 * * * /home/user/.youtube_downloader/youtube_downloader.sh --cron run "My Music Playlist"
```
**Download all playlists every Sunday at midnight:**
```bash
0 0 * * 0 /home/user/.youtube_downloader/youtube_downloader.sh --cron run
```
#### Setting Up Cron
1. **Open cron editor:**
```bash
crontab -e
```
2. **Add a cron job** (example: daily at 2 AM):
```bash
0 2 * * * /home/user/.youtube_downloader/youtube_downloader.sh --cron run
```
3. **Save and exit** (in nano: `Ctrl+X`, then `Y`, then `Enter`)
4. **Verify cron job:**
```bash
crontab -l
```
#### Important Notes for Cron
- **Use absolute paths** – cron doesn't have the same environment as your shell
- **Specify full path** to the script: `/home/user/.youtube_downloader/youtube_downloader.sh`
- **Colors are disabled** automatically in cron mode (detected via `--cron` flag)
- **All output logged** to `$HOME/.youtube_downloader/downloader.log`
- **Ensure permissions** – make the script executable: `chmod +x youtube_downloader.sh`
---
## Database
The script maintains a **SQLite database** at `~/.youtube_downloader/downloads.db` with the following tables:
### `downloaded_videos`
Tracks all downloaded videos to prevent duplicates.
| Column | Type | Purpose |
|--------|------|---------|
| `id` | INTEGER | Unique record ID |
| `video_id` | TEXT | YouTube video ID (unique) |
| `title` | TEXT | Video title |
| `url` | TEXT | Full YouTube URL |
| `playlist_name` | TEXT | Which playlist it came from |
| `download_date` | TIMESTAMP | When it was downloaded |
| `file_path` | TEXT | Full path to saved file |
| `file_size` | INTEGER | File size in bytes |
| `status` | TEXT | Status (always `'completed'` for valid downloads) |
### `download_sessions`
Logs each download run with statistics.
| Column | Type | Purpose |
|--------|------|---------|
| `id` | INTEGER | Session ID |
| `playlist_name` | TEXT | Which playlist was downloaded |
| `playlist_url` | TEXT | Playlist URL |
| `session_start` | TIMESTAMP | When the session started |
| `session_end` | TIMESTAMP | When the session finished |
| `videos_downloaded` | INTEGER | Count of newly downloaded videos |
| `videos_skipped` | INTEGER | Count of already-downloaded videos |
| `errors` | INTEGER | Count of download failures |
---
## Logs
All activity is logged to **`~/.youtube_downloader/downloader.log`**
### Log Levels
- **[INFO]** – General information (downloads, moves, session start/end)
- **[ERROR]** – Critical issues (failed downloads, missing directories)
- **[WARNING]** – Non-critical issues (missing destinations, disabled playlists)
- **[DEBUG]** – Detailed diagnostic info (only if `debug: true` in config)
### Viewing Logs
```bash
# View entire log
cat ~/.youtube_downloader/downloader.log
# View last 50 lines
tail -50 ~/.youtube_downloader/downloader.log
# Follow logs in real-time (while script is running)
tail -f ~/.youtube_downloader/downloader.log
# View errors only
grep "ERROR" ~/.youtube_downloader/downloader.log
```
---
## Common Tasks
### Add a New Playlist
1. Get the playlist URL from YouTube
2. Edit `~/.config/youtube_downloader/config.json`
3. Add a new object to the `playlists` array:
```json
{
"name": "New Playlist Name",
"url": "https://www.youtube.com/playlist?list=PLAYLIST_ID",
"destination": "/path/to/save/videos",
"enabled": true
}
```
4. Save and run: `./youtube_downloader.sh run "New Playlist Name"`
### Disable a Playlist Temporarily
Set `"enabled": false` for that playlist in `config.json`. The playlist will be skipped during automatic runs but can still be downloaded manually by name.
### Change Download Location
Edit the `destination` field for a playlist in `config.json`. New videos will go to the new location; old videos remain where they were.
### Download Audio Only
In `config.json`, set:
```json
"audio_only": true,
"video_format": "best"
```
The script will extract audio only and save as MP3/M4A.
### Reduce File Sizes
Change `video_format` to download lower quality:
```json
"video_format": "worst[ext=mp4]"
```
Or cap at 720p:
```json
"video_format": "bestvideo[height<=720]+bestaudio"
```
### Clear Download History
**WARNING: This is irreversible!**
```bash
./youtube_downloader.sh reset
```
Then confirm with `yes`.
---
## Troubleshooting
### Issue: "Config file not found"
**Solution:** Ensure `~/.config/youtube_downloader/config.json` exists and is readable:
```bash
ls -la ~/.config/youtube_downloader/config.json
```
Create it if missing (see [Configuration Guide](#configuration-guide)).
---
### Issue: "jq is not installed"
**Solution:** Install jq:
```bash
# Ubuntu/Debian
sudo apt-get install jq
# macOS
brew install jq
# Fedora
sudo dnf install jq
```
---
### Issue: "yt-dlp is not installed"
**Solution:** Install yt-dlp:
```bash
# Ubuntu/Debian
sudo apt-get install yt-dlp
# macOS
brew install yt-dlp
# Or via pip (any OS)
pip install yt-dlp
```
---
### Issue: "Failed to fetch playlist"
**Possible causes:**
- **Invalid playlist URL** – Double-check the URL format
- **Private/restricted playlist** – YouTube playlists must be public
- **Network issue** – Check internet connection
- **Rate-limited** – Try again later
**Check logs:**
```bash
tail -20 ~/.youtube_downloader/downloader.log
```
---
### Issue: Cron job not running
**Debug steps:**
1. **Check cron is enabled:**
```bash
sudo systemctl status cron
```
2. **Verify cron job exists:**
```bash
crontab -l
```
3. **Check system mail for errors:**
```bash
mail
```
4. **Test the script manually:**
```bash
/home/user/.youtube_downloader/youtube_downloader.sh --cron run
```
5. **Add logging to cron job:**
```bash
0 2 * * * /home/user/.youtube_downloader/youtube_downloader.sh --cron run >> /home/user/.youtube_downloader/cron.log 2>&1
```
6. **Check permissions:**
```bash
chmod +x /home/user/.youtube_downloader/youtube_downloader.sh
```
---
### Issue: "Permission denied" when running script
**Solution:** Make the script executable:
```bash
chmod +x ~/.youtube_downloader/youtube_downloader.sh
```
---
### Issue: Videos not moving to destination folder
**Possible causes:**
- **Destination folder doesn't exist** – Script will create it automatically
- **Insufficient disk space** – Check available space
- **File permissions** – Ensure write access to destination folder
**Check permissions:**
```bash
ls -la /path/to/destination/
```
---
## Performance Tips
1. **Increase download delay** if hitting YouTube rate limits:
```json
"download_delay": 5
```
2. **Lower video quality** to save bandwidth:
```json
"video_format": "worst[ext=mp4]"
```
3. **Use audio-only mode** if you only need audio:
```json
"audio_only": true
```
4. **Schedule cron jobs during off-peak hours** to avoid network congestion
5. **Monitor database size** – it grows with each download:
```bash
du -sh ~/.youtube_downloader/downloads.db
```
---
## File Structure
```
~/Podcast-Pownloader/
├── podcast-downloader.sh # Main script
├── downloads.db # SQLite database
├── downloader.log # Activity log
├── cron.log # Cron job output (optional)
└── podcasts.json # Configuration file
```
---
## License
This project is provided as-is for personal use. Please respect YouTube's Terms of Service and copyright laws when downloading content.
---
## Contributing
Found a bug or have a feature request? Feel free to submit issues or improvements!
---
A robust Bash script for downloading YouTube playlists with automatic video and audio extraction, deduplication tracking, and comprehensive logging.
## Disclaimer
**This tool is for personal use only.** Users are responsible for:
- Complying with YouTube's Terms of Service
- Respecting copyright and intellectual property rights
- Not using the tool to circumvent any protections or restrictions
- Ensuring all downloads comply with local laws
This tool is provided for educational and personal use only. Users are responsible for ensuring their use complies with all applicable laws and regulations, including but not limited to copyright, intellectual property, and terms of service agreements. Downloading content from YouTube or other platforms may violate their terms of service or local laws depending on your jurisdiction and the nature of the content. Before using this script, verify that you have the legal right to download and store the content you intend to access. The authors and contributors of this script assume no liability for any misuse or legal consequences that may arise from its use. Always respect content creators' rights and obtain appropriate permissions when necessary.
The authors assume no liability for misuse of this tool.
---
## Overview
This script downloads YouTube playlists while maintaining a persistent database of downloaded content to prevent duplicates. It separates video and audio streams, processes them independently, and stores them in organized directories with full logging and error handling.
## Features
- **Playlist Processing**: Downloads all videos from a YouTube playlist with progress tracking
- **Dual Stream Extraction**: Separates video and audio into distinct files for flexibility
- **Deduplication**: Maintains a database of downloaded videos to skip previously processed content
- **Robust Error Handling**: Comprehensive validation, retry logic, and error recovery
- **Rate Limiting**: Configurable bandwidth throttling to respect server resources
- **Lock Mechanism**: File-based locking prevents concurrent download conflicts
- **Log Rotation**: Automatic log file rotation when size exceeds threshold
- **Time Estimation**: Calculates and displays estimated completion time before processing
- **Progress Tracking**: Real-time feedback with countdown between downloads
## Requirements
- Bash 4.0 or higher
- yt-dlp (YouTube downloader)
- Standard Unix utilities: grep, awk, find, mv, mkdir, stat, date
- Sufficient disk space for temporary and final storage
## Installation
1. Clone or download this script to your system
2. Create the configuration file at `/opt/podcast-pownloader/PODCAST_NAME/podcast_settings.conf` (if you change the config file location, remember to update the "CONFIG_FILE=" in the podcast_downloader.sh script)
3. Update the configuration with your specific paths and preferences (see Configuration section)
4. Make the script executable:
```
chmod +x podcast_downloader.sh
```
5. Ensure yt-dlp is [installed](https://github.com/yt-dlp/yt-dlp#installation)
## Configuration
Create a configuration file at the path specified in `CONFIG_FILE`. Required variables:
| Variable | Description | Example |
|----------|-------------|---------|
| `DOWNLOAD_DIR` | Temporary directory for intermediate files | `/tmp/` |
| `FINAL_DIR` | Directory where processed videos and audio are saved | `/location/to/save/directory` |
| `LOG_DIR` | Directory for storing log files | `/opt/podcast-pownloader/PODCAST_NAME/` |
| `LOG_FILE` | Path to the main activity log | `/opt/podcast-pownloader/PODCAST_NAME/.youtube_downloader.log` |
| `DOWNLOADS_LOG` | Path to the downloads database | `/opt/podcast-pownloader/PODCAST_NAME/downloaded_episodes.log` |
| `DOWNLOADS_LOCK` | Path to lock directory (must not exist) | `/opt/podcast-pownloader/PODCAST_NAME/.podcast_downloads.lock` |
| `MAX_LOG_SIZE` | Maximum log file size in bytes before rotation | `10485760` (10 MB) |
| `QUALITY` | yt-dlp format specification | `137+140` |
| `VIDEO_FORMAT` | Output video container format | `mp4` |
| `AUDIO_FORMAT` | Audio format (not currently used in output) | `m4a` |
| `SLEEP_INTERVAL` | Seconds to wait between downloads | `120` |
| `RATE_LIMIT` | Bandwidth limit for downloads | `3M` |
| `CONNECTION_TIMEOUT` | Socket timeout in seconds | `30` |
| `MAX_RETRIES` | Number of retries for failed downloads | `3` |
| `PLAYLIST_URL` | Default YouTube playlist URL | `https://www.youtube.com/playlist?list=YOUR_LIST_ID` |
## Usage
### Basic Usage
Run with a playlist URL as an argument:
```
./podcast_downloader.sh "https://www.youtube.com/playlist?list=YOUR_LIST_ID"
```
### Using Configuration Default
If `PLAYLIST_URL` is set in the configuration file, run without arguments:
```
./podcast_downloader.sh
```
## Output Structure
### Directory Layout
```
FINAL_DIR/
video_title.mp4
video_title.mp3
another_video.mp4
another_video.mp3
LOG_DIR/
.youtube_downloader.log
.youtube_downloader.log.1234567890
downloaded_episodes.log
```
### Downloads Database Format
The `downloaded_episodes.log` file uses pipe-delimited format:
```
VIDEO_ID|TITLE|FILENAME|DOWNLOADED_DATE
dQw4w9WgXcQ|Example Video|Example Video.mp4|2024-01-15 14:30:45
```
### Log Format
```
[2024-01-15 14:30:45] Starting download for playlist: https://www.youtube.com/playlist?list=...
[2024-01-15 14:30:47] Found 150 videos to process
[2024-01-15 14:31:02] [1/150] Downloading: 'Example Video'
[2024-01-15 14:35:10] [1/150] Successfully processed 'Example Video'
```
## Functions
### Core Download Functions
- `download_playlist()`: Main function that processes all videos in a playlist
- `process_downloaded_video()`: Moves downloaded files to final directory and updates database
- `is_downloaded()`: Checks if a video ID exists in the downloads database
- `record_download()`: Adds a new entry to the downloads database
### Utility Functions
- `acquire_lock()` / `release_lock()`: Manages concurrent access protection
- `validate_config()`: Verifies all required configuration variables are set
- `sanitize_filename()`: Removes invalid characters from filenames
- `log_message()`: Writes timestamped messages to log file with automatic rotation
- `rotate_log()`: Rotates log file when size limit is exceeded
- `estimate_download_time()`: Calculates approximate total processing time
- `format_time()`: Converts seconds to human-readable duration format
- `wait_for_next()`: Displays countdown between downloads
- `show_log()`: Displays the last 20 downloads from the database
## Error Handling
The script implements multiple error handling strategies:
- Configuration validation at startup ensures all required variables are present
- Lock-based synchronization prevents concurrent execution issues
- Retry logic with configurable limits handles transient network failures
- File existence checks before processing prevent data loss
- Filename deduplication adds video ID suffix if collision is detected
- Graceful cleanup of temporary files via trap handlers
## Exit Codes
- `0`: Successful completion
- `1`: Configuration error, missing files, or download failure
## Logging
All operations are logged with timestamps. The script provides:
- Download start and completion messages
- Per-video progress indicators with current count and total count
- Detailed error messages with context
- Skipped video notifications
- Database write confirmations
- Total execution time
Access logs at the path specified by `LOG_FILE`. Historical logs are retained with timestamp suffixes when rotation occurs.
## Notes
- The script creates separate video (mp4) and audio (mp3) files for each download
- Filenames are automatically sanitized to remove invalid filesystem characters
- Duplicate file names are handled by appending the video ID to the base name
- Lock mechanism uses directory creation atomicity for reliability
- The script respects rate limiting to avoid overwhelming download sources
- Clean shutdown is performed even if interrupted, removing temporary files
## Troubleshooting
**Configuration file not found**: Verify the path in `CONFIG_FILE` variable matches your actual configuration file location.
**Failed to fetch playlist**: Check your internet connection and ensure the playlist URL is correct and publicly accessible.
**Lock timeout**: If the script repeatedly fails to acquire lock, check for stale lock directories at the `DOWNLOADS_LOCK` path and remove manually if necessary.
**Insufficient disk space**: Ensure `DOWNLOAD_DIR` and `FINAL_DIR` have adequate free space. Temporary files are cleaned up automatically after processing.
**Invalid video ID format**: This warning indicates yt-dlp returned unexpected data. Verify the playlist is valid and contains standard YouTube videos.