wget一行命令就能镜像一个网站HTTrack有图形界面点几下就能拖到本地还有必要自己写Python脚本吗：单站的离线备份和50个站群定期备份以及目标站有反爬要带CookieUA和JS渲染页面纯静态工具拿不到内容是完全不同的三种场景

wget一行命令就能镜像一个网站，HTTrack有图形界面点几下就能把整站拖到本地，那还有必要自己写Python脚本吗？如果只是下载一个纯静态的小网站做离线备份，确实没必要。但如果是50个站群要定期备份、目标站有反爬需要带Cookie和UA、页面是JS渲染的静态工具根本拿不到内容、下载完还要自动比对哪些页面更新了——这些场景下，wget和HTTrack的默认配置根本跑不通，要么缺参数要么缺灵活性，最后还是得自己写脚本把采集、渲染、校验串起来

搜"批量下载网站工具"的人，场景差距非常大：有人只想把竞争对手的站扒下来做离线分析，有人要给自己的站群做定期全站备份，有人要批量采集几十个站的文章内容和图片做素材库，还有人是做SEO要导出全站URL和页面结构。需求不同，工具选错了要么下不全、要么效率低、要么被目标站封IP。下面按四种场景把工具和方案拆开讲。

一、wget：一行命令镜像一个网站

wget 是Linux自带的命令行下载工具，也是整站镜像最经典的方式。一条命令就能递归下载整个站点，并自动转换链接为本地相对路径，让离线浏览时链接不出错。

# 基础版：镜像整个网站wget --mirror --convert-links --adjust-extension \--page-requisites --no-parent \https://example.com/# 限制版：只下载3层深度、只抓同域名wget -r -l 3 -np -k -p --domains=example.com \https://example.com/# 伪装版：模拟浏览器、控制速度、排除特定目录wget --mirror -k -p \--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0" \--wait=1 --limit-rate=200k \--exclude-directories=/tag,/category,/wp-admin \https://example.com/

二、HTTrack：图形界面，点点鼠标就能下

HTTrack是Windows/Linux/Mac全平台支持的整站下载工具，有图形界面也有命令行模式。比wget的优势是：可视化配置、项目管理（可以暂停/续传）、自动处理编码和文件名、支持代理和Cookie。

HTTrack命令行模式，适合脚本调用

# Windows HTTrack 命令行httrack "https://example.com/" -O "./mirror/example" \-v -r3 -c4 \--user-agent "Mozilla/5.0 Chrome/120.0" \+*.html +*.css +*.js +*.png +*.jpg \-*.mp4 -*.zip -*.exe# Linux 安装 HTTracksudo apt install httrack -y   # Debian/Ubuntusudo yum install httrack -y   # CentOS/RHEL

三、Python脚本：批量下载多个站、只下想要的内容

wget和HTTrack能搞定单个站的完整镜像，但管着10个站群要做定期备份、或者只想下载文章正文不下载导航栏和侧边栏——这时候就需要自己写脚本了。下面是一套完整的批量下载方案，三部分串起来用。

第一步：爬取全站URL列表

# url_collector.py — 递归爬取全站URLimport requestsfrom bs4 import BeautifulSoupfrom urllib.parse import urljoin, urlparseimport timedef collect_urls(start_url, max_pages=500):"""从首页出发递归收集全站内链"""domain = urlparse(start_url).netlocvisited = set()to_visit = [start_url]all_urls = set()headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0'}while to_visit and len(visited) < max_pages:url = to_visit.pop(0)if url in visited:continuetry:resp = requests.get(url, headers=headers, timeout=10)if resp.status_code != 200:visited.add(url)continuevisited.add(url)all_urls.add(url)print(f'[{len(visited)}/{max_pages}] {url}')soup = BeautifulSoup(resp.text, 'html.parser')for a in soup.find_all('a', href=True):href = a['href']full_url = urljoin(url, href)parsed = urlparse(full_url)# 只收同域名、不是资源文件的链接if parsed.netloc == domain and full_url not in visited:# 排除图片/CSS/JS/PDF等ext = parsed.path.split('.')[-1].lower() if '.' in parsed.path else ''if ext not in ('jpg', 'jpeg', 'png', 'gif', 'css', 'js', 'pdf', 'zip'):to_visit.append(full_url)time.sleep(0.5)  # 礼貌间隔except Exception as e:print(f'[SKIP] {url}: {e}')visited.add(url)return list(all_urls)# 使用urls = collect_urls('https://example.com/', max_pages=200)print(f'Collected {len(urls)} URLs')# 保存URL列表with open('urls.txt', 'w', encoding='utf-8') as f:f.write('\n'.join(urls))

第二步：批量下载页面并提取正文

# batch_downloader.py — 批量下载+正文提取import osimport requestsfrom bs4 import BeautifulSoupfrom concurrent.futures import ThreadPoolExecutor, as_completedimport redef download_and_extract(url, output_dir, site_name):"""下载单个页面并提取正文内容"""headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0','Accept': 'text/html,application/xhtml+xml','Accept-Language': 'zh-CN,zh;q=0.9',}try:resp = requests.get(url, headers=headers, timeout=15)resp.encoding = resp.apparent_encoding or 'utf-8'soup = BeautifulSoup(resp.text, 'html.parser')# 提取标题title = soup.title.string if soup.title else 'Untitled'title = re.sub(r'[\\/:*?"<>|]', '_', title)[:100]# 移除不需要的元素for tag in soup.find_all(['script', 'style', 'nav', 'footer', 'header', 'aside']):tag.decompose()# 提取正文body = soup.find('body')if body:# 尝试找文章主体区域article = body.find('article') or body.find(class_=re.compile(r'content|post|article|entry', re.I))content = article or bodytext = content.get_text('\n', strip=True)else:text = soup.get_text('\n', strip=True)# 清理多余空行text = re.sub(r'\n{3,}', '\n\n', text)# 保存为HTML（保留原始结构）html_path = os.path.join(output_dir, f'{site_name}', 'html')os.makedirs(html_path, exist_ok=True)safe_name = re.sub(r'[\\/:*?"<>|]', '_', url.split('/')[-1] or 'index')with open(f'{html_path}/{safe_name}.html', 'w', encoding='utf-8') as f:f.write(resp.text)# 保存为TXT（纯文本）txt_path = os.path.join(output_dir, f'{site_name}', 'txt')os.makedirs(txt_path, exist_ok=True)with open(f'{txt_path}/{safe_name}.txt', 'w', encoding='utf-8') as f:f.write(f'{title}\n\n{text}')return {'url': url, 'title': title, 'size': len(text)}except Exception as e:return {'url': url, 'error': str(e)}def batch_download(url_list, output_dir, site_name, workers=5):"""并发批量下载"""os.makedirs(f'{output_dir}/{site_name}', exist_ok=True)success = 0failed = 0with ThreadPoolExecutor(max_workers=workers) as executor:futures = {executor.submit(download_and_extract, url, output_dir, site_name): urlfor url in url_list}for i, future in enumerate(as_completed(futures)):result = future.result()if 'error' in result:failed += 1print(f'[{i+1}/{len(url_list)}] FAIL: {result["url"][:60]}')else:success += 1print(f'[{i+1}/{len(url_list)}] OK: {result["title"][:40]}')print(f'\nDone. Success: {success}, Failed: {failed}')return success, failed# 使用示例# batch_download(url_list, './downloads', 'example_site', workers=5)

第三步：多站点批量调度 + 增量更新

# multi_site_scheduler.py — 多站点批量调度import osimport jsonimport hashlibfrom datetime import datetime# 站点配置SITES = [{'name': 'site_a', 'url': 'https://example-a.com/', 'max_pages': 200},{'name': 'site_b', 'url': 'https://example-b.com/', 'max_pages': 150},{'name': 'site_c', 'url': 'https://example-c.com/', 'max_pages': 300},]OUTPUT_DIR = './site_backups'STATE_FILE = './download_state.json'def load_state():"""加载上次下载的状态（用于增量更新）"""if os.path.exists(STATE_FILE):with open(STATE_FILE, 'r', encoding='utf-8') as f:return json.load(f)return {}def save_state(state):with open(STATE_FILE, 'w', encoding='utf-8') as f:json.dump(state, f, ensure_ascii=False, indent=2)def should_download(url, state):"""判断是否需要重新下载（内容有变化）"""if url not in state:return True# 超过24小时重新检查last_check = state[url].get('last_check', '')if last_check < datetime.now().strftime('%Y%m%d'):return Truereturn Falsedef backup_all_sites():"""批量备份所有站点"""state = load_state()total_downloaded = 0for site in SITES:print(f'\n{"="*50}')print(f'Processing: {site["name"]} ({site["url"]})')print(f'{"="*50}')# 爬取全站URLurls = collect_urls(site['url'], max_pages=site['max_pages'])print(f'Found {len(urls)} URLs')# 筛选需要更新的URLnew_urls = [u for u in urls if should_download(u, state)]print(f'Need to download: {len(new_urls)} (skipped {len(urls) - len(new_urls)})')if new_urls:# 批量下载success, _ = batch_download(new_urls, OUTPUT_DIR, site['name'], workers=5)total_downloaded += success# 更新状态for u in new_urls:state[u] = {'last_check': datetime.now().strftime('%Y%m%d%H%M')}save_state(state)print(f'{site["name"]} done.')print(f'\n{"="*50}')print(f'All sites backed up. Total downloaded: {total_downloaded} pages')print(f'{"="*50}')if __name__ == '__main__':backup_all_sites()

四、JS渲染的网站：Puppeteer/Playwright无头浏览器方案

越来越多的网站是React/Vue/Next.js等前端框架渲染的，HTML源码里只有一个空div，真实内容全靠JS动态生成。wget、HTTrack、requests对这些站全都无效。这时候需要用无头浏览器：

// puppeteer_downloader.js — 无头浏览器整站下载const puppeteer = require('puppeteer');const fs = require('fs');const path = require('path');async function downloadRenderedPage(url, outputDir) {const browser = await puppeteer.launch({ headless: 'new' });const page = await browser.newPage();await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0');await page.goto(url, {waitUntil: 'networkidle2',  // 等网络请求基本停止timeout: 30000});// 等待额外的渲染时间await page.evaluate(() => new Promise(r => setTimeout(r, 2000)));// 获取渲染后的完整HTMLconst html = await page.content();// 截图（可选）await page.screenshot({path: `${outputDir}/screenshot.png`,fullPage: true});// 保存HTMLfs.mkdirSync(outputDir, { recursive: true });fs.writeFileSync(`${outputDir}/index.html`, html);// 提取纯文本const text = await page.evaluate(() => document.body.innerText);fs.writeFileSync(`${outputDir}/content.txt`, text);await browser.close();console.log(`Downloaded: ${url}`);}// 批量下载多个JS渲染的页面async function batchRenderDownload(urlList, baseOutputDir) {for (const url of urlList) {const siteName = new URL(url).hostname;const outputDir = path.join(baseOutputDir, siteName,url.replace(/[^a-z0-9]/gi, '_').substring(0, 60));try {await downloadRenderedPage(url, outputDir);} catch (e) {console.error(`Failed: ${url} - ${e.message}`);}}}

五、六款工具横向对比，选对的别选贵的

六、下载过程中容易碰到的问题

七、定时自动备份，配个cron就全自动了

站群定期备份不需要人盯着。把上面的Python调度脚本配合crontab，每天凌晨自动跑一次，增量更新只下载变化的页面：

# 每天凌晨3点自动备份所有站点0 3 * * * cd /opt/site-backup && python multi_site_scheduler.py >> backup.log 2>&1# 每周日全量备份一次（不管增量状态）0 4 * * 0 cd /opt/site-backup && python multi_site_scheduler.py --full >> backup_full.log 2>&1# 备份完成后清理30天前的旧文件0 5 * * * find /opt/site-backup/site_backups -mtime +30 -delete

最后梳理一下

批量下载网站这件事，工具选对方向比工具选对品牌重要：

说穿了，选wget还是写Python脚本，本质差别不在功能上——wget能干的事Python都能干，但反过来Python能干的事wget不一定干得了。如果需求只是"下载一个站"，wget够用；如果需求是"管理一群站的下载任务、定期执行、增量更新、异常处理、通知提醒"，那就得写脚本了。