自动化框架

Selenium、Puppeteer、Playwright、Scrapy 等自动化与采集框架的接入方法。所有示例把 USERNAME、PASSWORD 替换为你的凭据即可运行。

Scrapy

在 settings.py 启用代理中间件,或按请求设置:

# 方式一:全局(settings.py)
HTTPPROXY_ENABLED = True

# 在 Spider 中为每个请求指定代理
def start_requests(self):
    proxy = "http://USERNAME-pool-flow-region-US:[email protected]:23333"
    for url in self.start_urls:
        yield scrapy.Request(url, meta={"proxy": proxy})

逐请求轮换是默认行为,天然适配 Scrapy 的并发模型;需要按会话固定 IP 时在 username 中加入 sid。

Selenium(Python)

Chrome 命令行参数不支持带认证的代理,推荐用 selenium-wire 处理账密:

# pip install selenium-wire
from seleniumwire import webdriver

options = {
    "proxy": {
        "http": "http://USERNAME-pool-flow-region-US:[email protected]:23333",
        "https": "http://USERNAME-pool-flow-region-US:[email protected]:23333",
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)
driver.get("https://ipinfo.io")
print(driver.page_source)
driver.quit()

也可以改用白名单模式:提取免认证的 IP:端口 后,直接用原生 --proxy-server 参数。

Puppeteer(Node.js)

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({
    args: ["--proxy-server=http://proxy.eproxies.net:23333"],
  });
  const page = await browser.newPage();
  await page.authenticate({
    username: "USERNAME-pool-flow-region-US",
    password: "PASSWORD",
  });
  await page.goto("https://ipinfo.io/json");
  console.log(await page.evaluate(() => document.body.innerText));
  await browser.close();
})();

Playwright(Node.js / Python)

// Node.js
const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({
    proxy: {
      server: "http://proxy.eproxies.net:23333",
      username: "USERNAME-pool-flow-region-US",
      password: "PASSWORD",
    },
  });
  const page = await browser.newPage();
  await page.goto("https://ipinfo.io/json");
  console.log(await page.textContent("body"));
  await browser.close();
})();

# Python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://proxy.eproxies.net:23333",
        "username": "USERNAME-pool-flow-region-US",
        "password": "PASSWORD",
    })
    page = browser.new_page()
    page.goto("https://ipinfo.io/json")
    print(page.text_content("body"))
    browser.close()

AI Agent / LLM 数据管线

用 LLM 驱动浏览器或采集数据时(LangChain 工具、Browser Use、自研 agent 等),把上述任一框架的代理配置传入即可。要点:

给每个并行 agent 分配独立 sid,互不干扰(多会话示例)。
让 agent 在任务开始时先请求 https://ipinfo.io 校验出口 IP,再执行业务。
完整参数语法一页速查:Agent 速查表。

爬虫采集工具

图形化采集工具的官方配置教程:Octoparse · Scrapy · ScrapeStorm · SellerSprite · Helium 10