Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Big puppeteer-extra upgrade. #649

Open
wants to merge 205 commits into
base: master
Choose a base branch
from
Open

Big puppeteer-extra upgrade. #649

wants to merge 205 commits into from

Conversation

UrielCh
Copy link

@UrielCh UrielCh commented Jun 10, 2022

One of my Biggest PR.

This PR contains the following changes:

Improvements

  • All workflows are working
  • All plugin are written in typescript
  • Dependency loader rewrite.
  • Enforce typing
  • Works on windows
  • Add default export everywhere
export default (pluginConfig?: Partial<PluginOptions>) =>new Plugin(pluginConfig)
  • puppeteer-extra loader can inject option to plugins
  PuppeteerExtra.setPluginOptions(name: PluginName, options: any): void {
  • puppeteer-extra-plugin-stealth loader can inject option to evasions
puppeteer.use(plugin({
  evasionsOptions: {
    "navigator.languages": { languages : ["es-ES", "es"]},
    "navigator.hardwareConcurrency": { hardwareConcurrency: 12 },
 }}));

in this case typing enforce all Plugin constuctors.

  • fix invalid interface PluginData
  • split puppeteer-extra/index.ts and extract a PuppeteerExtra.ts
  • clean tsconfig.json "include" from "include": ["the word/*.ts", "test.ts", "nonexistingstuf.ts"...] to ["index.ts"]
  • clean tsconfig.json "excude" from "excude": ["stuff from the include"] to nothink
  • start preparing an offline version of the tests so test will not break from a day to an other.
  • add PuppeteerPage, PuppeteerTarget, PuppeteerBrowser, PuppeteerConnectOptions, PuppeteerRequest, PuppeteerClickOptions, PuppeteerNavigationOptions, PuppeteerResponse type exports in puppeteer-extra-plugin to symplify typing in plugins.
  • 'puppeteer-extra-plugin-stealth/stealthtests' works on windows hosts
  • drop external ava config file in favor of an ava section in package.json
  • And a few some more changes, that I'm trying to list here.

Breaking changes:

  • _getMissingDependencies(plugins: any) removed in favor of get dependencies()
    dependencies simply return a list of plugin names, _getMissingDependencies was a mess.
    Display warning if older plugins are loaded.
  • get dependencies() retrun an Array instead of a Set support Set or Array

Regression

  • the linter is mostly disabled, I have some issues with yarn + lerna workspaces.
  • I live explicit typescript version 14 in dependency instead of *.
  • drop long time deprecated puppeteer-firefox

TODO

  • improve clean script, some generated source are not removed by clean task
  • fix pptr 14 support

Changes I self-restrain

  • drop yarn in favor of pnpm

@berstend
Copy link
Owner

berstend commented Jul 3, 2022

@UrielCh I've published playwright-extra in #664

It contains new/refactored code, clean deps, new test runners and a few other features like this one:

import { chromium } from 'playwright-extra'

import StealthPlugin from 'puppeteer-extra-plugin-stealth'
chromium.use(StealthPlugin())

// New way to overwrite the default options of stealth evasion plugins
// https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth/evasions
chromium.plugins.setDependencyDefaults('stealth/evasions/webgl.vendor', {
  vendor: 'Bob',
  renderer: 'Alice'
})

Once the dust has settled doing housekeeping in the repo (especially updating dependencies/linting, etc) is in order 😄

I used your PR as a reference while implementing the new code 👍

@berstend berstend marked this pull request as draft July 8, 2022 08:42
@UrielCh UrielCh marked this pull request as ready for review August 3, 2022 09:50
@UrielCh
Copy link
Author

UrielCh commented Aug 3, 2022

Hi,
Can you integrate my "ambient-dts-fix-path": "node ../post-replace.js"
and drop replace-in-files; your replace-in-files are platform dependent.

@ppyrzanowski
Copy link

Any updates on this pull request? Windows compatibility is still not available, and proper typescript rewrites to the plugins and rest would improve overall dev experience. This PR would resolve both of my main pain points.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
package: core Affecting a core package plugin: puppeteer-extra PuppeteerExtra Plugin related plugin: recaptcha 🏴 reCAPTCHA plugin related plugin: stealth ㊙️ Detection evasion related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants