Skip to content

Latest commit

 

History

History
146 lines (127 loc) · 5.03 KB

README.md

File metadata and controls

146 lines (127 loc) · 5.03 KB

Scrapper

STATUS [ACTIVE]

UPDATE 06/20/2024

At this time Intermarche,SystemeU and Leclerc use Datadome protection

  • Intermarche -> Impossible for me to bypass the new version of Datadome -> Target waiting
  • SystemeU -> Same to Intermarche , bypass with proxy and IP Rotating is possible
  • Leclerc -> Same to Intermarche

PRESHOT 2024 TARGET EVOLUTION

  • SystemeU -> Update the version of the DataDome Solution
  • Auchan and Carrefour add DataDome Solution
  • Monoprix no protection
  • Leclerc need to rebuild the pathing of the website to use correctly the DataDome solution

PRESHOT 2024 TOOL EVOLUTION

  • php-webdriver -> Maybe Deprecated soon for WebScraping
  • puppeteer -> need more update for hide the headless mode (waiting)
  • playwright -> microsoft tool (Ubuntu 20.* or newer)
  • selenium -> next test for scrapping target (Famous tool)

Disclaimer

  • This tool is not for collect personal information
  • Please respect the RGPDs rules

What is a scrapper

A tool to collect any information of website pages :For example javascript,html,css sources It's possible to look the content of a website pages with the browser with this tips :

  • firefox view-source:https://www.mozilla.org/fr/ or CTRL^U
  • CRTL+MAJ+I for web inspector -> Console, possibility to change the display conten

Or with special library and framework like :

  • Selenium (Python,...)
  • Goutte (Symfony)
  • Scrapy (Python)

Or API :

  • ScrapFly
  • ScraperAPI

How

Why

  • For my project PriceComparator
  • Developpement of your own tools is important to understand and learn many things.

Paths

Paths 
dev
└── JSON_updates.php
project
├── infos_programs.php
└── project.php
src
├── control_google_.js
├── DatadomeBreaker/
├── libJSON/
├── scrape.js
├── scrape_su.js
├── scrapper_auchan.php
├── scrapper_carrefour.php
├── scrapper_intermarche.php
├── scrapper_leclerc.php
├── scrapper_monoprix.php
├── scrapper.php
├── scrapper_systemeu.php
├── test_extra_puppeteer.js
└── test_rq_submod.js
your_project
├── process_p.php
├── proofs/
├── README.md
└── usage.php
composer.json
package.json
README.md

Usage

LIKE A PACKAGE :

  • curl -sS https://getcomposer.org/installer | php7.2 OR #2:
    • php7.2 composer.phar update
  • #2 -> composer install :
    • composer update
  • In move your_project folder in the root of your project for test the functions

LIKE A PROJECT :

  • composer require php-webdriver/php-webdriver
  • project.php for known how the different tools works
  • scrapper*.php the differents files for scraping mission
  • vendor add lib for php-webdriver
  • node_modules(hide with .gitignore) for node.js module
  • *.json/*.txt for different test to build program to efficient scraping

php project/project.php --info it's a good start

Version

V1.5.1

  • Basic version of scrapper :

    • http, https
    • html content generate by JS -> puppeteer
    • cloudflare security
    • text in tag with another tag $\color{green}\textsf{(V2.0\ scrapper.php)}$

  • Specific version for specific website :

    • The french supermarket compagny :
      • Leclerc [BLOCKED]:
        • parse specific JS -> json
        • usage of https of basic version :
        • NoBot Solutions DataDome Solution
        • Try Bypass NoBot Solutions with knownledge of all stores (libJSON/leclercs.json) (works before Datadome Solution buy)
      • Carrefour :
        • parse specific JS -> json
        • usage of php-webdriver
        • NoBot Solutions -> Cloudflare
      • Auchan :
        • parse text in html tag
        • usage of php-webdriver
        • NoBot Solutions
      • Monoprix :
        • parse specific JS -> json
        • usage of puppeteer or php-webdriver is possible
        • products for all stores in the target country
        • NoBot Solutions
      • Intermaché [BLOCKED] :
        • parse specific JS -> json
        • usage of php-webdriver
        • NoBot Solutions -> DataDome Solution
      • SystemeU [BLOCKED]:
        • parse specific JS -> json (products only on the display page)
        • usage of puppeteer or php-webdriver IMPOSSIBLE
        • NoBot Solutions -> DataDome Solution
        • Necessary to use puppeteer-extra-plugin-stealth -> not enough
        • Try Bypass with src/libJSON/* (scrape2() in scrape_su.js) but blocked again

Features