Website Clone Crawler (with Python Code)

Jan 7, 2025#python脚本71

AI Translation

This post is translated from Chinese into English through AI.View Original

AI-generated summary

The script is a website mirroring tool that creates a directory structure under a "sites" folder, organizing downloaded files by type (e.g., images, CSS). It modifies HTML links to point to local files and generates a ZIP archive containing all downloaded content. The process includes making HTTP requests, saving files, and processing HTML tags to identify file references. The main function takes a URL as input and handles the mirroring process, saving the modified HTML and creating the ZIP file.

Main functionality:
Creates a 'sites' folder in the current directory
Creates a subfolder in the 'sites' folder according to the domain name
Downloaded files are categorized by type (e.g., images folder for images, css folder for stylesheets, etc.)
Automatically modifies links in the HTML file to point to local files
Finally generates a ZIP archive containing all downloaded files

For example, if you download https://example.com:
The HTML file will be saved as sites/examplecom/examplecom.htm
All files will be compressed into sites/examplecom.zip
Images will be stored in the sites/examplecom/images/ directory
CSS files will be stored in the sites/examplecom/css/ directory
JavaScript files will be stored in the sites/examplecom/js/ directory

During execution, it will display in the console:
: Detected file links
Downloaded file save paths
Final locations of the generated HTML and ZIP files