Senior Web Crawler Engineer

Responsibilities

1. Responsible for scraping, cleaning, parsing, and storing web data.
2. Proficient in Python programming language, familiar with JavaScript, and experienced in building backend services with FastAPI.
3. Skilled in using databases, including Redis, PostgreSQL, and MySQL.
4. Familiar with deploying services using Docker.
5. Knowledgeable in various scraping technologies, including anti-scraping, proxies, and CAPTCHA handling.
6. Proficient in using regular expressions and XPath to extract information from structured and unstructured data.
7. Mastery of common anti-scraping mechanisms and expertise in various anti-scraping bypass techniques, including but not limited to: hook, JS protection, obfuscation, and reverse analysis.
8. Participate in the design and optimization of distributed web scraping systems, familiar with scheduling strategies for collection systems, real-time monitoring of scraper progress, and alert feedback.

Requirements

1. Programming skills: Proficient in Python, familiar with JavaScript, and able to use FastAPI to build backend services.
2. Database: Proficient in using PostgreSQL/MySQL for data storage and Redis for caching and deduplication.
3. Proficient in XPath/regular expression parsing of structured and unstructured data.
4. Knowledgeable in building proxy IP pools, request disguise, and dynamic rendering techniques.
5. Experience in anti-scraping bypass techniques (Hook technology, JS reverse engineering, obfuscation code analysis, etc.) is preferred.
6. Candidates with experience in reverse engineering of cryptographic algorithms are preferred.
7. Proficient in using Docker for containerized service deployment.