Crawler Engineer

PURPOSE OF THE JOB

Summarize why this job exists and the contribution it makes to the overall business of the company.

Responsible for the company's self-developed data collection capability construction, focusing on the incremental supplementation and verification of high-value corporate business data (daily collection scale ≤ 10,000), through Python/Playwright browser automation, front-end analysis, and interface reverse engineering, to build a stable and monitorable collection link, supporting the construction of enterprise master files and data quality improvement. Ensure the collection runs stably 7×24, collaborating with data governance and application teams to complete multi-source verification and database specification docking, and evaluate external RPA/third-party data sources as supplements. The current stage focuses on feasibility verification and production implementation, without pursuing distributed crawler clusters and large-scale proxy pool construction.

PRINCIPAL ACCOUNTABILITIES

Describe the major activities of the job, emphasizing the end results which must be achieved (suggest listing from the most important task).

1. Collection system construction and maintenance

   • Design and implement browser automation collection services based on Python + Playwright.
   • Build core modules such as enterprise update library, task queue, failure retry, and collection monitoring and alarms.
   • Responsible for the structured parsing, cleaning, and storage of collection results, docking with ODS / enterprise master file data specifications.
   • Ensure the collection link runs stably 7×24, responsible for success rate, timeliness, and data quality.

2. Front-end analysis and interface reverse engineering

   • Proficient in using Chrome DevTools (Network / Sources / Application) for breakpoint debugging, request capturing, and call stack analysis.
   • Independently analyze Ajax / Fetch / XHR request links, restore parameter sources, Header, Cookie, Token, and calling sequence.
   • Locate interface signature/encryption logic (sign, timestamp, nonce, token, etc.), reproduce and implement stable interface calls in Python.
   • Handle non-standard HTTP collection scenarios such as WebSocket, GraphQL, dynamic interfaces (interface addresses/parameters change with pages or sessions).
   • Analyze JS obfuscation, Webpack / Vite packaged code, restore key logic through breakpoints, Hook, or de-obfuscation methods.
   • Preferably adopt "interface collection instead of page collection": when reverse engineering is feasible, use Requests / httpx to replace browser rendering, reducing costs and improving stability and concurrency.
   • Establish a migration mechanism from page collection to interface collection: first validate feasibility on the page, then solidify the interface solution through reverse engineering, and have the ability to downgrade and fallback when the interface fails.

3. Anti-scraping and stability response

   • Handle common risk control measures such as slider verification, image verification codes, and behavior verification.
   • Design strategies for proxy IP rotation, request frequency control, and browser fingerprint randomization.
   • Respond to webdriver detection, environment fingerprinting, request replay verification, and other front-end protection mechanisms.
   • Quickly respond to changes in target site pages, DOM/JS changes, interface changes, and field rule changes, shortening fault recovery time.
   • Establish classification of collection exceptions and automatic/semi-automatic retry mechanisms.
   • Establish fault location, script repair, downgrade fallback, and data replenishment mechanisms to ensure timely recovery from collection interruptions caused by changes in pages, interfaces, or field rules.

4. Data quality and business collaboration

   • Collect high-value fields such as basic corporate information, change records, shareholders/personnel, and operational abnormalities according to business priorities.
   • Collaborate with data governance and data application teams, participate in multi-source data verification, field mapping, and master file alignment.
   • Output daily operation reports on collection coverage, accuracy, and failure reasons.
   • Ensure that the data collection plan complies with the usage rules of the target site, internal data compliance, and information security requirements, establishing a sustainable collection mechanism between efficiency, stability, and compliance.

TECHNICAL EVOLUTION

• Evaluate and interface with external RPA/third-party data sources, complementing self-developed collection.
• Productize reverse engineering results (configurable interface templates, signature modules, failure alarms).
• Ensure continuous iteration of collection capabilities; gradually optimize architecture based on business scale, currently not pursuing distributed crawler clusters and large-scale proxy pools.
• Continuously optimize collection strategies, scheduling mechanisms, and operating costs under limited proxy, computing, and operation resources, improving collection stability, success rate, and resource utilization efficiency.

1. Participate in irregular releases, evening duty, or emergency support.

2. Other job-related tasks assigned by supervisors or the company.

MAJOR PROBLEMS / CHALLENGES

Describe 2 or 3 of the most difficult challenges or problems typically encountered in this job.

1. Anti-scraping and risk control confrontation

   • Target sites generally have multi-layer risk control such as slider verification, image verification codes, and behavior verification.
   • IP bans (403), frequency limits, webdriver detection, environment fingerprints, etc., lead to collection interruptions.
   • Need to maintain stable collection under limited proxy resources, balancing cost and success rate.

2. Front-end reverse engineering and interface stability

   • Interface signature/encryption logic hidden in obfuscated JS or Webpack packaged code, making reverse engineering difficult and maintenance costly.
   • Scenarios such as WebSocket, GraphQL, and dynamic interfaces increase the complexity of analysis and reproduction.
   • Frequent changes in target site interfaces or JS logic lead to interface solutions easily becoming invalid, requiring quick location and repair or downgrade fallback.

3. Page and data structure changes

   • Changes in target site pages, DOM/JS changes, and field rule changes lead to parsing failures.
   • Need to complete fault location, script repair, and data replenishment within a short time.

4. Data quality and business alignment

   • High-value corporate incremental updates (daily collection ≤ 10,000) need to balance coverage, accuracy, and timeliness.

5. Resource and architecture constraints

   • Independently complete the full chain construction from verification to production.
   • Currently not building distributed crawler clusters and large-scale proxy pools, need to ensure stable operation 7×24 under lightweight architecture.

6. Compliance and risk control

   • Collection behavior must comply with the target site's rules and the company's internal data compliance requirements.
   • Need to find sustainable solutions between efficiency, stability, and compliance.

JOB REQUIREMENTS

EXPERIENCE, KNOWLEDGE, SKILLS, ABILITIES

List the experience, knowledge, skills, and abilities necessary to do the job in a fully competent manner.

1. Education and experience

   • Bachelor's degree or above in computer science or related fields.
   • More than 3 years of data collection/crawler development experience, able to independently manage a collection link from 0 to 1.

2. Technical knowledge and skills

   • Proficient in Python, familiar with asynchronous programming, HTTP protocol, HTML/CSS/XPath/CSS Selector parsing.
   • Proficient in using Playwright (or Puppeteer/Selenium) for browser automation, with Headless deployment experience.
   • Practical experience in anti-scraping confrontation: verification codes, IP bans (403), frequency limits, Cookie/Session management.
   • Familiar with proxy IP usage and rotation strategies, able to independently troubleshoot collection link failures.
   • Capable of designing and implementing task scheduling, queues, retries, and monitoring alarms.
   • Good coding standards, with awareness of logs, alarms, and documentation.

3. Front-end analysis and reverse engineering capabilities

   • Proficient in using Chrome DevTools for debugging, capturing requests, call stack analysis, and local override.
   • Able to independently analyze Ajax/Fetch request links and restore the complete data loading process.
   • Experience in interface signature reverse engineering, able to locate encrypt/sign/token generation logic and reproduce it on the server.
   • Able to handle collection scenarios such as WebSocket, GraphQL, and dynamic interfaces.
   • Able to read and analyze JS obfuscation, Webpack packaged code, and extract key encryption/signature functions.
   • Experience in prioritizing interface collection practices: using HTTP clients instead of browser automation when reverse engineering is feasible.

JOB LEVEL

Mid to Senior Level

EMPLOYMENT TYPE

Full-time

JOB FUNCTION

Engineer and Information Technology

INDUSTRY

Information Services