Mid-Senior Backend Engineer (Crawling Direction)

Responsibilities

• Develop and maintain asynchronous crawlers and backend services using Rust.

• Ensure the adaptability and stability of crawlers for public or authorized data sources.

• Handle pagination, rate limiting, retries, task scheduling, and data integrity issues.

• Maintain RabbitMQ, PostgreSQL, Redis, and handle task recovery and data restoration.

• Complete service deployment, rollback, and online troubleshooting in a Kubernetes environment.

• Use AI tools to assist in development, testing, and documentation.

Requirements

• Over 3 years of backend development experience, with actual Rust project experience preferred.

• Familiarity with Rust asynchronous programming, with Tokio project experience preferred.

• Experience in web crawling, familiar with HTTP, pagination, rate limiting, and task scheduling.

• Ability to use TypeScript / Node.js, and knowledge of browser automation tools like Playwright.

• Ability to read and modify common Python scripts.

• Familiarity with Linux, Docker, Kubernetes, capable of independent deployment and troubleshooting.

• Familiarity with at least two of PostgreSQL, Redis, RabbitMQ.

• Proficient in using AI programming tools, with awareness of code validation, security, and documentation.

Bonus Points

• Experience with large-scale web crawlers, Redis Cluster, TimescaleDB, pgvector, BM25, or production incident handling.