Senior Web Crawling Engineer
AI SummaryWritten from the original listing. Read the full description below before applying.
The Senior Web Crawling Engineer will develop and maintain the web crawling infrastructure for data extraction and analytics platforms, focusing on building scalable crawlers and optimizing performance.
Hexanix One — IT Services and IT Consulting, headquartered in Ahmedabad, Gujarat.
About The Role
We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.
Key Responsibilities
- Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology
- Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers
- Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems
- Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)
- Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation
- Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)
- Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy websites
Required Qualifications
- 5+ years of strong Python development experience
- Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions)
- Deep understanding of Scrapy internals (Twisted, asynchronous networking)
- Strong experience with XPath, CSS selectors, JSONPath
- Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions
- Experience handling dynamic websites and API-based scraping
Nice to Have
- Experience integrating Scrapy with Playwright or Selenium
- Knowledge of bot detection techniques and mitigation
- Experience building REST APIs for crawl orchestration
- Familiarity with cloud deployment and DevOps practices
What We Offer
- Competitive compensation package
- Hybrid work model with flexible hours
- Professional development opportunities
- Team building events and activities
- Generous PTO and holiday schedule
Employment type
Full-time
Seniority level
Mid-Senior level
Job function
Engineering and Information Technology
Industries
IT Services and IT Consulting
Collected automatically from in.linkedin.com on Aug 20, 2026.Report an issue with this listing