Engineer

Join NielsenIQ as an Engineer specializing in Python Web Scraping in Pune or Chennai. This role is crucial for our Data Impact division, which leverages vast amounts of e-commerce data to provide innovative monitoring and action tools for businesses. We are a fast-growing, international startup with a dynamic and complementary team, offering real job and career opportunities in a friendly, trust-based environment that encourages autonomy and challenges.

Responsibilities:

• Capture massive data from the web and mobile terminals.
• Design architectures for data extraction, deduplication, classification, clustering, and filtering.
• Develop distributed web crawlers and independently resolve development issues.
• Research and implement algorithms for web page information extraction to enhance data capture efficiency and quality.
• Analyze and warehouse crawled data, and monitor crawler systems for anomalies.
• Design and develop data collection strategies and anti-shielding rules.
• Develop core algorithms based on system data processing flows and business requirements.

Qualifications:

• Proficiency in Python and familiarity with web scraping frameworks like Scrapy.
• 1-12 years of experience in web scraping and data crawling projects.
• Expertise in vertical search crawlers, distributed web crawlers, data parsing, cleaning, storage, and anti-crawler technologies.
• Proficiency in Linux basic operations.
• Experience in distributed crawler architecture design and IP farms/proxies is a plus.
• A strong foundation in data structures and algorithms is preferred.
• Familiarity with data storage and processing technologies, including RDBMS and non-structured databases.
• Experience with frameworks like SSH, multi-threading, and network communication programming.
• Hands-on experience crawling e-commerce platforms is a significant advantage.

Additional Information & Benefits:

• Flexible and rewarding work environment with peer-to-peer recognition.
• Wellness plans for you and your family.
• Financial wellness tools.
• Career development opportunities for upskilling.
• Flexible working environment.
• Volunteer time off.
• LinkedIn Learning access.
• Employee Assistance Program (EAP).

NIQ is committed to diversity, equity, and inclusion, reflecting the diverse markets we serve. We are an Equal Opportunity/Affirmative Action-Employer and encourage candidates who share our mission to apply. Learn more about our commitment at the NIQ News Center.