Software Engineer
Full Job Description
About Anyscale
At Anyscale, we are on a mission to democratize distributed computing for software developers of all skill levels. We commercialize Ray, an open-source project creating libraries for scalable machine learning, trusted by companies like OpenAI, Uber, Spotify, and Instacart.
About The Role
Ray provides a universal API for building distributed applications where data is critical to usability and performance. We are seeking strong engineers to build, optimize, and scale Ray's Datasets library and data processing capabilities.
The Ray Data Team
This team develops the Ray Datasets library used in production at Amazon and Alibaba. Built on Apache Arrow and Ray Core (C++ backend), we interact closely with schedulers, memory/I/O subsystems, and ML libraries like Train, RLlib, and Serve.
Key Projects
- Optimize large-scale performance of Ray Datasets using Arrow primitives.
- Integrate seamlessly with diverse ML training and data sources.
- Ensure stability through rigorous stress testing infrastructure.
- Pioneer streaming workloads integration (e.g., Beam on Ray).
Responsibilities
- Develop high-quality open-source software to simplify distributed programming.
- Implement architectural improvements to Ray core and Datasets.
- Enhance testing processes for smoother releases.
- Promote work through talks, tutorials, and blog posts.
We'd Love To Hear From You If
- You have at least 5 years of relevant experience.
- You possess a solid background in algorithms, data structures, and system design.
- You have built scalable, fault-tolerant distributed systems.
- Experience with database internals (Spark or Dask) is required; streaming expertise is a plus.
Company
Anyscale
Anyscale empowers Python developers to build and run all their AI workflows—from data preparation to training and inference—at any scale, trusted by leading teams at Canva, TripAdvisor, Physical Intel...