Founding Data Pipeline Engineer at Autoplay

Join an ambitious team pushing the frontiers of AI tech with the aim of reimagining the way humans interact with software. We are backed by Betaworks, the same accelerator that incubated Hugging Face, and closed a $2.1M pre-seed round.

Autoplay builds AI agents that help users navigate any software to achieve their goals in real time. We are a deep tech company that prioritizes user experience. Our mission is to build a product that seamlessly integrates into users’ lives, making software feel like magic.

At Autoplay, we believe that the greatest innovations are achieved when software is used to its fullest potential. However, the current interaction between humans and software is riddled with friction, leading to frustration and a lack of confidence. Our goal is to move software use into a ‘happy place’ where users are excited to engage and optimize it, allowing them to explore and push their creativity to new heights.

As a Senior Data Pipeline Engineer, you will play a critical role in architecting, building, and maintaining large-scale data pipelines capable of efficiently handling vast amounts of heavy video files. You will leverage technologies such as Apache Airflow, Apache Spark, and cloud-native solutions on GCP and AWS to optimize performance, scalability, and reliability. Your work will be integral to ensuring smooth data ingestion, transformation, and processing to support our AI-driven platform.

• Bachelor’s Degree in Computer Science, Engineering, or a related field
• 5+ years of experience designing and building scalable data pipelines
• Extensive experience with Airflow for workflow automation
• Strong expertise in Apache Spark for large-scale data processing
• Proven experience working with heavy video data pipelines, including transcoding, storage optimization, and streaming
• Hands-on experience with DevOps practices for data engineering, including CI/CD, containerization (Docker, Kubernetes), and infrastructure as code (Terraform, CloudFormation)
• Experience working with cloud platforms (GCP and AWS), including S3, GCS, BigQuery, Redshift, Lambda, Pub/Sub, and Kafka
• Strong understanding of distributed computing, data partitioning, and performance optimization

Technical Skills:

• End-to-End Data Architecture Design – Ability to design scalable and fault-tolerant data architectures that can handle increasing workloads as the company grows.
• DataOps & CI/CD for Data Pipelines – Experience with automating data workflows, testing, and deployment using tools like Jenkins, GitHub Actions, and Terraform.
• Streaming Data Processing – Knowledge of Kafka, Flink, or Spark Streaming for real-time data ingestion and processing.
• Metadata Management & Data Lineage – Ability to track, log, and monitor data flows using tools like Apache Atlas or dbt.
• Cost Optimization & Performance Tuning – Experience optimizing cloud costs (e.g., storage, compute, egress costs) while keeping pipelines efficient and responsive.
• Experience with Distributed Databases – Understanding of NoSQL, columnar storage (e.g., BigTable, Cassandra, ClickHouse) for efficient querying.
• Data Privacy & Compliance – Familiarity with GDPR, CCPA, HIPAA if working with sensitive data.

Non-negotiables:

• You are curious, resourceful, and love solving complex technical challenges
• You have a strong builder’s mindset and enjoy taking ownership of large-scale systems
• You thrive in an early-stage startup environment, working collaboratively and adapting quickly to evolving requirements

Preferred Skills:

• Experience with video compression, encoding, and streaming protocols (e.g., HLS, MPEG-DASH, FFmpeg, WebRTC)
• Proficiency in Python, Scala, or Java for data pipeline development
• Familiarity with data lake architectures and Delta Lake or Iceberg
• Strong understanding of real-time data processing frameworks (e.g., Flink, Spark Streaming, Kafka Streams)

• Design, build, and maintain scalable, high-performance data pipelines optimized for handling large-scale video files
• Develop ETL workflows using Apache Airflow for scheduling and orchestration
• Implement distributed data processing solutions leveraging Apache Spark
• Optimize storage, retrieval, and processing of heavy video datasets in cloud environments (GCP and AWS)
• Collaborate closely with AI, backend, frontend, and DevOps teams to ensure seamless data integration and availability
• Establish robust monitoring, logging, and alerting mechanisms for data pipelines to ensure high availability and reliability
• Implement best practices for data governance, security, and compliance
• Research and implement cutting-edge data engineering techniques to improve pipeline efficiency and cost-effectiveness