Solution Engineer – Data Pipelines & AI

About the role

Digital Divide Data is seeking a Solution Engineer to design, build, and operate data ingestion pipelines and AI‑powered services. You will work with modern cloud technologies to transform raw CSV/TSV files into structured, queryable datasets that power intelligent applications.

Key responsibilities

• Design and implement robust data ingestion pipelines, converting raw files into standardized schemas.
• Model custom entities and map diverse data sources to target schemas.
• Build and maintain data environments using SQLite, Google Cloud SQL, and BigQuery.
• Develop Dockerised services for data loading, embedding generation, and data‑serving applications.
• Deploy and operate production workloads on Google Cloud Run, ensuring scalability, reliability, and high availability.
• Work across Redis, VPC networking, Artifact Registry, and Cloud Storage.
• Implement secure cloud architectures with Cloud IAM, VPC, and Cloud Run ingress controls.
• Automate workflows and deployments via GitHub Actions, including scheduled ingestion and container builds.
• Integrate AI capabilities such as natural‑language querying, semantic search, embeddings, and LLM‑driven workflows.
• Contribute to emerging agentic architectures and protocols like Model Context Protocol (MCP).

Required profile

• 2+ years of experience in data engineering, cloud engineering, or backend development.
• Strong Python development skills with practical Pandas experience.
• Solid experience with Docker and containerised applications.
• Deep knowledge of Google Cloud Platform services (compute, managed databases, data warehousing, networking, IAM).
• Proficient SQL skills and experience with relational and analytical data stores.
• Experience designing, building, and consuming REST APIs.
• Self‑motivated engineering mindset with ability to troubleshoot complex problems and own end‑to‑end delivery.

Required skills

• Python
• Pandas
• Docker
• Google Cloud Platform (Compute, Cloud SQL, BigQuery, VPC, IAM, Cloud Run, Cloud Storage, Artifact Registry)
• SQLite
• SQL
• REST APIs
• GitHub Actions
• Redis
• AI/ML – embeddings, Sentence Transformers, LLM applications
• Flask
• Jinja

What we offer

• Opportunity to build the technical foundation for data‑driven, AI‑enhanced applications.
• Hands‑on experience with cutting‑edge cloud and AI technologies.
• Collaborative environment focused on continuous learning and innovation.