Python AI Engineer - AI Indexing & Search Pipeline

About the role

A Python AI Engineer is sought to build and advance an AI indexing and search pipeline. You will design and implement end-to-end pipelines for data crawling, content processing, vector embedding generation and indexing, powering high-performance semantic search.

Key responsibilities

• Develop, optimize and maintain the Python-based Batch AI Indexer system and its end-to-end data pipeline.
• Crawl content from web sources (homepage, Naver blog and others) and optimize text chunking and preprocessing for language models.
• Use OpenAI Embeddings for vectorization, integrate Elasticsearch 8.x (dense_vector, Nori analyzer) and use FAISS for local experiments.
• Implement incremental indexing, full re-indexing, batch state management with Redis and batch history with PostgreSQL, plus retry and error-handling logic.
• Schedule jobs with APScheduler or crontab, deploy batch apps on Linux with systemd and venv, and handle logging, monitoring and troubleshooting.
• Work with Backend, AI and DevOps teams to deploy and scale the system.

Required profile

• 2-4+ years of hands-on Python development experience.
• Web scraping and crawling libraries (BeautifulSoup, Scrapy, Playwright or Selenium).
• E-commerce development experience preferred.

Required skills

• Elasticsearch 8.x (vector search, kNN, custom analyzers) and embedding models (OpenAI, HuggingFace).
• Redis and PostgreSQL; batch processing, idempotency and error handling.
• Linux, systemd, crontab/APScheduler, Git and logging frameworks.
• Nice to have: Korean NLP, LangChain or LlamaIndex, Docker, Kubernetes or CI/CD.