Department Manager, AI - Data Engineer
About this position
Responsibilities
• Develop and maintain robust data pipelines to ingest, process, and transform raw data into formats suitable for LLM training.
• Conduct meeting with users to understand the data requirements and perform database design based on data understanding and requirements with consideration for performance
• Maintain data dictionary, relationship and its interpretation
• Analyze problem and find resolution, as well as work closely with administrators to monitor performance and advise any necessary infrastructure changes
• Work with business domain experts, data scientists and application developers to identify data that is relevant for analysis
• Develop big data solutions for batch processing and near real-time streaming
• Own end-to-end data ETL/ELT process framework from Data Source to Data warehouse
• Select and integrate appropriate tools and frameworks required to provide requested capabilities
• Design and develop BI solutions
• Hands-on development mentality, with a willingness to troubleshoot and solve complex problems
• Keep abreast of new developments in the big data ecosystem and learn new technologies
• Ability to effectively work independently and handle multiple priorities
Requirements
• Bachelor degree or higher in Computer Science, Computer Engineering, Information Technology, Management Information System or an IT related field.
• 3+ year's experiences in Data Management or Data Engineer (Retail or E-Commerce business is preferrable)
• Expert experience in query language (SQL), Databrick SQL, PostgreSQL
• Experience in Big Data Technologies like Hadoop, Apache Spark, Databrick
• Experience in Python is a must
• Experience in Generative AI is a must
• Knowledge in machine/statistical learning, data mining is a plus
• Strong analytical, problem solving, communication and interpersonal skills.
• Having good attitude toward team working and willing to work hard.