Work Set-up: Remote
Schedule: 10am-6pm SGT
Responsibilities:
- Architect and Design Data Solutions: Lead the design and architecture of scalable,
secure, and efficient data pipelines for both batch and real-time data processing on
AWS. This includes data ingestion, transformation, storage, and consumption layers. - Confluent Kafka Expertise: Design, implement, and optimize highly performant and
reliable data streaming solutions using Confluent Platform (Kafka, ksqlDB, Kafka
Connect, Schema Registry). Ensure efficient data flow for real-time analytics and AI
applications. - AWS Cloud Native Development: Develop and deploy data solutions leveraging a wide
range of AWS services, including but not limited to: - Data Storage: S3 (Data Lake), RDS, DynamoDB, Redshift, Lake Formation.
- Data Processing: Glue, EMR (Spark), Lambda, Kinesis, MSK (for Kafka integration).
- Orchestration: AWS Step Functions, Airflow (on EC2 or MWAA)
- Analytics & ML: Athena, QuickSight, SageMaker (for MLOps integration).
Required Skills and Qualifications:
- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related
quantitative field. - 3 to 5 years of experience in data engineering, with a significant focus on cloud-based
solutions. - Strong expertise in AWS data services (S3, Glue, EMR, Redshift, Kinesis, Lambda, etc.).
- Extensive hands-on experience with Confluent Platform/Apache Kafka for building
real-time data streaming applications. - Proficiency in programming languages such as Python, PySpark, Scala, or Java.
- Expertise in SQL and experience with various database systems (relational and NoSQL).
- Solid understanding of data warehousing, data lakes, and data modeling concepts (star
schema, snowflake schema, etc.). - Experience with CI/CD pipelines and DevOps practices (Git, Terraform, Jenkins, Azure
DevOps, or similar). - AWS Certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified
Preferred Qualifications (Nice to Have):
- Solutions Architect - Associate/Professional).
- Experience with other streaming technologies (e.g., Flink).
- Knowledge of containerization technologies (Docker, Kubernetes).
- Familiarity with Data Mesh or Data Fabric concepts.
- Experience with data visualization tools (e.g., Tableau, Power BI, QuickSight).
- Understanding of MLOps principles and tools.
- Candidate must have a working laptop
- Work permits
- American Samoa, Australia, Bangladesh, Brunei, Bhutan, Cocos (Keeling) Islands, China, Christmas Island, Fiji, Micronesia, Guam, Hong Kong, Indonesia, India, British Indian Ocean Territory, Japan, Cambodia, Kiribati, South Korea, Laos, Sri Lanka, Marshall Islands, Myanmar (Burma), Mongolia, Macao, Northern Mariana Islands, Maldives, Malaysia, New Caledonia, Norfolk Island, Nepal, Nauru, Niue, New Zealand, French Polynesia, Papua New Guinea, Philippines, Pakistan, Pitcairn Islands, Palau, Russia, Solomon Islands, Singapore, Thailand, Tokelau, Timor-Leste, Tonga, Tuvalu, Taiwan, U.S. Outlying Islands, Vietnam, Vanuatu, Wallis & Futuna, and Samoa
- Timezones
- APAC
- Education
- bachelor degree and postgraduate degree
- Experience
- 3+ years