Data Engineer

About the job Data Engineer Job Title: Data Engineer Experience: 8 to 10 Years ?Time Zone: IST Time?? Job Type: Remote Work Location: - ?Domain: - ?? Responsibilities: ? Data Pipeline Architecture: Design, develop, and optimize end-to-end data pipelines to extract, transform, and load (ETL) data from various sources into our data warehouse. Ensure data quality, reliability, and performance throughout the pipeline. ? Data Modeling and Schema Design: Work with data scientists, analysts, and stakeholders to understand data requirements and create scalable and efficient data models. Implement and maintain database schemas that facilitate easy data access and querying. ? Data Integration: Integrate data from diverse internal and external sources, including databases, APIs, and third-party systems. Build connectors and adaptors to ensure seamless data flow between systems. ? Performance Optimization: Continuously monitor and fine-tune the performance of data pipelines and databases... Identify bottlenecks and implement optimizations to enhance processing speed and resource utilization. ? Data Security and Governance: Implement robust security measures to safeguard sensitive data. Ensure compliance with data protection regulations and industry best practices for data governance and privacy. ? Data Transformation and Enrichment: Develop data transformation routines to enrich raw data and make it suitable for analytical processing. Apply data cleansing, aggregation, and normalization techniques as needed. ? Data Monitoring and Error Handling: Establish monitoring systems to detect data inconsistencies, anomalies, and errors. Develop automated alerts and error handling processes to ensure data integrity. ? Technology Evaluation and Implementation: Stay up-to-date with the latest data engineering technologies and best practices. Evaluate new tools and frameworks, and lead the implementation of suitable technologies to improve data processing efficiency and scalability. ? Documentation and Collaboration: Maintain comprehensive documentation for data engineering processes, data dictionaries, and workflows. Collaborate with cross-functional teams to understand their data needs and deliver effective solutions. ? Mentoring and Leadership: Mentor and provide guidance to junior data engineering team members. Act as a technical leader, driving innovation and best practices within the data engineering team. Requirements: ? Bachelor's or Master's degree in Computer Science, Engineering, or a related field. ? Proven track record of at least 8 years of experience in data engineering roles. ? Expertise in building and maintaining data pipelines using ETL tools like Apache Spark, Apache Airflow, or similar. ? Strong proficiency in SQL and database technologies (e.g., PostgreSQL, MySQL, NoSQL databases). ? Extensive experience with cloud-based data platforms, such as AWS, Azure, or Google Cloud Platform. ? Solid understanding of data modeling concepts and data warehousing principles. ? Proficiency in at least one programming language (e.g., Python, Java) for data manipulation and automation. ? Experience with data streaming technologies (e.g., Kafka, Kinesis) is a plus. ? Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes) is beneficial. ? Excellent problem-solving skills and the ability to work independently and as part of a team. ? Strong communication and interpersonal skills to collaborate effectively with stakeholders across the organization

Apply Job!