Data Engineer - (Migration & Profiling)
Location: Remote
BGV: Basic
Description
● 5-7 yrs of IT experience
● Good understanding of analytics tools for effective analysis of data
● Should be able to lead teams
● Should have been part of the production deployment team, Production Support team
● Experience with Big Data tools- Hadoop, Spark, Apache Beam, Kafka, etc.
● Experience with object-oriented/object function scripting languages: Python, Java, C++,
Scala, etc.
● Experience with any DW tools like BQ, Redshift, Synapse, or Snowflake
● Experience in ETL and Data Warehousing.
● Experience and firm understanding of relational and non-relational databases like MySQL,
MS SQL Server, Postgres, MongoDB, Cassandra, etc.
● Experience with cloud platforms like AWS, GCP and Azure.
● Experience with workflow management using tools like Apache Airflow.
Role
● Roles & Responsibilities
● Develop high-performance and scalable solutions using GCP that extract, transform, and
load big data.
● Designing and building production-grade data solutions from ingestion to consumption using
Java / Python
● Design and optimize data models on the GCP cloud using GCP data stores such as
BigQuery
● Should be able to handle the deployment process
● Optimizing data pipelines for performance and cost for large-scale data lakes.
● Writing complex, highly optimized queries across large data sets and creating data
processing layers.
● Closely interact with Data Engineers to identify the right tools to deliver product features by
performing POC
● Collaborative team player that interacts with business, BAs, and other Data/ML engineers
● Research new use cases for existing data.
● Preferred:
● Need to be Aware of Design Best practices for OLTP and OLAP Systems
● Should be part of team designing the DB and pipeline
● Should have exposure to Load testing methodologies, Debugging pipelines and Delta load
handling
● Worked on heterogeneous migration projects
● Role
●
● Develop high-performance and scalable solutions using GCP that extract, transform, and
load big data.
● Designing and building production-grade data solutions from ingestion to consumption using
Java / Python
● Design and optimize data models on the GCP cloud using GCP data stores such as
BigQuery
● Should be able to handle the deployment process
● Optimizing data pipelines for performance and cost for large-scale data lakes.
● Writing complex, highly optimized queries across large data sets and creating data
processing layers.
● Closely interact with Data Engineers to identify the right tools to deliver product features by
performing POC
● Collaborative team player that interacts with business, BAs, and other Data/ML engineers
● Research new use cases for existing data.
