Job

Location: 1001 Sagittarius Avenue, Franklin, IN 46131

Design, develop, and deploy the scalable, robust data pipelines on GCP (Google Cloud Platform) to facilitate the data movement, transformation, and data loading process for the data sourced from various systems.
Develop data ingestion pipelines using Cloud Storage, Cloud Functions, BigQuery, Cloud DataFLow, Cloud Composer, Pub/Sub, and Cloud Spanner to facilitate real-time data streaming from variety of sources like on-prem databases and applications and streamline the workflows to handle large volume of data.
Build extensive data quality framework to ensure accuracy, completeness, integrity, and consistency of data for analytics and decision-making process.
Work with GCP services such as BigQuery for large-scale data warehousing, analysis, Cloud Storage for secure storage capability, dataflow for handling real-time data processing, Pub/Sub for event-driven data ingestion, and CLoud Composer to scheduled/orchestrate the workflows and run data pipelines efficiently.
Implement data quality checks and validation processes in the ETL (Extract, Transform, and Load) pipelines built in GCP using Cloud Data Fusion for extraction, Dataflow for transformation with validation checks, BigQuery for loading, Cloud Logging for error logging, Cloud Monitoring for alerts, and Looker or Data Studio for visualization.
Work with different stakeholders in the data engineering team to collaborate on data process brainstorming ideas, architecture, code reviews, performance improvement measures, and best practices for the pipeline development.
Design, and develop the data governance policies to align with business standards and requirements.
Migrate the data from on-premise to cloud systems ensuring the appropriate data movement.
Establish monitoring and alerting using Cloud Monitoring (formerly Stackdriver) for real-time alerts, Cloud Logging for tracking and logging issues, and Cloud Functions to automate responses for quick resolution of detected issues.
Work with Transfer Service for Cloud Storage for efficient data transfer, Cloud Storage for temporary storage, BigQuery for analytics integration, and Cloud Pub/Sub for real-time data ingestion.

All the responsibilities mentioned above are in line with the professional background and requires an absolute minimum of a Bachelor’s degree in computer science, computer information systems, information technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.

Stier Solutions

Latest Jobs

GCP Data Engineer