Senior Big Data Engineer in Softserve

WE ARE
SoftServe is a 12000+ team across the world. At the moment we are growing so you can join one of several dozen projects we are developing right now. Our big, distributed team is working with a fantastic customer, a giant in the mining industry, located in Canada. Our customer is a diversified resource company committed to responsible mining and mineral development with business units focused on steelmaking coal, copper, zinc, and energy. Headquartered in Vancouver, British Columbia (B.C.), Canada, they own or have interests in 13 operating mines, a large metallurgical complex, and several major development projects in the Americas. The client has expertise across a wide range of activities related to exploration, development, mining, and minerals processing, including smelting and refining, health and safety, environmental protection, materials stewardship, recycling, and research.

• Modern technologies popular on the market
• Interesting customers
• Either alone on a subproject or in a team
• Both developing new functionality and improving the existing ones
• Practices and approaches to continuous improvement, continuous integration, and delivery processes

Experienced with Python min 3 years:

• pyspark
• pandas
• airflow

Experienced with the following technologies/frameworks:

• Azure Data Factory
• Databricks or HDInsight or Synapse Analytics
• Azure SQL DW (a.k.a. Synapse SQL Dedicated Pool)
• Azure Blob Storage and Azure Data Lake
• Azure SQL DB/Azure SQL MI/SQL Server
• Azure Functions
• Apache Spark
• Event Hubs or similar (Kafka)
• Azure Stream Analytics or similar (Spark, Flink, Storm)
• Azure Cosmos DB (nice to have)

• Experience in complex cloud services management concept
• Networking (virtual networks, firewalls, communication between services).
• Authentication and security (IAM, service principals and managed identities)
• Event-driven architecture (Event Grid, Azure Functions, Logic Apps)
• Hands-on experience with Hadoop ecosystem (e.g. Hive, Impala, Kafka, Spark, HBase, Presto)
• Hands-on experience with streaming platforms and stream ingestion services (Event Hubs, Kafka).7
• Hands-on experience with stream analytics concepts and technologies (Flink, Spark, Storm, Azure Stream Analytics)
• Database modelling, DWH design and implementation experience.
• ETL/ELT process design and implementation experience.
• Experienced in massive-parallel-processing (MPP) systems: Azure SQL DW (Synapse), GCP BigQuery, Impala, AWS Redshift
• Hands-on experience with NoSQL databases
• Experience in optimization of code, ETL processes and SQL queries.

• Deep knowledge of SQL language and database management techniques (row-level security, built-in database roles, partitioning, index maintenance)
• Experience with OLAP systems and reporting tools (Azure Analysis Service, Power BI
• Git and gitflow
• Experience with CI/CD process building tools (Azure DevOps,)
• Project management methodologies (Agile, Waterfall)
• Strong understanding of security concepts and leading the implementation of practices and approaches in protecting data, systems, and assets in the cloud
• Accustomed to working in Agile/Scrum team setup (preferred)

TOGETHER WE WILL

• Be part of challenging projects
• Be able to work with a diverse team of professionals with a great sense of humor on a challenging project and interesting customer