Big Data Engineer - Denmark
Want to know company name or location? Company managed [?]
Posted on : 08 March 2017
- Our company has embarked on the Digitization and Big Data journey and we have many exciting initiatives ongoing.
- We believe strongly in establishing our company internal capabilities within Big Data and want to proof the value of Big Data with high-performing teams and dedicated focus.
- As teams are formed to drive initiatives forward we need an ambitious and skilled Data Engineer to help us get a shared, well-supported, future-proof Big data foundation in place.
- Our new Data Engineer will be part of our Big Data initiatives and help structure and support the processes for collecting, storing, processing and analyzing big data. In the initial phases your focus will be on framing the Big Data Platform together with architects, data scientist and business analysts and identifying the tool set and structures required for a scalable solution supporting our many initiatives.
- Following the focus will be on testing, implementation, maintenance and monitoring the solution components with a continuous focus on exploration, optimization and improvement.
- You will also have co-responsibility for the Big Data Platforms integration with the wider architecture used across the company.
- You will be anchored in our company IT and it will be your job to ensure that many parallel Big Data initiatives in the company corporate business units are established on a uniform set of processes, supported by open source tools, delivered within frameworks on a shared platform supporting the various needs of business units. Initiatives will go through different stages of development through to production and they must be supported accordingly throughout this life-cycle.
- Your tasks include selecting and integrating Big Data tools and frameworks required to support Big Data initiatives;
- Implementing ETL processes to manage data sources and target destinations; Defining data retention policies;
- Developing, maintaining, testing and evaluating Data Scientists’ recipes with relevant toolsets; Monitoring and controlling versioning and code promotion; Monitoring performance and advising on changes to setup; and supporting Data Scientists and Business Analysts on daily operations related to tools and processes.
In this position, you need:
- A Bachelor’s or Master’s degree in computer science or software engineering and a minimum of 3 years of practical experience working with Big Data.
- To thrive with complex problem solving on a daily basis.
- Good oral and written communication skills in English.
- Experience with the use of distributed computing principles and experience with building stream-processing systems, using solutions such as Storm and Spark-Streaming.
- Proficiency with Hadoop and good knowledge of Big Data querying tools, such as Pig, Hive, and Impala, and NoSQL databases, such as HBase, Cassandra and MongoDB.
- Experience with integration of data from multiple data sources using nifi or similar, and with various messaging systems, such as Kafka or RabbitMQ.
- Knowledge of various ETL techniques and frameworks and experience in designing robust workflows.
- Good understanding of Lambda Architecture, along with its advantages and drawbacks and experience and desire to work with cloud computing environments.
- To prefer working in teams and collaborating with others to clarify requirements.
- You want to be ahead of the game and you see development potentials and possess the skills to drive these forward.
- A good sense of humor and the ability to communicate well with people from different organizational units.
- Our company is the world leader in biological solutions.
- Together with customers, partners and the global community, we improve industrial performance while preserving the planet’s resources and helping build better lives.
- As the world’s largest provider of enzyme and microbial technologies, our bio innovation enables higher agricultural yields, low-temperature washing, energy-efficient production, renewable fuel and many other benefits that we rely on today and in the future.