The Role
- As a big data engineer, you will be responsible for developing solutions and implementing requirements around our large-scale data processing platform in the Amazon cloud.
- This includes writing automated loaders that ingest data from various sources, developing modules to enrich raw data with additional attributes and designing algorithms to connect millions of data points every week
- In addition, you will work on algorithms to derive insights from our data set and implement tools to ensure a frictionless delivery of data to our customers all around the globe.
- Work as part of our big data processing team to bring our platform to the next level
- Enhance our data processing pipeline by implementing new algorithms on Apache Spark
- Develop new distance measures and rules to connect data points
Requirements
- At least two years of experience as Java developer
- Eligible to work in Germany and willing to re-locate to Frankfurt am Main
- Skills in scripting languages like Python
- Fluent in English
Nice to Have
- Previously worked in agile environments
- Experience with AWS infrastructure like S3, Redshift, EC2, etc.
- Experience working with big data and technologies like Hadoop, Spark, Oozie, Hive, etc.
- Experience with rule based expert systems and machine learning algorithms