Informations
Jobs
What is the article about?
Are you interested in data and cloud technology? Would you like to understand more about different roles in development teams? Are you looking for a new job? This blog post is about our way of working and our job profiles at Otto Group data.works.
Today, meet Sven Peeters, Data Engineer at Otto Group data.works since February 2021.
Hi Sven! Can you introduce your job in 3 sentences?
In short, as a Data Engineer I am responsible to design and implement new features for different data-driven services. Our products range from computing customer profiles in a large scale to hosting HTTP services which categorize content of webpages using machine learning. The data we provide is mainly used to enrich programmatic advertising with behavioral and contextual information.
Arbeitssituation
"I am responsible to design and implement new features for different data-driven services."
Sounds interesting! Give us some more details into your working day!
My daily work is manifold and hard to describe to the full extent in this short blog post. In my team, I work closely together with Machine Learning Engineers and Data Scientists. Hence, my tasks are often related to machine learning in addition to usual data and software engineering topics, which can be summarized in the following areas:
Besides coding and drafting cloud architectures, I spend some time every day sharing best practices and technology news or I just play a match of kicker with my great colleagues from data.works.
- Design and implementation of Big Data ETL-Pipelines which make predictions using Python, Java, Kubeflow Pipelines, BigQuery SQL, Dataflow and Kubernetes (GKE)
- Design and implementation of microservices using Java, Python, Cloud Functions, Cloud Run, AppEngine and Kubernetes (GKE)
- Management of infrastructure in the Google Cloud Platform with Terraform
- Maintenance including bug fixes, patching and support of all products previously built by me and my team
- Closely working together with our external clients and Otto Group companies for optimal customer satisfaction
Besides coding and drafting cloud architectures, I spend some time every day sharing best practices and technology news or I just play a match of kicker with my great colleagues from data.works.
Knowing that the interpretation of role profiles and titles can vary in different companies, we want to give a rough overview on how we define these roles.
What are you most proud of since starting in our team?
I am most proud of the rework of the full text search backend of our contextual targeting product. Due to heavily increasing load on the product our old PostgreSQL backend was stumbling and reached the end of scalability. Despite me being a Junior Data Engineer at that time, my team enabled me to take a leading position in the design and development of the new backend. This included choosing Apache Solr as our new full text search engine, designing the cloud architecture, and migrating or rewriting affected components of the old backend. The new backend has been online since April and runs like a charm.
You already mentioned many different technologies that you’re using.
What is your favorite one and why?
In our daily work we use many recent and impressive technologies like Kubeflow Pipelines, Argo or Apache Beam on Dataflow. Hence, choosing one that stands out is not easy. If I need to select one as my favorite, I would choose Dataflow in conjunction with Java. The serverless approach of Dataflow and the pipeline semantics of Apache Beam which unifies batch and stream processing offers me the opportunity to write Java code that processes a huge amount of data without spending hours to think about how I can scale the program and distribute the processed data over machines effectively. Jobs that would run days on a single machine are seamlessly executed in some minutes on over 100 machines in parallel. It’s always fun to see how my code gets distributed and executed in parallel over a huge number of machines in the Google Cloud Platform.
Sounds like you’re really passionate about your job! And what are you doing when you close your work laptop?
In my free time I like to play table tennis with friends in my local table tennis club. Besides playing table tennis, I have a huge passion for motorsport. I watch almost every session of each Formula 1 weekend and love to spend some time on virtual motorsport circuits.
To sum it up, what is the best about working at Otto Group data.works?
Otto Group data.works offers me the opportunity to pursue and learn a lot about my passion in big data processing and machine learning while developing smart and cool large scale data products in a young and dynamic team of data enthusiasts.
This article was originally published on Medium and can be read there as well.
Want to be part of our team?
0No comments yet.
Write a comment
Answer to: Reply directly to the topic
Written by
Sven Peeters
Data Engineer
Similar Articles
FabianApril 17, 2024Confluent Helps Power a Diverse and Scalable Online Shopping Experience for OTTO
05Find out how OTTO is taking its 10-year-old data infrastructure into a new era with the event streaming platform powered by Confluent and making itself future-proof.DevelopmentOperationsWorking methods
Christian, Jürgen, Konstantinos, Tobias, TungNovember 15, 2023Dagster: Data Engineering Best Practice during Slack Time
011In the inc(AI) team, we are always trying out different tools and methods. Slack time has proven to be a good way of doing this, in which we learn alone or together. The motivation for such Slack time naturally comes from the challenges that we have to solve during our work. Our strategy is to always look to the open source community first to find suitable solutions for us. Some time ago, we also investigated a new tool called Dagster in our Slack time.DevelopmentWorking methods
