Senior Data Engineer
- Architect, develop, and maintain scalable data infrastructures including relational databases, data lakes, and cloud-based data warehouses (e.g., Redshift, BigQuery, Snowflake)
- Work with big data technologies (e.g., Hadoop, Spark), NoSQL databases (e.g., MongoDB, Cassandra), and AWS cloud platform.
- Handle real-time streaming data using tools like Apache Kafka.
- Design real-time streaming systems that ingest, process, and analyse data continuously with minimal latency, using event brokers like Apache Kafka.
- Handling event-driven architectures where data changes are captured and processed immediately for real-time analytics or triggering actions.
- Work with stream processing frameworks such as Apache Flink, Apache Spark Streaming to perform transformations, aggregations, and validations on streaming data.
- Implementing scalable, fault-tolerant streaming pipelines that maintain data integrity despite high throughput and velocity.
- Design, implement, and optimize ETL/ELT pipelines to collect, clean, transform, and load data from various internal, external, structured and unstructured sources.
- Build and maintain scalable data architectures (e.g. data lakes/data warehouses).
- Collaborate with software engineers, product managers, platform engineers and support teams to ensure data accessibility, quality, and integrity.
- Work with our Security Manager to maintain data governance and security best practices.
- Monitor, troubleshoot, and optimise the performance of data pipelines and systems, using tools such as Kafka.
- Participate in code reviews and follow best practices for software engineering and DevOps in data environments.
- Work with AWS cloud platform to manage data infrastructure.
- Build robust, automated data workflows using orchestration tools such as Apache Airflow, ensuring dependency management, scheduling, and error handling.
- Manage large-scale datasets, optimise storage strategies, and ensure efficient querying and data retrieval performance.
- Create self-healing, anomaly-detecting pipelines that ensure data reliability and freshness across environments.
- Apply CI/CD practices and version control (Git) to data pipelines, promoting robust, testable, and collaborative data engineering workflows.
- Implement real-time data processing to support real-time analytics and decision-making.
- Handle ingestion, transformation, and enrichment of streaming data, ensuring fault tolerance and scalability.
- Bachelor's degree in Computer Science, Data Engineering, Information Systems or a related field.
- 5-7 years experience in software engineering, with 3+ years of experience in data engineering or similar role.
- Strong proficiency in SQL, including MySQL and PostgreSQL, proficiency with NoSQL databases and knowledge of Kafka and Java coding.
- Experience with data pipeline tools and frameworks (e.g., Apache Airflow, dbt, Luigi).
- Knowledge of cloud-native streaming tools like Amazon Kinesis
- Familiarity with data storage and warehousing technologies (e.g., Snowflake, Redshift, BigQuery, Databricks).
- Experience working with cloud platforms, ideally with AWS using Docker and Kubernetes.
- Strong understanding of data modelling, normalisation, and performance optimisation.
- Knowledge of real-time data processing (Kafka, Spark Streaming, Flink).
- Familiarity with CI/CD pipelines and infrastructure as code (Terraform, CloudFormation).
- Experience with Git management tools
- Ability to communicate and collaborate clearly and effectively
- Strong time management skills with the ability to prioritise workloads under pressure and meet deadlines
- A self-starter who is hands-on and knows how to find answers and work with ambiguity.
- Values-driven and practical in your approach.
- We are an accredited flexible company and were named in Flexa's Top 50 Most Flexible Companies:
- We utilise a hybrid working model, providing equipment for home working alongside one or two monthly visits to our beautiful central London office.
- We offer a highly competitive benefits package including private healthcare and rail discounts.
- We provide a wealth of career development opportunities with training that is individual, focused on improving your skills and helping you become the best professional you can be.
- Our team's health and wellness is genuinely important to us, so we offer a number of wellbeing seminars and membership to the #1 leading meditation app.
- A unique opportunity to work for a tech company that is helping the environment by revolutionising the way we travel.
Recommended Jobs
Band 8a/8b Locum Pharmacist (Haematology/Oncology) - London
Role: Locum Pharmacist (Haematology/Oncology) Banding: Band 8a/8b Location: London Start Date: ASAP Duration: 3 Months Hours: Full-time, 37.5 hours per week (Monday to Friday and we…
Electrician
ELECTRICIAN - NIGHTS - NORTHWOOD - LEADING UK DEPARTMENT STORE*** We are seeking a skilled Electrician to join our dynamic team. The ideal candidate will possess a strong foundation in electrical wo…
Mortgage Sales Manager - Middlesex
Mortgage Sales Manager – Middlesex – High Earnings Job Reference: 55006571 Location: Middlesex – Enfield, Harrow, Wembley, Pinner, Colindale, Maidenhead, Slough, Watford, Ruislip, Uxbridge S…
In-house Editor and Videographer
Who we are: Broadwick Live is a multifaceted international company with headquarters in London who create, develop, own and operate a diverse portfolio of venues, spaces, events and experiences. …
Retail Supervisor - Part Time
Retail Supervisor - Part Time, 24 hoursWe are looking to recruit a new retail supervisor for our shop in Livingston, West Lothian. We offer a permanent part time position working a variety of shifts,…
UK - Paid Search Account Manager
About You: You have a passion for all things paid search and want to further develop as an expert You want to be involved in all areas of the campaign, driving performance and working with a ra…
Associate Director - Tax
Associate Tax Director Top London based practice - Advisory £95,000 plus fantastic benefits Hybrid working Our client is a full service, tech enabled accountancy firm, supporting fast-grow…
Ongoing Locum RVN needed from mid Sept in Southwest London
Our client, a friendly and well-equipped small animal hospital in South West London, is looking for a Registered Veterinary Nurse to join their team on an ongoing basis from 15th September. Dates ava…
Finance Business Partner
Finance Business Partner London Up to £42,000 per annum Are you a commercially savvy finance professional looking to make a significant impact in a dynamic environment? Do you excel at b…
Procurement Manager
Tapestry is seeking a Procurement Manager to lead strategic sourcing initiatives across the EMEAI region. This role involves managing supplier relationships, conducting market analysis, and collaborat…