Member of Technical Staff (Index Engineer, Search Crawler)

Perplexity
London

AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.

Responsibilities

  • Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.

  • Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.

  • Work on storage formats, compaction, and update mechanisms that keep data accessible and current.

  • Keep pipelines reliable and predictable under high-throughput conditions.

  • Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.

  • Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.

  • Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.

  • Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.

  • Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.

Qualifications

  • 5+ years of experience building production backend or data infrastructure systems.

  • Strong programming experience in Go, C++, or Rust.

  • Experience with high-throughput data processing systems and very large datasets.

  • Experience building or operating databases, storage systems, data planes, or indexing pipelines.

  • Strong understanding of distributed systems, fault tolerance, consistency, and scalability.

  • Experience running production systems and handling operational incidents.

  • A systems-thinking mindset and the ability to reason about end-to-end data flows.

Posted 2026-11-18

Recommended Jobs

Member of Technical Staff, Training Performance Engineer

Cohere
London

Who are we? Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. …

View Details
Posted 2026-11-09

Commis Chef (Kitchen Support Expert) - London Marriott Hotel Grosvenor Square

Marriott
London

Commis Chef (Kitchen Support Expert)  London Marriott Hotel Grosvenor Square At Marriott Hotels, we believe our legacy is built by the people who bring it to life every day. As part of a team …

View Details
Posted 2026-11-09

Field Service Engineer (Swimming Pools)

Rise Technical Recruitment
Croydon, Greater London

Field Service Engineer (Swimming Pools) £30,000 - £40,000 + Van + Company Benefits Home based: Croydon, Sutton, Tonbridge or surrounding areas Are you a Field Service Engineer from a Swimming Pool …

View Details
Posted 2026-10-30

Strategic Account Executive (Hiring Immediately)

EngageTech
London

Description About EngageTech EngageTech helps enterprise B2B technology companies build a better pipeline. For nearly two decades we have combined elite SDRs with proprietary data and AI, with …

View Details
Posted 2026-11-18

VodafoneThree - Solution Architect - Web & CMS Platforms

Vodafone
London

Solution Architect - Web & CMS Platforms Location: London OR Newbury + Hybrid Salary: Excellent basic salary plus bonus and Vodafone benefits Working hours: Full time 37.5 hours per week – Mond…

View Details
Posted 2026-11-18

Au-pair in London, Job ID J219BF

Little Ones UK Ltd
Brixton, Greater London

A lovely family in Brixton is looking for an experienced Au-pair or Junior Nanny to join their household and assist in caring for their children. The ideal candidate will be someone with previous exp…

View Details
Posted 2026-11-06

Senior Project Manager (Hiring Immediately)

pracyva
London

Company Description Job Description Role: Senior Project Manager Duration: 6 Months Work Location: London Experience: 10+ years Job Description: Looking for a Project Manager who…

View Details
Posted 2026-11-15

Maintenance Operative

TXM Recruit
Wimbledon, Greater London

TXM Recruit are currently recruiting for a mobile Maintenance Operative to work for one of the UKs leading facilities companies on a plethora of their commercial contracts. Schedule - Monday to Fri…

View Details
Posted 2026-10-30

Merchant Acquiring Product Management, Global Payment Solutions

Bank of America Corporation
London

Job Title: Merchant Acquiring Product Management, Global Payment Solutions Corporate Title: Vice President Location: London, United Kingdom Company Overview At Bank of America, we are…

View Details
Posted 2026-10-21