Member of Technical Staff (Web Access Engineer, Crawler)

Perplexity · London · posted 14 Nov 2026
Full-timeEngineering5+ years

About the job

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems. Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale. Responsibilities • Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs. • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes. • Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably. • Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors. • Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests. • Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements. Qualifications • 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++. • Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions. • Understanding of CDN caching and content delivery, HTTP…

Read the full description on Perplexity's site ↗

Skills

Upload your CV on LitCareer to see which of these you match.

Similar jobs

Desktop Engineer (Electron)
Bjak · United Kingdom · 16 Nov 2026
0
Senior Associate, Data Centre Engineer
0200 Ares Operations · London, UK · 15 Nov 2026
0
Systems Engineer
0200 Ares Operations · London, UK · 15 Nov 2026
0
Vice President, Quantitative Engineer
0200 Ares Operations · London, UK · 15 Nov 2026
Senior Hypervisor Engineer
Nebius · London · 15 Nov 2026
R
Senior Security Architect
Rackspace US · UK-Hayes-Hyde Park Hayes · 15 Nov 2026

Collected from Perplexity's official careers site by LitCareer. Always apply on the employer's own site; LitCareer never charges you to apply.