Ilya Kreymer
Lead Software Engineer at Webrecorder Software
San Francisco, California, United States
Summary
🤩
Rockstar🎓
Top SchoolIlya Kreymer is a lead software engineer and open-source founder with 14+ years building high-fidelity web archiving systems and infrastructure from the Wayback Machine to Webrecorder and Conifer. Based in San Francisco, he combines backend, full-stack and DevOps expertise to deliver robust crawlers, replay tooling and archival data pipelines used by institutions and individuals worldwide. He founded and leads projects like webrecorder/browsertrix-crawler and pywb, contributing pragmatic fixes for POST handling, multipart support, resilient ARC processing, and scalable deployment patterns including Docker and S3 uploads. Comfortable shipping across code, infrastructure and product, he’s known for improving fault tolerance in large crawlers and simplifying complex archival workflows. An amateur accordionist who once toured playing music, he brings a maker’s curiosity and a user-focused ethos to preserving the web as open, reusable history.
14 years of coding experience
12 years of employment as a software developer
Brown University
Github Skills (48)
puppeteer10
javascript10
archiving10
web-crawler10
rewrite10
python10
webscraper10
http10
webscraping10
archives10
request-handler10
java10
crawling10
webar10
data-storage10
Programming languages (20)
Github contributions (5)
webrecorder/browsertrix-crawler
Oct 2020 - Jan 2023
Run a high-fidelity browser-based web archiving crawler in a single Docker container
Role in this project:
Full-stack Developer & DevOps Engineer
Full-stack Developer & DevOps EngineerContributions:88 releases, 239 reviews, 141 commits in 2 years 3 months
Contributions summary:Ilya primarily refactored the crawler and driver components, wrapping crawling functionality in a `Crawler` class and adding extensibility. They implemented features such as support for headless/non-headless operation, custom drivers, and collection names for `pywb`. Furthermore, the user made several infrastructure and build process improvements by switching to a more modern base image, added support for VNC-based profile creation, and implemented code to upload to S3. The user also introduced new command-line flags and improved the internal project logging.
crawlerdocker-containercrawlingwaczwarc
Rhizome-Conifer/conifer
Apr 2015 - Jun 2020
Collect and revisit web pages.
Role in this project:Back-end Developer
Back-end DeveloperContributions:1099 commits, 600 PRs, 1269 pushes in 5 years 3 months
Contributions summary:Ilya implemented features and fixed bugs related to uploading and managing web archive data within the Conifer repository. This involved modifying the `UploadController` to handle content length explicitly, and the `DownloadController` to set the content's type. Additional work included the development and integration of core functionalities, such as adding functionality to a remote browser API and updating the user interface related to these changes. Further contributions included improving the collection data model for better efficiency and stability.
webrecordercollectpythonpywbweb-archiving
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.