fix: add file size limit to FileModel to prevent multi-GB allocations#12060
Draft
warp-dev-github-integration[bot] wants to merge 1 commit into
Draft
fix: add file size limit to FileModel to prevent multi-GB allocations#12060warp-dev-github-integration[bot] wants to merge 1 commit into
warp-dev-github-integration[bot] wants to merge 1 commit into
Conversation
FileModel::open(), read_content_for_file(), and reload_file_paths() all use async_fs::read_to_string without any size check. When a user opens or the file-watcher detects changes to a very large file (logs, binaries, database dumps, etc.), the entire content is read into memory as a String. Sentry issue 7259255054 shows 24 GB of in-use memory with 99.27% from async_fs::read_to_string via the blocking::Executor thread pool. This change: - Adds a FileTooLarge variant to FileLoadError - Adds a MAX_READABLE_FILE_SIZE constant (256 MB) and check_file_size() helper to FileModel - Guards all three async_fs::read_to_string call sites with a file size check before reading - Updates the exhaustive match in read_binary_file_context to handle the new error variant Sentry: https://sentry.io/organizations/warpdotdev/issues/7259255054/ Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a 256 MB file size limit to
FileModelto prevent multi-gigabyte memory allocations when opening or reloading very large files.Problem
Sentry issue 7259255054 shows 24.18 GB of in-use memory with 99.27% from
async_fs::read_to_stringvia theblocking::Executorthread pool. The heap profile shows the allocation path:FileModel::open(),read_content_for_file(), andreload_file_paths()all useasync_fs::read_to_stringwithout any size check. When a user opens or the file-watcher detects changes to a very large file (logs, binaries, database dumps, etc.), the entire content is read into memory as aString.Changes
warp_util/src/file.rs: AddedFileTooLarge { size, limit }variant toFileLoadErrorwarp_files/src/lib.rs:MAX_READABLE_FILE_SIZEconstant (256 MB)check_file_size()async helperFileModel::open()with size check beforeread_to_stringFileModel::read_content_for_file()with size checkFileModel::reload_file_paths()with size check (logs warning and skips)app/src/ai/blocklist/action_model/execute.rs: AddedFileTooLargearm to exhaustive match inread_binary_file_contextValidation
cargo check --workspacepassescargo clippy -p warp_files -p warp_util -- -D warningspassesCHANGELOG-BUG-FIX: Fixed excessive memory usage when opening very large files by adding a 256 MB file size limit
Conversation: https://staging.warp.dev/conversation/99f355e9-9bd4-4dbf-ae68-c81bc1db01ae
Run: https://oz.staging.warp.dev/runs/019e8793-6d32-723a-bbc1-75d4c4694915
This PR was generated with Oz.