won't finish indexing big cpp codebases

[mattepiu]

Last working version: 6 May 2026

After that date, indexing freezes after a certain point. Log below:

level=info msg=parallel.extract.progress done=5960 total=5993
level=info msg=parallel.extract.progress done=5970 total=5993
level=info msg=parallel.extract.progress done=5980 total=5993
level=info msg=parallel.extract.progress done=5990 total=5993
level=info msg=parallel.extract.file.done pos=10 elapsed_ms=34024 defs=10989 path=Plugins/myplugin/1_3_2/Model.hpp
level=info msg=parallel.extract.file.done pos=8 elapsed_ms=34965 defs=11107 path=Plugins/myplugin/1_3_2/Model.hpp
level=info msg=parallel.extract.file.done pos=7 elapsed_ms=37006 defs=11107 path=Plugins/myplugin/1_3_2/Specified.hpp
level=info msg=parallel.extract.file.done pos=9 elapsed_ms=37354 defs=10989 path=Plugins/myplugin/1_3_2/ModelBody.hpp
level=info msg=parallel.extract.progress done=10 total=5993
level=info msg=parallel.extract.file.done pos=6 elapsed_ms=39378 defs=13630 path=Plugins/myplugin/1_3_2/myfile.hpp
level=info msg=parallel.extract.file.done pos=4 elapsed_ms=67671 defs=14390 path=Plugins/myplugin/1_3_2/Specified.hpp
level=info msg=parallel.extract.file.done pos=5 elapsed_ms=72565 defs=14390 path=Plugins/myplugin/1_3_2/Specified.hpp
level=info msg=parallel.extract.file.done pos=0 elapsed_ms=169575 defs=22423 path=Plugins/myplugin/1_3_2/MessageSpecified.hpp
level=info msg=parallel.extract.mem rss_mb=5141 peak_mb=5615 budget_mb=16226 per_worker_mb=811
level=info msg=parallel.extract.done nodes=267331 errors=2
level=info msg=pass.timing pass=parallel_extract elapsed_ms=171020
level=info msg=parallel.registry.start files=5993
level=info msg=parallel.registry.done entries=183470 defines=267331 imports=26
level=info msg=pass.timing pass=registry_build elapsed_ms=1366
level=info msg=pass.start pass=lsp_cross files=5993

[DeusData]

Hey, yes I already identified the issue. Fix will ship with 0.7.1. Sorry for the inconvenience

[EnziinSystem]

I think this type of problem has its root cause in the software architecture.

This tool indexes and builds graphs in RAM without knowing in advance how large the codebase will be, resulting in the program stopping indexing at some point if memory resources reach their limit.

I suggest considering a file binary-based indexing solution instead of RAM.

We don't need fast indexing, we just need fast DB access. As long as it can index fully, without rejecting any codebase. I am willing to wait more than 30 minutes for the indexing to take place.

Consider a file binary-based solution with arbitrary graph depth using FlatBuffer (https://flatbuffers.dev).

Thanks.

[DeusData]

@EnziinSystem nope this was a regression in the LSP pass. The fix will be shipped in the 0.7.1. Currently I am not thinking about switching the architecture as indexing speed is an important metric for me. The next release will also have improved memory management.

[EnziinSystem]

@EnziinSystem nope this was a regression in the LSP pass. The fix will be shipped in the 0.7.1. Currently I am not thinking about switching the architecture as indexing speed is an important metric for me. The next release will also have improved memory management.

I respect your decision, but please note that using FlatBuffer is not slow at all; I've saved and loaded 2D drawings with millions of objects in milliseconds.
Please consider this if you need it in the future.

[DeusData]

Fixed on main (02d2774), verified green across all platforms in CI.

The C/C++ cross-LSP resolver had four overload-lookup functions doing a linear scan over the entire project registry on every call site — O(calls × project_funcs) — which is why large headers (your Model.hpp at ~11k defs / 34s each) made indexing crawl to a halt. They now use the pre-built hash index (O(overloads)), with equivalence verified over 1606 lookups and a scale-tier regression guard (tests/test_cpp_index_hang.sh).

Thanks for the detailed report with per-file timings — that pinpointed the bottleneck exactly. 🙏