Data deduplication engine, supporting optional compression and public key encryption.
-
Updated
Aug 25, 2022 - Rust
#
data-deduplication
Here are 28 public repositories matching this topic...
Official Repository of "LLM × DATA" Survey Paper
data-transformation data-acquisition data-deduplication data-filtering vlm data-selection data-synthesis data-provenance llm data-mixing
-
Updated
Mar 24, 2026
🚢 Data Toolkit for Sailor Language Models
-
Updated
Feb 24, 2025 - Python
Self-contained C# library for data deduplication using Sqlite
compression storage nuget dedupe sqlite-database data-deduplication chunk compress deduplication chunk-data duplicate-data chunk-key
-
Updated
Apr 7, 2023 - C#
Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.
-
Updated
Sep 21, 2023 - Java
Optimal distributed data deduplication and supervised learning pipeline using Apache Spark
distributed-systems data-science spark hadoop data-deduplication data-engineering data-cleaning deduplication
-
Updated
Aug 19, 2020 - Scala
A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.
-
Updated
Jan 6, 2021 - Java
Sievio turns GitHub, local repos, and web PDFs into clean JSONL for LLM pretraining, fine-tuning, and RAG. It offers structure-aware chunking, reliable Unicode decoding, pluggable QC and safety checks, plus optional dataset cards and deduplication.
python data-deduplication dataset-creation data-pipelines repository-mining jsonl github-repos rag text-preprocessing quality-filtering code-mining llm llm-training llm-datasets
-
Updated
Dec 27, 2025 - Python
PolyDeDupe: Multi-Lingual Data Deduplication
-
Updated
Jun 8, 2026 - Python
A pure-JS, content-addressed, copy-on-write virtual filesystem for the browser, featuring: deduplication, filesystem universes (snapshots), events, and optional asynchronous sync.
browser localstorage pure-javascript data-deduplication indexeddb leightweight content-addressable-storage copy-on-write delta-sync webxdc virtual-filesystem
-
Updated
Jan 8, 2026 - JavaScript
Enterprise-grade SaaS platform for importing, cleaning, and managing large-scale mailing lists with advanced deduplication and enrichment.
-
Updated
Jan 1, 2026 - TypeScript
-
Updated
Jan 7, 2023 - Python
A calculator for storage and transmission of deduplicated data. Output: charts and tables
data-deduplication deduplication deduplication-calculator storage-deduplication-calculator network-deduplication-calculator
-
Updated
Jun 7, 2026
Этот проект представляет собой мощный инструмент для поиска и анализа дублирующихся файлов в указанной директории. Программа позволяет эффективно выявлять одинаковые файлы на основе их содержимого, используя алгоритм хеширования SHA-256. Она поддерживает настройку параметров, таких как минимальный размер файла для проверки и игнорирование определен
python hashing productivity multithreading data-deduplication file-system sha256 file-management system-utility cli-tool dev-tools file-deduplication file-comparison disk-cleanup command-line-utility duplicate-file-finder
-
Updated
Feb 14, 2025 - Python
Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class
-
Updated
Mar 26, 2026 - TypeScript
Entify is an early open-source workspace for entity resolution, record linkage, and data deduplication. It helps teams profile messy datasets, configure Splink-powered matching workflows, and review explainable match clusters.
identity-resolution record-linkage entity-resolution fuzzy-matching data-deduplication master-data-management data-quality duplicate-detection duckdb customer-360 splink semantic-blocking
-
Updated
Jun 11, 2026 - TypeScript
面向 Excel/CSV 的本地数据研判工具箱，支持多表碰撞、跨表检索、智能提取、模糊去重、差异对比和取最新行，并可打包为 Docker 离线部署包。
docker typescript csv nextjs excel xlsx spreadsheet fuzzy-matching data-deduplication agpl data-analysis data-cleaning tailwindcss data-matching data-workbench offline-deployment
-
Updated
Jun 11, 2026 - TypeScript
A web tool that compares two URL lists, identifies unique and matching domains, and creates a detailed report. It streamlines URL data analysis efficiently.
-
Updated
Jan 24, 2026 - JavaScript
ExisEcho API - sample fuzzy logic code & examples
-
Updated
Feb 7, 2026 - Python
Improve this page
Add a description, image, and links to the data-deduplication topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-deduplication topic, visit your repo's landing page and select "manage topics."