Support Yolov5(4.0)/Yolov5(5.0)/YoloR/YoloX/Yolov4/Yolov3/CenterNet/CenterFace/RetinaFace/Classify/Unet. use darknet/libtorch/pytorch/mxnet to onnx to tensorrt

Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready and real time inference.

Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.

Batch LLM Inference with Ray Data LLM: From Simple to Advanced

PipelineScheduler optimizes workload distribution between servers and edge devices, setting optimal batch sizes to maximize throughput and minimize latency amid content dynamics and network instability. It also addresses resource contention with spatiotemporal inference scheduling to reduce co-location interference.

RWKV7 推理服务 | 高并发批量推理，动态扩缩容，兼容OpenAI API，CUDA优化，低显存部署大模型推理 RWKV linear attention batch inference server — auto-queue, dynamic capacity, OpenAI compatible.

Fast self-hosted embedding engine for search, RAG, and reindexing workloads on NVIDIA GPUs. Built in Rust + TensorRT for teams that care about scale, cost, and control.

Ray Saturday Dec 2022 edition

Torchfusion is a very opinionated torch inference on datafusion.

Serve pytorch inference requests using batching with redis for faster performance.

Support batch inference of Grounding DINO. "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection"

High-throughput, multi-actor RL runtime for adaptive system modeling

Self-hosted batch LLM pipeline for analyzing customer feedback from Excel. Upload xlsx, describe the task, configure output fields — get structured results. Works with any OpenAI-compatible API and Ollama.

LightGBM Inference on Datafusion

简单的 Ollama JSONL 批量推理工具 / Simple Ollama JSONL batch inference tool.

This repository provides sample codes, which enable you to learn how to use auto-ml image classification, or object detection under Azure ML(AML) environment.

Neural network classifier with training, evaluation, calibration, and prediction using PyTorch.

🚀 Process JSON data in batches with `llm-batch`, leveraging sequential or parallel modes for efficient interaction with LLMs.

This repo simulates how an ML model moves to production in an industry setting. The goal is to build, deploy, monitor, and retrain a sentiment analysis model using Kubernetes (minikube) and FastAPI.

Batch object-detection pipeline that runs Ultralytics YOLO11 (detect/segment) over an image/video library on Backblaze B2 S3-compatible storage, writing COCO JSON, annotated frames, and per-class instance crops back to B2.