v0.5.0

What's Changed

• [CI/Build] CMakeLists: build all extensions' cmake targets at the same time by @dtrifiro in #5034
• [Kernel] Refactor CUTLASS kernels to always take scales that reside on the GPU by @tlrmchlsmth in #5137
• [Kernel] Update Cutlass fp8 configs by @varun-sundar-rabindranath in #5144
• [Minor] Fix the path typo in loader.py: save_sharded_states.py -> save_sharded_state.py by @dashanji in #5151
• [Bugfix] Fix call to init_logger in openai server by @NadavShmayo in #4765
• [Feature][Kernel] Support bitsandbytes quantization and QLoRA by @chenqianfzh in #4776
• [Bugfix] Remove deprecated @abstractproperty by @zhuohan123 in #5174
• [Bugfix]: Fix issues related to prefix caching example (#5177) by @Delviet in #5180
• [BugFix] Prevent LLM.encode for non-generation Models by @robertgshaw2-neuralmagic in #5184
• Update test_ignore_eos by @simon-mo in #4898
• [Frontend][OpenAI] Support for returning max_model_len on /v1/models response by @Avinash-Raj in #4643
• [Kernel][ROCm][AMD] enable fused topk_softmax kernel for moe layer by @divakar-amd in #4927
• [Misc] Simplify code and fix type annotations in conftest.py by @DarkLight1337 in #5118
• [Core] Support image processor by @DarkLight1337 in #4197
• [Core] Remove unnecessary copies in flash attn backend by @Yard1 in #5138
• [Kernel] Pass a device pointer into the quantize kernel for the scales by @tlrmchlsmth in #5159
• [CI/BUILD] enable intel queue for longer CPU tests by @zhouyuan in #4113
• [Misc]: Implement CPU/GPU swapping in BlockManagerV2 by @Kaiyang-Chen in #3834
• New CI template on AWS stack by @khluu in #5110
• [FRONTEND] OpenAI tools support named functions by @br3no in #5032
• [Bugfix] Support prompt_logprobs==0 by @toslunar in #5217
• [Bugfix] Add warmup for prefix caching example by @zhuohan123 in #5235
• [Kernel] Enhance MoE benchmarking & tuning script by @WoosukKwon in #4921
• [Bugfix]: During testing, use pytest monkeypatch for safely overriding the env var that indicates the vLLM backend by @afeldman-nm in #5210
• [Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor by @zifeitong in #5229
• [CI/Build] Add inputs tests by @DarkLight1337 in #5215
• [Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend by @DamonFool in #5249
• [Kernel] Add back batch size 1536 and 3072 to MoE tuning by @WoosukKwon in #5242
• [CI/Build] Simplify model loading for HfRunner by @DarkLight1337 in #5251
• [CI/Build] Reducing CPU CI execution time by @bigPYJ1151 in #5241
• [CI] mark AMD test as softfail to prevent blockage by @simon-mo in #5256
• [Misc] Add transformers version to collect_env.py by @mgoin in #5259
• [Misc] update collect env by @youkaichao in #5261
• [Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True by @zifeitong in #5226
• [Misc] Add CustomOp interface for device portability by @WoosukKwon in #5255
• [Misc] Fix docstring of get_attn_backend by @WoosukKwon in #5271
• [Frontend] OpenAI API server: Add add_special_tokens to ChatCompletionRequest (default False) by @tomeras91 in #5278
• [CI] Add nightly benchmarks by @simon-mo in #5260
• [misc] benchmark_serving.py -- add ITL results and tweak TPOT results by @tlrmchlsmth in #5263
• [Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size by @tlrmchlsmth in #5157
• [Model] Correct Mixtral FP8 checkpoint loading by @comaniac in #5231
• [BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM by @DriverSong in #5207
• [Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 by @pcmoritz in #5238
• [Docs] Add Sequoia as sponsors by @simon-mo in #5287
• [Speculative Decoding] Add ProposerWorkerBase abstract class by @njhill in #5252
• [BugFix] Fix log message about default max model length by @njhill in #5284
• [Bugfix] Make EngineArgs use named arguments for config construction by @mgoin in #5285
• [Bugfix][Frontend/Core] Don't log exception when AsyncLLMEngine gracefully shuts down. by @wuisawesome in #5290
• [Misc] Skip for logits_scale == 1.0 by @WoosukKwon in #5291
• [Docs] Add Ray Summit CFP by @simon-mo in #5295
• [CI] Disable flash_attn backend for spec decode by @simon-mo in #5286
• [Frontend][Core] Update Outlines Integration from FSM to Guide by @br3no in #4109
• [CI/Build] Update vision tests by @DarkLight1337 in #5307
• Bugfix: fix broken of download models from modelscope by @liuyhwangyh in #5233
• [Kernel] Retune Mixtral 8x22b configs for FP8 on H100 by @pcmoritz in #5294
• [Frontend] enable passing multiple LoRA adapters at once to generate() by @mgoldey in #5300
• [Core] Avoid copying prompt/output tokens if no penalties are used by @Yard1 in #5289
• [Core] Change LoRA embedding sharding to support loading methods by @Yard1 in #5038
• [Misc] Missing error message for custom ops import by @DamonFool in #5282
• [Feature][Frontend]: Add support for stream_options in ChatCompletionRequest by @Etelis in #5135
• [Misc][Utils] allow get_open_port to be called for multiple times by @youkaichao in #5333
• [Kernel] Switch fp8 layers to use the CUTLASS kernels by @tlrmchlsmth in #5183
• Remove Ray health check by @Yard1 in #4693
• Addition of lacked ignored_seq_groups in _schedule_chunked_prefill by @JamesLim-sy in #5296
• [Kernel] Dynamic Per-Token Activation Quantization by @dsikka in #5037
• [Frontend] Add OpenAI Vision API Support by @ywang96 in #5237
• [Misc] Remove unused cuda_utils.h in CPU backend by @DamonFool in #5345
• fix DbrxFusedNormAttention missing cache_config by @Calvinnncy97 in #5340
• [Bug Fix] Fix the support check for FP8 CUTLASS by @cli99 in #5352
• [Misc] Add args for selecting distributed executor to benchmarks by @BKitor in #5335
• [ROCm][AMD] Use pytorch sdpa math backend to do naive attention by @hongxiayang in #4965
• [CI/Test] improve robustness of test by replacing del with context manager (hf_runner) by @youkaichao in #5347
• [CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) by @youkaichao in #5357
• [Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale by @mgoin in #5353
• [Core][CUDA Graph] add output buffer for cudagraph to reduce memory footprint by @youkaichao in #5074
• [mis][ci/test] fix flaky test in tests/test_sharded_state_loader.py by @youkaichao in #5361
• [Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops by @bnellnm in #5047
• [Bugfix] Fix KeyError: 1 When Using LoRA adapters by @BlackBird-Coding in #5164
• [Misc] Update to comply with the new compressed-tensors config by @dsikka in #5350
• [Frontend][Misc] Enforce Pixel Values as Input Type for VLMs in API Server by @ywang96 in #5374
• [misc][typo] fix typo ...

v0.4.3

Highlights

Model Support

LLM

• Added support for Falcon (#5069)
• Added support for IBM Granite Code models (#4636)
• Added blocksparse flash attention kernel and Phi-3-Small model (#4799)
• Added Snowflake arctic model implementation (#4652, #4889, #4690)
• Supported Dynamic RoPE scaling (#4638)
• Supported for long context lora (#4787)

Embedding Models

• Intial support for Embedding API with e5-mistral-7b-instruct (#3734)
• Cross-attention KV caching and memory-management towards encoder-decoder model support (#4837)

Vision Language Model

• Add base class for vision-language models (#4809)
• Consolidate prompt arguments to LLM engines (#4328)
• LLaVA model refactor (#4910)

Hardware Support

AMD

• Add fused_moe Triton configs (#4951)
• Add support for Punica kernels (#3140)
• Extending the set of AMD tests with Regression, Basic Correctness, Distributed, Engine, Llava Tests (#4797)

Production Engine

Batch API

• Support OpenAI batch file format (#4794)

Making Ray Optional

• Add MultiprocessingGPUExecutor (#4539)
• Eliminate parallel worker per-step task scheduling overhead (#4894)

Automatic Prefix Caching

• Accelerating the hashing function by avoiding deep copies (#4696)

Speculative Decoding

• CUDA graph support (#4295)
• Enable TP>1 speculative decoding (#4840)
• Improve n-gram efficiency (#4724)

Performance Optimization

Quantization

• Add GPTQ Marlin 2:4 sparse structured support (#4790)
• Initial Activation Quantization Support (#4525)
• Marlin prefill performance improvement (about better on average) (#4983)
• Automatically Detect SparseML models (#5119)

Better Attention Kernel

• Use flash-attn for decoding (#3648)

FP8

• Improve FP8 linear layer performance (#4691)
• Add w8a8 CUTLASS kernels (#4749)
• Support for CUTLASS kernels in CUDA graphs (#4954)
• Load FP8 kv-cache scaling factors from checkpoints (#4893)
• Make static FP8 scaling more robust (#4570)
• Refactor FP8 kv-cache with NVIDIA float8_e4m3 support (#4535)

Optimize Distributed Communication

• change python dict to pytorch tensor (#4607)
• change python dict to pytorch tensor for blocks to swap (#4659)
• improve paccess check (#4992)
• remove vllm-nccl (#5091)
• support both cpu and device tensor in broadcast tensor dict (#4660)

Extensible Architecture

Pipeline Parallelism

• refactor custom allreduce to support multiple tp groups (#4754)
• refactor pynccl to hold multiple communicators (#4591)
• Support PP PyNCCL Groups (#4988)

What's Changed

• Disable cuda version check in vllm-openai image by @zhaoyang-star in #4530
• [Bugfix] Fix asyncio.Task not being subscriptable by @DarkLight1337 in #4623
• [CI] use ccache actions properly in release workflow by @simon-mo in #4629
• [CI] Add retry for agent lost by @cadedaniel in #4633
• Update lm-format-enforcer to 0.10.1 by @noamgat in #4631
• [Kernel] Make static FP8 scaling more robust by @pcmoritz in #4570
• [Core][Optimization] change python dict to pytorch tensor by @youkaichao in #4607
• [Build/CI] Fixing 'docker run' to re-enable AMD CI tests. by @Alexei-V-Ivanov-AMD in #4642
• [Bugfix] Fixed error in slice_lora_b for MergedQKVParallelLinearWithLora by @FurtherAI in #4609
• [Core][Optimization] change copy-on-write from dict[int, list] to list by @youkaichao in #4648
• [Bug fix][Core] fixup ngram not setup correctly by @leiwen83 in #4551
• [Core][Distributed] support both cpu and device tensor in broadcast tensor dict by @youkaichao in #4660
• [Core] Optimize sampler get_logprobs by @rkooo567 in #4594
• [CI] Make mistral tests pass by @rkooo567 in #4596
• [Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi by @DefTruth in #4573
• [Misc] Add get_name method to attention backends by @WoosukKwon in #4685
• [Core] Faster startup for LoRA enabled models by @Yard1 in #4634
• [Core][Optimization] change python dict to pytorch tensor for blocks to swap by @youkaichao in #4659
• [CI/Test] fix swap test for multi gpu by @youkaichao in #4689
• [Misc] Use vllm-flash-attn instead of flash-attn by @WoosukKwon in #4686
• [Dynamic Spec Decoding] Auto-disable by the running queue size by @comaniac in #4592
• [Speculative decoding] [Bugfix] Fix overallocation in ngram + spec logprobs by @cadedaniel in #4672
• [Bugfix] Fine-tune gptq_marlin configs to be more similar to marlin by @alexm-neuralmagic in #4626
• [Frontend] add tok/s speed metric to llm class when using tqdm by @MahmoudAshraf97 in #4400
• [Frontend] Move async logic outside of constructor by @DarkLight1337 in #4674
• [Misc] Remove unnecessary ModelRunner imports by @WoosukKwon in #4703
• [Misc] Set block size at initialization & Fix test_model_runner by @WoosukKwon in #4705
• [ROCm] Add support for Punica kernels on AMD GPUs by @kliuae in #3140
• [Bugfix] Fix CLI arguments in OpenAI server docs by @DarkLight1337 in #4709
• [Bugfix] Update grafana.json by @robertgshaw2-neuralmagic in #4711
• [Bugfix] Add logs for all model dtype casting by @mgoin in #4717
• [Model] Snowflake arctic model implementation by @sfc-gh-hazhang in #4652
• [Kernel] [FP8] Improve FP8 linear layer performance by @pcmoritz in #4691
• [Kernel] Refactor FP8 kv-cache with NVIDIA float8_e4m3 support by @comaniac in #4535
• [Core][Distributed] refactor pynccl to hold multiple communicators by @youkaichao in #4591
• [Misc] Keep only one implementation of the create_dummy_prompt function. by @AllenDou in #4716
• chunked-prefill-doc-syntax by @simon-mo in #4603
• [Core]fix type annotation for swap_blocks by @jikunshang in #4726
• [Misc] Apply a couple g++ cleanups by @stevegrubb in #4719
• [Core] Fix circular reference which leaked llm instance in local dev env by @rkooo567 in #4737
• [Bugfix] Fix CLI arguments in OpenAI server docs by @AllenDou in #4729
• [Speculative decoding] CUDA graph support by @heeju-kim2 in #4295
• [CI] Nits for bad initialization of SeqGroup in testing by @robertgshaw2-neuralmagic in #4748
• [Core][Test] fix function name typo in custom allreduce by @youkaichao in #4750
• [Model][Misc] Add e5-mistral-7b-instruct and Embedding API by @CatherineSue in #3734
• [Model] Add support for IBM Granite Code models by @yikangshen in #4636
• [CI/Build] Tweak Marlin Nondeterminism Issues In CI by @robertgshaw2-neuralmagic in #4713
• [CORE] Improvement in ranks code by @SwapnilDreams100 in #4718
• [Core][Distributed] refactor custom allreduce to support multiple tp groups by @youkaichao in #4754
• [CI/Build] Move test_utils.py to tests/utils.py by @DarkLight1337 in #4425
• [Scheduler] Warning upon preemption and Swapping by @rkooo567 in #4647
• [Misc] Enhance attention selector by @WoosukKwon in #4751
• [Frontend] [Core] perf: Automatically detect vLLM-tensorized model, update tensorizer to version 2.9.0 by @sangstar in #4208
• [Speculative decoding] Improve n-gram efficiency by @comaniac in #4724
• [Kernel] Use flash-attn for decoding by @skrider in #3648
• [Bugfix] Fix dynamic FP8 quantization for Mixtral by @pcmoritz in #4793
• [Doc] Shorten README by removing supported model list by @zhuohan123 in #4796
• [Doc] Add API reference for offline inference by @DarkLight1337 in #4710
• [Doc] Add meetups to the doc by @zhuohan123 in #4798
• [Core][Hash][Automatic Prefix caching] Accelerating the hashing function by avoiding deep copies by @KuntaiDu in #4696
• [Bugfix][Doc] Fix CI failure in...

v0.4.2

Highlights

Features

• Chunked prefill is ready for testing! It improves inter-token latency in high load scenario by chunking the prompt processing and priortizes decode (#4580)
• Speculative decoding functionalities: logprobs (#4378), ngram (#4237)
• Support FlashInfer as attention backend (#4353)

Models and Enhancements

• Add support for Phi-3-mini (#4298, #4372, #4380)
• Add more histogram metrics (#2764, #4523)
• Full tensor parallelism for LoRA layers (#3524)
• Expanding Marlin kernel to support all GPTQ models (#3922, #4466, #4533)

Dependency Upgrade

• Upgrade to torch==2.3.0 (#4454)
• Upgrade to tensorizer==2.9.0 (#4467)
• Expansion of AMD test suite (#4267)

Progress and Dev Experience

• Centralize and document all environment variables (#4548, #4574)
• Progress towards fully typed codebase (#4337, #4427, #4555, #4450)
• Progress towards pipeline parallelism (#4512, #4444, #4566)
• Progress towards multiprocessing based executors (#4348, #4402, #4419)
• Progress towards FP8 support (#4343, #4332, 4527)

What's Changed

• [Core][Distributed] use existing torch.cuda.device context manager by @youkaichao in #4318
• [Misc] Update ShareGPT Dataset Sampling in Serving Benchmark by @ywang96 in #4279
• [Bugfix] Fix marlin kernel crash on H100 by @alexm-nm in #4218
• [Doc] Add note for docker user by @youkaichao in #4340
• [Misc] Use public API in benchmark_throughput by @zifeitong in #4300
• [Model] Adds Phi-3 support by @caiom in #4298
• [Core] Move ray_utils.py from engine to executor package by @njhill in #4347
• [Bugfix][Model] Refactor OLMo model to support new HF format in transformers 4.40.0 by @Isotr0py in #4324
• [CI/Build] Adding functionality to reset the node's GPUs before processing. by @Alexei-V-Ivanov-AMD in #4213
• [Doc] README Phi-3 name fix. by @caiom in #4372
• [Core]refactor aqlm quant ops by @jikunshang in #4351
• [Mypy] Typing lora folder by @rkooo567 in #4337
• [Misc] Optimize flash attention backend log by @esmeetu in #4368
• [Core] Add shutdown() method to ExecutorBase by @njhill in #4349
• [Core] Move function tracing setup to util function by @njhill in #4352
• [ROCm][Hardware][AMD][Doc] Documentation update for ROCm by @hongxiayang in #4376
• [Bugfix] Fix parameter name in get_tokenizer by @DarkLight1337 in #4107
• [Frontend] Add --log-level option to api server by @normster in #4377
• [CI] Disable non-lazy string operation on logging by @rkooo567 in #4326
• [Core] Refactoring sampler and support prompt logprob for chunked prefill by @rkooo567 in #4309
• [Misc][Refactor] Generalize linear_method to be quant_method by @comaniac in #4373
• [Misc] add RFC issue template by @youkaichao in #4401
• [Core] Introduce DistributedGPUExecutor abstract class by @njhill in #4348
• [Kernel] Optimize FP8 support for MoE kernel / Mixtral via static scales by @pcmoritz in #4343
• [Frontend][Bugfix] Disallow extra fields in OpenAI API by @DarkLight1337 in #4355
• [Misc] Fix logger format typo by @esmeetu in #4396
• [ROCm][Hardware][AMD] Enable group query attention for triton FA by @hongxiayang in #4406
• [Kernel] Full Tensor Parallelism for LoRA Layers by @FurtherAI in #3524
• [Model] Phi-3 4k sliding window temp. fix by @caiom in #4380
• [Bugfix][Core] Fix get decoding config from ray by @esmeetu in #4335
• [Bugfix] Abort requests when the connection to /v1/completions is interrupted by @chestnut-Q in #4363
• [BugFix] Fix min_tokens when eos_token_id is None by @njhill in #4389
• ✨ support local cache for models by @prashantgupta24 in #4374
• [BugFix] Fix return type of executor execute_model methods by @njhill in #4402
• [BugFix] Resolved Issues For LinearMethod --> QuantConfig by @robertgshaw2-neuralmagic in #4418
• [Misc] fix typo in llm_engine init logging by @DefTruth in #4428
• Add more Prometheus metrics by @ronensc in #2764
• [CI] clean docker cache for neuron by @simon-mo in #4441
• [mypy][5/N] Support all typing on model executor by @rkooo567 in #4427
• [Kernel] Marlin Expansion: Support AutoGPTQ Models with Marlin by @robertgshaw2-neuralmagic in #3922
• [CI] hotfix: soft fail neuron test by @simon-mo in #4458
• [Core][Distributed] use cpu group to broadcast metadata in cpu by @youkaichao in #4444
• [Misc] Upgrade to torch==2.3.0 by @mgoin in #4454
• [Bugfix][Kernel] Fix compute_type for MoE kernel by @WoosukKwon in #4463
• [Core]Refactor gptq_marlin ops by @jikunshang in #4466
• [BugFix] fix num_lookahead_slots missing in async executor by @leiwen83 in #4165
• [Doc] add visualization for multi-stage dockerfile by @prashantgupta24 in #4456
• [Kernel] Support Fp8 Checkpoints (Dynamic + Static) by @robertgshaw2-neuralmagic in #4332
• [Frontend] Support complex message content for chat completions endpoint by @fgreinacher in #3467
• [Frontend] [Core] Tensorizer: support dynamic num_readers, update version by @alpayariyak in #4467
• [Bugfix][Minor] Make ignore_eos effective by @bigPYJ1151 in #4468
• fix_tokenizer_snapshot_download_bug by @kingljl in #4493
• Unable to find Punica extension issue during source code installation by @kingljl in #4494
• [Core] Centralize GPU Worker construction by @njhill in #4419
• [Misc][Typo] type annotation fix by @HarryWu99 in #4495
• [Misc] fix typo in block manager by @Juelianqvq in #4453
• Allow user to define whitespace pattern for outlines by @robcaulk in #4305
• [Misc]Add customized information for models by @jeejeelee in #4132
• [Test] Add ignore_eos test by @rkooo567 in #4519
• [Bugfix] Fix the fp8 kv_cache check error that occurs when failing to obtain the CUDA version. by @AnyISalIn in #4173
• [Bugfix] Fix 307 Redirect for /metrics by @robertgshaw2-neuralmagic in #4523
• [Doc] update(example model): for OpenAI compatible serving by @fpaupier in #4503
• [Bugfix] Use random seed if seed is -1 by @sasha0552 in #4531
• [CI/Build][Bugfix] VLLM_USE_PRECOMPILED should skip compilation by @tjohnson31415 in #4534
• [Speculative decoding] Add ngram prompt lookup decoding by @leiwen83 in #4237
• [Core] Enable prefix caching with block manager v2 enabled by @leiwen83 in #4142
• [Core] Add multiproc_worker_utils for multiprocessing-based workers by @njhill in #4357
• [Kernel] Update fused_moe tuning script for FP8 by @pcmoritz in #4457
• [Bugfix] Add validation for seed by @sasha0552 in #4529
• [Bugfix][Core] Fix and refactor logging stats by @esmeetu in #4336
• [Core][Distributed] fix pynccl del error by @youkaichao in #4508
• [Misc] Remove Mixtral device="cuda" declarations by @pcmoritz in #4543
• [Misc] Fix expert_ids shape in MoE by @WoosukKwon in #4517
• [MISC] Rework logger to enable pythonic custom logging configuration to be provided by @tdg5 in #4273
• [Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens i...

v0.4.1

Highlights

Features

• Support and enhance CommandR+ (#3829), minicpm (#3893), Meta Llama 3 (#4175, #4182), Mixtral 8x22b (#4073, #4002)
• Support private model registration, and updating our support policy (#3871, 3948)
• Support PyTorch 2.2.1 and Triton 2.2.0 (#4061, #4079, #3805, #3904, #4271)
• Add option for using LM Format Enforcer for guided decoding (#3868)
• Add option for optionally initialize tokenizer and detokenizer (#3748)
• Add option for load model using tensorizer (#3476)

Enhancements

• vLLM is now mostly type checked by mypy (#3816, #4006, #4161, #4043)
• Progress towards chunked prefill scheduler (#3550, #3853, #4280, #3884)
• Progress towards speculative decoding (#3250, #3706, #3894)
• Initial support with dynamic per-tensor scaling via FP8 (#4118)

Hardwares

• Intel CPU inference backend is added (#3993, #3634)
• AMD backend is enhanced with Triton kernel and e4m3fn KV cache (#3643, #3290)

What's Changed

• [Kernel] Layernorm performance optimization by @mawong-amd in #3662
• [Doc] Update installation doc for build from source and explain the dependency on torch/cuda version by @youkaichao in #3746
• [CI/Build] Make Marlin Tests Green by @robertgshaw2-neuralmagic in #3753
• [Misc] Minor fixes in requirements.txt by @WoosukKwon in #3769
• [Misc] Some minor simplifications to detokenization logic by @njhill in #3670
• [Misc] Fix Benchmark TTFT Calculation for Chat Completions by @ywang96 in #3768
• [Speculative decoding 4/9] Lookahead scheduling for speculative decoding by @cadedaniel in #3250
• [Misc] Add support for new autogptq checkpoint_format by @Qubitium in #3689
• [Misc] [CI/Build] Speed up block manager CPU-only unit tests ~10x by opting-out of GPU cleanup by @cadedaniel in #3783
• [Hardware][Intel] Add CPU inference backend by @bigPYJ1151 in #3634
• [HotFix] [CI/Build] Minor fix for CPU backend CI by @bigPYJ1151 in #3787
• [Frontend][Bugfix] allow using the default middleware with a root path by @A-Mahla in #3788
• [Doc] Fix vLLMEngine Doc Page by @ywang96 in #3791
• [CI/Build] fix TORCH_CUDA_ARCH_LIST in wheel build by @youkaichao in #3801
• Fix crash when try torch.cuda.set_device in worker by @leiwen83 in #3770
• [Bugfix] Add __init__.py files for vllm/core/block/ and vllm/spec_decode/ by @mgoin in #3798
• [CI/Build] 0.4.0.post1, fix sm 7.0/7.5 binary by @youkaichao in #3803
• [Speculative decoding] Adding configuration object for speculative decoding by @cadedaniel in #3706
• [BugFix] Use different mechanism to get vllm version in is_cpu() by @njhill in #3804
• [Doc] Update README.md by @robertgshaw2-neuralmagic in #3806
• [Doc] Update contribution guidelines for better onboarding by @michaelfeil in #3819
• [3/N] Refactor scheduler for chunked prefill scheduling by @rkooo567 in #3550
• Enable scaled FP8 (e4m3fn) KV cache on ROCm (AMD GPU) by @AdrianAbeyta in #3290
• [Misc] Publish 3rd meetup slides by @WoosukKwon in #3835
• Fixes the argument for local_tokenizer_group by @sighingnow in #3754
• [Core] Enable hf_transfer by default if available by @michaelfeil in #3817
• [Bugfix] Add kv_scale input parameter to CPU backend by @WoosukKwon in #3840
• [Core] [Frontend] Make detokenization optional by @mgerstgrasser in #3749
• [Bugfix] Fix args in benchmark_serving by @CatherineSue in #3836
• [Benchmark] Refactor sample_requests in benchmark_throughput by @gty111 in #3613
• [Core] manage nccl via a pypi package & upgrade to pt 2.2.1 by @youkaichao in #3805
• [Hardware][CPU] Update cpu torch to match default of 2.2.1 by @mgoin in #3854
• [Model] Cohere CommandR+ by @saurabhdash2512 in #3829
• [Core] improve robustness of pynccl by @youkaichao in #3860
• [Doc]Add asynchronous engine arguments to documentation. by @SeanGallen in #3810
• [CI/Build] fix pip cache with vllm_nccl & refactor dockerfile to build wheels by @youkaichao in #3859
• [Misc] Add pytest marker to opt-out of global test cleanup by @cadedaniel in #3863
• [Misc] Fix linter issues in examples/fp8/quantizer/quantize.py by @cadedaniel in #3864
• [Bugfix] Fixing requirements.txt by @noamgat in #3865
• [Misc] Define common requirements by @WoosukKwon in #3841
• Add option to completion API to truncate prompt tokens by @tdoublep in #3144
• [Chunked Prefill][4/n] Chunked prefill scheduler. by @rkooo567 in #3853
• [Bugfix] Fix incorrect output on OLMo models in Tensor Parallelism by @Isotr0py in #3869
• [CI/Benchmark] add more iteration and use multiple percentiles for robust latency benchmark by @youkaichao in #3889
• [Core] enable out-of-tree model register by @youkaichao in #3871
• [WIP][Core] latency optimization by @youkaichao in #3890
• [Bugfix] Fix Llava inference with Tensor Parallelism. by @Isotr0py in #3883
• [Model] add minicpm by @SUDA-HLT-ywfang in #3893
• [Bugfix] Added Command-R GPTQ support by @egortolmachev in #3849
• [Bugfix] Enable Proper attention_bias Usage in Llama Model Configuration by @Ki6an in #3767
• [Hotfix][CI/Build][Kernel] CUDA 11.8 does not support layernorm optimizations by @mawong-amd in #3782
• [BugFix][Model] Fix commandr RoPE max_position_embeddings by @esmeetu in #3919
• [Core] separate distributed_init from worker by @youkaichao in #3904
• [Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable hardware-agnostic speculative decoding" by @cadedaniel in #3837
• [Bugfix] Fix KeyError on loading GPT-NeoX by @jsato8094 in #3925
• [ROCm][Hardware][AMD] Use Triton Kernel for default FA on ROCm by @jpvillam-amd in #3643
• [Misc] Avoid loading incorrect LoRA config by @jeejeelee in #3777
• [Benchmark] Add cpu options to bench scripts by @PZD-CHINA in #3915
• [Bugfix] fix utils.py/merge_dict func TypeError: 'type' object is not subscriptable by @zhaotyer in #3955
• [Bugfix] Fix logits processor when prompt_logprobs is not None by @huyiwen in #3899
• [Bugfix] handle prompt_logprobs in _apply_min_tokens_penalty by @tjohnson31415 in #3876
• [Bugfix][ROCm] Add numba to Dockerfile.rocm by @WoosukKwon in #3962
• [Model][AMD] ROCm support for 256 head dims for Gemma by @jamestwhedbee in #3972
• [Doc] Add doc to state our model support policy by @youkaichao in #3948
• [Bugfix] Remove key sorting for guided_json parameter in OpenAi compatible Server by @dmarasco in #3945
• [Doc] Fix getting stared to use publicly available model by @fpaupier in #3963
• [Bugfix] handle hf_config with architectures == None by @tjohnson31415 in #3982
• [WIP][Core][Refactor] move vllm/model_executor/parallel_utils into vllm/distributed and vllm/device_communicators by @youkaichao in #3950
• [Core][5/N] Fully working chunked prefill e2e by @rkooo567 in #3884
• [Core][Model] Use torch.compile to accelerate layernorm in commandr by @youkaichao in #3985
• [Test] Add xformer and flash attn tests by @rkooo567 in #3961
• [Misc] refactor ops and cache_ops layer by @jikunshang in #3913
• [Doc][Installation] delete python setup.py develop by @youkaichao in #3989
• [Ke...

v0.4.0.post1, restore sm70/75 support

Highlight

v0.4.0 lacks support for sm70/75 support. We did a hotfix for it.

What's Changed

• [Kernel] Layernorm performance optimization by @mawong-amd in #3662
• [Doc] Update installation doc for build from source and explain the dependency on torch/cuda version by @youkaichao in #3746
• [CI/Build] Make Marlin Tests Green by @robertgshaw2-neuralmagic in #3753
• [Misc] Minor fixes in requirements.txt by @WoosukKwon in #3769
• [Misc] Some minor simplifications to detokenization logic by @njhill in #3670
• [Misc] Fix Benchmark TTFT Calculation for Chat Completions by @ywang96 in #3768
• [Speculative decoding 4/9] Lookahead scheduling for speculative decoding by @cadedaniel in #3250
• [Misc] Add support for new autogptq checkpoint_format by @Qubitium in #3689
• [Misc] [CI/Build] Speed up block manager CPU-only unit tests ~10x by opting-out of GPU cleanup by @cadedaniel in #3783
• [Hardware][Intel] Add CPU inference backend by @bigPYJ1151 in #3634
• [HotFix] [CI/Build] Minor fix for CPU backend CI by @bigPYJ1151 in #3787
• [Frontend][Bugfix] allow using the default middleware with a root path by @A-Mahla in #3788
• [Doc] Fix vLLMEngine Doc Page by @ywang96 in #3791
• [CI/Build] fix TORCH_CUDA_ARCH_LIST in wheel build by @youkaichao in #3801
• Fix crash when try torch.cuda.set_device in worker by @leiwen83 in #3770
• [Bugfix] Add __init__.py files for vllm/core/block/ and vllm/spec_decode/ by @mgoin in #3798
• [CI/Build] 0.4.0.post1, fix sm 7.0/7.5 binary by @youkaichao in #3803

New Contributors

• @mawong-amd made their first contribution in #3662
• @Qubitium made their first contribution in #3689
• @bigPYJ1151 made their first contribution in #3634
• @A-Mahla made their first contribution in #3788

Full Changelog: v0.4.0...v0.4.0.post1

v0.4.0

Major changes

Models

• New models: Command+R(#3433), Qwen2 MoE(#3346), DBRX(#3660), XVerse (#3610), Jais (#3183).
• New vision language model: LLaVA (#3042)

Production features

• Automatic prefix caching (#2762, #3703) supporting long system prompt to be automatically cached across requests. Use the flag --enable-prefix-caching to turn it on.
• Support json_object in OpenAI server for arbitrary JSON, --use-delay flag to improve time to first token across many requests, and min_tokens to EOS suppression.
• Progress in chunked prefill scheduler (#3236, #3538), and speculative decoding (#3103).
• Custom all reduce kernel has been re-enabled after more robustness fixes.
• Replaced cupy dependency due to its bugs.

Hardware

• Improved Neuron support for AWS Inferentia.
• CMake based build system for extensibility.

Ecosystem

• Extensive serving benchmark refactoring (#3277)
• Usage statistics collection (#2852)

What's Changed

• allow user chose log level by --log-level instead of fixed 'info'. by @AllenDou in #3109
• Reorder kv dtype check to avoid nvcc not found error on AMD platform by @cloudhan in #3104
• Add Automatic Prefix Caching by @SageMoore in #2762
• Add vLLM version info to logs and openai API server by @jasonacox in #3161
• [FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark by @zhuohan123 in #3158
• Make it easy to profile workers with nsight by @pcmoritz in #3162
• [DOC] add setup document to support neuron backend by @liangfu in #2777
• [Minor Fix] Remove unused code in benchmark_prefix_caching.py by @gty111 in #3171
• Add document for vllm paged attention kernel. by @pian13131 in #2978
• enable --gpu-memory-utilization in benchmark_throughput.py by @AllenDou in #3175
• [Minor fix] The domain dns.google may cause a socket.gaierror exception by @ttbachyinsda in #3176
• Push logprob generation to LLMEngine by @Yard1 in #3065
• Add health check, make async Engine more robust by @Yard1 in #3015
• Fix the openai benchmarking requests to work with latest OpenAI apis by @wangchen615 in #2992
• [ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs by @hongxiayang in #3123
• Store eos_token_id in Sequence for easy access by @njhill in #3166
• [Fix] Avoid pickling entire LLMEngine for Ray workers by @njhill in #3207
• [Tests] Add block manager and scheduler tests by @rkooo567 in #3108
• [Testing] Fix core tests by @cadedaniel in #3224
• A simple addition of dynamic_ncols=True by @chujiezheng in #3242
• Add GPTQ support for Gemma by @TechxGenus in #3200
• Update requirements-dev.txt to include package for benchmarking scripts. by @wangchen615 in #3181
• Separate attention backends by @WoosukKwon in #3005
• Measure model memory usage by @mgoin in #3120
• Possible fix for conflict between Automated Prefix Caching (#2762) and multi-LoRA support (#1804) by @jacobthebanana in #3263
• Fix auto prefix bug by @ElizaWszola in #3239
• Connect engine healthcheck to openai server by @njhill in #3260
• Feature add lora support for Qwen2 by @whyiug in #3177
• [Minor Fix] Fix comments in benchmark_serving by @gty111 in #3252
• [Docs] Fix Unmocked Imports by @ywang96 in #3275
• [FIX] Make flash_attn optional by @WoosukKwon in #3269
• Move model filelocks from /tmp/ to ~/.cache/vllm/locks/ dir by @mgoin in #3241
• [FIX] Fix prefix test error on main by @zhuohan123 in #3286
• [Speculative decoding 3/9] Worker which speculates, scores, and applies rejection sampling by @cadedaniel in #3103
• Enhance lora tests with more layer and rank variations by @tterrysun in #3243
• [ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA by @dllehr-amd in #3262
• [BugFix] Fix get tokenizer when using ray by @esmeetu in #3301
• [Fix] Fix best_of behavior when n=1 by @njhill in #3298
• Re-enable the 80 char line width limit by @zhuohan123 in #3305
• [docs] Add LoRA support information for models by @pcmoritz in #3299
• Add distributed model executor abstraction by @zhuohan123 in #3191
• [ROCm] Fix warp and lane calculation in blockReduceSum by @kliuae in #3321
• Support Mistral Model Inference with transformers-neuronx by @DAIZHENWEI in #3153
• docs: Add BentoML deployment doc by @Sherlock113 in #3336
• Fixes #1556 double free by @br3no in #3347
• Add kernel for GeGLU with approximate GELU by @WoosukKwon in #3337
• [Fix] fix quantization arg when using marlin by @DreamTeamWangbowen in #3319
• add hf_transfer to requirements.txt by @RonanKMcGovern in #3031
• fix bias in if, ambiguous by @hliuca in #3259
• [Minor Fix] Use cupy-cuda11x in CUDA 11.8 build by @chenxu2048 in #3256
• Add missing kernel for CodeLlama-34B on A/H100 (no tensor parallelism) when using Multi-LoRA. by @orsharir in #3350
• Add batched RoPE kernel by @tterrysun in #3095
• Fix lint by @Yard1 in #3388
• [FIX] Simpler fix for async engine running on ray by @zhuohan123 in #3371
• [Hotfix] [Debug] test_openai_server.py::test_guided_regex_completion by @simon-mo in #3383
• allow user to chose which vllm's merics to display in grafana by @AllenDou in #3393
• [Kernel] change benchmark script so that result can be directly used; tune moe kernel in A100/H100 with tp=2,4,8 by @youkaichao in #3389
• Install flash_attn in Docker image by @tdoublep in #3396
• Add args for mTLS support by @declark1 in #3410
• [issue templates] add some issue templates by @youkaichao in #3412
• Fix assertion failure in Qwen 1.5 with prefix caching enabled by @chenxu2048 in #3373
• fix marlin config repr by @qeternity in #3414
• Feature: dynamic shared mem moe_align_block_size_kernel by @akhoroshev in #3376
• [Misc] add HOST_IP env var by @youkaichao in #3419
• Add chat templates for Falcon by @Dinghow in #3420
• Add chat templates for ChatGLM by @Dinghow in #3418
• Fix dist.broadcast stall without group argument by @GindaChen in #3408
• Fix tie_word_embeddings for Qwen2. by @fyabc in #3344
• [Fix] Add args for mTLS support by @declark1 in #3430
• Fixes the misuse/mixuse of time.time()/time.monotonic() by @sighingnow in #3220
• [Misc] add error message in non linux platform by @youkaichao in #3438
• Fix issue templates by @hmellor in #3436
• fix document error for value and v_vec illustration by @laneeeee in #3421
• Asynchronous tokenization by @Yard1 in #2879
• Removed Extraneous Print Message From OAI Server by @robertgshaw2-neuralmagic in #3440
• [Misc] PR templates by @youkaichao in #3413
• Fixes the incorrect argument in the prefix-prefill test cases by @sighingnow in #3246
• Replace lstrip() with removeprefix() to fix Ruff linter warning by @ronensc in #2958
• Fix Baichuan chat template by @Dinghow in #3340
• ...

v0.3.3

Major changes

• StarCoder2 support
• Performance optimization and LoRA support for Gemma
• 2/3/8-bit GPTQ support
• Integrate Marlin Kernels for Int4 GPTQ inference
• Performance optimization for MoE kernel
• [Experimental] AWS Inferentia2 support
• [Experimental] Structured output (JSON, Regex) in OpenAI Server

What's Changed

• Update a comment in benchmark_serving.py by @ronensc in #2934
• Added early stopping to completion APIs by @Maxusmusti in #2939
• Migrate MistralForCausalLM to LlamaForCausalLM by @esmeetu in #2868
• Use Llama RMSNorm for Gemma by @WoosukKwon in #2974
• chore(vllm): codespell for spell checking by @mspronesti in #2820
• Optimize GeGLU layer in Gemma by @WoosukKwon in #2975
• [FIX] Fix issue #2904 by @44670 in #2983
• Remove Flash Attention in test env by @WoosukKwon in #2982
• Include tokens from prompt phase in counter_generation_tokens by @ronensc in #2802
• Fix nvcc not found in vllm-openai image by @zhaoyang-star in #2781
• [Fix] Fix assertion on Mistral YaRN model len by @WoosukKwon in #2984
• Port metrics from aioprometheus to prometheus_client by @hmellor in #2730
• Add LogProbs for Chat Completions in OpenAI by @jlcmoore in #2918
• Optimized fused MoE Kernel, take 2 by @pcmoritz in #2979
• [Minor] Remove gather_cached_kv kernel by @WoosukKwon in #3043
• [Minor] Remove unused config file by @esmeetu in #3039
• Fix using CuPy for eager mode by @esmeetu in #3037
• Fix stablelm by @esmeetu in #3038
• Support Orion model by @dachengai in #2539
• fix get_ip error in pure ipv6 environment by @Jingru in #2931
• [Minor] Fix type annotation in fused moe by @WoosukKwon in #3045
• Support logit bias for OpenAI API by @dylanwhawk in #3027
• [Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM by @WoosukKwon in #3046
• Enables GQA support in the prefix prefill kernels by @sighingnow in #3007
• multi-lora documentation fix by @ElefHead in #3064
• Restrict prometheus_client >= 0.18.0 to prevent errors when importing pkgs by @AllenDou in #3070
• Support inference with transformers-neuronx by @liangfu in #2569
• Add LoRA support for Gemma by @WoosukKwon in #3050
• Add Support for 2/3/8-bit GPTQ Quantization Models by @chu-tianxiang in #2330
• Fix: AttributeError in OpenAI-compatible server by @jaywonchung in #3018
• add cache_config's info to prometheus metrics. by @AllenDou in #3100
• Support starcoder2 architecture by @sh0416 in #3089
• Fix building from source on WSL by @aliencaocao in #3112
• [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams by @njhill in #3099
• Add guided decoding for OpenAI API server by @felixzhu555 in #2819
• Fix: Output text is always truncated in some models by @HyperdriveHustle in #3016
• Remove exclude_unset in streaming response by @sh0416 in #3143
• docs: Add tutorial on deploying vLLM model with KServe by @terrytangyuan in #2586
• fix relative import path of protocol.py by @Huarong in #3134
• Integrate Marlin Kernels for Int4 GPTQ inference by @robertgshaw2-neuralmagic in #2497
• Bump up to v0.3.3 by @WoosukKwon in #3129

New Contributors

• @Maxusmusti made their first contribution in #2939
• @44670 made their first contribution in #2983
• @jlcmoore made their first contribution in #2918
• @dachengai made their first contribution in #2539
• @dylanwhawk made their first contribution in #3027
• @ElefHead made their first contribution in #3064
• @AllenDou made their first contribution in #3070
• @jaywonchung made their first contribution in #3018
• @sh0416 made their first contribution in #3089
• @aliencaocao made their first contribution in #3112
• @felixzhu555 made their first contribution in #2819
• @HyperdriveHustle made their first contribution in #3016
• @terrytangyuan made their first contribution in #2586
• @Huarong made their first contribution in #3134

Full Changelog: v0.3.2...v0.3.3

v0.3.2

Major Changes

This version adds support for the OLMo and Gemma Model, as well as seed parameter.

What's Changed

• Defensively copy sampling_params by @njhill in #2881
• multi-LoRA as extra models in OpenAI server by @jvmncs in #2775
• Add code-revision config argument for Hugging Face Hub by @mbm-ai in #2892
• [Minor] Small fix to make distributed init logic in worker looks cleaner by @zhuohan123 in #2905
• [Test] Add basic correctness test by @zhuohan123 in #2908
• Support OLMo models. by @Isotr0py in #2832
• Add warning to prevent changes to benchmark api server by @simon-mo in #2858
• Fix vllm:prompt_tokens_total metric calculation by @ronensc in #2869
• [ROCm] include gfx908 as supported by @jamestwhedbee in #2792
• [FIX] Fix beam search test by @zhuohan123 in #2930
• Make vLLM logging formatting optional by @Yard1 in #2877
• Add metrics to RequestOutput by @Yard1 in #2876
• Add Gemma model by @xiangxu-google in #2964
• Upgrade transformers to v4.38.0 by @WoosukKwon in #2965
• [FIX] Add Gemma model to the doc by @zhuohan123 in #2966
• [ROCm] Upgrade transformers to v4.38.0 by @WoosukKwon in #2967
• Support per-request seed by @njhill in #2514
• Bump up version to v0.3.2 by @zhuohan123 in #2968

New Contributors

• @jvmncs made their first contribution in #2775
• @mbm-ai made their first contribution in #2892
• @Isotr0py made their first contribution in #2832
• @jamestwhedbee made their first contribution in #2792

Full Changelog: v0.3.1...v0.3.2

v0.3.1

Major Changes

This version fixes the following major bugs:

• Memory leak with distributed execution. (Solved by using CuPY for collective communication).
• Support for Python 3.8.

Also with many smaller bug fixes listed below.

What's Changed

• Fixes assertion failure in prefix caching: the lora index mapping should respect prefix_len. by @sighingnow in #2688
• fix some bugs about parameter description by @zspo in #2689
• [Minor] Fix test_cache.py CI test failure by @pcmoritz in #2684
• Add unit test for Mixtral MoE layer by @pcmoritz in #2677
• Refactor Prometheus and Add Request Level Metrics by @rib-2 in #2316
• Add Internlm2 by @Leymore in #2666
• Fix compile error when using rocm by @zhaoyang-star in #2648
• fix python 3.8 syntax by @simon-mo in #2716
• Update README for meetup slides by @simon-mo in #2718
• Use revision when downloading the quantization config file by @Pernekhan in #2697
• remove hardcoded device="cuda" to support more device by @jikunshang in #2503
• fix length_penalty default value to 1.0 by @zspo in #2667
• Add one example to run batch inference distributed on Ray by @c21 in #2696
• docs: update langchain serving instructions by @mspronesti in #2736
• Set&Get llm internal tokenizer instead of the TokenizerGroup by @dancingpipi in #2741
• Remove eos tokens from output by default by @zcnrex in #2611
• add requirement: triton >= 2.1.0 by @whyiug in #2746
• [Minor] Fix benchmark_latency by @WoosukKwon in #2765
• [ROCm] Fix some kernels failed unit tests by @hongxiayang in #2498
• Set local logging level via env variable by @gardberg in #2774
• [ROCm] Fixup arch checks for ROCM by @dllehr-amd in #2627
• Add fused top-K softmax kernel for MoE by @WoosukKwon in #2769
• fix issue when model parameter is not a model id but path of the model. by @liuyhwangyh in #2489
• [Minor] More fix of test_cache.py CI test failure by @LiuXiaoxuanPKU in #2750
• [ROCm] Fix build problem resulted from previous commit related to FP8 kv-cache support by @hongxiayang in #2790
• Add documentation on how to do incremental builds by @pcmoritz in #2796
• [Ray] Integration compiled DAG off by default by @rkooo567 in #2471
• Disable custom all reduce by default by @WoosukKwon in #2808
• [ROCm] support Radeon™ 7900 series (gfx1100) without using flash-attention by @hongxiayang in #2768
• Add documentation section about LoRA by @pcmoritz in #2834
• Refactor 2 awq gemm kernels into m16nXk32 by @zcnrex in #2723
• Serving Benchmark Refactoring by @ywang96 in #2433
• [CI] Ensure documentation build is checked in CI by @simon-mo in #2842
• Refactor llama family models by @esmeetu in #2637
• Revert "Refactor llama family models" by @pcmoritz in #2851
• Use CuPy for CUDA graphs by @WoosukKwon in #2811
• Remove Yi model definition, please use LlamaForCausalLM instead by @pcmoritz in #2854
• Add LoRA support for Mixtral by @tterrysun in #2831
• Migrate InternLMForCausalLM to LlamaForCausalLM by @pcmoritz in #2860
• Fix internlm after #2860 by @pcmoritz in #2861
• [Fix] Fix memory profiling when GPU is used by multiple processes by @WoosukKwon in #2863
• Fix docker python version by @NikolaBorisov in #2845
• Migrate AquilaForCausalLM to LlamaForCausalLM by @esmeetu in #2867
• Don't use cupy NCCL for AMD backends by @WoosukKwon in #2855
• Align LoRA code between Mistral and Mixtral (fixes #2875) by @pcmoritz in #2880
• [BugFix] Fix GC bug for LLM class by @WoosukKwon in #2882
• Fix decilm.py by @pcmoritz in #2883
• [ROCm] Dockerfile fix for flash-attention build by @hongxiayang in #2885
• Prefix Caching- fix t4 triton error by @caoshiyi in #2517
• Bump up to v0.3.1 by @WoosukKwon in #2887

New Contributors

• @sighingnow made their first contribution in #2688
• @rib-2 made their first contribution in #2316
• @Leymore made their first contribution in #2666
• @Pernekhan made their first contribution in #2697
• @jikunshang made their first contribution in #2503
• @c21 made their first contribution in #2696
• @zcnrex made their first contribution in #2611
• @whyiug made their first contribution in #2746
• @gardberg made their first contribution in #2774
• @dllehr-amd made their first contribution in #2627
• @rkooo567 made their first contribution in #2471
• @ywang96 made their first contribution in #2433
• @tterrysun made their first contribution in #2831

Full Changelog: v0.3.0...v0.3.1

v0.3.0

Major Changes

• Experimental multi-lora support
• Experimental prefix caching support
• FP8 KV Cache support
• Optimized MoE performance and Deepseek MoE support
• CI tested PRs
• Support batch completion in server

What's Changed

• Miner fix of type hint by @beginlner in #2340
• Build docker image with shared objects from "build" step by @payoto in #2237
• Ensure metrics are logged regardless of requests by @ichernev in #2347
• Changed scheduler to use deques instead of lists by @NadavShmayo in #2290
• Fix eager mode performance by @WoosukKwon in #2377
• [Minor] Remove unused code in attention by @WoosukKwon in #2384
• Add baichuan chat template jinjia file by @EvilPsyCHo in #2390
• [Speculative decoding 1/9] Optimized rejection sampler by @cadedaniel in #2336
• Fix ipv4 ipv6 dualstack by @yunfeng-scale in #2408
• [Minor] Rename phi_1_5 to phi by @WoosukKwon in #2385
• [DOC] Add additional comments for LLMEngine and AsyncLLMEngine by @litone01 in #1011
• [Minor] Fix the format in quick start guide related to Model Scope by @zhuohan123 in #2425
• Add gradio chatbot for openai webserver by @arkohut in #2307
• [BUG] RuntimeError: deque mutated during iteration in abort_seq_group by @chenxu2048 in #2371
• Allow setting fastapi root_path argument by @chiragjn in #2341
• Address Phi modeling update 2 by @huiwy in #2428
• Update a more user-friendly error message, offering more considerate advice for beginners, when using V100 GPU #1901 by @chuanzhubin in #2374
• Update quickstart.rst with small clarifying change (fix typo) by @nautsimon in #2369
• Aligning top_p and top_k Sampling by @chenxu2048 in #1885
• [Minor] Fix err msg by @WoosukKwon in #2431
• [Minor] Optimize cuda graph memory usage by @esmeetu in #2437
• [CI] Add Buildkite by @simon-mo in #2355
• Announce the second vLLM meetup by @WoosukKwon in #2444
• Allow buildkite to retry build on agent lost by @simon-mo in #2446
• Fix weigit loading for GQA with TP by @zhangch9 in #2379
• CI: make sure benchmark script exit on error by @simon-mo in #2449
• ci: retry on build failure as well by @simon-mo in #2457
• Add StableLM3B model by @ita9naiwa in #2372
• OpenAI refactoring by @FlorianJoncour in #2360
• [Experimental] Prefix Caching Support by @caoshiyi in #1669
• fix stablelm.py tensor-parallel-size bug by @YingchaoX in #2482
• Minor fix in prefill cache example by @JasonZhu1313 in #2494
• fix: fix some args desc by @zspo in #2487
• [Neuron] Add an option to build with neuron by @liangfu in #2065
• Don't download both safetensor and bin files. by @NikolaBorisov in #2480
• [BugFix] Fix abort_seq_group by @beginlner in #2463
• refactor completion api for readability by @simon-mo in #2499
• Support OpenAI API server in benchmark_serving.py by @hmellor in #2172
• Simplify broadcast logic for control messages by @zhuohan123 in #2501
• [Bugfix] fix load local safetensors model by @esmeetu in #2512
• Add benchmark serving to CI by @simon-mo in #2505
• Add group as an argument in broadcast ops by @GindaChen in #2522
• [Fix] Keep scheduler.running as deque by @njhill in #2523
• migrate pydantic from v1 to v2 by @joennlae in #2531
• [Speculative decoding 2/9] Multi-step worker for draft model by @cadedaniel in #2424
• Fix "Port could not be cast to integer value as " by @pcmoritz in #2545
• Add qwen2 by @JustinLin610 in #2495
• Fix progress bar and allow HTTPS in benchmark_serving.py by @hmellor in #2552
• Add a 1-line docstring to explain why calling context_attention_fwd twice in test_prefix_prefill.py by @JasonZhu1313 in #2553
• [Feature] Simple API token authentication by @taisazero in #1106
• Add multi-LoRA support by @Yard1 in #1804
• lint: format all python file instead of just source code by @simon-mo in #2567
• [Bugfix] fix crash if max_tokens=None by @NikolaBorisov in #2570
• Added include_stop_str_in_output and length_penalty parameters to OpenAI API by @galatolofederico in #2562
• [Doc] Fix the syntax error in the doc of supported_models. by @keli-wen in #2584
• Support Batch Completion in Server by @simon-mo in #2529
• fix names and license by @JustinLin610 in #2589
• [Fix] Use a correct device when creating OptionalCUDAGuard by @sh1ng in #2583
• [ROCm] add support to ROCm 6.0 and MI300 by @hongxiayang in #2274
• Support for Stable LM 2 by @dakotamahan-stability in #2598
• Don't build punica kernels by default by @pcmoritz in #2605
• AWQ: Up to 2.66x higher throughput by @casper-hansen in #2566
• Use head_dim in config if exists by @xiangxu-google in #2622
• Custom all reduce kernels by @hanzhi713 in #2192
• [Minor] Fix warning on Ray dependencies by @WoosukKwon in #2630
• Speed up Punica compilation by @WoosukKwon in #2632
• Small async_llm_engine refactor by @andoorve in #2618
• Update Ray version requirements by @simon-mo in #2636
• Support FP8-E5M2 KV Cache by @zhaoyang-star in #2279
• Fix error when tp > 1 by @zhaoyang-star in #2644
• No repeated IPC open by @hanzhi713 in #2642
• ROCm: Allow setting compilation target by @rlrs in #2581
• DeepseekMoE support with Fused MoE kernel by @zwd003 in #2453
• Fused MOE for Mixtral by @pcmoritz in #2542
• Fix 'Actor methods cannot be called directly' when using --engine-use-ray by @HermitSun in #2664
• Add swap_blocks unit tests by @sh1ng in #2616
• Fix a small typo (tenosr -> tensor) by @pcmoritz in #2672
• [Minor] Fix false warning when TP=1 by @WoosukKwon in #2674
• Add quantized mixtral support by @WoosukKwon in #2673
• Bump up version to v0.3.0 by @zhuohan123 in #2656

New Contributors

• @payoto made their first contribution in #2237
• @NadavShmayo made their first contribution in #2290
• @EvilPsyCHo made their first contribution in #2390
• @litone01 made their first contribution in #1011
• @arkohut made their first contribution in #2307
• @chiragjn made their first contribution in #2341
• @huiwy made their first contribution in #2428
• @chuanzhubin made their first contribution in #2374
• @nautsimon made their first contribution in #2369
• @zhangch9 made their first contribution in #2379
• @ita9naiwa made their first contribution in #2372
• @caoshiyi made their first contribution in https://gi...