Implementing a 35-Billion Parameter Mixture of Experts Model for Small Teams: A Practical Guide (Open Source)
In recent months, we embarked on an ambitious project: deploying a large language model (LLM) capable of serving a team of twenty individuals, all hosted on a single desktop machine. This article shares our experience, insights, and technical details, demonstrating how it’s possible to run a substantial AI model without reliance on cloud infrastructure—using entirely open-source tools and hardware.
Hardware and Model Selection
Our setup is built around high-performance, single-node infrastructure:
- NVIDIA DGX Spark: Equipped with 128GB of unified GPU memory, providing the computational backbone.
- Model: Ornith-1.5-35B-A3B, a Mixture of Experts (MoE) architecture with 35 billion total parameters and approximately 3 billion active parameters during inference. This architecture optimizes performance by activating only a subset of parameters per token, balancing quality and resource efficiency.
- Checkpoint: An official 4-bit NVFP4 version, approximately 22 GiB in size, ready to deploy without the need for further quantization.
- Software Framework: vLLM 0.24, which supports multi-token prediction and speculative decoding without requiring a separate draft model.
Functional Capabilities
Our deployment offers robust features suitable for team workflows:
- Per-User Response Speed: Individual responses reach approximately 79 tokens per second, delivering near-instant interactions within chat interfaces.
- Multi-User Throughput: With twenty concurrent users, the system maintains around 25 tokens per second each, totaling roughly 490 tokens/sec.
- Rich Context Handling: Supports a maximum context window of 262,144 tokens, enabling long conversations or extensive document analysis.
- Multimodal and Tool Integration: Supports vision inputs and tool calling, extending the model’s versatility.
- Reasoning and Accuracy: Achieves high accuracy on reasoning benchmarks: 97% on MATH-500, 98% on GSM8K, and 81% MMLU-Pro. While not state-of-the-art, these are sufficient for daily coding, documentation, and question-answering tasks, with no team complaints thus far.
How the Team Uses the System
Our team leverages various interfaces to maximize productivity:
- OpenCode: Terminal-based coding environment tied directly to the model.
- WebUI: Chat interface for conversational queries, PDF parsing, and embeddings—all running locally.
- Data Privacy: No data leaves our premises, ensuring confidentiality for proprietary code and sensitive documents—a key motivator for such a local deployment.
Multi-User Management and Infrastructure
Operating a single model for multiple users introduces unique challenges:
- User Identification: Each individual or device receives a unique API key.
- Usage Monitoring: Input/output tokens are tracked live per user, providing real-time usage data.
- Analytics & Alerts: Data is pushed to Grafana dashboards; alerts trigger if unauthorized or excessive API calls occur.
- Security: A lightweight Python gateway mediates all requests, preventing direct access to vLLM and maintaining control over resource utilization. This transforms the setup from a simple project into a manageable infrastructure.
Lessons Learned and Gotchas
For those considering similar projects, we encountered practical hurdles:
- Memory Management: With a unified GPU, be cautious not to overcommit. We capped concurrent memory usage at 70% to maintain stability under load.
- Concurrency Limits: Increasing user slots beyond twenty reduced throughput due to queuing delays. Balancing user count and performance is crucial.
- Software Configuration: Certain applications like WebUI default to fixed model settings, ignoring environment variables after initial setup—costing us unexpected debugging time.
All scripts, dashboards, and benchmarking tools are openly available on GitHub, fostering transparency and community collaboration:
If you’re contemplating a similar self-hosted solution for a small team, I welcome questions and discussions. I’m particularly interested in approaches others take for per-user tracking and resource management—I’ve evaluated options like LiteLLM but prefer a lightweight, fully-understood setup.
Visual Overview
In conclusion, running a large language model for team use on a single machine is not only feasible but manageable with the right hardware, open-source tools, and careful configuration. This approach grants organizations full control over their AI environment, ensuring privacy, cost-efficiency, and customization—making it an attractive option for small teams aiming to integrate sophisticated AI capabilities internally.
