Meta Introduces Muse Spark Open Multimodal MoE Model

Meta has released Muse Spark Open, an open-weight multimodal mixture-of-experts model containing approximately 816 billion total parameters, with about 42 billion active per token. It takes text, images, audio, and video inputs, generating text responses across a context window extending up to one million tokens. Designed for agentic coding workflows, it is trained to read codebases, debug, write code, execute tool calls, and inspect visual media like screenshots or video recordings. The model features 62 layers, with each MoE layer containing 256 experts, of which eight are selected per token. Visual inputs are processed via a 1.8B-parameter perception encoder, while audio is parsed through a convolutional pyramid encoder into tokens representing 80-millisecond slices. Checkpoints are available in BF16 and NVFP4 formats.

Key takeaways

• Muse Spark Open features ~816 billion total parameters, activating ~42 billion per token via 256 experts per layer (8 selected per token), and supports a 1M-token context.
• The model handles text, images, audio, and video inputs, outputting text designed for agentic tasks like multi-file refactoring, debugging, and tool execution.
• Perception processing utilizes a 1.8B-parameter visual encoder alongside an audio encoder outputting one token every 80 milliseconds (12.5 tokens per second).
• Model weights are available in BF16 (~1.5 TB) and NVFP4 formats, supporting deployment on clusters of NVIDIA H200 GPUs or single B200 nodes.