🎵 [ComfyUI-MuScriptor] Transcribe Any Music into Multi-Instrument MIDI inside ComfyUI!

Bring state-of-the-art AI audio-to-MIDI transcription directly into your ComfyUI generation pipelines. Powered by Kyutai and Mirelo's MuScriptor model, this custom node lets you convert raw audio (AI-generated or uploaded) into precise, multi-track MIDI files and JSON note data.

🚀 Introduction

Whether you are generating AI music with AudioLM, MusicGen, or Stable Audio, or working with real-world audio tracks, turning audio into usable MIDI has always been a holy grail for AI producers, sound designers, and visual artists.

ComfyUI-MuScriptor integrates the cutting-edge MuScriptor multi-instrument music transcription architecture into ComfyUI. With this node, you can feed any audio input (via ComfyUI's standard AUDIO type or a local path) and get high-quality .mid files ready to drag into your DAW (Ableton, FL Studio, Logic Pro, Reaper) or drive audio-reactive visual generators!

✨ Key Features

• 🎼 Multi-Instrument Audio-to-MIDI Transcription: Accurately transcribes complex polyphonic music into separate MIDI instrument events (Piano, Strings, Guitar, Drums, Bass, etc.).

• 🔌 Native ComfyUI Integration: Directly connects with AUDIO outputs from audio generation nodes or local audio file paths.

• ⚡ Multiple Model Sizes: Choose between 3 model variants depending on your VRAM and speed needs:

  • small (103M parameters) — Ultra-fast, ideal for low VRAM or quick drafts.

  • medium (307M parameters) — Default, excellent balance of precision and speed.

  • large (1.4B parameters) — Maximum transcription accuracy for complex multi-instrumental tracks.

• 🎯 Instrument Filtering: Target specific instruments! Restrict the transcription output to only transcribe specific instruments (e.g. acoustic_piano, violin, drums).

• 📊 Dual Output System:

  • midi_path — Saved .mid file in your ComfyUI output directory.

  • notes_json — Structured JSON array of all note events (pitch, start_time, end_time, instrument) for custom scripting or animation drivers.

• 🚀 Smart In-Memory Caching: Models are loaded dynamically into GPU/RAM and cached between workflow runs for instant execution on subsequent generations.

💡 Creative Use Cases for AI Artists & Producers

1. AI Music Generation to DAW Workflow

Generate an audio track using AI audio nodes, feed it into MuScriptor Transcribe Audio to MIDI, and export the MIDI into your DAW. Swap the synthetic AI audio with premium VST instruments (Kontakt, Serum, Diva, Superior Drummer) for studio-grade production quality!

2. Audio-Reactive Visuals & Motion Graphics

Use the notes_json output to drive keyframes in ComfyUI animation workflows (AnimateDiff, Deforum, or custom Python/TouchDesigner scripts). Trigger visual effects, camera zooms, or light flashes on specific musical beats and instrument hits!

3. Remixing & Sampling

Isolate melodies, chord progressions, or drum grooves from existing sample packs or recordings without spending hours manually writing MIDI notes by ear.

🎛️ Node Inputs & Settings Guide

The main node is located under the MuScriptor category: MuScriptor Transcribe Audio to MIDI.

📥 Inputs

ParameterTypeDescriptionaudioAUDIO (optional)Standard ComfyUI AUDIO link (takes priority if connected).audio_pathSTRING (optional)Local file path to any audio file (.wav, .mp3, .flac, etc.).model_sizeENUMChoose small (103M), medium (307M, default), or large (1.4B).deviceENUMHardware target: auto (auto-detect CUDA/MPS/CPU), cuda, cpu, or mps.dtypeENUMPrecision: auto, float32, float16, or bfloat16.use_samplingBOOLEANEnable temperature sampling instead of greedy decoding (Default: False).temperatureFLOATControls sampling randomness (0.0 to 2.0).cfg_coefFLOATClassifier-Free Guidance coefficient (Default: 1.0).beam_sizeINTBeam search width (1 = greedy decoding, ≥2 = beam search for higher accuracy).prelude_forcingBOOLEANEnforces chunk boundary alignment for seamless long-track transcription (Default: True).instrumentsSTRINGOptional list of allowed instruments separated by commas or newlines (e.g. acoustic_piano, violin).filename_prefixSTRINGPrefix for the output .mid file saved in ComfyUI's output folder.

📤 Outputs

OutputTypeDescriptionmidi_pathSTRINGAbsolute path to the generated .mid file.notes_jsonSTRINGJSON string array of decoded notes: [{"pitch": 60, "start_time": 0.5, "end_time": 1.2, "instrument": "acoustic_piano"}, ...].

📦 Step-by-Step Installation

1. Clone into custom_nodes

Open your terminal inside your ComfyUI directory:

bash

cd ComfyUI/custom_nodes git clone https://github.com/martyyz-ai/ComfyUI-MuScriptor.git comfyui-muscriptor

2. Install Requirements

Install the required Python packages into your ComfyUI environment:

bash

pip install -r comfyui-muscriptor/requirements.txt

3. HuggingFace Authentication (Required)

The MuScriptor model weights are hosted on HuggingFace under a gated repository. Access takes less than a minute:

1. Visit the MuScriptor HuggingFace Page and click Accept Terms.

2. Log in via your terminal:

   bash

   hf auth login or set your environment variable:

   bash

   export HF_TOKEN=your_huggingface_token_here

🌟 Pro-Tips for Best Results

1. Clean Audio = Better MIDI: Transcribing stems or clean multi-instrument tracks produces sharper MIDI than heavily compressed or noisy audio.

2. Beam Search for Complex Pieces: If transcribing fast piano solos or complex orchestra arrangements, increase beam_size to 2 or 4.

3. Instrument Constraining: If you only want the piano melody from a full track, specify acoustic_piano in the instruments box to prevent the model from misassigning notes to other instruments.

🛠️ Credits & Links

• ComfyUI Node Author: martyyz-ai

• GitHub Repository: ComfyUI-MuScriptor on GitHub

• Model Creators: Kyutai & Mirelo

• License: MIT License

If you find this node useful for your creative workflows, drop a like or review on Civitai and star the GitHub repository! Happy generating! 🎹🎶