musicgen_melody

mindnlp.transformers.models.musicgen_melody.configuration_musicgen_melody

Musicgen Melody model configuration

mindnlp.transformers.models.musicgen_melody.configuration_musicgen_melody.MusicgenMelodyConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [MusicgenMelodyModel]. It is used to instantiate a Musicgen Melody model according to the specified arguments, defining the text encoder, audio encoder and Musicgen Melody decoder configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Musicgen Melody facebook/musicgen-melody architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
num_chroma	Number of chroma bins to use. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
chroma_length	Maximum chroma duration if audio is used to condition the model. Corresponds to the maximum duration used during training. TYPE: `int`, *optional*, defaults to 235 DEFAULT: 235
kwargs	Dictionary of keyword arguments. Notably: - **text_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that defines the text encoder config. - **audio_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that defines the audio encoder config. - **decoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that defines the decoder config. TYPE: *optional* DEFAULT: {}

>>> from transformers import (
...     MusicgenMelodyConfig,
...     MusicgenMelodyDecoderConfig,
...     T5Config,
...     EncodecConfig,
...     MusicgenMelodyForConditionalGeneration,
... )

>>> # Initializing text encoder, audio encoder, and decoder model configurations
>>> text_encoder_config = T5Config()
>>> audio_encoder_config = EncodecConfig()
>>> decoder_config = MusicgenMelodyDecoderConfig()

>>> configuration = MusicgenMelodyConfig.from_sub_models_config(
...     text_encoder_config, audio_encoder_config, decoder_config
... )

>>> # Initializing a MusicgenMelodyForConditionalGeneration (with random weights) from the facebook/musicgen-melody style configuration
>>> model = MusicgenMelodyForConditionalGeneration(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
>>> config_text_encoder = model.config.text_encoder
>>> config_audio_encoder = model.config.audio_encoder
>>> config_decoder = model.config.decoder

>>> # Saving the model, including its configuration
>>> model.save_pretrained("musicgen_melody-model")

>>> # loading model and config from pretrained folder
>>> musicgen_melody_config = MusicgenMelodyConfig.from_pretrained("musicgen_melody-model")
>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("musicgen_melody-model", config=musicgen_melody_config)

Source code in mindnlp\transformers\models\musicgen_melody\configuration_musicgen_melody.py

class MusicgenMelodyConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`MusicgenMelodyModel`]. It is used to instantiate a
    Musicgen Melody model according to the specified arguments, defining the text encoder, audio encoder and Musicgen Melody decoder
    configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Musicgen Melody
    [facebook/musicgen-melody](https://huggingface.co/facebook/musicgen-melody) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        num_chroma (`int`, *optional*, defaults to 12): Number of chroma bins to use.
        chroma_length (`int`, *optional*, defaults to 235):
            Maximum chroma duration if audio is used to condition the model. Corresponds to the maximum duration used during training.
        kwargs (*optional*):
            Dictionary of keyword arguments. Notably:

                - **text_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that
                  defines the text encoder config.
                - **audio_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that
                  defines the audio encoder config.
                - **decoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that defines
                  the decoder config.

    Example:

    ```python
    >>> from transformers import (
    ...     MusicgenMelodyConfig,
    ...     MusicgenMelodyDecoderConfig,
    ...     T5Config,
    ...     EncodecConfig,
    ...     MusicgenMelodyForConditionalGeneration,
    ... )

    >>> # Initializing text encoder, audio encoder, and decoder model configurations
    >>> text_encoder_config = T5Config()
    >>> audio_encoder_config = EncodecConfig()
    >>> decoder_config = MusicgenMelodyDecoderConfig()

    >>> configuration = MusicgenMelodyConfig.from_sub_models_config(
    ...     text_encoder_config, audio_encoder_config, decoder_config
    ... )

    >>> # Initializing a MusicgenMelodyForConditionalGeneration (with random weights) from the facebook/musicgen-melody style configuration
    >>> model = MusicgenMelodyForConditionalGeneration(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config
    >>> config_text_encoder = model.config.text_encoder
    >>> config_audio_encoder = model.config.audio_encoder
    >>> config_decoder = model.config.decoder

    >>> # Saving the model, including its configuration
    >>> model.save_pretrained("musicgen_melody-model")

    >>> # loading model and config from pretrained folder
    >>> musicgen_melody_config = MusicgenMelodyConfig.from_pretrained("musicgen_melody-model")
    >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("musicgen_melody-model", config=musicgen_melody_config)
    ```"""

    model_type = "musicgen_melody"
    is_composition = True

    def __init__(
        self,
        num_chroma=12,
        chroma_length=235,
        **kwargs,
    ):
        super().__init__(**kwargs)
        if "text_encoder" not in kwargs or "audio_encoder" not in kwargs or "decoder" not in kwargs:
            raise ValueError("Config has to be initialized with text_encoder, audio_encoder and decoder config")

        text_encoder_config = kwargs.pop("text_encoder")
        text_encoder_model_type = text_encoder_config.pop("model_type")

        audio_encoder_config = kwargs.pop("audio_encoder")
        audio_encoder_model_type = audio_encoder_config.pop("model_type")

        decoder_config = kwargs.pop("decoder")

        self.text_encoder = AutoConfig.for_model(text_encoder_model_type, **text_encoder_config)
        self.audio_encoder = AutoConfig.for_model(audio_encoder_model_type, **audio_encoder_config)
        self.decoder = MusicgenMelodyDecoderConfig(**decoder_config)
        self.is_encoder_decoder = False

        self.num_chroma = num_chroma
        self.chroma_length = chroma_length

    @classmethod
    def from_sub_models_config(
        cls,
        text_encoder_config: PretrainedConfig,
        audio_encoder_config: PretrainedConfig,
        decoder_config: MusicgenMelodyDecoderConfig,
        **kwargs,
    ):
        r"""
        Instantiate a [`MusicgenMelodyConfig`] (or a derived class) from text encoder, audio encoder and decoder
        configurations.

        Returns:
            [`MusicgenMelodyConfig`]: An instance of a configuration object
        """

        return cls(
            text_encoder=text_encoder_config.to_dict(),
            audio_encoder=audio_encoder_config.to_dict(),
            decoder=decoder_config.to_dict(),
            **kwargs,
        )

    @property
    # This is a property because you might want to change the codec model on the fly
    def sampling_rate(self):
        return self.audio_encoder.sampling_rate

    @property
    def _attn_implementation(self):
        # This property is made private for now (as it cannot be changed and a PreTrainedModel.use_attn_implementation method needs to be implemented.)
        if hasattr(self, "_attn_implementation_internal"):
            if self._attn_implementation_internal is None:
                # `config.attn_implementation` should never be None, for backward compatibility.
                return "eager"
            else:
                return self._attn_implementation_internal
        else:
            return "eager"

    @_attn_implementation.setter
    def _attn_implementation(self, value):
        self._attn_implementation_internal = value
        self.decoder._attn_implementation = value

mindnlp.transformers.models.musicgen_melody.configuration_musicgen_melody.MusicgenMelodyConfig.from_sub_models_config(text_encoder_config, audio_encoder_config, decoder_config, **kwargs) classmethod

Instantiate a [MusicgenMelodyConfig] (or a derived class) from text encoder, audio encoder and decoder configurations.

RETURNS	DESCRIPTION
	[MusicgenMelodyConfig]: An instance of a configuration object

Source code in mindnlp\transformers\models\musicgen_melody\configuration_musicgen_melody.py

@classmethod
def from_sub_models_config(
    cls,
    text_encoder_config: PretrainedConfig,
    audio_encoder_config: PretrainedConfig,
    decoder_config: MusicgenMelodyDecoderConfig,
    **kwargs,
):
    r"""
    Instantiate a [`MusicgenMelodyConfig`] (or a derived class) from text encoder, audio encoder and decoder
    configurations.

    Returns:
        [`MusicgenMelodyConfig`]: An instance of a configuration object
    """

    return cls(
        text_encoder=text_encoder_config.to_dict(),
        audio_encoder=audio_encoder_config.to_dict(),
        decoder=decoder_config.to_dict(),
        **kwargs,
    )

mindnlp.transformers.models.musicgen_melody.configuration_musicgen_melody.MusicgenMelodyDecoderConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of an [MusicgenMelodyDecoder]. It is used to instantiate a Musicgen Melody decoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Musicgen Melody facebook/musicgen-melody architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the MusicgenMelodyDecoder model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [MusicgenMelodyDecoder]. TYPE: `int`, *optional*, defaults to 2048 DEFAULT: 2048
max_position_embeddings	The maximum sequence length that this model might ever be used with. Typically, set this to something large just in case (e.g., 512 or 1024 or 2048). TYPE: `int`, *optional*, defaults to 2048 DEFAULT: 2048
num_hidden_layers	Number of decoder layers. TYPE: `int`, *optional*, defaults to 24 DEFAULT: 24
ffn_dim	Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer block. TYPE: `int`, *optional*, defaults to 4096 DEFAULT: 4096
num_attention_heads	Number of attention heads for each attention layer in the Transformer block. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
layerdrop	The LayerDrop probability for the decoder. See the LayerDrop paper for more details. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
use_cache	Whether the model should return the last key/values attentions (not used by all models) TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
activation_function	The non-linear activation function (function or string) in the decoder and pooler. If string, "gelu", "relu", "silu" and "gelu_new" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_size	Dimensionality of the layers and the pooler layer. TYPE: `int`, *optional*, defaults to 1024 DEFAULT: 1024
dropout	The dropout probability for all fully connected layers in the embeddings, text_encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
activation_dropout	The dropout ratio for activations inside the fully connected layer. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
initializer_factor	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
scale_embedding	Scale embeddings by diving by sqrt(hidden_size). TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
num_codebooks	The number of parallel codebooks forwarded to the model. TYPE: `int`, *optional*, defaults to 4 DEFAULT: 4
audio_channels	Number of audio channels used by the model (either mono or stereo). Stereo models generate a separate audio stream for the left/right output channels. Mono models generate a single audio stream output. TYPE: `int`, *optional*, defaults to 1 DEFAULT: 1
pad_token_id	The id of the padding token. TYPE: `int`, *optional*, defaults to 2048 DEFAULT: 2048
bos_token_id	The id of the beginning-of-sequence token. TYPE: `int`, *optional*, defaults to 2048 DEFAULT: 2048
eos_token_id	The id of the end-of-sequence token. TYPE: `int`, *optional* DEFAULT: None
tie_word_embeddings	Whether to tie word embeddings with the text encoder. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

Source code in mindnlp\transformers\models\musicgen_melody\configuration_musicgen_melody.py

class MusicgenMelodyDecoderConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of an [`MusicgenMelodyDecoder`]. It is used to instantiate a
    Musicgen Melody decoder according to the specified arguments, defining the model architecture. Instantiating a
    configuration with the defaults will yield a similar configuration to that of the Musicgen Melody
    [facebook/musicgen-melody](https://huggingface.co/facebook/musicgen-melody) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 2048):
            Vocabulary size of the MusicgenMelodyDecoder model. Defines the number of different tokens that can be
            represented by the `inputs_ids` passed when calling [`MusicgenMelodyDecoder`].
        max_position_embeddings (`int`, *optional*, defaults to 2048):
            The maximum sequence length that this model might ever be used with. Typically, set this to something large
            just in case (e.g., 512 or 1024 or 2048).
        num_hidden_layers (`int`, *optional*, defaults to 24):
            Number of decoder layers.
        ffn_dim (`int`, *optional*, defaults to 4096):
            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer block.
        num_attention_heads (`int`, *optional*, defaults to 16):
            Number of attention heads for each attention layer in the Transformer block.
        layerdrop (`float`, *optional*, defaults to 0.0):
            The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
            for more details.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether the model should return the last key/values attentions (not used by all models)
        activation_function (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the decoder and pooler. If string, `"gelu"`,
            `"relu"`, `"silu"` and `"gelu_new"` are supported.
        hidden_size (`int`, *optional*, defaults to 1024):
            Dimensionality of the layers and the pooler layer.
        dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, text_encoder, and pooler.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        activation_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for activations inside the fully connected layer.
        initializer_factor (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        scale_embedding (`bool`, *optional*, defaults to `False`):
            Scale embeddings by diving by sqrt(hidden_size).
        num_codebooks (`int`, *optional*, defaults to 4):
            The number of parallel codebooks forwarded to the model.
        audio_channels (`int`, *optional*, defaults to 1):
            Number of audio channels used by the model (either mono or stereo). Stereo models generate a separate
            audio stream for the left/right output channels. Mono models generate a single audio stream output.
        pad_token_id (`int`, *optional*, defaults to 2048): The id of the *padding* token.
        bos_token_id (`int`, *optional*, defaults to 2048): The id of the *beginning-of-sequence* token.
        eos_token_id (`int`, *optional*): The id of the *end-of-sequence* token.
        tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie word embeddings with the text encoder.
    """

    model_type = "musicgen_melody_decoder"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=2048,
        max_position_embeddings=2048,
        num_hidden_layers=24,
        ffn_dim=4096,
        num_attention_heads=16,
        layerdrop=0.0,
        use_cache=True,
        activation_function="gelu",
        hidden_size=1024,
        dropout=0.1,
        attention_dropout=0.0,
        activation_dropout=0.0,
        initializer_factor=0.02,
        scale_embedding=False,
        num_codebooks=4,
        audio_channels=1,
        pad_token_id=2048,
        bos_token_id=2048,
        eos_token_id=None,
        tie_word_embeddings=False,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.ffn_dim = ffn_dim
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.dropout = dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.activation_function = activation_function
        self.initializer_factor = initializer_factor
        self.layerdrop = layerdrop
        self.use_cache = use_cache
        self.scale_embedding = scale_embedding  # scale factor will be sqrt(d_model) if True
        self.num_codebooks = num_codebooks

        if audio_channels not in [1, 2]:
            raise ValueError(f"Expected 1 (mono) or 2 (stereo) audio channels, got {audio_channels} channels.")
        self.audio_channels = audio_channels

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody

MindSpore Musicgen Melody model.

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyAttention

Bases: Module

Multi-headed attention from 'Attention Is All You Need' paper

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        is_decoder: bool = False,
        bias: bool = True,
        is_causal: bool = False,
        config: Optional[MusicgenMelodyConfig] = None,
    ):
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        self.config = config

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5
        self.is_decoder = is_decoder
        self.is_causal = is_causal

        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

    def _shape(self, tensor: mindspore.Tensor, seq_len: int, bsz: int):
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).swapaxes(1, 2)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        key_value_states: Optional[mindspore.Tensor] = None,
        past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        layer_head_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
        """Input shape: Batch x Time x Channel"""

        # if key_value_states are provided this layer is used as a cross-attention layer
        # for the decoder
        is_cross_attention = key_value_states is not None

        bsz, tgt_len, _ = hidden_states.shape

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scaling
        # get key, value proj
        # `past_key_value[0].shape[2] == key_value_states.shape[1]`
        # is checking that the `sequence_length` of the `past_key_value` is the same as
        # the provided `key_value_states` to support prefix tuning
        if (
            is_cross_attention
            and past_key_value is not None
            and past_key_value[0].shape[2] == key_value_states.shape[1]
        ):
            # reuse k,v, cross_attentions
            key_states = past_key_value[0]
            value_states = past_key_value[1]
        elif is_cross_attention:
            # cross_attentions
            key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
            value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
        elif past_key_value is not None:
            # reuse k, v, self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
            key_states = ops.cat([past_key_value[0], key_states], dim=2)
            value_states = ops.cat([past_key_value[1], value_states], dim=2)
        else:
            # self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        if self.is_decoder:
            # if cross_attention save Tuple(mindspore.Tensor, mindspore.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(mindspore.Tensor, mindspore.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_states, value_states)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.reshape(*proj_shape)
        value_states = value_states.reshape(*proj_shape)

        src_len = key_states.shape[1]
        attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

        if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.shape}"
            )

        if attention_mask is not None:
            if attention_mask.shape != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = nn.functional.softmax(attn_weights, dim=-1)

        if layer_head_mask is not None:
            if layer_head_mask.shape != (self.num_heads,):
                raise ValueError(
                    f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                    f" {layer_head_mask.shape}"
                )
            attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        if output_attentions:
            # this operation is a bit awkward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to be reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)

        attn_output = ops.bmm(attn_probs, value_states)

        if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.shape}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.swapaxes(1, 2)

        # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
        # partitioned across GPUs when using tensor-parallelism.
        attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

        attn_output = self.out_proj(attn_output)

        return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyAttention.forward(hidden_states, key_value_states=None, past_key_value=None, attention_mask=None, layer_head_mask=None, output_attentions=False)

Input shape: Batch x Time x Channel

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def forward(
    self,
    hidden_states: mindspore.Tensor,
    key_value_states: Optional[mindspore.Tensor] = None,
    past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    layer_head_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
    """Input shape: Batch x Time x Channel"""

    # if key_value_states are provided this layer is used as a cross-attention layer
    # for the decoder
    is_cross_attention = key_value_states is not None

    bsz, tgt_len, _ = hidden_states.shape

    # get query proj
    query_states = self.q_proj(hidden_states) * self.scaling
    # get key, value proj
    # `past_key_value[0].shape[2] == key_value_states.shape[1]`
    # is checking that the `sequence_length` of the `past_key_value` is the same as
    # the provided `key_value_states` to support prefix tuning
    if (
        is_cross_attention
        and past_key_value is not None
        and past_key_value[0].shape[2] == key_value_states.shape[1]
    ):
        # reuse k,v, cross_attentions
        key_states = past_key_value[0]
        value_states = past_key_value[1]
    elif is_cross_attention:
        # cross_attentions
        key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
        value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
    elif past_key_value is not None:
        # reuse k, v, self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
        key_states = ops.cat([past_key_value[0], key_states], dim=2)
        value_states = ops.cat([past_key_value[1], value_states], dim=2)
    else:
        # self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

    if self.is_decoder:
        # if cross_attention save Tuple(mindspore.Tensor, mindspore.Tensor) of all cross attention key/value_states.
        # Further calls to cross_attention layer can then reuse all cross-attention
        # key/value_states (first "if" case)
        # if uni-directional self-attention (decoder) save Tuple(mindspore.Tensor, mindspore.Tensor) of
        # all previous decoder key/value_states. Further calls to uni-directional self-attention
        # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
        # if encoder bi-directional self-attention `past_key_value` is always `None`
        past_key_value = (key_states, value_states)

    proj_shape = (bsz * self.num_heads, -1, self.head_dim)
    query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
    key_states = key_states.reshape(*proj_shape)
    value_states = value_states.reshape(*proj_shape)

    src_len = key_states.shape[1]
    attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

    if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
        raise ValueError(
            f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
            f" {attn_weights.shape}"
        )

    if attention_mask is not None:
        if attention_mask.shape != (bsz, 1, tgt_len, src_len):
            raise ValueError(
                f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
            )
        attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    if layer_head_mask is not None:
        if layer_head_mask.shape != (self.num_heads,):
            raise ValueError(
                f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                f" {layer_head_mask.shape}"
            )
        attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    if output_attentions:
        # this operation is a bit awkward, but it's required to
        # make sure that attn_weights keeps its gradient.
        # In order to do so, attn_weights have to be reshaped
        # twice and have to be reused in the following
        attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
    else:
        attn_weights_reshaped = None

    attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)

    attn_output = ops.bmm(attn_probs, value_states)

    if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
        raise ValueError(
            f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
            f" {attn_output.shape}"
        )

    attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
    attn_output = attn_output.swapaxes(1, 2)

    # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
    # partitioned across GPUs when using tensor-parallelism.
    attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

    attn_output = self.out_proj(attn_output)

    return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyDecoder

Bases: MusicgenMelodyPreTrainedModel

Transformer decoder consisting of config.num_hidden_layers layers. Each layer is a [MusicgenMelodyDecoderLayer]

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyDecoder(MusicgenMelodyPreTrainedModel):
    """
    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MusicgenMelodyDecoderLayer`]
    """

    def __init__(self, config: MusicgenMelodyDecoderConfig):
        super().__init__(config)
        self.dropout = config.dropout
        self.layerdrop = config.layerdrop
        self.max_target_positions = config.max_position_embeddings
        self.d_model = config.hidden_size
        self.num_codebooks = config.num_codebooks
        self.embed_scale = math.sqrt(config.hidden_size) if config.scale_embedding else 1.0

        embed_dim = config.vocab_size + 1
        self.embed_tokens = nn.ModuleList(
            [nn.Embedding(embed_dim, config.hidden_size) for _ in range(config.num_codebooks)]
        )

        self.embed_positions = MusicgenMelodySinusoidalPositionalEmbedding(
            config.max_position_embeddings,
            config.hidden_size,
        )

        self.layers = nn.ModuleList([MusicgenMelodyDecoderLayer(config) for _ in range(config.num_hidden_layers)])
        self.layer_norm = nn.LayerNorm(config.hidden_size)
        self.attn_implementation = config._attn_implementation

        self.gradient_checkpointing = False
        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        return self.embed_tokens

    def set_input_embeddings(self, value):
        self.embed_tokens = value

    # Ignore copy
    def forward(
        self,
        input_ids: mindspore.Tensor = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPast]:
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        use_cache = use_cache if use_cache is not None else self.config.use_cache
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # retrieve input_ids and inputs_embeds
        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
        elif input_ids is not None:
            # (bsz * codebooks, seq_len) -> (bsz, codebooks, seq_len)
            input = input_ids.reshape(-1, self.num_codebooks, input_ids.shape[-1])
            bsz, num_codebooks, seq_len = input.shape
            input_shape = (bsz, seq_len)
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
            input = inputs_embeds[:, :, -1:]
        else:
            raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")

        # past_key_values_length
        past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0

        if inputs_embeds is None:
            inputs_embeds = sum(self.embed_tokens[codebook](input[:, codebook]) for codebook in range(num_codebooks))

        if encoder_hidden_states is not None:
            # take care of attention masks
            if encoder_attention_mask is not None and attention_mask is None:
                attention_mask = ops.ones(inputs_embeds.shape[:2])

            if attention_mask is not None:
                if encoder_attention_mask is None:
                    encoder_attention_mask = ops.ones(encoder_hidden_states.shape[:2], dtype=attention_mask.dtype)
                attention_mask = ops.cat([encoder_attention_mask, attention_mask.to(encoder_attention_mask.dtype)], dim=1)

            # fuse encoder_hidden_states and inputs_embeds
            inputs_embeds = ops.cat([encoder_hidden_states, inputs_embeds], dim=1)

        input_shape = inputs_embeds.shape[:-1]

        attention_mask = _prepare_4d_causal_attention_mask(
            attention_mask, input_shape, inputs_embeds, past_key_values_length
        )

        # embed positions
        positions = self.embed_positions(inputs_embeds, past_key_values_length)

        hidden_states = inputs_embeds + positions

        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing`. Setting `use_cache=False`..."
                )
                use_cache = False

        # decoder layers
        all_hidden_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None
        next_decoder_cache = () if use_cache else None

        # check if head_mask has a correct number of layers specified if desired
        if head_mask is not None:
            if head_mask.shape[0] != len(self.layers):
                raise ValueError(
                    f"The `head_mask` should be specified for {len(self.layers)} layers, but it is for"
                    f" {head_mask.shape[0]}."
                )

        for idx, decoder_layer in enumerate(self.layers):
            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            if output_hidden_states:
                all_hidden_states += (hidden_states,)
            dropout_probability = random.uniform(0, 1)
            if self.training and (dropout_probability < self.layerdrop):
                continue

            past_key_value = past_key_values[idx] if past_key_values is not None else None

            if self.gradient_checkpointing and self.training:
                layer_outputs = self._gradient_checkpointing_func(
                    decoder_layer.forward,
                    hidden_states,
                    attention_mask,
                    head_mask[idx] if head_mask is not None else None,
                    None,
                    output_attentions,
                    use_cache,
                )
            else:
                layer_outputs = decoder_layer(
                    hidden_states,
                    attention_mask=attention_mask,
                    layer_head_mask=(head_mask[idx] if head_mask is not None else None),
                    past_key_value=past_key_value,
                    output_attentions=output_attentions,
                    use_cache=use_cache,
                )
            hidden_states = layer_outputs[0]

            if use_cache:
                next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)

            if output_attentions:
                all_attentions += (layer_outputs[1],)

        hidden_states = self.layer_norm(hidden_states)

        # add hidden states from the last decoder layer
        if output_hidden_states:
            all_hidden_states += (hidden_states,)

        next_cache = next_decoder_cache if use_cache else None
        if not return_dict:
            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_attentions] if v is not None)
        return BaseModelOutputWithPast(
            last_hidden_state=hidden_states,
            past_key_values=next_cache,
            hidden_states=all_hidden_states,
            attentions=all_attentions,
        )

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyDecoderLayer

Bases: Module

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyDecoderLayer(nn.Module):
    def __init__(self, config: MusicgenMelodyDecoderConfig):
        super().__init__()
        self.embed_dim = config.hidden_size

        self.self_attn = MUSICGEN_MELODY_ATTENTION_CLASSES[config._attn_implementation](
            embed_dim=self.embed_dim,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=True,
            bias=False,
            is_causal=True,
            config=config,
        )
        self.dropout = config.dropout
        self.activation_fn = ACT2FN[config.activation_function]
        self.activation_dropout = config.activation_dropout

        self.self_attn_layer_norm = nn.LayerNorm(self.embed_dim)

        self.fc1 = nn.Linear(self.embed_dim, config.ffn_dim, bias=False)
        self.fc2 = nn.Linear(config.ffn_dim, self.embed_dim, bias=False)
        self.final_layer_norm = nn.LayerNorm(self.embed_dim)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        layer_head_mask: Optional[mindspore.Tensor] = None,
        past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
        output_attentions: Optional[bool] = False,
        use_cache: Optional[bool] = True,
    ) -> mindspore.Tensor:
        """
        Args:
            hidden_states (`mindspore.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
            attention_mask (`mindspore.Tensor`): attention mask of size
                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
            layer_head_mask (`mindspore.Tensor`): mask for attention heads in a given layer of size `(attention_heads,)`.
            past_key_value (`Tuple(mindspore.Tensor)`): cached past key and value projection states
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
                returned tensors for more detail.
        """
        residual = hidden_states
        hidden_states = self.self_attn_layer_norm(hidden_states)

        # Self Attention
        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
        # add present self-attn cache to positions 1,2 of present_key_value tuple
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            past_key_value=self_attn_past_key_value,
            attention_mask=attention_mask,
            layer_head_mask=layer_head_mask,
            output_attentions=output_attentions,
        )
        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.final_layer_norm(hidden_states)
        hidden_states = self.activation_fn(self.fc1(hidden_states))
        hidden_states = nn.functional.dropout(hidden_states, p=self.activation_dropout, training=self.training)
        hidden_states = self.fc2(hidden_states)
        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)

        if output_attentions:
            outputs += (self_attn_weights,)

        if use_cache:
            outputs += (present_key_value,)

        return outputs

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyDecoderLayer.forward(hidden_states, attention_mask=None, layer_head_mask=None, past_key_value=None, output_attentions=False, use_cache=True)

PARAMETER	DESCRIPTION
hidden_states	input to the layer of shape (batch, seq_len, embed_dim) TYPE: `mindspore.Tensor`
attention_mask	attention mask of size (batch, 1, tgt_len, src_len) where padding elements are indicated by very large negative values. TYPE: `mindspore.Tensor` DEFAULT: None
layer_head_mask	mask for attention heads in a given layer of size (attention_heads,). TYPE: `mindspore.Tensor` DEFAULT: None
past_key_value	cached past key and value projection states TYPE: `Tuple(mindspore.Tensor)` DEFAULT: None
output_attentions	Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail. TYPE: `bool`, *optional* DEFAULT: False

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    layer_head_mask: Optional[mindspore.Tensor] = None,
    past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
    output_attentions: Optional[bool] = False,
    use_cache: Optional[bool] = True,
) -> mindspore.Tensor:
    """
    Args:
        hidden_states (`mindspore.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
        attention_mask (`mindspore.Tensor`): attention mask of size
            `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
        layer_head_mask (`mindspore.Tensor`): mask for attention heads in a given layer of size `(attention_heads,)`.
        past_key_value (`Tuple(mindspore.Tensor)`): cached past key and value projection states
        output_attentions (`bool`, *optional*):
            Whether or not to return the attentions tensors of all attention layers. See `attentions` under
            returned tensors for more detail.
    """
    residual = hidden_states
    hidden_states = self.self_attn_layer_norm(hidden_states)

    # Self Attention
    # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
    self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
    # add present self-attn cache to positions 1,2 of present_key_value tuple
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
        hidden_states=hidden_states,
        past_key_value=self_attn_past_key_value,
        attention_mask=attention_mask,
        layer_head_mask=layer_head_mask,
        output_attentions=output_attentions,
    )
    hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)
    hidden_states = residual + hidden_states

    # Fully Connected
    residual = hidden_states
    hidden_states = self.final_layer_norm(hidden_states)
    hidden_states = self.activation_fn(self.fc1(hidden_states))
    hidden_states = nn.functional.dropout(hidden_states, p=self.activation_dropout, training=self.training)
    hidden_states = self.fc2(hidden_states)
    hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)
    hidden_states = residual + hidden_states

    outputs = (hidden_states,)

    if output_attentions:
        outputs += (self_attn_weights,)

    if use_cache:
        outputs += (present_key_value,)

    return outputs

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForCausalLM

Bases: MusicgenMelodyPreTrainedModel

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyForCausalLM(MusicgenMelodyPreTrainedModel):
    def __init__(self, config: MusicgenMelodyDecoderConfig):
        super().__init__(config)

        self.model = MusicgenMelodyModel(config)

        self.num_codebooks = config.num_codebooks
        self.lm_heads = nn.ModuleList(
            [nn.Linear(config.hidden_size, config.vocab_size, bias=False) for _ in range(config.num_codebooks)]
        )

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        return self.model.decoder.embed_tokens

    def set_input_embeddings(self, value):
        self.model.decoder.embed_tokens = value

    def get_output_embeddings(self):
        return self.lm_heads

    def set_output_embeddings(self, new_embeddings):
        self.lm_heads = new_embeddings

    def set_decoder(self, decoder):
        self.model.decoder = decoder

    def get_decoder(self):
        return self.model.decoder

    # Ignore copy
    def forward(
        self,
        input_ids: mindspore.Tensor = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
    ) -> Union[Tuple, MusicgenMelodyOutputWithPast]:
        r"""
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length, num_codebooks)`, *optional*):
            Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
            `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
            are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
        Returns:
        """

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if (labels is not None) and (input_ids is None and inputs_embeds is None):
            input_ids = shift_tokens_right(labels, self.config.pad_token_id, self.config.bos_token_id)

        outputs = self.model(
            input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
            head_mask=head_mask,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]

        lm_logits = ops.stack([head(hidden_states) for head in self.lm_heads], dim=1)

        loss = None
        if labels is not None:
            # since encoder hidden states have been concatenated to the decoder hidden states,
            # we take the last timestamps corresponding to labels
            logits = lm_logits[:, :, -labels.shape[1] :]

            loss_fct = CrossEntropyLoss()
            loss = ops.zeros([])

            # per codebook cross-entropy
            # ref: https://github.com/facebookresearch/audiocraft/blob/69fea8b290ad1b4b40d28f92d1dfc0ab01dbab85/audiocraft/solvers/musicgen.py#L242-L243
            # -100 labels are ignored
            labels = labels.masked_fill(labels == self.config.pad_token_id, -100)

            # per codebook cross-entropy
            for codebook in range(self.config.num_codebooks):
                codebook_logits = logits[:, codebook].view(-1, logits.shape[-1])
                codebook_labels = labels[..., codebook].view(-1)
                loss += loss_fct(codebook_logits, codebook_labels)

            loss = loss / self.config.num_codebooks

        # (bsz, num_codebooks, seq_len, vocab_size) -> (bsz * num_codebooks, seq_len, vocab_size)
        lm_logits = lm_logits.reshape(-1, *lm_logits.shape[2:])

        if not return_dict:
            output = (lm_logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return MusicgenMelodyOutputWithPast(
            loss=loss,
            logits=lm_logits,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

    # Ignore copy
    def prepare_inputs_for_generation(
        self,
        input_ids,
        attention_mask=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
        head_mask=None,
        past_key_values=None,
        use_cache=True,
        delay_pattern_mask=None,
        guidance_scale=None,
        **kwargs,
    ):
        if delay_pattern_mask is None:
            input_ids, delay_pattern_mask = self.build_delay_pattern_mask(
                input_ids,
                pad_token_id=self.generation_config.pad_token_id,
                max_length=self.generation_config.max_length,
            )

        # apply the delay pattern mask
        input_ids = self.apply_delay_pattern_mask(input_ids, delay_pattern_mask)

        if guidance_scale is not None and guidance_scale > 1:
            # for classifier free guidance we need to replicate the decoder args across the batch dim (we'll split these
            # before sampling)
            input_ids = input_ids.tile((2, 1))
            if attention_mask is not None:
                attention_mask = attention_mask.tile((2, 1))

            if encoder_hidden_states is not None:
                encoder_hidden_states = ops.concatenate(
                    [encoder_hidden_states, ops.zeros_like(encoder_hidden_states)], dim=0
                )

            if encoder_attention_mask is not None:
                encoder_attention_mask = ops.concatenate(
                    encoder_attention_mask, ops.zeros_like(encoder_attention_mask), dim=0
                )

        if past_key_values is not None:
            input_ids = input_ids[:, -1:]

            # we only want to use conditional signal in the 1st generation step but keeping the attention mask
            encoder_hidden_states = None

        return {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "encoder_hidden_states": encoder_hidden_states,
            "encoder_attention_mask": encoder_attention_mask,
            "head_mask": head_mask,
            "past_key_values": past_key_values,
            "use_cache": use_cache,
        }

    def build_delay_pattern_mask(self, input_ids: mindspore.Tensor, pad_token_id: int, max_length: int = None):
        """Build a delayed pattern mask to the input_ids. Each codebook is offset by the previous codebook by
        one, giving a delayed pattern mask at the start of sequence and end of sequence. Take the example where there
        are 4 codebooks and a max sequence length of 8, we have the delayed pattern mask of shape `(codebooks,
        seq_len)`:
        - [P, -1, -1, -1, -1, P, P, P]
        - [P, P, -1, -1, -1, -1, P, P]
        - [P, P, P, -1, -1, -1, -1, P]
        - [P, P, P, P, -1, -1, -1, -1]
        where P is the special padding token id and -1 indicates that the token is valid for prediction. If we include
        a prompt (decoder input ids), the -1 positions indicate where new tokens should be predicted. Otherwise, the
        mask is set to the value in the prompt:
        - [P, a, b, -1, -1, P, P, P]
        - [P, P, c, d, -1, -1, P, P]
        - [P, P, P, e, f, -1, -1, P]
        - [P, P, P, P, g, h, -1, -1]
        where a-h indicate the input prompt (decoder input ids) that are offset by 1. Now, we only override the -1
        tokens in our prediction.
        """
        # (bsz * num_codebooks, seq_len) -> (bsz, num_codebooks, seq_len)
        input_ids = input_ids.reshape(-1, self.num_codebooks, input_ids.shape[-1])
        bsz, num_codebooks, seq_len = input_ids.shape

        max_length = max_length if max_length is not None else self.generation_config.max_length
        input_ids_shifted = (
            ops.ones((bsz, num_codebooks, max_length), dtype=mindspore.int64) * -1
        )

        channel_codebooks = num_codebooks // 2 if self.config.audio_channels == 2 else num_codebooks
        # we only apply the mask if we have a large enough seq len - otherwise we return as is
        if max_length < 2 * channel_codebooks - 1:
            return input_ids.reshape(bsz * num_codebooks, -1), input_ids_shifted.reshape(bsz * num_codebooks, -1)

        # fill the shifted ids with the prompt entries, offset by the codebook idx
        for codebook in range(channel_codebooks):
            if self.config.audio_channels == 1:
                # mono channel - loop over the codebooks one-by-one
                input_ids_shifted[:, codebook, codebook : seq_len + codebook] = input_ids[:, codebook]
            else:
                # left/right channels are interleaved in the generated codebooks, so handle one then the other
                input_ids_shifted[:, 2 * codebook, codebook : seq_len + codebook] = input_ids[:, 2 * codebook]
                input_ids_shifted[:, 2 * codebook + 1, codebook : seq_len + codebook] = input_ids[:, 2 * codebook + 1]

        # construct a pattern mask that indicates the positions of padding tokens for each codebook
        # first fill the upper triangular part (the EOS padding)
        delay_pattern = ops.triu(
            ops.ones((channel_codebooks, max_length)), diagonal=max_length - channel_codebooks + 1
        )
        # then fill the lower triangular part (the BOS padding)
        delay_pattern = delay_pattern + ops.tril(ops.ones((channel_codebooks, max_length)))

        if self.config.audio_channels == 2:
            # for left/right channel we need to duplicate every row of the pattern mask in an interleaved fashion
            delay_pattern = ops.repeat_interleave(delay_pattern, 2, dim=0)

        mask = ~(delay_pattern.bool())
        input_ids = mask * input_ids_shifted + ~mask * pad_token_id

        # find the first position to start generating - this is the first place we have the -1 token
        # and will always be in the first codebook (since it has no codebook offset)
        first_codebook_ids = input_ids[:, 0, :]
        start_ids = ops.nonzero((first_codebook_ids == -1))[:, 1]
        if len(start_ids) > 0:
            first_start_id = min(start_ids)
        else:
            # we have no tokens that need to be filled - return entire matrix of input ids
            first_start_id = seq_len

        # (bsz * num_codebooks, seq_len) -> (bsz, num_codebooks, seq_len)
        pattern_mask = input_ids.reshape(bsz * num_codebooks, -1)
        input_ids = input_ids[..., :first_start_id].reshape(bsz * num_codebooks, -1)
        return input_ids, pattern_mask

    @staticmethod
    def apply_delay_pattern_mask(input_ids, decoder_pad_token_mask):
        """Apply a delay pattern mask to the decoder input ids, only preserving predictions where
        the mask is set to -1, and otherwise setting to the value detailed in the mask."""
        seq_len = input_ids.shape[-1]
        decoder_pad_token_mask = decoder_pad_token_mask[..., :seq_len]
        input_ids = ops.where(decoder_pad_token_mask == -1, input_ids, decoder_pad_token_mask)
        return input_ids

    @no_grad()
    # Ignore copy
    def generate(
        self,
        inputs: Optional[mindspore.Tensor] = None,
        generation_config: Optional[GenerationConfig] = None,
        logits_processor: Optional[LogitsProcessorList] = None,
        stopping_criteria: Optional[StoppingCriteriaList] = None,
        synced_gpus: Optional[bool] = None,
        streamer: Optional["BaseStreamer"] = None,
        **kwargs,
    ):
        """

        Generates sequences of token ids for models with a language modeling head.

        <Tip warning={true}>

        Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the
        model's default generation configuration. You can override any `generation_config` by passing the corresponding
        parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`.

        For an overview of generation strategies and code examples, check out the [following
        guide](./generation_strategies).

        </Tip>

        Parameters:
            inputs (`mindspore.Tensor` of varying shape depending on the modality, *optional*):
                The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
                method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
                should be in the format `input_ids`. For encoder-decoder models *inputs* can represent any of
                `input_ids`, `input_values`, `input_features`, or `pixel_values`.
            generation_config (`~generation.GenerationConfig`, *optional*):
                The generation configuration to be used as base parametrization for the generation call. `**kwargs`
                passed to generate matching the attributes of `generation_config` will override them. If
                `generation_config` is not provided, the default will be used, which had the following loading
                priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model
                configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s
                default values, whose documentation should be checked to parameterize generation.
            logits_processor (`LogitsProcessorList`, *optional*):
                Custom logits processors that complement the default logits processors built from arguments and
                generation config. If a logit processor is passed that is already created with the arguments or a
                generation config an error is thrown. This feature is intended for advanced users.
            stopping_criteria (`StoppingCriteriaList`, *optional*):
                Custom stopping criteria that complement the default stopping criteria built from arguments and a
                generation config. If a stopping criteria is passed that is already created with the arguments or a
                generation config an error is thrown. This feature is intended for advanced users.
            synced_gpus (`bool`, *optional*, defaults to `False`):
                Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
            streamer (`BaseStreamer`, *optional*):
                Streamer object that will be used to stream the generated sequences. Generated tokens are passed
                through `streamer.put(token_ids)` and the streamer is responsible for any further processing.
            kwargs (`Dict[str, Any]`, *optional*):
                Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be
                forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
                specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.

        Return:
            [`~utils.ModelOutput`] or `mindspore.Tensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True`
            or when `config.return_dict_in_generate=True`) or a `mindspore.Tensor`.

                If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
                [`~utils.ModelOutput`] types are:

                    - [`~generation.GenerateDecoderOnlyOutput`],
                    - [`~generation.GenerateBeamDecoderOnlyOutput`]

                If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
                [`~utils.ModelOutput`] types are:

                    - [`~generation.GenerateEncoderDecoderOutput`],
                    - [`~generation.GenerateBeamEncoderDecoderOutput`]
        """
        # 1. Handle `generation_config` and kwargs that might update it, and validate the resulting objects
        if generation_config is None:
            generation_config = self.generation_config

        generation_config = copy.deepcopy(generation_config)
        model_kwargs = generation_config.update(**kwargs)  # All unused kwargs must be model kwargs
        generation_config.validate()
        self._validate_model_kwargs(model_kwargs.copy())

        # 2. Set generation parameters if not already defined
        logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
        stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()

        requires_attention_mask = "encoder_outputs" not in model_kwargs
        kwargs_has_attention_mask = model_kwargs.get("attention_mask", None) is not None

        # 3. Define model inputs`
        input_ids, model_input_name, model_kwargs = self._prepare_model_inputs(
            inputs, generation_config.bos_token_id, model_kwargs
        )
        batch_size = input_ids.shape[0] // self.num_codebooks
        self._prepare_special_tokens(generation_config, kwargs_has_attention_mask)

        # 4. Define other model kwargs
        model_kwargs["use_cache"] = generation_config.use_cache
        model_kwargs["guidance_scale"] = generation_config.guidance_scale

        if model_kwargs.get("attention_mask", None) is None and requires_attention_mask:
            model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
                input_ids, generation_config._pad_token_tensor, generation_config._eos_token_tensor
            )

        # 5. Prepare `max_length` depending on other stopping criteria.
        input_ids_length = input_ids.shape[-1]
        has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
        has_default_min_length = kwargs.get("min_length") is None and generation_config.min_length is not None
        generation_config = self._prepare_generated_length(
            generation_config=generation_config,
            has_default_max_length=has_default_max_length,
            has_default_min_length=has_default_min_length,
            model_input_name=model_input_name,
            inputs_tensor=input_ids,
            input_ids_length=input_ids_length,
        )

        # 6. Prepare `input_ids` which will be used for auto-regressive generation
        # Build the delay pattern mask for offsetting each codebook prediction by 1 (this behaviour is specific to Musicgen)
        input_ids, delay_pattern_mask = self.build_delay_pattern_mask(
            input_ids,
            pad_token_id=generation_config._decoder_start_token_tensor,
            max_length=generation_config.max_length,
        )

        if streamer is not None:
            streamer.put(input_ids.cpu())

        # stash the delay mask so that we don't have to recompute it in each forward pass
        model_kwargs["delay_pattern_mask"] = delay_pattern_mask

        # 7. determine generation mode
        generation_mode = generation_config.get_generation_mode()

        # 8. prepare batched CFG externally (to enable coexistance with the unbatched CFG)
        if generation_config.guidance_scale is not None and generation_config.guidance_scale > 1:
            logits_processor.append(ClassifierFreeGuidanceLogitsProcessor(generation_config.guidance_scale))
            generation_config.guidance_scale = None

        # 9. prepare distribution pre_processing samplers
        logits_processor = self._get_logits_processor(
            generation_config=generation_config,
            input_ids_seq_length=input_ids_length,
            encoder_input_ids=input_ids,
            prefix_allowed_tokens_fn=None,
            logits_processor=logits_processor,
        )

        # 10. prepare stopping criteria
        stopping_criteria = self._get_stopping_criteria(
            generation_config=generation_config, stopping_criteria=stopping_criteria
        )

        if generation_mode in (GenerationMode.SAMPLE, GenerationMode.GREEDY_SEARCH):
            # expand input_ids with `num_return_sequences` additional sequences per batch
            input_ids, model_kwargs = self._expand_inputs_for_generation(
                input_ids=input_ids,
                expand_size=generation_config.num_return_sequences,
                **model_kwargs,
            )

            # 11. run sample
            outputs = self._sample(
                input_ids,
                logits_processor=logits_processor,
                stopping_criteria=stopping_criteria,
                generation_config=generation_config,
                synced_gpus=synced_gpus,
                streamer=streamer,
                **model_kwargs,
            )

        else:
            raise ValueError(
                "Got incompatible mode for generation, should be one of greedy or sampling. "
                "Ensure that beam search is de-activated by setting `num_beams=1` and `num_beam_groups=1`."
            )

        if generation_config.return_dict_in_generate:
            output_ids = outputs.sequences
        else:
            output_ids = outputs

        # apply the pattern mask to the final ids
        output_ids = self.apply_delay_pattern_mask(output_ids, model_kwargs["delay_pattern_mask"])

        # revert the pattern delay mask by filtering the pad token id
        output_ids = output_ids[output_ids != generation_config._pad_token_tensor].reshape(
            batch_size, self.num_codebooks, -1
        )

        if generation_config.return_dict_in_generate:
            outputs.sequences = output_ids
            return outputs
        else:
            return output_ids

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForCausalLM.apply_delay_pattern_mask(input_ids, decoder_pad_token_mask) staticmethod

Apply a delay pattern mask to the decoder input ids, only preserving predictions where the mask is set to -1, and otherwise setting to the value detailed in the mask.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@staticmethod
def apply_delay_pattern_mask(input_ids, decoder_pad_token_mask):
    """Apply a delay pattern mask to the decoder input ids, only preserving predictions where
    the mask is set to -1, and otherwise setting to the value detailed in the mask."""
    seq_len = input_ids.shape[-1]
    decoder_pad_token_mask = decoder_pad_token_mask[..., :seq_len]
    input_ids = ops.where(decoder_pad_token_mask == -1, input_ids, decoder_pad_token_mask)
    return input_ids

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForCausalLM.build_delay_pattern_mask(input_ids, pad_token_id, max_length=None)

Build a delayed pattern mask to the input_ids. Each codebook is offset by the previous codebook by one, giving a delayed pattern mask at the start of sequence and end of sequence. Take the example where there are 4 codebooks and a max sequence length of 8, we have the delayed pattern mask of shape (codebooks, seq_len): - [P, -1, -1, -1, -1, P, P, P] - [P, P, -1, -1, -1, -1, P, P] - [P, P, P, -1, -1, -1, -1, P] - [P, P, P, P, -1, -1, -1, -1] where P is the special padding token id and -1 indicates that the token is valid for prediction. If we include a prompt (decoder input ids), the -1 positions indicate where new tokens should be predicted. Otherwise, the mask is set to the value in the prompt: - [P, a, b, -1, -1, P, P, P] - [P, P, c, d, -1, -1, P, P] - [P, P, P, e, f, -1, -1, P] - [P, P, P, P, g, h, -1, -1] where a-h indicate the input prompt (decoder input ids) that are offset by 1. Now, we only override the -1 tokens in our prediction.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def build_delay_pattern_mask(self, input_ids: mindspore.Tensor, pad_token_id: int, max_length: int = None):
    """Build a delayed pattern mask to the input_ids. Each codebook is offset by the previous codebook by
    one, giving a delayed pattern mask at the start of sequence and end of sequence. Take the example where there
    are 4 codebooks and a max sequence length of 8, we have the delayed pattern mask of shape `(codebooks,
    seq_len)`:
    - [P, -1, -1, -1, -1, P, P, P]
    - [P, P, -1, -1, -1, -1, P, P]
    - [P, P, P, -1, -1, -1, -1, P]
    - [P, P, P, P, -1, -1, -1, -1]
    where P is the special padding token id and -1 indicates that the token is valid for prediction. If we include
    a prompt (decoder input ids), the -1 positions indicate where new tokens should be predicted. Otherwise, the
    mask is set to the value in the prompt:
    - [P, a, b, -1, -1, P, P, P]
    - [P, P, c, d, -1, -1, P, P]
    - [P, P, P, e, f, -1, -1, P]
    - [P, P, P, P, g, h, -1, -1]
    where a-h indicate the input prompt (decoder input ids) that are offset by 1. Now, we only override the -1
    tokens in our prediction.
    """
    # (bsz * num_codebooks, seq_len) -> (bsz, num_codebooks, seq_len)
    input_ids = input_ids.reshape(-1, self.num_codebooks, input_ids.shape[-1])
    bsz, num_codebooks, seq_len = input_ids.shape

    max_length = max_length if max_length is not None else self.generation_config.max_length
    input_ids_shifted = (
        ops.ones((bsz, num_codebooks, max_length), dtype=mindspore.int64) * -1
    )

    channel_codebooks = num_codebooks // 2 if self.config.audio_channels == 2 else num_codebooks
    # we only apply the mask if we have a large enough seq len - otherwise we return as is
    if max_length < 2 * channel_codebooks - 1:
        return input_ids.reshape(bsz * num_codebooks, -1), input_ids_shifted.reshape(bsz * num_codebooks, -1)

    # fill the shifted ids with the prompt entries, offset by the codebook idx
    for codebook in range(channel_codebooks):
        if self.config.audio_channels == 1:
            # mono channel - loop over the codebooks one-by-one
            input_ids_shifted[:, codebook, codebook : seq_len + codebook] = input_ids[:, codebook]
        else:
            # left/right channels are interleaved in the generated codebooks, so handle one then the other
            input_ids_shifted[:, 2 * codebook, codebook : seq_len + codebook] = input_ids[:, 2 * codebook]
            input_ids_shifted[:, 2 * codebook + 1, codebook : seq_len + codebook] = input_ids[:, 2 * codebook + 1]

    # construct a pattern mask that indicates the positions of padding tokens for each codebook
    # first fill the upper triangular part (the EOS padding)
    delay_pattern = ops.triu(
        ops.ones((channel_codebooks, max_length)), diagonal=max_length - channel_codebooks + 1
    )
    # then fill the lower triangular part (the BOS padding)
    delay_pattern = delay_pattern + ops.tril(ops.ones((channel_codebooks, max_length)))

    if self.config.audio_channels == 2:
        # for left/right channel we need to duplicate every row of the pattern mask in an interleaved fashion
        delay_pattern = ops.repeat_interleave(delay_pattern, 2, dim=0)

    mask = ~(delay_pattern.bool())
    input_ids = mask * input_ids_shifted + ~mask * pad_token_id

    # find the first position to start generating - this is the first place we have the -1 token
    # and will always be in the first codebook (since it has no codebook offset)
    first_codebook_ids = input_ids[:, 0, :]
    start_ids = ops.nonzero((first_codebook_ids == -1))[:, 1]
    if len(start_ids) > 0:
        first_start_id = min(start_ids)
    else:
        # we have no tokens that need to be filled - return entire matrix of input ids
        first_start_id = seq_len

    # (bsz * num_codebooks, seq_len) -> (bsz, num_codebooks, seq_len)
    pattern_mask = input_ids.reshape(bsz * num_codebooks, -1)
    input_ids = input_ids[..., :first_start_id].reshape(bsz * num_codebooks, -1)
    return input_ids, pattern_mask

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForCausalLM.forward(input_ids=None, attention_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, head_mask=None, past_key_values=None, inputs_embeds=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

labels (mindspore.Tensor of shape (batch_size, sequence_length, num_codebooks), optional): Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size] All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size] Returns:

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def forward(
    self,
    input_ids: mindspore.Tensor = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
) -> Union[Tuple, MusicgenMelodyOutputWithPast]:
    r"""
    labels (`mindspore.Tensor` of shape `(batch_size, sequence_length, num_codebooks)`, *optional*):
        Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
        `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
        are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
    Returns:
    """

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if (labels is not None) and (input_ids is None and inputs_embeds is None):
        input_ids = shift_tokens_right(labels, self.config.pad_token_id, self.config.bos_token_id)

    outputs = self.model(
        input_ids,
        attention_mask=attention_mask,
        encoder_hidden_states=encoder_hidden_states,
        encoder_attention_mask=encoder_attention_mask,
        head_mask=head_mask,
        past_key_values=past_key_values,
        inputs_embeds=inputs_embeds,
        use_cache=use_cache,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = outputs[0]

    lm_logits = ops.stack([head(hidden_states) for head in self.lm_heads], dim=1)

    loss = None
    if labels is not None:
        # since encoder hidden states have been concatenated to the decoder hidden states,
        # we take the last timestamps corresponding to labels
        logits = lm_logits[:, :, -labels.shape[1] :]

        loss_fct = CrossEntropyLoss()
        loss = ops.zeros([])

        # per codebook cross-entropy
        # ref: https://github.com/facebookresearch/audiocraft/blob/69fea8b290ad1b4b40d28f92d1dfc0ab01dbab85/audiocraft/solvers/musicgen.py#L242-L243
        # -100 labels are ignored
        labels = labels.masked_fill(labels == self.config.pad_token_id, -100)

        # per codebook cross-entropy
        for codebook in range(self.config.num_codebooks):
            codebook_logits = logits[:, codebook].view(-1, logits.shape[-1])
            codebook_labels = labels[..., codebook].view(-1)
            loss += loss_fct(codebook_logits, codebook_labels)

        loss = loss / self.config.num_codebooks

    # (bsz, num_codebooks, seq_len, vocab_size) -> (bsz * num_codebooks, seq_len, vocab_size)
    lm_logits = lm_logits.reshape(-1, *lm_logits.shape[2:])

    if not return_dict:
        output = (lm_logits,) + outputs[1:]
        return ((loss,) + output) if loss is not None else output

    return MusicgenMelodyOutputWithPast(
        loss=loss,
        logits=lm_logits,
        past_key_values=outputs.past_key_values,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForCausalLM.generate(inputs=None, generation_config=None, logits_processor=None, stopping_criteria=None, synced_gpus=None, streamer=None, **kwargs)

Generates sequences of token ids for models with a language modeling head.

Most generation-controlling parameters are set in generation_config which, if not passed, will be set to the model's default generation configuration. You can override any generation_config by passing the corresponding parameters to generate(), e.g. .generate(inputs, num_beams=4, do_sample=True).

For an overview of generation strategies and code examples, check out the following guide.

PARAMETER	DESCRIPTION
inputs	The sequence used as a prompt for the generation or as model inputs to the encoder. If None the method initializes it with bos_token_id and a batch size of 1. For decoder-only models inputs should be in the format input_ids. For encoder-decoder models inputs can represent any of input_ids, input_values, input_features, or pixel_values. TYPE: `mindspore.Tensor` of varying shape depending on the modality, *optional* DEFAULT: None
generation_config	The generation configuration to be used as base parametrization for the generation call. **kwargs passed to generate matching the attributes of generation_config will override them. If generation_config is not provided, the default will be used, which had the following loading priority: 1) from the generation_config.json model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [~generation.GenerationConfig]'s default values, whose documentation should be checked to parameterize generation. TYPE: `~generation.GenerationConfig`, *optional* DEFAULT: None
logits_processor	Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. TYPE: `LogitsProcessorList`, *optional* DEFAULT: None
stopping_criteria	Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. TYPE: `StoppingCriteriaList`, *optional* DEFAULT: None
synced_gpus	Whether to continue running the while loop until max_length (needed for ZeRO stage 3) TYPE: `bool`, *optional*, defaults to `False` DEFAULT: None
streamer	Streamer object that will be used to stream the generated sequences. Generated tokens are passed through streamer.put(token_ids) and the streamer is responsible for any further processing. TYPE: `BaseStreamer`, *optional* DEFAULT: None
kwargs	Ad hoc parametrization of generate_config and/or additional model-specific kwargs that will be forwarded to the forward function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_. TYPE: `Dict[str, Any]`, *optional* DEFAULT: {}

Return

[~utils.ModelOutput] or mindspore.Tensor: A [~utils.ModelOutput] (if return_dict_in_generate=True or when config.return_dict_in_generate=True) or a mindspore.Tensor.

If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
[`~utils.ModelOutput`] types are:

    - [`~generation.GenerateDecoderOnlyOutput`],
    - [`~generation.GenerateBeamDecoderOnlyOutput`]

If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[`~utils.ModelOutput`] types are:

    - [`~generation.GenerateEncoderDecoderOutput`],
    - [`~generation.GenerateBeamEncoderDecoderOutput`]

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@no_grad()
# Ignore copy
def generate(
    self,
    inputs: Optional[mindspore.Tensor] = None,
    generation_config: Optional[GenerationConfig] = None,
    logits_processor: Optional[LogitsProcessorList] = None,
    stopping_criteria: Optional[StoppingCriteriaList] = None,
    synced_gpus: Optional[bool] = None,
    streamer: Optional["BaseStreamer"] = None,
    **kwargs,
):
    """

    Generates sequences of token ids for models with a language modeling head.

    <Tip warning={true}>

    Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the
    model's default generation configuration. You can override any `generation_config` by passing the corresponding
    parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`.

    For an overview of generation strategies and code examples, check out the [following
    guide](./generation_strategies).

    </Tip>

    Parameters:
        inputs (`mindspore.Tensor` of varying shape depending on the modality, *optional*):
            The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
            method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
            should be in the format `input_ids`. For encoder-decoder models *inputs* can represent any of
            `input_ids`, `input_values`, `input_features`, or `pixel_values`.
        generation_config (`~generation.GenerationConfig`, *optional*):
            The generation configuration to be used as base parametrization for the generation call. `**kwargs`
            passed to generate matching the attributes of `generation_config` will override them. If
            `generation_config` is not provided, the default will be used, which had the following loading
            priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model
            configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s
            default values, whose documentation should be checked to parameterize generation.
        logits_processor (`LogitsProcessorList`, *optional*):
            Custom logits processors that complement the default logits processors built from arguments and
            generation config. If a logit processor is passed that is already created with the arguments or a
            generation config an error is thrown. This feature is intended for advanced users.
        stopping_criteria (`StoppingCriteriaList`, *optional*):
            Custom stopping criteria that complement the default stopping criteria built from arguments and a
            generation config. If a stopping criteria is passed that is already created with the arguments or a
            generation config an error is thrown. This feature is intended for advanced users.
        synced_gpus (`bool`, *optional*, defaults to `False`):
            Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
        streamer (`BaseStreamer`, *optional*):
            Streamer object that will be used to stream the generated sequences. Generated tokens are passed
            through `streamer.put(token_ids)` and the streamer is responsible for any further processing.
        kwargs (`Dict[str, Any]`, *optional*):
            Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be
            forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
            specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.

    Return:
        [`~utils.ModelOutput`] or `mindspore.Tensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True`
        or when `config.return_dict_in_generate=True`) or a `mindspore.Tensor`.

            If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
            [`~utils.ModelOutput`] types are:

                - [`~generation.GenerateDecoderOnlyOutput`],
                - [`~generation.GenerateBeamDecoderOnlyOutput`]

            If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
            [`~utils.ModelOutput`] types are:

                - [`~generation.GenerateEncoderDecoderOutput`],
                - [`~generation.GenerateBeamEncoderDecoderOutput`]
    """
    # 1. Handle `generation_config` and kwargs that might update it, and validate the resulting objects
    if generation_config is None:
        generation_config = self.generation_config

    generation_config = copy.deepcopy(generation_config)
    model_kwargs = generation_config.update(**kwargs)  # All unused kwargs must be model kwargs
    generation_config.validate()
    self._validate_model_kwargs(model_kwargs.copy())

    # 2. Set generation parameters if not already defined
    logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
    stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()

    requires_attention_mask = "encoder_outputs" not in model_kwargs
    kwargs_has_attention_mask = model_kwargs.get("attention_mask", None) is not None

    # 3. Define model inputs`
    input_ids, model_input_name, model_kwargs = self._prepare_model_inputs(
        inputs, generation_config.bos_token_id, model_kwargs
    )
    batch_size = input_ids.shape[0] // self.num_codebooks
    self._prepare_special_tokens(generation_config, kwargs_has_attention_mask)

    # 4. Define other model kwargs
    model_kwargs["use_cache"] = generation_config.use_cache
    model_kwargs["guidance_scale"] = generation_config.guidance_scale

    if model_kwargs.get("attention_mask", None) is None and requires_attention_mask:
        model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
            input_ids, generation_config._pad_token_tensor, generation_config._eos_token_tensor
        )

    # 5. Prepare `max_length` depending on other stopping criteria.
    input_ids_length = input_ids.shape[-1]
    has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
    has_default_min_length = kwargs.get("min_length") is None and generation_config.min_length is not None
    generation_config = self._prepare_generated_length(
        generation_config=generation_config,
        has_default_max_length=has_default_max_length,
        has_default_min_length=has_default_min_length,
        model_input_name=model_input_name,
        inputs_tensor=input_ids,
        input_ids_length=input_ids_length,
    )

    # 6. Prepare `input_ids` which will be used for auto-regressive generation
    # Build the delay pattern mask for offsetting each codebook prediction by 1 (this behaviour is specific to Musicgen)
    input_ids, delay_pattern_mask = self.build_delay_pattern_mask(
        input_ids,
        pad_token_id=generation_config._decoder_start_token_tensor,
        max_length=generation_config.max_length,
    )

    if streamer is not None:
        streamer.put(input_ids.cpu())

    # stash the delay mask so that we don't have to recompute it in each forward pass
    model_kwargs["delay_pattern_mask"] = delay_pattern_mask

    # 7. determine generation mode
    generation_mode = generation_config.get_generation_mode()

    # 8. prepare batched CFG externally (to enable coexistance with the unbatched CFG)
    if generation_config.guidance_scale is not None and generation_config.guidance_scale > 1:
        logits_processor.append(ClassifierFreeGuidanceLogitsProcessor(generation_config.guidance_scale))
        generation_config.guidance_scale = None

    # 9. prepare distribution pre_processing samplers
    logits_processor = self._get_logits_processor(
        generation_config=generation_config,
        input_ids_seq_length=input_ids_length,
        encoder_input_ids=input_ids,
        prefix_allowed_tokens_fn=None,
        logits_processor=logits_processor,
    )

    # 10. prepare stopping criteria
    stopping_criteria = self._get_stopping_criteria(
        generation_config=generation_config, stopping_criteria=stopping_criteria
    )

    if generation_mode in (GenerationMode.SAMPLE, GenerationMode.GREEDY_SEARCH):
        # expand input_ids with `num_return_sequences` additional sequences per batch
        input_ids, model_kwargs = self._expand_inputs_for_generation(
            input_ids=input_ids,
            expand_size=generation_config.num_return_sequences,
            **model_kwargs,
        )

        # 11. run sample
        outputs = self._sample(
            input_ids,
            logits_processor=logits_processor,
            stopping_criteria=stopping_criteria,
            generation_config=generation_config,
            synced_gpus=synced_gpus,
            streamer=streamer,
            **model_kwargs,
        )

    else:
        raise ValueError(
            "Got incompatible mode for generation, should be one of greedy or sampling. "
            "Ensure that beam search is de-activated by setting `num_beams=1` and `num_beam_groups=1`."
        )

    if generation_config.return_dict_in_generate:
        output_ids = outputs.sequences
    else:
        output_ids = outputs

    # apply the pattern mask to the final ids
    output_ids = self.apply_delay_pattern_mask(output_ids, model_kwargs["delay_pattern_mask"])

    # revert the pattern delay mask by filtering the pad token id
    output_ids = output_ids[output_ids != generation_config._pad_token_tensor].reshape(
        batch_size, self.num_codebooks, -1
    )

    if generation_config.return_dict_in_generate:
        outputs.sequences = output_ids
        return outputs
    else:
        return output_ids

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration

Bases: PreTrainedModel

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyForConditionalGeneration(PreTrainedModel):
    config_class = MusicgenMelodyConfig
    main_input_name = "input_ids"
    supports_gradient_checkpointing = True
    _supports_flash_attn_2 = True
    _supports_sdpa = True

    def __init__(
        self,
        config: MusicgenMelodyConfig = None,
        text_encoder: Optional[PreTrainedModel] = None,
        audio_encoder: Optional[PreTrainedModel] = None,
        decoder: Optional[MusicgenMelodyForCausalLM] = None,
    ):
        if config is None and None in (text_encoder, audio_encoder, decoder):
            raise ValueError(
                "Either a configuration has to be provided, or all three of text encoder, audio encoder and Musicgen Melody decoder."
            )
        if config is None:
            config = MusicgenMelodyConfig.from_sub_models_config(
                text_encoder.config, audio_encoder.config, decoder.config
            )
        else:
            if not isinstance(config, self.config_class):
                raise ValueError(f"Config: {config} has to be of type {self.config_class}")

        # initialize with config
        super().__init__(config)

        if text_encoder is None:
            text_encoder = AutoModelForTextEncoding.from_config(config.text_encoder)

        if audio_encoder is None:
            audio_encoder = AutoModel.from_config(config.audio_encoder)

        if decoder is None:
            decoder = MusicgenMelodyForCausalLM(config.decoder)

        self.text_encoder = text_encoder
        self.audio_encoder = audio_encoder
        self.decoder = decoder

        # make sure that the individual model's config refers to the shared config
        # so that the updates to the config will be synced
        self.text_encoder.config = self.config.text_encoder
        self.audio_encoder.config = self.config.audio_encoder
        self.decoder.config = self.config.decoder

        # text encoder outputs might need to be projected to different dimension for decoder
        if self.text_encoder.config.hidden_size != self.decoder.config.hidden_size:
            self.enc_to_dec_proj = nn.Linear(self.text_encoder.config.hidden_size, self.decoder.config.hidden_size)

        # audio encoder outputs after chroma extraction might need to be projected to different dimension for decoder
        if self.config.num_chroma != self.decoder.config.hidden_size:
            self.audio_enc_to_dec_proj = nn.Linear(self.config.num_chroma, self.decoder.config.hidden_size)

        if self.text_encoder.get_output_embeddings() is not None:
            raise ValueError(
                f"The encoder {self.text_encoder} should not have a LM Head. Please use a model without and LM Head"
            )

        # Initialize projection layers weights and tie text encoder and decoder weights if set accordingly
        self.post_init()

    def _init_weights(self, module):
        # MusicgenMelodyForConditionalGeneration is made of PreTrainedModels that have already been initialized
        # Projection layers still need to be initialized.
        std = self.decoder.config.initializer_factor
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0.0, std=std)
            if module.bias is not None:
                nn.init.zeros_(module.bias)

    def tie_weights(self):
        # tie text encoder & decoder if needed
        if self.config.tie_encoder_decoder:
            # tie text encoder and decoder base model
            decoder_base_model_prefix = self.decoder.base_model_prefix
            tied_weights = self._tie_encoder_decoder_weights(
                self.text_encoder,
                self.decoder._modules[decoder_base_model_prefix],
                self.decoder.base_model_prefix,
                "text_encoder",
            )
            # Setting a dynamic variable instead of `_tied_weights_keys` because it's a class
            # attributed not an instance member, therefore modifying it will modify the entire class
            # Leading to issues on subsequent calls by different tests or subsequent calls.
            self._dynamic_tied_weights_keys = tied_weights

    def get_text_encoder(self):
        return self.text_encoder

    def get_encoder(self):
        # get the text encoder to compute the conditionning hidden-states for generation
        return self.get_text_encoder()

    def get_decoder(self):
        return self.decoder

    def get_input_embeddings(self):
        return self.text_encoder.get_input_embeddings()

    def get_output_embeddings(self):
        return self.decoder.get_output_embeddings()

    def set_output_embeddings(self, new_embeddings):
        return self.decoder.set_output_embeddings(new_embeddings)

    @classmethod
    # Copied from transformers.models.musicgen.modeling_musicgen.MusicgenForConditionalGeneration.from_sub_models_pretrained with Musicgen->MusicgenMelody, musicgen-small->musicgen-melody
    def from_sub_models_pretrained( # pylint: disable=keyword-arg-before-vararg
        cls,
        text_encoder_pretrained_model_name_or_path: str = None,
        audio_encoder_pretrained_model_name_or_path: str = None,
        decoder_pretrained_model_name_or_path: str = None,
        *model_args,
        **kwargs,
    ) -> PreTrainedModel:
        r"""
        Instantiate a text encoder, an audio encoder, and a MusicGen decoder from one, two or three base classes of the
        library from pretrained model checkpoints.


        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
        the model, you need to first set it back in training mode with `model.train()`.

        Params:
            text_encoder_pretrained_model_name_or_path (`str`, *optional*):
                Information necessary to initiate the text encoder. Can be either:

                    - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                    - A path to a *directory* containing model weights saved using
                      [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

            audio_encoder_pretrained_model_name_or_path (`str`, *optional*):
                Information necessary to initiate the audio encoder. Can be either:

                    - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                    - A path to a *directory* containing model weights saved using
                      [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

            decoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`):
                Information necessary to initiate the decoder. Can be either:

                    - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                    - A path to a *directory* containing model weights saved using
                      [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

            model_args (remaining positional arguments, *optional*):
                All remaining positional arguments will be passed to the underlying model's `__init__` method.

            kwargs (remaining dictionary of keyword arguments, *optional*):
                Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
                `output_attentions=True`).

                - To update the text encoder configuration, use the prefix *text_encoder_* for each configuration
                  parameter.
                - To update the audio encoder configuration, use the prefix *audio_encoder_* for each configuration
                  parameter.
                - To update the decoder configuration, use the prefix *decoder_* for each configuration parameter.
                - To update the parent model configuration, do not use a prefix for each configuration parameter.

                Behaves differently depending on whether a `config` is provided or automatically loaded.

        Example:

        ```python
        >>> from transformers import MusicgenMelodyForConditionalGeneration

        >>> # initialize a musicgen model from a t5 text encoder, encodec audio encoder, and musicgen decoder
        >>> model = MusicgenMelodyForConditionalGeneration.from_sub_models_pretrained(
        ...     text_encoder_pretrained_model_name_or_path="google-t5/t5-base",
        ...     audio_encoder_pretrained_model_name_or_path="facebook/encodec_24khz",
        ...     decoder_pretrained_model_name_or_path="facebook/musicgen-melody",
        ... )
        >>> # saving model after fine-tuning
        >>> model.save_pretrained("./musicgen-ft")
        >>> # load fine-tuned model
        >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("./musicgen-ft")
        ```"""

        kwargs_text_encoder = {
            argument[len("text_encoder_") :]: value
            for argument, value in kwargs.items()
            if argument.startswith("text_encoder_")
        }

        kwargs_audio_encoder = {
            argument[len("audio_encoder_") :]: value
            for argument, value in kwargs.items()
            if argument.startswith("audio_encoder_")
        }

        kwargs_decoder = {
            argument[len("decoder_") :]: value for argument, value in kwargs.items() if argument.startswith("decoder_")
        }

        # remove text encoder, audio encoder and decoder kwargs from kwargs
        for key in kwargs_text_encoder.keys():
            del kwargs["text_encoder_" + key]
        for key in kwargs_audio_encoder.keys():
            del kwargs["audio_encoder_" + key]
        for key in kwargs_decoder.keys():
            del kwargs["decoder_" + key]

        # Load and initialize the encoder and decoder
        # The distinction between encoder and decoder at the model level is made
        # by the value of the flag `is_decoder` that we need to set correctly.
        text_encoder = kwargs_text_encoder.pop("model", None)
        if text_encoder is None:
            if text_encoder_pretrained_model_name_or_path is None:
                raise ValueError(
                    "If `text_encoder_model` is not defined as an argument, a `text_encoder_pretrained_model_name_or_path` has "
                    "to be defined."
                )

            if "config" not in kwargs_text_encoder:
                encoder_config, kwargs_text_encoder = AutoConfig.from_pretrained(
                    text_encoder_pretrained_model_name_or_path, **kwargs_text_encoder, return_unused_kwargs=True
                )

                if encoder_config.is_decoder is True or encoder_config.add_cross_attention is True:
                    logger.info(
                        f"Initializing {text_encoder_pretrained_model_name_or_path} as a text_encoder model "
                        "from a decoder model. Cross-attention and casual mask are disabled."
                    )
                    encoder_config.is_decoder = False
                    encoder_config.add_cross_attention = False

                kwargs_text_encoder["config"] = encoder_config

            text_encoder = AutoModel.from_pretrained(
                text_encoder_pretrained_model_name_or_path, *model_args, **kwargs_text_encoder
            )

        audio_encoder = kwargs_audio_encoder.pop("model", None)
        if audio_encoder is None:
            if audio_encoder_pretrained_model_name_or_path is None:
                raise ValueError(
                    "If `audio_encoder_model` is not defined as an argument, an `audio_encoder_pretrained_model_name_or_path` has "
                    "to be defined."
                )

            if "config" not in kwargs_audio_encoder:
                encoder_config, kwargs_audio_encoder = AutoConfig.from_pretrained(
                    audio_encoder_pretrained_model_name_or_path, **kwargs_audio_encoder, return_unused_kwargs=True
                )

                if encoder_config.is_decoder is True or encoder_config.add_cross_attention is True:
                    logger.info(
                        f"Initializing {audio_encoder_pretrained_model_name_or_path} as an audio_encoder model "
                        "from a decoder model. Cross-attention and casual mask are disabled."
                    )
                    encoder_config.is_decoder = False
                    encoder_config.add_cross_attention = False

                kwargs_audio_encoder["config"] = encoder_config

            audio_encoder = AutoModel.from_pretrained(
                audio_encoder_pretrained_model_name_or_path, *model_args, **kwargs_audio_encoder
            )

        decoder = kwargs_decoder.pop("model", None)
        if decoder is None:
            if decoder_pretrained_model_name_or_path is None:
                raise ValueError(
                    "If `decoder_model` is not defined as an argument, a `decoder_pretrained_model_name_or_path` has "
                    "to be defined."
                )

            if "config" not in kwargs_decoder:
                decoder_config, kwargs_decoder = AutoConfig.from_pretrained(
                    decoder_pretrained_model_name_or_path, **kwargs_decoder, return_unused_kwargs=True
                )

                if isinstance(decoder_config, MusicgenMelodyConfig):
                    decoder_config = decoder_config.decoder

                if decoder_config.is_decoder is False or decoder_config.add_cross_attention is False:
                    logger.info(
                        f"Initializing {decoder_pretrained_model_name_or_path} as a decoder model. Cross attention"
                        f" layers are added to {decoder_pretrained_model_name_or_path} and randomly initialized if"
                        f" {decoder_pretrained_model_name_or_path}'s architecture allows for cross attention layers."
                    )
                    decoder_config.is_decoder = True
                    decoder_config.add_cross_attention = True

                kwargs_decoder["config"] = decoder_config

            if kwargs_decoder["config"].is_decoder is False or kwargs_decoder["config"].add_cross_attention is False:
                logger.warning(
                    f"Decoder model {decoder_pretrained_model_name_or_path} is not initialized as a decoder. "
                    f"In order to initialize {decoder_pretrained_model_name_or_path} as a decoder, "
                    "make sure that the attributes `is_decoder` and `add_cross_attention` of `decoder_config` "
                    "passed to `.from_sub_models_pretrained(...)` are set to `True` or do not pass a "
                    "`decoder_config` to `.from_sub_models_pretrained(...)`"
                )

            decoder = MusicgenMelodyForCausalLM.from_pretrained(
                decoder_pretrained_model_name_or_path, **kwargs_decoder
            )

        # instantiate config with corresponding kwargs
        config = MusicgenMelodyConfig.from_sub_models_config(
            text_encoder.config, audio_encoder.config, decoder.config, **kwargs
        )
        return cls(text_encoder=text_encoder, audio_encoder=audio_encoder, decoder=decoder, config=config)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        input_features: Optional[mindspore.Tensor] = None,
        decoder_input_ids: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Tuple[Tuple[mindspore.Tensor]] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        decoder_inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        **kwargs,
    ) -> Union[Tuple, MusicgenMelodyOutputWithPast]:
        r"""
        Returns:

        Examples:
        ```python
        >>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration
        >>> import torch

        >>> processor = AutoProcessor.from_pretrained("facebook/musicgen-melody")
        >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")

        >>> inputs = processor(
        ...     text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
        ...     padding=True,
        ...     return_tensors="pt",
        ... )

        >>> pad_token_id = model.generation_config.pad_token_id
        >>> decoder_input_ids = (
        ...     ops.ones((inputs.input_ids.shape[0] * model.decoder.num_codebooks, 1), dtype=mindspore.int64)
        ...     * pad_token_id
        ... )

        >>> logits = model(**inputs, decoder_input_ids=decoder_input_ids).logits
        >>> logits.shape  # (bsz * num_codebooks, encoder_len + tgt_len, vocab_size)
        ops.Size([8, 249, 2048])
        ```"""

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        kwargs_text_encoder = {
            argument[len("text_encoder_")]: value
            for argument, value in kwargs.items()
            if argument.startswith("text_encoder_")
        }

        kwargs_decoder = {
            argument[len("decoder_") :]: value for argument, value in kwargs.items() if argument.startswith("decoder_")
        }

        if encoder_hidden_states is None:
            if inputs_embeds is not None or input_ids is not None:
                encoder_outputs = self.text_encoder(
                    input_ids=input_ids,
                    attention_mask=attention_mask,
                    inputs_embeds=inputs_embeds,
                    output_attentions=output_attentions,
                    output_hidden_states=output_hidden_states,
                    return_dict=return_dict,
                    **kwargs_text_encoder,
                )

                encoder_hidden_states = encoder_outputs[0]

                # optionally project encoder_hidden_states
                if self.text_encoder.config.hidden_size != self.decoder.config.hidden_size:
                    encoder_hidden_states = self.enc_to_dec_proj(encoder_hidden_states)

            if attention_mask is not None and encoder_hidden_states is not None:
                encoder_hidden_states = encoder_hidden_states * attention_mask[..., None]

            # set a default audio conditional hidden states if text is not None
            if encoder_hidden_states is not None and input_features is None:
                input_features = ops.zeros(
                    (encoder_hidden_states.shape[0], 1, self.config.num_chroma),
                    dtype=self.dtype,
                )
                input_features[:, :, 0] = 1

            if input_features is not None:
                audio_hidden_states = input_features

                # optionally project audio_hidden_states ->
                # (batch_size, seq_len, num_chroma) -> (batch_size, seq_len, hidden_size)
                if self.config.num_chroma != self.decoder.config.hidden_size:
                    audio_hidden_states = self.audio_enc_to_dec_proj(audio_hidden_states)

                # pad or truncate to config.chroma_length
                if audio_hidden_states.shape[1] < self.config.chroma_length:
                    n_repeat = int(math.ceil(self.config.chroma_length / audio_hidden_states.shape[1]))
                    audio_hidden_states = audio_hidden_states.tile((1, n_repeat, 1))
                else:
                    logger.warning(
                        f"The conditional audio signal is of length {audio_hidden_states.shape[1]}, which exceeds"
                        f"the maximum chroma duration of {self.config.chroma_length}."
                        f"The audio will be truncated to {self.config.chroma_length} frames."
                    )
                audio_hidden_states = audio_hidden_states[:, : self.config.chroma_length]

                if encoder_hidden_states is not None:
                    encoder_hidden_states = ops.cat([audio_hidden_states, encoder_hidden_states], dim=1)
                else:
                    encoder_hidden_states = audio_hidden_states

        if (labels is not None) and (decoder_input_ids is None and decoder_inputs_embeds is None):
            decoder_input_ids = shift_tokens_right(
                labels, self.config.decoder.pad_token_id, self.config.decoder.bos_token_id
            )

        # Decode
        decoder_outputs = self.decoder(
            input_ids=decoder_input_ids,
            attention_mask=decoder_attention_mask,
            encoder_hidden_states=encoder_hidden_states,
            inputs_embeds=decoder_inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            use_cache=use_cache,
            past_key_values=past_key_values,
            return_dict=return_dict,
            labels=labels,
            **kwargs_decoder,
        )

        if not return_dict:
            return decoder_outputs + (encoder_hidden_states,)

        return MusicgenMelodyOutputWithPast(
            loss=decoder_outputs.loss,
            logits=decoder_outputs.logits,
            past_key_values=decoder_outputs.past_key_values,
            hidden_states=decoder_outputs.hidden_states,
            attentions=decoder_outputs.attentions,
            encoder_hidden_states=encoder_hidden_states,
        )

    def prepare_inputs_for_generation(
        self,
        decoder_input_ids,
        encoder_hidden_states=None,
        past_key_values=None,
        attention_mask=None,
        decoder_attention_mask=None,
        decoder_head_mask=None,
        use_cache=None,
        decoder_delay_pattern_mask=None,
        guidance_scale=None,
        **kwargs,
    ):
        if decoder_delay_pattern_mask is None:
            decoder_input_ids, decoder_delay_pattern_mask = self.decoder.build_delay_pattern_mask(
                decoder_input_ids,
                self.generation_config.pad_token_id,
                max_length=self.generation_config.max_length,
            )

        # apply the delay pattern mask
        decoder_input_ids = self.decoder.apply_delay_pattern_mask(decoder_input_ids, decoder_delay_pattern_mask)

        if guidance_scale is not None and guidance_scale > 1:
            # for classifier free guidance we need to replicate the decoder args across the batch dim (we'll split these
            # before sampling)
            decoder_input_ids = decoder_input_ids.tile((2, 1))
            if decoder_attention_mask is not None:
                decoder_attention_mask = decoder_attention_mask.tile((2, 1))

        if past_key_values is not None:
            past_length = past_key_values[0][0].shape[2]

            # Some generation methods already pass only the last input ID
            if decoder_input_ids.shape[1] > past_length:
                remove_prefix_length = past_length
            else:
                # Default to old behavior: keep only final ID
                remove_prefix_length = decoder_input_ids.shape[1] - 1

            decoder_input_ids = decoder_input_ids[:, remove_prefix_length:]

            # we only want to use conditional signal in the 1st generation step but keeping the attention mask
            encoder_hidden_states = None
            # we also have to update the attention mask

        return {
            "input_ids": None,  # encoder_hidden_states is defined. input_ids not needed
            "encoder_hidden_states": encoder_hidden_states,
            "past_key_values": past_key_values,
            "decoder_input_ids": decoder_input_ids,
            "attention_mask": attention_mask,
            "decoder_attention_mask": decoder_attention_mask,
            "decoder_head_mask": decoder_head_mask,
            "use_cache": use_cache,
        }

    # Copied from transformers.models.musicgen.modeling_musicgen.MusicgenForConditionalGeneration._prepare_decoder_input_ids_for_generation
    def _prepare_decoder_input_ids_for_generation(
        self,
        batch_size: int,
        model_input_name: str,
        model_kwargs: Dict[str, mindspore.Tensor],
        decoder_start_token_id: int = None,
        bos_token_id: int = None,
    ) -> Tuple[mindspore.Tensor, Dict[str, mindspore.Tensor]]:
        """Prepares `decoder_input_ids` for generation with encoder-decoder models"""

        # 1. Check whether the user has defined `decoder_input_ids` manually. To facilitate in terms of input naming,
        # we also allow the user to pass it under `input_ids`, if the encoder does not use it as the main input.
        if model_kwargs is not None and "decoder_input_ids" in model_kwargs:
            decoder_input_ids = model_kwargs.pop("decoder_input_ids")
        elif "input_ids" in model_kwargs and model_input_name != "input_ids":
            decoder_input_ids = model_kwargs.pop("input_ids")
        else:
            decoder_input_ids = None

        # 2. Encoder-decoder models expect the `decoder_input_ids` to start with a special token. Let's ensure that.
        decoder_start_token_id = self._get_decoder_start_token_id(decoder_start_token_id, bos_token_id)
        decoder_input_ids_start = (
            ops.ones((batch_size * self.decoder.num_codebooks, 1), dtype=mindspore.int64)
            * decoder_start_token_id
        )

        # no user input -> use decoder_start_token_id as decoder_input_ids
        if decoder_input_ids is None:
            decoder_input_ids = decoder_input_ids_start

        # user input but doesn't start with decoder_start_token_id -> prepend decoder_start_token_id (and adjust
        # decoder_attention_mask if provided)
        elif (decoder_input_ids[..., 0] != decoder_start_token_id).all().item():
            decoder_input_ids = ops.cat([decoder_input_ids_start, decoder_input_ids], dim=-1)
            if "decoder_attention_mask" in model_kwargs:
                decoder_attention_mask = model_kwargs["decoder_attention_mask"]
                decoder_attention_mask = ops.cat(
                    (ops.ones_like(decoder_attention_mask)[:, :1], decoder_attention_mask),
                    dim=-1,
                )
                model_kwargs["decoder_attention_mask"] = decoder_attention_mask

        return decoder_input_ids, model_kwargs

    def _prepare_encoder_hidden_states_kwargs_for_generation(
        self,
        inputs_tensor: mindspore.Tensor,
        model_kwargs,
        model_input_name: Optional[str],
        generation_config: GenerationConfig,
    ) -> Dict[str, Any]:
        encoder_hidden_states = None
        # attention mask is consumed once to produce text conditional hidden states through the text encoder
        encoder_attention_mask = model_kwargs.pop("attention_mask")
        guidance_scale = generation_config.guidance_scale

        # 1. condition on text
        if inputs_tensor is not None:
            encoder = self.get_text_encoder()

            # Prepare args and kwargs from model kwargs.
            irrelevant_prefix = ["decoder_", "use_cache"]
            encoder_kwargs = {
                argument: value
                for argument, value in model_kwargs.items()
                if not any(argument.startswith(p) for p in irrelevant_prefix)
            }
            encoder_signature = set(inspect.signature(encoder.forward).parameters)
            encoder_accepts_wildcard = "kwargs" in encoder_signature or "model_kwargs" in encoder_signature
            if not encoder_accepts_wildcard:
                encoder_kwargs = {
                    argument: value for argument, value in encoder_kwargs.items() if argument in encoder_signature
                }
            encoder_kwargs["output_attentions"] = generation_config.output_attentions
            encoder_kwargs["output_hidden_states"] = generation_config.output_hidden_states

            # make sure that encoder returns `ModelOutput`
            model_input_name = model_input_name if model_input_name is not None else self.text_encoder.main_input_name
            encoder_kwargs["return_dict"] = True
            encoder_kwargs[model_input_name] = inputs_tensor
            if encoder_attention_mask is not None:
                encoder_kwargs["attention_mask"] = encoder_attention_mask
            encoder_hidden_states = encoder(**encoder_kwargs).last_hidden_state

            # optionally project encoder_hidden_states
            if self.text_encoder.config.hidden_size != self.decoder.config.hidden_size:
                encoder_hidden_states = self.enc_to_dec_proj(encoder_hidden_states)

            # for classifier free guidance we need to add a 'null' input to our encoder hidden states
            if guidance_scale is not None and guidance_scale > 1:
                encoder_hidden_states = ops.concatenate(
                    [encoder_hidden_states, ops.zeros_like(encoder_hidden_states)], dim=0
                )
                if encoder_attention_mask is not None:
                    encoder_attention_mask = ops.concatenate(
                        [encoder_attention_mask, ops.zeros_like(encoder_attention_mask)], dim=0
                    )
            if encoder_attention_mask is not None:
                encoder_hidden_states = encoder_hidden_states * encoder_attention_mask[..., None]

        # 2. condition on audio
        audio_hidden_states = model_kwargs.get("input_features", None)

        if inputs_tensor is not None:
            if audio_hidden_states is not None:
                null_audio_hidden_states = ops.zeros_like(audio_hidden_states)
            else:
                null_audio_hidden_states = ops.zeros(
                    (inputs_tensor.shape[0], 1, self.config.num_chroma), dtype=self.dtype
                )
            null_audio_hidden_states[:, :, 0] = 1

            if audio_hidden_states is None:
                audio_hidden_states = null_audio_hidden_states

        if audio_hidden_states is not None:
            # for classifier free guidance we need to add a 'null' input to our audio hidden states
            if guidance_scale is not None and guidance_scale > 1:
                audio_hidden_states = ops.concatenate([audio_hidden_states, null_audio_hidden_states], dim=0)

            # optionally project audio_hidden_states ->
            # (batch_size, seq_len, num_chroma) -> (batch_size, seq_len, hidden_size)
            if self.config.num_chroma != self.decoder.config.hidden_size:
                audio_hidden_states = self.audio_enc_to_dec_proj(audio_hidden_states)

            # pad or truncate to config.chroma_length
            if audio_hidden_states.shape[1] < self.config.chroma_length:
                n_repeat = int(math.ceil(self.config.chroma_length / audio_hidden_states.shape[1]))
                audio_hidden_states = audio_hidden_states.tile((1, n_repeat, 1))
            audio_hidden_states = audio_hidden_states[:, : self.config.chroma_length]

            if encoder_hidden_states is not None:
                encoder_hidden_states = ops.cat([audio_hidden_states, encoder_hidden_states], dim=1)
            else:
                encoder_hidden_states = audio_hidden_states

        model_kwargs["encoder_hidden_states"] = encoder_hidden_states

        return model_kwargs

    def prepare_decoder_input_ids_from_labels(self, labels: mindspore.Tensor):
        return shift_tokens_right(labels, self.config.decoder.pad_token_id, self.config.decoder.bos_token_id)

    def resize_token_embeddings(self, *args, **kwargs):
        raise NotImplementedError(
            "Resizing the embedding layers via the EncoderDecoderModel directly is not supported. Please use the"
            " respective methods of the wrapped objects (model.encoder.resize_token_embeddings(...) or"
            " model.decoder.resize_token_embeddings(...))"
        )

    def _maybe_initialize_input_ids_for_generation(
        self,
        inputs: Optional[mindspore.Tensor] = None,
        bos_token_id: Optional[int] = None,
        model_kwargs: Optional[Dict[str, mindspore.Tensor]] = None,
    ) -> mindspore.Tensor:
        """Initializes input ids for generation, if necessary."""
        if inputs is not None:
            return inputs

        if bos_token_id is None:
            raise ValueError("`bos_token_id` has to be defined when no `input_ids` are provided.")

        # If there is some tensor in `model_kwargs`, we can infer the batch size from it. This is helpful with
        # soft-prompting or in multimodal implementations built on top of decoder-only language models.
        batch_size = 1
        for value in model_kwargs.values():
            if isinstance(value, mindspore.Tensor):
                batch_size = value.shape[0]
                break
        return ops.ones((batch_size, 1), dtype=mindspore.int64) * bos_token_id

    def freeze_audio_encoder(self):
        """
        Freeze the audio encoder weights.
        """
        for param in self.audio_encoder.parameters():
            param.requires_grad = False
        self.audio_encoder._requires_grad = False

    def freeze_text_encoder(self):
        """
        Freeze the text encoder weights.
        """
        for param in self.text_encoder.parameters():
            param.requires_grad = False
        self.text_encoder._requires_grad = False

    # Copied from transformers.models.musicgen.modeling_musicgen.MusicgenForConditionalGeneration._get_decoder_start_token_id
    def _get_decoder_start_token_id(
        self, decoder_start_token_id: Union[int, List[int]] = None, bos_token_id: int = None
    ) -> int:
        decoder_start_token_id = (
            decoder_start_token_id
            if decoder_start_token_id is not None
            else self.generation_config.decoder_start_token_id
        )
        bos_token_id = bos_token_id if bos_token_id is not None else self.generation_config.bos_token_id

        if decoder_start_token_id is not None:
            return decoder_start_token_id
        elif bos_token_id is not None:
            return bos_token_id
        raise ValueError(
            "`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation."
        )

    @no_grad()
    def generate(
        self,
        inputs: Optional[mindspore.Tensor] = None,
        generation_config: Optional[GenerationConfig] = None,
        logits_processor: Optional[LogitsProcessorList] = None,
        stopping_criteria: Optional[StoppingCriteriaList] = None,
        synced_gpus: Optional[bool] = None,
        streamer: Optional["BaseStreamer"] = None,
        **kwargs,
    ):
        """

        Generates sequences of token ids for models with a language modeling head.

        <Tip warning={true}>

        Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the
        model's default generation configuration. You can override any `generation_config` by passing the corresponding
        parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`.

        For an overview of generation strategies and code examples, check out the [following
        guide](./generation_strategies).

        </Tip>

        Parameters:
            inputs (`mindspore.Tensor` of varying shape depending on the modality, *optional*):
                The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
                method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
                should be in the format `input_ids`. For encoder-decoder models *inputs* can represent any of
                `input_ids`, `input_values`, `input_features`, or `pixel_values`.
            generation_config (`~generation.GenerationConfig`, *optional*):
                The generation configuration to be used as base parametrization for the generation call. `**kwargs`
                passed to generate matching the attributes of `generation_config` will override them. If
                `generation_config` is not provided, the default will be used, which had the following loading
                priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model
                configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s
                default values, whose documentation should be checked to parameterize generation.
            logits_processor (`LogitsProcessorList`, *optional*):
                Custom logits processors that complement the default logits processors built from arguments and
                generation config. If a logit processor is passed that is already created with the arguments or a
                generation config an error is thrown. This feature is intended for advanced users.
            stopping_criteria (`StoppingCriteriaList`, *optional*):
                Custom stopping criteria that complement the default stopping criteria built from arguments and a
                generation config. If a stopping criteria is passed that is already created with the arguments or a
                generation config an error is thrown. This feature is intended for advanced users.
            synced_gpus (`bool`, *optional*, defaults to `False`):
                Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
            streamer (`BaseStreamer`, *optional*):
                Streamer object that will be used to stream the generated sequences. Generated tokens are passed
                through `streamer.put(token_ids)` and the streamer is responsible for any further processing.
            kwargs (`Dict[str, Any]`, *optional*):
                Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be
                forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
                specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.

        Return:
            [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True`
            or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor`.

                If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
                [`~utils.ModelOutput`] types are:

                    - [`~generation.GenerateDecoderOnlyOutput`],
                    - [`~generation.GenerateBeamDecoderOnlyOutput`]

                If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
                [`~utils.ModelOutput`] types are:

                    - [`~generation.GenerateEncoderDecoderOutput`],
                    - [`~generation.GenerateBeamEncoderDecoderOutput`]
        """
        # 1. Handle `generation_config` and kwargs that might update it, and validate the resulting objects
        if generation_config is None:
            generation_config = self.generation_config

        generation_config = copy.deepcopy(generation_config)
        model_kwargs = generation_config.update(**kwargs)  # All unused kwargs must be model kwargs
        generation_config.validate()
        self._validate_model_kwargs(model_kwargs.copy())

        # 2. Set generation parameters if not already defined
        logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
        stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()

        requires_attention_mask = "encoder_outputs" not in model_kwargs
        kwargs_has_attention_mask = model_kwargs.get("attention_mask", None) is not None

        # 3. Define model inputs
        inputs_tensor, model_input_name, model_kwargs = self._prepare_model_inputs(
            inputs, generation_config.bos_token_id, model_kwargs
        )
        batch_size = inputs_tensor.shape[0]
        self._prepare_special_tokens(generation_config, kwargs_has_attention_mask)

        # 4. Define other model kwargs
        model_kwargs["use_cache"] = generation_config.use_cache
        model_kwargs["guidance_scale"] = generation_config.guidance_scale

        if model_kwargs.get("attention_mask", None) is None and requires_attention_mask:
            model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
                inputs_tensor, generation_config._pad_token_tensor, generation_config._eos_token_tensor
            )

        if "encoder_hidden_states" not in model_kwargs:
            # encoder_hidden_states are created and added to `model_kwargs`
            model_kwargs = self._prepare_encoder_hidden_states_kwargs_for_generation(
                inputs_tensor, model_kwargs, model_input_name, generation_config
            )

        # 5. Prepare `input_ids` which will be used for auto-regressive generation
        input_ids, model_kwargs = self._prepare_decoder_input_ids_for_generation(
            batch_size=batch_size,
            model_input_name=model_input_name,
            model_kwargs=model_kwargs,
            decoder_start_token_id=generation_config._decoder_start_token_tensor,
            bos_token_id=generation_config._bos_token_tensor,
        )

        # 6. Prepare `max_length` depending on other stopping criteria.
        input_ids_length = input_ids.shape[-1]
        has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
        has_default_min_length = kwargs.get("min_length") is None and generation_config.min_length is not None
        generation_config = self._prepare_generated_length(
            generation_config=generation_config,
            has_default_max_length=has_default_max_length,
            has_default_min_length=has_default_min_length,
            model_input_name=model_input_name,
            inputs_tensor=inputs_tensor,
            input_ids_length=input_ids_length,
        )

        # build the delay pattern mask for offsetting each codebook prediction by 1 (this behaviour is specific to MusicGen)
        input_ids, decoder_delay_pattern_mask = self.decoder.build_delay_pattern_mask(
            input_ids,
            pad_token_id=generation_config._decoder_start_token_tensor,
            max_length=generation_config.max_length,
        )
        # stash the delay mask so that we don't have to recompute in each forward pass
        model_kwargs["decoder_delay_pattern_mask"] = decoder_delay_pattern_mask

        # input_ids are ready to be placed on the streamer (if used)
        if streamer is not None:
            streamer.put(input_ids.cpu())

        # 7. determine generation mode
        generation_mode = generation_config.get_generation_mode()

        # 8. prepare batched CFG externally (to enable coexistance with the unbatched CFG)
        if generation_config.guidance_scale is not None and generation_config.guidance_scale > 1:
            logits_processor.append(ClassifierFreeGuidanceLogitsProcessor(generation_config.guidance_scale))
            generation_config.guidance_scale = None

        # 9. prepare distribution pre_processing samplers
        logits_processor = self._get_logits_processor(
            generation_config=generation_config,
            input_ids_seq_length=input_ids_length,
            encoder_input_ids=inputs_tensor,
            prefix_allowed_tokens_fn=None,
            logits_processor=logits_processor,
        )

        # 10. prepare stopping criteria
        stopping_criteria = self._get_stopping_criteria(
            generation_config=generation_config, stopping_criteria=stopping_criteria
        )

        if generation_mode in (GenerationMode.SAMPLE, GenerationMode.GREEDY_SEARCH):
            # 11. prepare logits warper
            prepared_logits_warper = (
                self._get_logits_warper(generation_config)
                if generation_config.do_sample
                else None
            )

            # expand input_ids with `num_return_sequences` additional sequences per batch
            input_ids, model_kwargs = self._expand_inputs_for_generation(
                input_ids=input_ids,
                expand_size=generation_config.num_return_sequences,
                is_encoder_decoder=self.config.is_encoder_decoder,
                **model_kwargs,
            )

            # 12. run sample
            outputs = self._sample(
                input_ids,
                logits_processor=logits_processor,
                logits_warper=prepared_logits_warper,
                stopping_criteria=stopping_criteria,
                generation_config=generation_config,
                synced_gpus=synced_gpus,
                streamer=streamer,
                **model_kwargs,
            )

        else:
            raise ValueError(
                "Got incompatible mode for generation, should be one of greedy or sampling. "
                "Ensure that beam search is de-activated by setting `num_beams=1` and `num_beam_groups=1`."
            )

        if generation_config.return_dict_in_generate:
            output_ids = outputs.sequences
        else:
            output_ids = outputs

        # apply the pattern mask to the final ids
        output_ids = self.decoder.apply_delay_pattern_mask(output_ids, model_kwargs["decoder_delay_pattern_mask"])

        # revert the pattern delay mask by filtering the pad token id
        output_ids = output_ids[output_ids != generation_config._pad_token_tensor].reshape(
            batch_size, self.decoder.num_codebooks, -1
        )

        # append the frame dimension back to the audio codes
        output_ids = output_ids[None, ...]

        audio_scales = model_kwargs.get("audio_scales")
        if audio_scales is None:
            audio_scales = [None] * batch_size

        if self.decoder.config.audio_channels == 1:
            output_values = self.audio_encoder.decode(
                output_ids,
                audio_scales=audio_scales,
            ).audio_values
        else:
            codec_outputs_left = self.audio_encoder.decode(output_ids[:, :, ::2, :], audio_scales=audio_scales)
            output_values_left = codec_outputs_left.audio_values

            codec_outputs_right = self.audio_encoder.decode(output_ids[:, :, 1::2, :], audio_scales=audio_scales)
            output_values_right = codec_outputs_right.audio_values

            output_values = ops.cat([output_values_left, output_values_right], dim=1)

        if generation_config.return_dict_in_generate:
            outputs.sequences = output_values
            return outputs
        else:
            return output_values

    def _update_model_kwargs_for_generation(
        self,
        outputs: ModelOutput,
        model_kwargs: Dict[str, Any],
        is_encoder_decoder: bool = False,
        model_inputs: Optional[Dict[str, Any]] = None,
    ) -> Dict[str, Any]:
        # update past_key_values
        cache_name, cache = self._extract_past_from_model_output(outputs)
        model_kwargs[cache_name] = cache

        if getattr(outputs, "state", None) is not None:
            model_kwargs["state"] = outputs.state

        # update token_type_ids with last value
        if "token_type_ids" in model_kwargs:
            token_type_ids = model_kwargs["token_type_ids"]
            model_kwargs["token_type_ids"] = ops.cat([token_type_ids, token_type_ids[:, -1].unsqueeze(-1)], dim=-1)

        # update decoder attention mask
        if "decoder_attention_mask" in model_kwargs:
            decoder_attention_mask = model_kwargs["decoder_attention_mask"]
            model_kwargs["decoder_attention_mask"] = ops.cat(
                [decoder_attention_mask, ops.ones((decoder_attention_mask.shape[0], 1), dtype=decoder_attention_mask.dtype)],
                dim=-1,
            )

        return model_kwargs

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration.forward(input_ids=None, attention_mask=None, input_features=None, decoder_input_ids=None, decoder_attention_mask=None, past_key_values=None, encoder_hidden_states=None, inputs_embeds=None, decoder_inputs_embeds=None, labels=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, **kwargs)

Examples:

>>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration
>>> import torch

>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-melody")
>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")

>>> inputs = processor(
...     text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
...     padding=True,
...     return_tensors="pt",
... )

>>> pad_token_id = model.generation_config.pad_token_id
>>> decoder_input_ids = (
...     ops.ones((inputs.input_ids.shape[0] * model.decoder.num_codebooks, 1), dtype=mindspore.int64)
...     * pad_token_id
... )

>>> logits = model(**inputs, decoder_input_ids=decoder_input_ids).logits
>>> logits.shape  # (bsz * num_codebooks, encoder_len + tgt_len, vocab_size)
ops.Size([8, 249, 2048])

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    input_features: Optional[mindspore.Tensor] = None,
    decoder_input_ids: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_values: Tuple[Tuple[mindspore.Tensor]] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    decoder_inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    **kwargs,
) -> Union[Tuple, MusicgenMelodyOutputWithPast]:
    r"""
    Returns:

    Examples:
    ```python
    >>> from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration
    >>> import torch

    >>> processor = AutoProcessor.from_pretrained("facebook/musicgen-melody")
    >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")

    >>> inputs = processor(
    ...     text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    ...     padding=True,
    ...     return_tensors="pt",
    ... )

    >>> pad_token_id = model.generation_config.pad_token_id
    >>> decoder_input_ids = (
    ...     ops.ones((inputs.input_ids.shape[0] * model.decoder.num_codebooks, 1), dtype=mindspore.int64)
    ...     * pad_token_id
    ... )

    >>> logits = model(**inputs, decoder_input_ids=decoder_input_ids).logits
    >>> logits.shape  # (bsz * num_codebooks, encoder_len + tgt_len, vocab_size)
    ops.Size([8, 249, 2048])
    ```"""

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    kwargs_text_encoder = {
        argument[len("text_encoder_")]: value
        for argument, value in kwargs.items()
        if argument.startswith("text_encoder_")
    }

    kwargs_decoder = {
        argument[len("decoder_") :]: value for argument, value in kwargs.items() if argument.startswith("decoder_")
    }

    if encoder_hidden_states is None:
        if inputs_embeds is not None or input_ids is not None:
            encoder_outputs = self.text_encoder(
                input_ids=input_ids,
                attention_mask=attention_mask,
                inputs_embeds=inputs_embeds,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
                **kwargs_text_encoder,
            )

            encoder_hidden_states = encoder_outputs[0]

            # optionally project encoder_hidden_states
            if self.text_encoder.config.hidden_size != self.decoder.config.hidden_size:
                encoder_hidden_states = self.enc_to_dec_proj(encoder_hidden_states)

        if attention_mask is not None and encoder_hidden_states is not None:
            encoder_hidden_states = encoder_hidden_states * attention_mask[..., None]

        # set a default audio conditional hidden states if text is not None
        if encoder_hidden_states is not None and input_features is None:
            input_features = ops.zeros(
                (encoder_hidden_states.shape[0], 1, self.config.num_chroma),
                dtype=self.dtype,
            )
            input_features[:, :, 0] = 1

        if input_features is not None:
            audio_hidden_states = input_features

            # optionally project audio_hidden_states ->
            # (batch_size, seq_len, num_chroma) -> (batch_size, seq_len, hidden_size)
            if self.config.num_chroma != self.decoder.config.hidden_size:
                audio_hidden_states = self.audio_enc_to_dec_proj(audio_hidden_states)

            # pad or truncate to config.chroma_length
            if audio_hidden_states.shape[1] < self.config.chroma_length:
                n_repeat = int(math.ceil(self.config.chroma_length / audio_hidden_states.shape[1]))
                audio_hidden_states = audio_hidden_states.tile((1, n_repeat, 1))
            else:
                logger.warning(
                    f"The conditional audio signal is of length {audio_hidden_states.shape[1]}, which exceeds"
                    f"the maximum chroma duration of {self.config.chroma_length}."
                    f"The audio will be truncated to {self.config.chroma_length} frames."
                )
            audio_hidden_states = audio_hidden_states[:, : self.config.chroma_length]

            if encoder_hidden_states is not None:
                encoder_hidden_states = ops.cat([audio_hidden_states, encoder_hidden_states], dim=1)
            else:
                encoder_hidden_states = audio_hidden_states

    if (labels is not None) and (decoder_input_ids is None and decoder_inputs_embeds is None):
        decoder_input_ids = shift_tokens_right(
            labels, self.config.decoder.pad_token_id, self.config.decoder.bos_token_id
        )

    # Decode
    decoder_outputs = self.decoder(
        input_ids=decoder_input_ids,
        attention_mask=decoder_attention_mask,
        encoder_hidden_states=encoder_hidden_states,
        inputs_embeds=decoder_inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        use_cache=use_cache,
        past_key_values=past_key_values,
        return_dict=return_dict,
        labels=labels,
        **kwargs_decoder,
    )

    if not return_dict:
        return decoder_outputs + (encoder_hidden_states,)

    return MusicgenMelodyOutputWithPast(
        loss=decoder_outputs.loss,
        logits=decoder_outputs.logits,
        past_key_values=decoder_outputs.past_key_values,
        hidden_states=decoder_outputs.hidden_states,
        attentions=decoder_outputs.attentions,
        encoder_hidden_states=encoder_hidden_states,
    )

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration.freeze_audio_encoder()

Freeze the audio encoder weights.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def freeze_audio_encoder(self):
    """
    Freeze the audio encoder weights.
    """
    for param in self.audio_encoder.parameters():
        param.requires_grad = False
    self.audio_encoder._requires_grad = False

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration.freeze_text_encoder()

Freeze the text encoder weights.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def freeze_text_encoder(self):
    """
    Freeze the text encoder weights.
    """
    for param in self.text_encoder.parameters():
        param.requires_grad = False
    self.text_encoder._requires_grad = False

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration.from_sub_models_pretrained(text_encoder_pretrained_model_name_or_path=None, audio_encoder_pretrained_model_name_or_path=None, decoder_pretrained_model_name_or_path=None, *model_args, **kwargs) classmethod

Instantiate a text encoder, an audio encoder, and a MusicGen decoder from one, two or three base classes of the library from pretrained model checkpoints.

The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with model.train().

PARAMETER	DESCRIPTION
text_encoder_pretrained_model_name_or_path	Information necessary to initiate the text encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. TYPE: `str`, *optional* DEFAULT: None
audio_encoder_pretrained_model_name_or_path	Information necessary to initiate the audio encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. TYPE: `str`, *optional* DEFAULT: None
decoder_pretrained_model_name_or_path	Information necessary to initiate the decoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. TYPE: `str`, *optional*, defaults to `None` DEFAULT: None
model_args	All remaining positional arguments will be passed to the underlying model's __init__ method. TYPE: remaining positional arguments, *optional* DEFAULT: ()
kwargs	Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). To update the text encoder configuration, use the prefix text_encoder_ for each configuration parameter. To update the audio encoder configuration, use the prefix audio_encoder_ for each configuration parameter. To update the decoder configuration, use the prefix decoder_ for each configuration parameter. To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a config is provided or automatically loaded. TYPE: remaining dictionary of keyword arguments, *optional* DEFAULT: {}

>>> from transformers import MusicgenMelodyForConditionalGeneration

>>> # initialize a musicgen model from a t5 text encoder, encodec audio encoder, and musicgen decoder
>>> model = MusicgenMelodyForConditionalGeneration.from_sub_models_pretrained(
...     text_encoder_pretrained_model_name_or_path="google-t5/t5-base",
...     audio_encoder_pretrained_model_name_or_path="facebook/encodec_24khz",
...     decoder_pretrained_model_name_or_path="facebook/musicgen-melody",
... )
>>> # saving model after fine-tuning
>>> model.save_pretrained("./musicgen-ft")
>>> # load fine-tuned model
>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("./musicgen-ft")

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@classmethod
# Copied from transformers.models.musicgen.modeling_musicgen.MusicgenForConditionalGeneration.from_sub_models_pretrained with Musicgen->MusicgenMelody, musicgen-small->musicgen-melody
def from_sub_models_pretrained( # pylint: disable=keyword-arg-before-vararg
    cls,
    text_encoder_pretrained_model_name_or_path: str = None,
    audio_encoder_pretrained_model_name_or_path: str = None,
    decoder_pretrained_model_name_or_path: str = None,
    *model_args,
    **kwargs,
) -> PreTrainedModel:
    r"""
    Instantiate a text encoder, an audio encoder, and a MusicGen decoder from one, two or three base classes of the
    library from pretrained model checkpoints.


    The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
    the model, you need to first set it back in training mode with `model.train()`.

    Params:
        text_encoder_pretrained_model_name_or_path (`str`, *optional*):
            Information necessary to initiate the text encoder. Can be either:

                - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                - A path to a *directory* containing model weights saved using
                  [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

        audio_encoder_pretrained_model_name_or_path (`str`, *optional*):
            Information necessary to initiate the audio encoder. Can be either:

                - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                - A path to a *directory* containing model weights saved using
                  [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

        decoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`):
            Information necessary to initiate the decoder. Can be either:

                - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                - A path to a *directory* containing model weights saved using
                  [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.

        model_args (remaining positional arguments, *optional*):
            All remaining positional arguments will be passed to the underlying model's `__init__` method.

        kwargs (remaining dictionary of keyword arguments, *optional*):
            Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
            `output_attentions=True`).

            - To update the text encoder configuration, use the prefix *text_encoder_* for each configuration
              parameter.
            - To update the audio encoder configuration, use the prefix *audio_encoder_* for each configuration
              parameter.
            - To update the decoder configuration, use the prefix *decoder_* for each configuration parameter.
            - To update the parent model configuration, do not use a prefix for each configuration parameter.

            Behaves differently depending on whether a `config` is provided or automatically loaded.

    Example:

    ```python
    >>> from transformers import MusicgenMelodyForConditionalGeneration

    >>> # initialize a musicgen model from a t5 text encoder, encodec audio encoder, and musicgen decoder
    >>> model = MusicgenMelodyForConditionalGeneration.from_sub_models_pretrained(
    ...     text_encoder_pretrained_model_name_or_path="google-t5/t5-base",
    ...     audio_encoder_pretrained_model_name_or_path="facebook/encodec_24khz",
    ...     decoder_pretrained_model_name_or_path="facebook/musicgen-melody",
    ... )
    >>> # saving model after fine-tuning
    >>> model.save_pretrained("./musicgen-ft")
    >>> # load fine-tuned model
    >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("./musicgen-ft")
    ```"""

    kwargs_text_encoder = {
        argument[len("text_encoder_") :]: value
        for argument, value in kwargs.items()
        if argument.startswith("text_encoder_")
    }

    kwargs_audio_encoder = {
        argument[len("audio_encoder_") :]: value
        for argument, value in kwargs.items()
        if argument.startswith("audio_encoder_")
    }

    kwargs_decoder = {
        argument[len("decoder_") :]: value for argument, value in kwargs.items() if argument.startswith("decoder_")
    }

    # remove text encoder, audio encoder and decoder kwargs from kwargs
    for key in kwargs_text_encoder.keys():
        del kwargs["text_encoder_" + key]
    for key in kwargs_audio_encoder.keys():
        del kwargs["audio_encoder_" + key]
    for key in kwargs_decoder.keys():
        del kwargs["decoder_" + key]

    # Load and initialize the encoder and decoder
    # The distinction between encoder and decoder at the model level is made
    # by the value of the flag `is_decoder` that we need to set correctly.
    text_encoder = kwargs_text_encoder.pop("model", None)
    if text_encoder is None:
        if text_encoder_pretrained_model_name_or_path is None:
            raise ValueError(
                "If `text_encoder_model` is not defined as an argument, a `text_encoder_pretrained_model_name_or_path` has "
                "to be defined."
            )

        if "config" not in kwargs_text_encoder:
            encoder_config, kwargs_text_encoder = AutoConfig.from_pretrained(
                text_encoder_pretrained_model_name_or_path, **kwargs_text_encoder, return_unused_kwargs=True
            )

            if encoder_config.is_decoder is True or encoder_config.add_cross_attention is True:
                logger.info(
                    f"Initializing {text_encoder_pretrained_model_name_or_path} as a text_encoder model "
                    "from a decoder model. Cross-attention and casual mask are disabled."
                )
                encoder_config.is_decoder = False
                encoder_config.add_cross_attention = False

            kwargs_text_encoder["config"] = encoder_config

        text_encoder = AutoModel.from_pretrained(
            text_encoder_pretrained_model_name_or_path, *model_args, **kwargs_text_encoder
        )

    audio_encoder = kwargs_audio_encoder.pop("model", None)
    if audio_encoder is None:
        if audio_encoder_pretrained_model_name_or_path is None:
            raise ValueError(
                "If `audio_encoder_model` is not defined as an argument, an `audio_encoder_pretrained_model_name_or_path` has "
                "to be defined."
            )

        if "config" not in kwargs_audio_encoder:
            encoder_config, kwargs_audio_encoder = AutoConfig.from_pretrained(
                audio_encoder_pretrained_model_name_or_path, **kwargs_audio_encoder, return_unused_kwargs=True
            )

            if encoder_config.is_decoder is True or encoder_config.add_cross_attention is True:
                logger.info(
                    f"Initializing {audio_encoder_pretrained_model_name_or_path} as an audio_encoder model "
                    "from a decoder model. Cross-attention and casual mask are disabled."
                )
                encoder_config.is_decoder = False
                encoder_config.add_cross_attention = False

            kwargs_audio_encoder["config"] = encoder_config

        audio_encoder = AutoModel.from_pretrained(
            audio_encoder_pretrained_model_name_or_path, *model_args, **kwargs_audio_encoder
        )

    decoder = kwargs_decoder.pop("model", None)
    if decoder is None:
        if decoder_pretrained_model_name_or_path is None:
            raise ValueError(
                "If `decoder_model` is not defined as an argument, a `decoder_pretrained_model_name_or_path` has "
                "to be defined."
            )

        if "config" not in kwargs_decoder:
            decoder_config, kwargs_decoder = AutoConfig.from_pretrained(
                decoder_pretrained_model_name_or_path, **kwargs_decoder, return_unused_kwargs=True
            )

            if isinstance(decoder_config, MusicgenMelodyConfig):
                decoder_config = decoder_config.decoder

            if decoder_config.is_decoder is False or decoder_config.add_cross_attention is False:
                logger.info(
                    f"Initializing {decoder_pretrained_model_name_or_path} as a decoder model. Cross attention"
                    f" layers are added to {decoder_pretrained_model_name_or_path} and randomly initialized if"
                    f" {decoder_pretrained_model_name_or_path}'s architecture allows for cross attention layers."
                )
                decoder_config.is_decoder = True
                decoder_config.add_cross_attention = True

            kwargs_decoder["config"] = decoder_config

        if kwargs_decoder["config"].is_decoder is False or kwargs_decoder["config"].add_cross_attention is False:
            logger.warning(
                f"Decoder model {decoder_pretrained_model_name_or_path} is not initialized as a decoder. "
                f"In order to initialize {decoder_pretrained_model_name_or_path} as a decoder, "
                "make sure that the attributes `is_decoder` and `add_cross_attention` of `decoder_config` "
                "passed to `.from_sub_models_pretrained(...)` are set to `True` or do not pass a "
                "`decoder_config` to `.from_sub_models_pretrained(...)`"
            )

        decoder = MusicgenMelodyForCausalLM.from_pretrained(
            decoder_pretrained_model_name_or_path, **kwargs_decoder
        )

    # instantiate config with corresponding kwargs
    config = MusicgenMelodyConfig.from_sub_models_config(
        text_encoder.config, audio_encoder.config, decoder.config, **kwargs
    )
    return cls(text_encoder=text_encoder, audio_encoder=audio_encoder, decoder=decoder, config=config)

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyForConditionalGeneration.generate(inputs=None, generation_config=None, logits_processor=None, stopping_criteria=None, synced_gpus=None, streamer=None, **kwargs)

Generates sequences of token ids for models with a language modeling head.

Most generation-controlling parameters are set in generation_config which, if not passed, will be set to the model's default generation configuration. You can override any generation_config by passing the corresponding parameters to generate(), e.g. .generate(inputs, num_beams=4, do_sample=True).

For an overview of generation strategies and code examples, check out the following guide.

PARAMETER	DESCRIPTION
inputs	The sequence used as a prompt for the generation or as model inputs to the encoder. If None the method initializes it with bos_token_id and a batch size of 1. For decoder-only models inputs should be in the format input_ids. For encoder-decoder models inputs can represent any of input_ids, input_values, input_features, or pixel_values. TYPE: `mindspore.Tensor` of varying shape depending on the modality, *optional* DEFAULT: None
generation_config	The generation configuration to be used as base parametrization for the generation call. **kwargs passed to generate matching the attributes of generation_config will override them. If generation_config is not provided, the default will be used, which had the following loading priority: 1) from the generation_config.json model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [~generation.GenerationConfig]'s default values, whose documentation should be checked to parameterize generation. TYPE: `~generation.GenerationConfig`, *optional* DEFAULT: None
logits_processor	Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. TYPE: `LogitsProcessorList`, *optional* DEFAULT: None
stopping_criteria	Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. TYPE: `StoppingCriteriaList`, *optional* DEFAULT: None
synced_gpus	Whether to continue running the while loop until max_length (needed for ZeRO stage 3) TYPE: `bool`, *optional*, defaults to `False` DEFAULT: None
streamer	Streamer object that will be used to stream the generated sequences. Generated tokens are passed through streamer.put(token_ids) and the streamer is responsible for any further processing. TYPE: `BaseStreamer`, *optional* DEFAULT: None
kwargs	Ad hoc parametrization of generate_config and/or additional model-specific kwargs that will be forwarded to the forward function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_. TYPE: `Dict[str, Any]`, *optional* DEFAULT: {}

Return

[~utils.ModelOutput] or torch.LongTensor: A [~utils.ModelOutput] (if return_dict_in_generate=True or when config.return_dict_in_generate=True) or a torch.FloatTensor.

If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
[`~utils.ModelOutput`] types are:

    - [`~generation.GenerateDecoderOnlyOutput`],
    - [`~generation.GenerateBeamDecoderOnlyOutput`]

If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[`~utils.ModelOutput`] types are:

    - [`~generation.GenerateEncoderDecoderOutput`],
    - [`~generation.GenerateBeamEncoderDecoderOutput`]

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@no_grad()
def generate(
    self,
    inputs: Optional[mindspore.Tensor] = None,
    generation_config: Optional[GenerationConfig] = None,
    logits_processor: Optional[LogitsProcessorList] = None,
    stopping_criteria: Optional[StoppingCriteriaList] = None,
    synced_gpus: Optional[bool] = None,
    streamer: Optional["BaseStreamer"] = None,
    **kwargs,
):
    """

    Generates sequences of token ids for models with a language modeling head.

    <Tip warning={true}>

    Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the
    model's default generation configuration. You can override any `generation_config` by passing the corresponding
    parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`.

    For an overview of generation strategies and code examples, check out the [following
    guide](./generation_strategies).

    </Tip>

    Parameters:
        inputs (`mindspore.Tensor` of varying shape depending on the modality, *optional*):
            The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
            method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
            should be in the format `input_ids`. For encoder-decoder models *inputs* can represent any of
            `input_ids`, `input_values`, `input_features`, or `pixel_values`.
        generation_config (`~generation.GenerationConfig`, *optional*):
            The generation configuration to be used as base parametrization for the generation call. `**kwargs`
            passed to generate matching the attributes of `generation_config` will override them. If
            `generation_config` is not provided, the default will be used, which had the following loading
            priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model
            configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s
            default values, whose documentation should be checked to parameterize generation.
        logits_processor (`LogitsProcessorList`, *optional*):
            Custom logits processors that complement the default logits processors built from arguments and
            generation config. If a logit processor is passed that is already created with the arguments or a
            generation config an error is thrown. This feature is intended for advanced users.
        stopping_criteria (`StoppingCriteriaList`, *optional*):
            Custom stopping criteria that complement the default stopping criteria built from arguments and a
            generation config. If a stopping criteria is passed that is already created with the arguments or a
            generation config an error is thrown. This feature is intended for advanced users.
        synced_gpus (`bool`, *optional*, defaults to `False`):
            Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
        streamer (`BaseStreamer`, *optional*):
            Streamer object that will be used to stream the generated sequences. Generated tokens are passed
            through `streamer.put(token_ids)` and the streamer is responsible for any further processing.
        kwargs (`Dict[str, Any]`, *optional*):
            Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be
            forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
            specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.

    Return:
        [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True`
        or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor`.

            If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
            [`~utils.ModelOutput`] types are:

                - [`~generation.GenerateDecoderOnlyOutput`],
                - [`~generation.GenerateBeamDecoderOnlyOutput`]

            If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
            [`~utils.ModelOutput`] types are:

                - [`~generation.GenerateEncoderDecoderOutput`],
                - [`~generation.GenerateBeamEncoderDecoderOutput`]
    """
    # 1. Handle `generation_config` and kwargs that might update it, and validate the resulting objects
    if generation_config is None:
        generation_config = self.generation_config

    generation_config = copy.deepcopy(generation_config)
    model_kwargs = generation_config.update(**kwargs)  # All unused kwargs must be model kwargs
    generation_config.validate()
    self._validate_model_kwargs(model_kwargs.copy())

    # 2. Set generation parameters if not already defined
    logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
    stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()

    requires_attention_mask = "encoder_outputs" not in model_kwargs
    kwargs_has_attention_mask = model_kwargs.get("attention_mask", None) is not None

    # 3. Define model inputs
    inputs_tensor, model_input_name, model_kwargs = self._prepare_model_inputs(
        inputs, generation_config.bos_token_id, model_kwargs
    )
    batch_size = inputs_tensor.shape[0]
    self._prepare_special_tokens(generation_config, kwargs_has_attention_mask)

    # 4. Define other model kwargs
    model_kwargs["use_cache"] = generation_config.use_cache
    model_kwargs["guidance_scale"] = generation_config.guidance_scale

    if model_kwargs.get("attention_mask", None) is None and requires_attention_mask:
        model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
            inputs_tensor, generation_config._pad_token_tensor, generation_config._eos_token_tensor
        )

    if "encoder_hidden_states" not in model_kwargs:
        # encoder_hidden_states are created and added to `model_kwargs`
        model_kwargs = self._prepare_encoder_hidden_states_kwargs_for_generation(
            inputs_tensor, model_kwargs, model_input_name, generation_config
        )

    # 5. Prepare `input_ids` which will be used for auto-regressive generation
    input_ids, model_kwargs = self._prepare_decoder_input_ids_for_generation(
        batch_size=batch_size,
        model_input_name=model_input_name,
        model_kwargs=model_kwargs,
        decoder_start_token_id=generation_config._decoder_start_token_tensor,
        bos_token_id=generation_config._bos_token_tensor,
    )

    # 6. Prepare `max_length` depending on other stopping criteria.
    input_ids_length = input_ids.shape[-1]
    has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
    has_default_min_length = kwargs.get("min_length") is None and generation_config.min_length is not None
    generation_config = self._prepare_generated_length(
        generation_config=generation_config,
        has_default_max_length=has_default_max_length,
        has_default_min_length=has_default_min_length,
        model_input_name=model_input_name,
        inputs_tensor=inputs_tensor,
        input_ids_length=input_ids_length,
    )

    # build the delay pattern mask for offsetting each codebook prediction by 1 (this behaviour is specific to MusicGen)
    input_ids, decoder_delay_pattern_mask = self.decoder.build_delay_pattern_mask(
        input_ids,
        pad_token_id=generation_config._decoder_start_token_tensor,
        max_length=generation_config.max_length,
    )
    # stash the delay mask so that we don't have to recompute in each forward pass
    model_kwargs["decoder_delay_pattern_mask"] = decoder_delay_pattern_mask

    # input_ids are ready to be placed on the streamer (if used)
    if streamer is not None:
        streamer.put(input_ids.cpu())

    # 7. determine generation mode
    generation_mode = generation_config.get_generation_mode()

    # 8. prepare batched CFG externally (to enable coexistance with the unbatched CFG)
    if generation_config.guidance_scale is not None and generation_config.guidance_scale > 1:
        logits_processor.append(ClassifierFreeGuidanceLogitsProcessor(generation_config.guidance_scale))
        generation_config.guidance_scale = None

    # 9. prepare distribution pre_processing samplers
    logits_processor = self._get_logits_processor(
        generation_config=generation_config,
        input_ids_seq_length=input_ids_length,
        encoder_input_ids=inputs_tensor,
        prefix_allowed_tokens_fn=None,
        logits_processor=logits_processor,
    )

    # 10. prepare stopping criteria
    stopping_criteria = self._get_stopping_criteria(
        generation_config=generation_config, stopping_criteria=stopping_criteria
    )

    if generation_mode in (GenerationMode.SAMPLE, GenerationMode.GREEDY_SEARCH):
        # 11. prepare logits warper
        prepared_logits_warper = (
            self._get_logits_warper(generation_config)
            if generation_config.do_sample
            else None
        )

        # expand input_ids with `num_return_sequences` additional sequences per batch
        input_ids, model_kwargs = self._expand_inputs_for_generation(
            input_ids=input_ids,
            expand_size=generation_config.num_return_sequences,
            is_encoder_decoder=self.config.is_encoder_decoder,
            **model_kwargs,
        )

        # 12. run sample
        outputs = self._sample(
            input_ids,
            logits_processor=logits_processor,
            logits_warper=prepared_logits_warper,
            stopping_criteria=stopping_criteria,
            generation_config=generation_config,
            synced_gpus=synced_gpus,
            streamer=streamer,
            **model_kwargs,
        )

    else:
        raise ValueError(
            "Got incompatible mode for generation, should be one of greedy or sampling. "
            "Ensure that beam search is de-activated by setting `num_beams=1` and `num_beam_groups=1`."
        )

    if generation_config.return_dict_in_generate:
        output_ids = outputs.sequences
    else:
        output_ids = outputs

    # apply the pattern mask to the final ids
    output_ids = self.decoder.apply_delay_pattern_mask(output_ids, model_kwargs["decoder_delay_pattern_mask"])

    # revert the pattern delay mask by filtering the pad token id
    output_ids = output_ids[output_ids != generation_config._pad_token_tensor].reshape(
        batch_size, self.decoder.num_codebooks, -1
    )

    # append the frame dimension back to the audio codes
    output_ids = output_ids[None, ...]

    audio_scales = model_kwargs.get("audio_scales")
    if audio_scales is None:
        audio_scales = [None] * batch_size

    if self.decoder.config.audio_channels == 1:
        output_values = self.audio_encoder.decode(
            output_ids,
            audio_scales=audio_scales,
        ).audio_values
    else:
        codec_outputs_left = self.audio_encoder.decode(output_ids[:, :, ::2, :], audio_scales=audio_scales)
        output_values_left = codec_outputs_left.audio_values

        codec_outputs_right = self.audio_encoder.decode(output_ids[:, :, 1::2, :], audio_scales=audio_scales)
        output_values_right = codec_outputs_right.audio_values

        output_values = ops.cat([output_values_left, output_values_right], dim=1)

    if generation_config.return_dict_in_generate:
        outputs.sequences = output_values
        return outputs
    else:
        return output_values

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyOutputWithPast dataclass

Bases: ModelOutput

Base class for Musicgen Melody autoregressive outputs.

PARAMETER	DESCRIPTION
loss	Language modeling loss (for next-token prediction). TYPE: `mindspore.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided DEFAULT: None
logits	Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)` DEFAULT: None
past_key_values	Tuple of tuple(mindspore.Tensor) of length config.n_layers, with each tuple having 2 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head)) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding. TYPE: `tuple(tuple(mindspore.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True` DEFAULT: None
hidden_states	Tuple of mindspore.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. TYPE: `tuple(mindspore.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` DEFAULT: None
attentions	Tuple of mindspore.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. TYPE: `tuple(mindspore.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True` DEFAULT: None
encoder_hidden_states	Sequence of conditional hidden-states representing the concatenation of the projeted text encoder output and the projeted audio encoder output. Used as a conditional signal. TYPE: `mindspore.Tensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional* DEFAULT: None

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@dataclass
class MusicgenMelodyOutputWithPast(ModelOutput):
    """
    Base class for Musicgen Melody autoregressive outputs.

    Args:
        loss (`mindspore.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided):
            Language modeling loss (for next-token prediction).
        logits (`mindspore.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
        past_key_values (`tuple(tuple(mindspore.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
            Tuple of `tuple(mindspore.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape
            `(batch_size, num_heads, sequence_length, embed_size_per_head)`)

            Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
            `past_key_values` input) to speed up sequential decoding.
        hidden_states (`tuple(mindspore.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
            Tuple of `mindspore.Tensor` (one for the output of the embeddings, if the model has an embedding layer, +
            one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

            Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
        attentions (`tuple(mindspore.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
            Tuple of `mindspore.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
            sequence_length)`.

            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
            heads.
        encoder_hidden_states (`mindspore.Tensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*):
            Sequence of conditional hidden-states representing the concatenation of the projeted text encoder output and the projeted audio encoder output.
            Used as a conditional signal.
    """

    loss: Optional[mindspore.Tensor] = None
    logits: mindspore.Tensor = None
    past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None
    hidden_states: Optional[Tuple[mindspore.Tensor]] = None
    attentions: Optional[Tuple[mindspore.Tensor]] = None
    encoder_hidden_states: Optional[mindspore.Tensor] = None

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodyPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodyPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """

    config_class = MusicgenMelodyDecoderConfig
    base_model_prefix = "model"
    supports_gradient_checkpointing = True
    _no_split_modules = ["MusicgenMelodyDecoderLayer", "MusicgenMelodyAttention"]

    def _init_weights(self, module):
        std = self.config.initializer_factor
        if isinstance(module, (nn.Linear, nn.Conv1d)):
            nn.init.normal_(module.weight, mean=0.0, std=std)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=std)
            if module.padding_idx is not None:
                module.weight[module.padding_idx] = 0

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodySinusoidalPositionalEmbedding

Bases: Module

This module produces sinusoidal positional embeddings of any length.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

class MusicgenMelodySinusoidalPositionalEmbedding(nn.Module):
    """This module produces sinusoidal positional embeddings of any length."""

    def __init__(self, num_positions: int, embedding_dim: int):
        super().__init__()
        self.embedding_dim = embedding_dim
        self.make_weights(num_positions, embedding_dim)

    def make_weights(self, num_embeddings: int, embedding_dim: int):
        emb_weights = self.get_embedding(num_embeddings, embedding_dim)
        if hasattr(self, "weights"):
            # in forward put the weights on the correct dtype and device of the param
            emb_weights = emb_weights.to(dtype=self.weights.dtype) # pylint: disable=access-member-before-definition

        self.weights = nn.Parameter(emb_weights)
        self.weights.requires_grad = False

    @staticmethod
    def get_embedding(num_embeddings: int, embedding_dim: int):
        """
        Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the
        description in Section 3.5 of "Attention Is All You Need".
        """
        half_dim = embedding_dim // 2
        emb = math.log(10000) / (half_dim - 1)
        emb = ops.exp(ops.arange(half_dim, dtype=mindspore.int64).float() * -emb)
        emb = ops.arange(num_embeddings, dtype=mindspore.int64).float().unsqueeze(1) * emb.unsqueeze(0)
        emb = ops.cat([ops.cos(emb), ops.sin(emb)], dim=1).view(num_embeddings, -1)
        if embedding_dim % 2 == 1:
            # zero pad
            emb = ops.cat([emb, ops.zeros(num_embeddings, 1)], dim=1)
        return emb.to(get_default_dtype())

    @no_grad()
    # Ignore copy
    def forward(self, inputs_embeds: mindspore.Tensor, past_key_values_length: int = 0):
        bsz, seq_len, _ = inputs_embeds.shape
        # Create the position ids from the input token ids.
        position_ids = (ops.arange(seq_len) + past_key_values_length)
        # expand embeddings if needed
        if seq_len > self.weights.shape[0]:
            self.make_weights(seq_len + self.offset, self.embedding_dim)
        return self.weights.index_select(0, position_ids.view(-1))

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.MusicgenMelodySinusoidalPositionalEmbedding.get_embedding(num_embeddings, embedding_dim) staticmethod

Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need".

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

@staticmethod
def get_embedding(num_embeddings: int, embedding_dim: int):
    """
    Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the
    description in Section 3.5 of "Attention Is All You Need".
    """
    half_dim = embedding_dim // 2
    emb = math.log(10000) / (half_dim - 1)
    emb = ops.exp(ops.arange(half_dim, dtype=mindspore.int64).float() * -emb)
    emb = ops.arange(num_embeddings, dtype=mindspore.int64).float().unsqueeze(1) * emb.unsqueeze(0)
    emb = ops.cat([ops.cos(emb), ops.sin(emb)], dim=1).view(num_embeddings, -1)
    if embedding_dim % 2 == 1:
        # zero pad
        emb = ops.cat([emb, ops.zeros(num_embeddings, 1)], dim=1)
    return emb.to(get_default_dtype())

mindnlp.transformers.models.musicgen_melody.modeling_musicgen_melody.shift_tokens_right(input_ids, pad_token_id, decoder_start_token_id)

Shift input ids one token to the right.

Source code in mindnlp\transformers\models\musicgen_melody\modeling_musicgen_melody.py

def shift_tokens_right(input_ids: mindspore.Tensor, pad_token_id: int, decoder_start_token_id: int):
    """
    Shift input ids one token to the right.
    """
    # swapaxes to get (bsz, num_codebooks, seq_len)
    input_ids = input_ids.swapaxes(1, 2)
    shifted_input_ids = input_ids.new_zeros(input_ids.shape)
    shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
    if decoder_start_token_id is None:
        raise ValueError("Make sure to set the decoder_start_token_id attribute of the model's configuration.")
    shifted_input_ids[..., 0] = decoder_start_token_id

    if pad_token_id is None:
        raise ValueError("Make sure to set the pad_token_id attribute of the model's configuration.")
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)

    return shifted_input_ids

mindnlp.transformers.models.musicgen_melody.feature_extraction_musicgen_melody

Feature extractor class for Musicgen Melody

mindnlp.transformers.models.musicgen_melody.feature_extraction_musicgen_melody.MusicgenMelodyFeatureExtractor

Bases: SequenceFeatureExtractor

Constructs a MusicgenMelody feature extractor.

This feature extractor inherits from [~feature_extraction_sequence_utils.SequenceFeatureExtractor] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts chroma features from audio processed by Demucs or directly from raw audio waveform.

PARAMETER	DESCRIPTION
feature_size	The feature dimension of the extracted features. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
sampling_rate	The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). TYPE: `int`, *optional*, defaults to 32000 DEFAULT: 32000
hop_length	Length of the overlaping windows for the STFT used to obtain the Mel Frequency coefficients. TYPE: `int`, *optional*, defaults to 4096 DEFAULT: 4096
chunk_length	The maximum number of chunks of sampling_rate samples used to trim and pad longer or shorter audio sequences. TYPE: `int`, *optional*, defaults to 30 DEFAULT: 30
n_fft	Size of the Fourier transform. TYPE: `int`, *optional*, defaults to 16384 DEFAULT: 16384
num_chroma	Number of chroma bins to use. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
padding_value	Padding value used to pad the audio. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
return_attention_mask	Whether to return the attention mask. Can be overwritten when calling the feature extractor. What are attention masks? For Whisper models, attention_mask should always be passed for batched inference, to avoid subtle bugs. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
stem_indices	Stem channels to extract if demucs outputs are passed. TYPE: `List[int]`, *optional*, defaults to `[3, 2]` DEFAULT: [3, 2]

Source code in mindnlp\transformers\models\musicgen_melody\feature_extraction_musicgen_melody.py

class MusicgenMelodyFeatureExtractor(SequenceFeatureExtractor):
    r"""
    Constructs a MusicgenMelody feature extractor.

    This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains
    most of the main methods. Users should refer to this superclass for more information regarding those methods.

    This class extracts chroma features from audio processed by [Demucs](https://github.com/adefossez/demucs/tree/main) or
    directly from raw audio waveform.

    Args:
        feature_size (`int`, *optional*, defaults to 12):
            The feature dimension of the extracted features.
        sampling_rate (`int`, *optional*, defaults to 32000):
            The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
        hop_length (`int`, *optional*, defaults to 4096):
            Length of the overlaping windows for the STFT used to obtain the Mel Frequency coefficients.
        chunk_length (`int`, *optional*, defaults to 30):
            The maximum number of chunks of `sampling_rate` samples used to trim and pad longer or shorter audio
            sequences.
        n_fft (`int`, *optional*, defaults to 16384):
            Size of the Fourier transform.
        num_chroma (`int`, *optional*, defaults to 12):
            Number of chroma bins to use.
        padding_value (`float`, *optional*, defaults to 0.0):
            Padding value used to pad the audio.
        return_attention_mask (`bool`, *optional*, defaults to `False`):
            Whether to return the attention mask. Can be overwritten when calling the feature extractor.

            [What are attention masks?](../glossary#attention-mask)

            <Tip>

            For Whisper models, `attention_mask` should always be passed for batched inference, to avoid subtle
            bugs.

            </Tip>
        stem_indices (`List[int]`, *optional*, defaults to `[3, 2]`):
            Stem channels to extract if demucs outputs are passed.
    """
    model_input_names = ["input_features"]

    def __init__(
        self,
        feature_size=12,
        sampling_rate=32000,
        hop_length=4096,
        chunk_length=30,
        n_fft=16384,
        num_chroma=12,
        padding_value=0.0,
        return_attention_mask=False,  # pad inputs to max length with silence token (zero) and no attention mask
        stem_indices=[3, 2],
        **kwargs,
    ):
        """
        Initializes a MusicgenMelodyFeatureExtractor object.

        Args:
            self: The object itself.
            feature_size (int, optional): The size of the feature. Defaults to 12.
            sampling_rate (int, optional): The sampling rate. Defaults to 32000.
            hop_length (int, optional): The hop length. Defaults to 4096.
            chunk_length (int, optional): The length of the chunk. Defaults to 30.
            n_fft (int, optional): The number of FFT points. Defaults to 16384.
            num_chroma (int, optional): The number of chroma. Defaults to 12.
            padding_value (float, optional): The padding value. Defaults to 0.0.
            return_attention_mask (bool, optional): Indicates whether to return attention mask. Defaults to False.
            stem_indices (list, optional): The stem indices. Defaults to [3, 2].
            **kwargs: Additional keyword arguments.

        Returns:
            None.

        Raises:
            ValueError: If the provided feature size, sampling rate, hop length, chunk length, n_fft, or num_chroma
                is invalid.
            TypeError: If the provided padding value is not a float.
            TypeError: If the provided return_attention_mask is not a boolean.
            TypeError: If stem_indices is not a list.
        """
        super().__init__(
            feature_size=feature_size,
            sampling_rate=sampling_rate,
            padding_value=padding_value,
            return_attention_mask=return_attention_mask,
            **kwargs,
        )
        self.n_fft = n_fft
        self.hop_length = hop_length
        self.chunk_length = chunk_length
        self.n_samples = chunk_length * sampling_rate
        self.sampling_rate = sampling_rate
        self.chroma_filters = mindspore.Tensor.from_numpy(
            chroma_filter_bank(sampling_rate=sampling_rate, num_frequency_bins=n_fft, tuning=0, num_chroma=num_chroma)
        ).float()
        self.spectrogram = Spectrogram(n_fft=n_fft, win_length=n_fft, hop_length=hop_length, power=2, center=True, pad=0, normalized=True)

        self.stem_indices = stem_indices

    def _ms_extract_fbank_features(self, waveform: mindspore.Tensor) -> mindspore.Tensor:
        """
        Compute the chroma spectrogram of the provided audio using the spectrogram implementation and the librosa chroma features.
        """
        # if wav length is not long enough, pad it
        wav_length = waveform.shape[-1]
        if wav_length < self.n_fft:
            pad = self.n_fft - wav_length
            rest = 0 if pad % 2 == 0 else 1
            waveform = ops.pad(waveform, (pad // 2, pad // 2 + rest), "constant", 0)

        # squeeze alongside channel dimension
        spec = mindspore.tensor(self.spectrogram(waveform.asnumpy()))

        # sum along the frequency dimension
        raw_chroma = ops.einsum("cf, ...ft->...ct", self.chroma_filters, spec)

        # normalise with max value
        norm_chroma = nn.functional.normalize(raw_chroma, p=float("inf"), dim=-2, eps=1e-6)

        # transpose time and chroma dimension -> (batch, time, chroma)
        norm_chroma = norm_chroma.swapaxes(1, 2)

        # replace max value alongside chroma dimension with 1 and replace the rest with 0
        idx = norm_chroma.argmax(-1, keepdims=True)
        norm_chroma[:] = 0
        norm_chroma = ops.scatter(norm_chroma, -1, idx, ops.ones_like(idx, dtype=norm_chroma.dtype))

        return norm_chroma

    def _extract_stem_indices(self, audio, sampling_rate=None):
        """
        Extracts stems from the output of the [Demucs](https://github.com/adefossez/demucs/tree/main) audio separation
        model, then converts to mono-channel and resample to the feature extractor sampling rate.

        Args:
            audio (`mindspore.Tensor` of shape `(batch_size, num_stems, channel_size, audio_length)`):
                The output of the Demucs model to be processed.
            sampling_rate (`int`, *optional*):
                Demucs sampling rate. If not specified, defaults to `44000`.
        """
        sampling_rate = 44000 if sampling_rate is None else sampling_rate

        # extract "vocals" and "others" sources from audio encoder (demucs) output
        # [batch_size, num_stems, channel_size, audio_length]
        wav = audio[:, mindspore.tensor(self.stem_indices)]

        # merge extracted stems to single waveform
        wav = wav.sum(1)

        # convert to mono-channel waveform
        wav = wav.mean(axis=1, keep_dims=True)

        # resample to model sampling rate
        # not equivalent to julius.resample
        if sampling_rate != self.sampling_rate:

            transform = Resample(sampling_rate, self.sampling_rate,
                             resample_method=ResampleMethod.SINC_INTERPOLATION,
                             rolloff=0.945, lowpass_filter_width=24)
            wav = mindspore.Tensor(transform(wav.asnumpy()))

        # [batch_size, 1, audio_length] -> [batch_size, audio_length]
        wav = wav.squeeze(1)

        return wav

    def __call__(
        self,
        audio: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
        truncation: bool = True,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_attention_mask: Optional[bool] = None,
        padding: Optional[str] = True,
        max_length: Optional[int] = None,
        sampling_rate: Optional[int] = None,
        **kwargs,
    ) -> BatchFeature:
        """
        Main method to featurize and prepare for the model one or several sequence(s).

        Args:
            audio (`mindspore.Tensor`, `np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[mindspore.Tensor]`,
                `List[List[float]]`):
                The sequence or batch of sequences to be padded. Each sequence can be a mindspore tensor, a numpy array,
                a list of float values, a list of numpy arrays, a list of mindspore tensors, or a list of list of float
                values. If `audio` is the output of Demucs, it has to be a mindspore tensor of shape
                `(batch_size, num_stems, channel_size, audio_length)`.
                Otherwise, it must be mono or stereo channel audio.
            truncation (`bool`, *optional*, default to `True`):
                Activates truncation to cut input sequences longer than *max_length* to *max_length*.
            pad_to_multiple_of (`int`, *optional*, defaults to None):
                If set will pad the sequence to a multiple of the provided value.

                This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
                `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
            return_tensors (`str` or [`~utils.TensorType`], *optional*):
                If set, will return tensors instead of list of python integers. Acceptable values are:

                - `'tf'`: Return TensorFlow `tf.constant` objects.
                - `'pt'`: Return PyTorch `mindspore.Tensor` objects.
                - `'np'`: Return Numpy `np.ndarray` objects.
            return_attention_mask (`bool`, *optional*):
                Whether to return the attention mask. If left to the default, will return the attention mask according
                to the specific feature_extractor's default.

                [What are attention masks?](../glossary#attention-mask)

                <Tip>
                For Musicgen Melody models, audio `attention_mask` is not necessary.
                </Tip>

            padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
                Select a strategy to pad the returned sequences (according to the model's padding side and padding
                index) among:

                - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                sequence if provided).
                - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
                acceptable input length for the model if that argument is not provided.
                - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
                lengths).
            max_length (`int`, *optional*):
                Maximum length of the returned list and optionally padding length (see above).
            sampling_rate (`int`, *optional*):
                The sampling rate at which the `audio` input was sampled. It is strongly recommended to pass
                `sampling_rate` at the forward call to prevent silent errors.
                Note that if `audio` is the output of Demucs, `sampling_rate` must be the sampling rate at which Demucs operates.
        """
        if sampling_rate is None:
            logger.warning_once(
                "It is strongly recommended to pass the `sampling_rate` argument to this function. "
                "Failing to do so can result in silent errors that might be hard to debug."
            )

        if isinstance(audio, mindspore.Tensor) and len(audio.shape) == 4:
            logger.warning_once(
                "`audio` is a 4-dimensional mindspore tensor and has thus been recognized as the output of `Demucs`. "
                "If this is not the case, make sure to read Musicgen Melody docstrings and "
                "to correct `audio` to get the right behaviour."
                "Link to the docstrings: https://huggingface.co/docs/transformers/main/en/model_doc/musicgen_melody"
            )
            audio = self._extract_stem_indices(audio, sampling_rate=sampling_rate)
        elif sampling_rate is not None and sampling_rate != self.sampling_rate:
            transform = Resample(sampling_rate, self.sampling_rate,
                             resample_method=ResampleMethod.SINC_INTERPOLATION,
                             rolloff=0.945, lowpass_filter_width=24)
            audio = transform(audio)

        is_batched = isinstance(audio, (np.ndarray, mindspore.Tensor)) and len(audio.shape) > 1
        is_batched = is_batched or (
            isinstance(audio, (list, tuple)) and (isinstance(audio[0], (mindspore.Tensor, np.ndarray, tuple, list)))
        )

        if is_batched and not isinstance(audio[0], mindspore.Tensor):
            audio = [mindspore.tensor(speech, dtype=mindspore.float32).unsqueeze(-1) for speech in audio]
        elif is_batched:
            audio = [speech.unsqueeze(-1) for speech in audio]
        elif not is_batched and not isinstance(audio, mindspore.Tensor):
            audio = mindspore.tensor(audio, dtype=mindspore.float32).unsqueeze(-1)

        if isinstance(audio[0], mindspore.Tensor) and audio[0].dtype is mindspore.float64:
            audio = [speech.to(mindspore.float32) for speech in audio]

        # always return batch
        if not is_batched:
            audio = [audio]

        if len(audio[0].shape) == 3:
            logger.warning_once(
                "`audio` has been detected as a batch of stereo signals. Will be convert to mono signals. "
                "If this is an undesired behaviour, make sure to read Musicgen Melody docstrings and "
                "to correct `audio` to get the right behaviour."
                "Link to the docstrings: https://huggingface.co/docs/transformers/main/en/model_doc/musicgen_melody"
            )
            # convert to mono-channel waveform
            audio = [stereo.mean(dim=0) for stereo in audio]

        batched_speech = BatchFeature({"input_features": audio})

        padded_inputs = self.pad(
            batched_speech,
            padding=padding,
            max_length=max_length if max_length else self.n_samples,
            truncation=truncation,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_tensors="ms",
        )

        input_features = self._ms_extract_fbank_features(padded_inputs["input_features"].squeeze(-1))

        padded_inputs["input_features"] = input_features

        if return_attention_mask:
            # rescale from raw audio length to spectrogram length
            padded_inputs["attention_mask"] = padded_inputs["attention_mask"][:, :: self.hop_length]

        if return_tensors is not None:
            padded_inputs = padded_inputs.convert_to_tensors(return_tensors)

        return padded_inputs

    def to_dict(self) -> Dict[str, Any]:
        """
        Serializes this instance to a Python dictionary. Returns:
            `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance.
        """
        output = copy.deepcopy(self.__dict__)
        output["feature_extractor_type"] = self.__class__.__name__
        if "mel_filters" in output:
            del output["mel_filters"]
        if "window" in output:
            del output["window"]
        if "chroma_filters" in output:
            del output["chroma_filters"]
        if "spectrogram" in output:
            del output["spectrogram"]
        return output

mindnlp.transformers.models.musicgen_melody.feature_extraction_musicgen_melody.MusicgenMelodyFeatureExtractor.__call__(audio, truncation=True, pad_to_multiple_of=None, return_tensors=None, return_attention_mask=None, padding=True, max_length=None, sampling_rate=None, **kwargs)

Main method to featurize and prepare for the model one or several sequence(s).

PARAMETER	DESCRIPTION
truncation	Activates truncation to cut input sequences longer than max_length to max_length. TYPE: `bool`, *optional*, default to `True` DEFAULT: True
pad_to_multiple_of	If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. TYPE: `int`, *optional*, defaults to None DEFAULT: None
return_tensors	If set, will return tensors instead of list of python integers. Acceptable values are: 'tf': Return TensorFlow tf.constant objects. 'pt': Return PyTorch mindspore.Tensor objects. 'np': Return Numpy np.ndarray objects. TYPE: `str` or [`~utils.TensorType`], *optional* DEFAULT: None
return_attention_mask	Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. What are attention masks? For Musicgen Melody models, audio attention_mask is not necessary. TYPE: `bool`, *optional* DEFAULT: None
padding	Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). 'max_length': Pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided. False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different lengths). TYPE: `bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True` DEFAULT: True
max_length	Maximum length of the returned list and optionally padding length (see above). TYPE: `int`, *optional* DEFAULT: None
sampling_rate	The sampling rate at which the audio input was sampled. It is strongly recommended to pass sampling_rate at the forward call to prevent silent errors. Note that if audio is the output of Demucs, sampling_rate must be the sampling rate at which Demucs operates. TYPE: `int`, *optional* DEFAULT: None

Source code in mindnlp\transformers\models\musicgen_melody\feature_extraction_musicgen_melody.py

def __call__(
    self,
    audio: Union[np.ndarray, List[float], List[np.ndarray], List[List[float]]],
    truncation: bool = True,
    pad_to_multiple_of: Optional[int] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    return_attention_mask: Optional[bool] = None,
    padding: Optional[str] = True,
    max_length: Optional[int] = None,
    sampling_rate: Optional[int] = None,
    **kwargs,
) -> BatchFeature:
    """
    Main method to featurize and prepare for the model one or several sequence(s).

    Args:
        audio (`mindspore.Tensor`, `np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[mindspore.Tensor]`,
            `List[List[float]]`):
            The sequence or batch of sequences to be padded. Each sequence can be a mindspore tensor, a numpy array,
            a list of float values, a list of numpy arrays, a list of mindspore tensors, or a list of list of float
            values. If `audio` is the output of Demucs, it has to be a mindspore tensor of shape
            `(batch_size, num_stems, channel_size, audio_length)`.
            Otherwise, it must be mono or stereo channel audio.
        truncation (`bool`, *optional*, default to `True`):
            Activates truncation to cut input sequences longer than *max_length* to *max_length*.
        pad_to_multiple_of (`int`, *optional*, defaults to None):
            If set will pad the sequence to a multiple of the provided value.

            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
            `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
        return_tensors (`str` or [`~utils.TensorType`], *optional*):
            If set, will return tensors instead of list of python integers. Acceptable values are:

            - `'tf'`: Return TensorFlow `tf.constant` objects.
            - `'pt'`: Return PyTorch `mindspore.Tensor` objects.
            - `'np'`: Return Numpy `np.ndarray` objects.
        return_attention_mask (`bool`, *optional*):
            Whether to return the attention mask. If left to the default, will return the attention mask according
            to the specific feature_extractor's default.

            [What are attention masks?](../glossary#attention-mask)

            <Tip>
            For Musicgen Melody models, audio `attention_mask` is not necessary.
            </Tip>

        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding
            index) among:

            - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
            sequence if provided).
            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
            acceptable input length for the model if that argument is not provided.
            - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
            lengths).
        max_length (`int`, *optional*):
            Maximum length of the returned list and optionally padding length (see above).
        sampling_rate (`int`, *optional*):
            The sampling rate at which the `audio` input was sampled. It is strongly recommended to pass
            `sampling_rate` at the forward call to prevent silent errors.
            Note that if `audio` is the output of Demucs, `sampling_rate` must be the sampling rate at which Demucs operates.
    """
    if sampling_rate is None:
        logger.warning_once(
            "It is strongly recommended to pass the `sampling_rate` argument to this function. "
            "Failing to do so can result in silent errors that might be hard to debug."
        )

    if isinstance(audio, mindspore.Tensor) and len(audio.shape) == 4:
        logger.warning_once(
            "`audio` is a 4-dimensional mindspore tensor and has thus been recognized as the output of `Demucs`. "
            "If this is not the case, make sure to read Musicgen Melody docstrings and "
            "to correct `audio` to get the right behaviour."
            "Link to the docstrings: https://huggingface.co/docs/transformers/main/en/model_doc/musicgen_melody"
        )
        audio = self._extract_stem_indices(audio, sampling_rate=sampling_rate)
    elif sampling_rate is not None and sampling_rate != self.sampling_rate:
        transform = Resample(sampling_rate, self.sampling_rate,
                         resample_method=ResampleMethod.SINC_INTERPOLATION,
                         rolloff=0.945, lowpass_filter_width=24)
        audio = transform(audio)

    is_batched = isinstance(audio, (np.ndarray, mindspore.Tensor)) and len(audio.shape) > 1
    is_batched = is_batched or (
        isinstance(audio, (list, tuple)) and (isinstance(audio[0], (mindspore.Tensor, np.ndarray, tuple, list)))
    )

    if is_batched and not isinstance(audio[0], mindspore.Tensor):
        audio = [mindspore.tensor(speech, dtype=mindspore.float32).unsqueeze(-1) for speech in audio]
    elif is_batched:
        audio = [speech.unsqueeze(-1) for speech in audio]
    elif not is_batched and not isinstance(audio, mindspore.Tensor):
        audio = mindspore.tensor(audio, dtype=mindspore.float32).unsqueeze(-1)

    if isinstance(audio[0], mindspore.Tensor) and audio[0].dtype is mindspore.float64:
        audio = [speech.to(mindspore.float32) for speech in audio]

    # always return batch
    if not is_batched:
        audio = [audio]

    if len(audio[0].shape) == 3:
        logger.warning_once(
            "`audio` has been detected as a batch of stereo signals. Will be convert to mono signals. "
            "If this is an undesired behaviour, make sure to read Musicgen Melody docstrings and "
            "to correct `audio` to get the right behaviour."
            "Link to the docstrings: https://huggingface.co/docs/transformers/main/en/model_doc/musicgen_melody"
        )
        # convert to mono-channel waveform
        audio = [stereo.mean(dim=0) for stereo in audio]

    batched_speech = BatchFeature({"input_features": audio})

    padded_inputs = self.pad(
        batched_speech,
        padding=padding,
        max_length=max_length if max_length else self.n_samples,
        truncation=truncation,
        pad_to_multiple_of=pad_to_multiple_of,
        return_attention_mask=return_attention_mask,
        return_tensors="ms",
    )

    input_features = self._ms_extract_fbank_features(padded_inputs["input_features"].squeeze(-1))

    padded_inputs["input_features"] = input_features

    if return_attention_mask:
        # rescale from raw audio length to spectrogram length
        padded_inputs["attention_mask"] = padded_inputs["attention_mask"][:, :: self.hop_length]

    if return_tensors is not None:
        padded_inputs = padded_inputs.convert_to_tensors(return_tensors)

    return padded_inputs

mindnlp.transformers.models.musicgen_melody.feature_extraction_musicgen_melody.MusicgenMelodyFeatureExtractor.__init__(feature_size=12, sampling_rate=32000, hop_length=4096, chunk_length=30, n_fft=16384, num_chroma=12, padding_value=0.0, return_attention_mask=False, stem_indices=[3, 2], **kwargs)

Initializes a MusicgenMelodyFeatureExtractor object.

PARAMETER	DESCRIPTION
self	The object itself.
feature_size	The size of the feature. Defaults to 12. TYPE: int DEFAULT: 12
sampling_rate	The sampling rate. Defaults to 32000. TYPE: int DEFAULT: 32000
hop_length	The hop length. Defaults to 4096. TYPE: int DEFAULT: 4096
chunk_length	The length of the chunk. Defaults to 30. TYPE: int DEFAULT: 30
n_fft	The number of FFT points. Defaults to 16384. TYPE: int DEFAULT: 16384
num_chroma	The number of chroma. Defaults to 12. TYPE: int DEFAULT: 12
padding_value	The padding value. Defaults to 0.0. TYPE: float DEFAULT: 0.0
return_attention_mask	Indicates whether to return attention mask. Defaults to False. TYPE: bool DEFAULT: False
stem_indices	The stem indices. Defaults to [3, 2]. TYPE: list DEFAULT: [3, 2]
**kwargs	Additional keyword arguments. DEFAULT: {}

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the provided feature size, sampling rate, hop length, chunk length, n_fft, or num_chroma is invalid.
TypeError	If the provided padding value is not a float.
TypeError	If the provided return_attention_mask is not a boolean.
TypeError	If stem_indices is not a list.

Source code in mindnlp\transformers\models\musicgen_melody\feature_extraction_musicgen_melody.py

def __init__(
    self,
    feature_size=12,
    sampling_rate=32000,
    hop_length=4096,
    chunk_length=30,
    n_fft=16384,
    num_chroma=12,
    padding_value=0.0,
    return_attention_mask=False,  # pad inputs to max length with silence token (zero) and no attention mask
    stem_indices=[3, 2],
    **kwargs,
):
    """
    Initializes a MusicgenMelodyFeatureExtractor object.

    Args:
        self: The object itself.
        feature_size (int, optional): The size of the feature. Defaults to 12.
        sampling_rate (int, optional): The sampling rate. Defaults to 32000.
        hop_length (int, optional): The hop length. Defaults to 4096.
        chunk_length (int, optional): The length of the chunk. Defaults to 30.
        n_fft (int, optional): The number of FFT points. Defaults to 16384.
        num_chroma (int, optional): The number of chroma. Defaults to 12.
        padding_value (float, optional): The padding value. Defaults to 0.0.
        return_attention_mask (bool, optional): Indicates whether to return attention mask. Defaults to False.
        stem_indices (list, optional): The stem indices. Defaults to [3, 2].
        **kwargs: Additional keyword arguments.

    Returns:
        None.

    Raises:
        ValueError: If the provided feature size, sampling rate, hop length, chunk length, n_fft, or num_chroma
            is invalid.
        TypeError: If the provided padding value is not a float.
        TypeError: If the provided return_attention_mask is not a boolean.
        TypeError: If stem_indices is not a list.
    """
    super().__init__(
        feature_size=feature_size,
        sampling_rate=sampling_rate,
        padding_value=padding_value,
        return_attention_mask=return_attention_mask,
        **kwargs,
    )
    self.n_fft = n_fft
    self.hop_length = hop_length
    self.chunk_length = chunk_length
    self.n_samples = chunk_length * sampling_rate
    self.sampling_rate = sampling_rate
    self.chroma_filters = mindspore.Tensor.from_numpy(
        chroma_filter_bank(sampling_rate=sampling_rate, num_frequency_bins=n_fft, tuning=0, num_chroma=num_chroma)
    ).float()
    self.spectrogram = Spectrogram(n_fft=n_fft, win_length=n_fft, hop_length=hop_length, power=2, center=True, pad=0, normalized=True)

    self.stem_indices = stem_indices

mindnlp.transformers.models.musicgen_melody.feature_extraction_musicgen_melody.MusicgenMelodyFeatureExtractor.to_dict()

Serializes this instance to a Python dictionary. Returns: Dict[str, Any]: Dictionary of all the attributes that make up this configuration instance.

Source code in mindnlp\transformers\models\musicgen_melody\feature_extraction_musicgen_melody.py

def to_dict(self) -> Dict[str, Any]:
    """
    Serializes this instance to a Python dictionary. Returns:
        `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance.
    """
    output = copy.deepcopy(self.__dict__)
    output["feature_extractor_type"] = self.__class__.__name__
    if "mel_filters" in output:
        del output["mel_filters"]
    if "window" in output:
        del output["window"]
    if "chroma_filters" in output:
        del output["chroma_filters"]
    if "spectrogram" in output:
        del output["spectrogram"]
    return output

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody

Text/audio processor class for MusicGen Melody

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor

Bases: ProcessorMixin

Constructs a MusicGen Melody processor which wraps a Wav2Vec2 feature extractor - for raw audio waveform processing - and a T5 tokenizer into a single processor class.

[MusicgenProcessor] offers all the functionalities of [MusicgenMelodyFeatureExtractor] and [T5Tokenizer]. See [~MusicgenProcessor.__call__] and [~MusicgenProcessor.decode] for more information.

PARAMETER	DESCRIPTION
feature_extractor	An instance of [MusicgenMelodyFeatureExtractor]. The feature extractor is a required input. TYPE: `MusicgenMelodyFeatureExtractor`
tokenizer	An instance of [T5Tokenizer]. The tokenizer is a required input. TYPE: `T5Tokenizer`

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

class MusicgenMelodyProcessor(ProcessorMixin):
    r"""
    Constructs a MusicGen Melody processor which wraps a Wav2Vec2 feature extractor - for raw audio waveform
    processing - and a T5 tokenizer into a single processor class.

    [`MusicgenProcessor`] offers all the functionalities of [`MusicgenMelodyFeatureExtractor`] and [`T5Tokenizer`]. See
    [`~MusicgenProcessor.__call__`] and [`~MusicgenProcessor.decode`] for more information.

    Args:
        feature_extractor (`MusicgenMelodyFeatureExtractor`):
            An instance of [`MusicgenMelodyFeatureExtractor`]. The feature extractor is a required input.
        tokenizer (`T5Tokenizer`):
            An instance of [`T5Tokenizer`]. The tokenizer is a required input.
    """
    feature_extractor_class = "MusicgenMelodyFeatureExtractor"
    tokenizer_class = ("T5Tokenizer", "T5TokenizerFast")

    def __init__(self, feature_extractor, tokenizer):
        """
        Initializes a MusicgenMelodyProcessor object.

        Args:
            self (MusicgenMelodyProcessor): The instance of the MusicgenMelodyProcessor class.
            feature_extractor: An object that performs feature extraction on melodies.
            tokenizer: An object that tokenizes melodies.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(feature_extractor, tokenizer)

    # Copied from transformers.models.musicgen.processing_musicgen.MusicgenProcessor.get_decoder_prompt_ids
    def get_decoder_prompt_ids(self, task=None, language=None, no_timestamps=True):
        """
        This method retrieves decoder prompt IDs for the MusicgenMelodyProcessor class.

        Args:
            self: The instance of the class.
            task (str): Specifies the task for which decoder prompt IDs are requested. Default is None.
            language (str): Specifies the language for which decoder prompt IDs are requested. Default is None.
            no_timestamps (bool): Indicates whether to exclude timestamps in the decoder prompt IDs. Default is True.

        Returns:
            None: This method does not return a value but processes the input parameters to retrieve decoder prompt IDs.

        Raises:
            None.
        """
        return self.tokenizer.get_decoder_prompt_ids(task=task, language=language, no_timestamps=no_timestamps)

    def __call__(self, audio=None, text=None, **kwargs):
        """
        Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the `audio`
        and `kwargs` arguments to MusicgenMelodyFeatureExtractor's [`~MusicgenMelodyFeatureExtractor.__call__`] if `audio`
        is not `None` to pre-process the audio. It also forwards the `text` and `kwargs` arguments to
        PreTrainedTokenizer's [`~PreTrainedTokenizer.__call__`] if `text` is not `None`. Please refer to the doctsring
        of the above two methods for more information.

        Args:
            audio (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`):
                The audio or batch of audios to be prepared. Each audio can be NumPy array or PyTorch tensor. In case
                of a NumPy array/PyTorch tensor, each audio should be a mono-stereo signal of shape (T), where T is the
                sample length of the audio.
            text (`str`, `List[str]`, `List[List[str]]`):
                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
            kwargs (*optional*):
                Remaining dictionary of keyword arguments that will be passed to the feature extractor and/or the
                tokenizer.

        Returns:
            [`BatchEncoding`]:
                A [`BatchEncoding`] with the following fields:

                - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`.
                - **input_features** -- Audio input features to be fed to a model. Returned when `audio` is not `None`.
                - **attention_mask** -- List of token indices specifying which tokens should be attended to by the model
                when `text` is not `None`.

                When only `audio` is specified, returns the timestamps attention mask.
        """
        sampling_rate = kwargs.pop("sampling_rate", None)

        if audio is None and text is None:
            raise ValueError("You need to specify either an `audio` or `text` input to process.")

        if text is not None:
            inputs = self.tokenizer(text, **kwargs)
        if audio is not None:
            audio_inputs = self.feature_extractor(audio, sampling_rate=sampling_rate, **kwargs)

        if text is None:
            return audio_inputs
        elif audio is None:
            return inputs
        else:
            inputs["input_features"] = audio_inputs["input_features"]
            return inputs

    # Copied from transformers.models.musicgen.processing_musicgen.MusicgenProcessor.batch_decode with padding_mask->attention_mask
    def batch_decode(self, *args, **kwargs):
        """
        This method is used to decode either batches of audio outputs from the MusicGen model, or batches of token ids
        from the tokenizer. In the case of decoding token ids, this method forwards all its arguments to T5Tokenizer's
        [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information.
        """
        audio_values = kwargs.pop("audio", None)
        attention_mask = kwargs.pop("attention_mask", None)

        if len(args) > 0:
            audio_values = args[0]
            args = args[1:]

        if audio_values is not None:
            return self._decode_audio(audio_values, attention_mask=attention_mask)
        else:
            return self.tokenizer.batch_decode(*args, **kwargs)

    # Copied from transformers.models.musicgen.processing_musicgen.MusicgenProcessor.decode
    def decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to T5Tokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the
        docstring of this method for more information.
        """
        return self.tokenizer.decode(*args, **kwargs)

    # Copied from transformers.models.musicgen.processing_musicgen.MusicgenProcessor._decode_audio with padding_mask->attention_mask
    def _decode_audio(self, audio_values, attention_mask = None) -> List[np.ndarray]:
        """
        This method strips any padding from the audio values to return a list of numpy audio arrays.
        """
        audio_values = to_numpy(audio_values)
        bsz, channels, seq_len = audio_values.shape

        if attention_mask is None:
            return list(audio_values)

        attention_mask = to_numpy(attention_mask)

        # match the sequence length of the padding mask to the generated audio arrays by padding with the **non-padding**
        # token (so that the generated audio values are **not** treated as padded tokens)
        difference = seq_len - attention_mask.shape[-1]
        padding_value = 1 - self.feature_extractor.padding_value
        attention_mask = np.pad(attention_mask, ((0, 0), (0, difference)), "constant", constant_values=padding_value)

        audio_values = audio_values.tolist()
        for i in range(bsz):
            sliced_audio = np.asarray(audio_values[i])[
                attention_mask[i][None, :] != self.feature_extractor.padding_value
            ]
            audio_values[i] = sliced_audio.reshape(channels, -1)

        return audio_values

    def get_unconditional_inputs(self, num_samples=1, return_tensors="ms"):
        """
        Helper function to get null inputs for unconditional generation, enabling the model to be used without the
        feature extractor or tokenizer.

        Args:
            num_samples (int, *optional*):
                Number of audio samples to unconditionally generate.

        Example:
            ```python
            >>> from transformers import MusicgenMelodyForConditionalGeneration, MusicgenMelodyProcessor
            ...
            >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")
            ...
            >>> # get the unconditional (or 'null') inputs for the model
            >>> processor = MusicgenMelodyProcessor.from_pretrained("facebook/musicgen-melody")
            >>> unconditional_inputs = processor.get_unconditional_inputs(num_samples=1)
            ...
            >>> audio_samples = model.generate(**unconditional_inputs, max_new_tokens=256)
            ```
        """
        inputs = self.tokenizer([""] * num_samples, return_tensors=return_tensors, return_attention_mask=True)
        inputs["attention_mask"][:] = 0

        return inputs

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.__call__(audio=None, text=None, **kwargs)

Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the audio and kwargs arguments to MusicgenMelodyFeatureExtractor's [~MusicgenMelodyFeatureExtractor.__call__] if audio is not None to pre-process the audio. It also forwards the text and kwargs arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.__call__] if text is not None. Please refer to the doctsring of the above two methods for more information.

PARAMETER	DESCRIPTION
audio	The audio or batch of audios to be prepared. Each audio can be NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each audio should be a mono-stereo signal of shape (T), where T is the sample length of the audio. TYPE: `np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]` DEFAULT: None
text	The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_split_into_words=True (to lift the ambiguity with a batch of sequences). TYPE: `str`, `List[str]`, `List[List[str]]` DEFAULT: None
kwargs	Remaining dictionary of keyword arguments that will be passed to the feature extractor and/or the tokenizer. TYPE: *optional* DEFAULT: {}

RETURNS

DESCRIPTION

[BatchEncoding]: A [BatchEncoding] with the following fields: input_ids -- List of token ids to be fed to a model. Returned when text is not None. input_features -- Audio input features to be fed to a model. Returned when audio is not None. attention_mask -- List of token indices specifying which tokens should be attended to by the model when text is not None. When only audio is specified, returns the timestamps attention mask.

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def __call__(self, audio=None, text=None, **kwargs):
    """
    Main method to prepare for the model one or several sequences(s) and audio(s). This method forwards the `audio`
    and `kwargs` arguments to MusicgenMelodyFeatureExtractor's [`~MusicgenMelodyFeatureExtractor.__call__`] if `audio`
    is not `None` to pre-process the audio. It also forwards the `text` and `kwargs` arguments to
    PreTrainedTokenizer's [`~PreTrainedTokenizer.__call__`] if `text` is not `None`. Please refer to the doctsring
    of the above two methods for more information.

    Args:
        audio (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`):
            The audio or batch of audios to be prepared. Each audio can be NumPy array or PyTorch tensor. In case
            of a NumPy array/PyTorch tensor, each audio should be a mono-stereo signal of shape (T), where T is the
            sample length of the audio.
        text (`str`, `List[str]`, `List[List[str]]`):
            The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
            (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
            `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
        kwargs (*optional*):
            Remaining dictionary of keyword arguments that will be passed to the feature extractor and/or the
            tokenizer.

    Returns:
        [`BatchEncoding`]:
            A [`BatchEncoding`] with the following fields:

            - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`.
            - **input_features** -- Audio input features to be fed to a model. Returned when `audio` is not `None`.
            - **attention_mask** -- List of token indices specifying which tokens should be attended to by the model
            when `text` is not `None`.

            When only `audio` is specified, returns the timestamps attention mask.
    """
    sampling_rate = kwargs.pop("sampling_rate", None)

    if audio is None and text is None:
        raise ValueError("You need to specify either an `audio` or `text` input to process.")

    if text is not None:
        inputs = self.tokenizer(text, **kwargs)
    if audio is not None:
        audio_inputs = self.feature_extractor(audio, sampling_rate=sampling_rate, **kwargs)

    if text is None:
        return audio_inputs
    elif audio is None:
        return inputs
    else:
        inputs["input_features"] = audio_inputs["input_features"]
        return inputs

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.__init__(feature_extractor, tokenizer)

Initializes a MusicgenMelodyProcessor object.

PARAMETER	DESCRIPTION
self	The instance of the MusicgenMelodyProcessor class. TYPE: MusicgenMelodyProcessor
feature_extractor	An object that performs feature extraction on melodies.
tokenizer	An object that tokenizes melodies.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def __init__(self, feature_extractor, tokenizer):
    """
    Initializes a MusicgenMelodyProcessor object.

    Args:
        self (MusicgenMelodyProcessor): The instance of the MusicgenMelodyProcessor class.
        feature_extractor: An object that performs feature extraction on melodies.
        tokenizer: An object that tokenizes melodies.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(feature_extractor, tokenizer)

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.batch_decode(*args, **kwargs)

This method is used to decode either batches of audio outputs from the MusicGen model, or batches of token ids from the tokenizer. In the case of decoding token ids, this method forwards all its arguments to T5Tokenizer's [~PreTrainedTokenizer.batch_decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def batch_decode(self, *args, **kwargs):
    """
    This method is used to decode either batches of audio outputs from the MusicGen model, or batches of token ids
    from the tokenizer. In the case of decoding token ids, this method forwards all its arguments to T5Tokenizer's
    [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information.
    """
    audio_values = kwargs.pop("audio", None)
    attention_mask = kwargs.pop("attention_mask", None)

    if len(args) > 0:
        audio_values = args[0]
        args = args[1:]

    if audio_values is not None:
        return self._decode_audio(audio_values, attention_mask=attention_mask)
    else:
        return self.tokenizer.batch_decode(*args, **kwargs)

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.decode(*args, **kwargs)

This method forwards all its arguments to T5Tokenizer's [~PreTrainedTokenizer.decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to T5Tokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the
    docstring of this method for more information.
    """
    return self.tokenizer.decode(*args, **kwargs)

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.get_decoder_prompt_ids(task=None, language=None, no_timestamps=True)

This method retrieves decoder prompt IDs for the MusicgenMelodyProcessor class.

PARAMETER	DESCRIPTION
self	The instance of the class.
task	Specifies the task for which decoder prompt IDs are requested. Default is None. TYPE: str DEFAULT: None
language	Specifies the language for which decoder prompt IDs are requested. Default is None. TYPE: str DEFAULT: None
no_timestamps	Indicates whether to exclude timestamps in the decoder prompt IDs. Default is True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
None	This method does not return a value but processes the input parameters to retrieve decoder prompt IDs.

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def get_decoder_prompt_ids(self, task=None, language=None, no_timestamps=True):
    """
    This method retrieves decoder prompt IDs for the MusicgenMelodyProcessor class.

    Args:
        self: The instance of the class.
        task (str): Specifies the task for which decoder prompt IDs are requested. Default is None.
        language (str): Specifies the language for which decoder prompt IDs are requested. Default is None.
        no_timestamps (bool): Indicates whether to exclude timestamps in the decoder prompt IDs. Default is True.

    Returns:
        None: This method does not return a value but processes the input parameters to retrieve decoder prompt IDs.

    Raises:
        None.
    """
    return self.tokenizer.get_decoder_prompt_ids(task=task, language=language, no_timestamps=no_timestamps)

mindnlp.transformers.models.musicgen_melody.processing_musicgen_melody.MusicgenMelodyProcessor.get_unconditional_inputs(num_samples=1, return_tensors='ms')

Helper function to get null inputs for unconditional generation, enabling the model to be used without the feature extractor or tokenizer.

PARAMETER	DESCRIPTION
num_samples	Number of audio samples to unconditionally generate. TYPE: int, *optional* DEFAULT: 1

Example

>>> from transformers import MusicgenMelodyForConditionalGeneration, MusicgenMelodyProcessor
...
>>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")
...
>>> # get the unconditional (or 'null') inputs for the model
>>> processor = MusicgenMelodyProcessor.from_pretrained("facebook/musicgen-melody")
>>> unconditional_inputs = processor.get_unconditional_inputs(num_samples=1)
...
>>> audio_samples = model.generate(**unconditional_inputs, max_new_tokens=256)

Source code in mindnlp\transformers\models\musicgen_melody\processing_musicgen_melody.py

def get_unconditional_inputs(self, num_samples=1, return_tensors="ms"):
    """
    Helper function to get null inputs for unconditional generation, enabling the model to be used without the
    feature extractor or tokenizer.

    Args:
        num_samples (int, *optional*):
            Number of audio samples to unconditionally generate.

    Example:
        ```python
        >>> from transformers import MusicgenMelodyForConditionalGeneration, MusicgenMelodyProcessor
        ...
        >>> model = MusicgenMelodyForConditionalGeneration.from_pretrained("facebook/musicgen-melody")
        ...
        >>> # get the unconditional (or 'null') inputs for the model
        >>> processor = MusicgenMelodyProcessor.from_pretrained("facebook/musicgen-melody")
        >>> unconditional_inputs = processor.get_unconditional_inputs(num_samples=1)
        ...
        >>> audio_samples = model.generate(**unconditional_inputs, max_new_tokens=256)
        ```
    """
    inputs = self.tokenizer([""] * num_samples, return_tensors=return_tensors, return_attention_mask=True)
    inputs["attention_mask"][:] = 0

    return inputs