Post
1691
Super Exciting New Paper By Meta🤖🧠🚀
Discrete Flow Matching:
Introduces a new framework/algorithm for generating text/code without having to predict auto-regressively or one “word” at a time as traditional GPT models do. It generates all parts of the text/code at once.
The algorithm does this by slowly transforming random noise (source) into meaningful text (data). It learns how to transform samples along a path created between source and target using a "probability velocity" that describes how probabilities change over time. During generation, DFM starts with a random sample and iteratively updates it using this learned velocity, gradually transforming it into a sample from the target distribution. This allows for non-autoregressive generation.
They were able to scale models of up to 1.7B parameters achieving impressive scores on HumanEval and MBPP for coding, significantly closing the gap between autoregressive models and discrete flow models.
Though in its infancy, it sure does hold a promising future as leading research scientists argue non-autoregressive methods yield better reasoning.
Discrete Flow Matching:
Introduces a new framework/algorithm for generating text/code without having to predict auto-regressively or one “word” at a time as traditional GPT models do. It generates all parts of the text/code at once.
The algorithm does this by slowly transforming random noise (source) into meaningful text (data). It learns how to transform samples along a path created between source and target using a "probability velocity" that describes how probabilities change over time. During generation, DFM starts with a random sample and iteratively updates it using this learned velocity, gradually transforming it into a sample from the target distribution. This allows for non-autoregressive generation.
They were able to scale models of up to 1.7B parameters achieving impressive scores on HumanEval and MBPP for coding, significantly closing the gap between autoregressive models and discrete flow models.
Though in its infancy, it sure does hold a promising future as leading research scientists argue non-autoregressive methods yield better reasoning.