Running falcon ai model on mojo lang

Question

How can we run https://huggingface.co/tiiuae/falcon-180B the one of the current best models on mojo lang which is faster than python 35k? When we just copy this:

`from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-180b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

`

we got error as below:

`

error: TODO: expressions are not yet supported at the file scope level
model = "tiiuae/falcon-180b"
^
hello.mojo:7:1: error: use of unknown declaration 'model'
model = "tiiuae/falcon-180b"
^~~~~
hello.mojo:9:1: error: TODO: expressions are not yet supported at the file scope level
tokenizer = AutoTokenizer.from_pretrained(model)
^
hello.mojo:3:6: error: unable to locate module 'transformers'
from transformers import AutoTokenizer, AutoModelForCausalLM
     ^
hello.mojo:10:1: error: TODO: expressions are not yet supported at the file scope level
pipeline = transformers.pipeline(
^
hello.mojo:18:1: error: TODO: expressions are not yet supported at the file scope level
sequences = pipeline(
^
hello.mojo:18:13: error: use of unknown declaration 'pipeline'
sequences = pipeline(
            ^~~~~~~~
hello.mojo:26:12: error: use of unknown declaration 'sequences'
for seq in sequences:
           ^~~~~~~~~
hello.mojo:27:5: error: TODO: expressions are not yet supported at the file scope level
    print(f"Result: {seq['generated_text']}")
    ^
hello.mojo:27:12: error: expected ')' in call argument list
    print(f"Result: {seq['generated_text']}")
           ^
hello.mojo:27:12: error: statements must start at the beginning of a line
    print(f"Result: {seq['generated_text']}")
           ^
mojo: error: failed to parse the provided Mojo

`

When the code from hugging face is copied and pasted in mojo environment, model should be running.

Answer 1

i'm having the same issue, i think they haven't rolled out many of the feature

Answer 2

first you need to install all the dependencies in your base enviroment if you are using conda because that's from where mojo reads python packages then try this piece of code :

from python import Python
def main():
    let t = Python.import_module("transformers")
    let torch = Python.import_module("torch")
    let model = "tiiuae/falcon-180b"
    
    let tokenizer = t.AutoTokenizer.from_pretrained(model)
    let pipeline = t.pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        torch_dtype=torch.bfloat16,
        trust_remote_code=True,
        device_map="auto",
    )
    let sequences = pipeline(
        "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
        max_length=200,
        do_sample=True,
        top_k=10,
        num_return_sequences=1,
        eos_token_id=tokenizer.eos_token_id,
    )
    for seq in sequences:
        let seq_printed=seq['generated_text']
        print("Result: ", seq_printed)