New Package: LLM Scorer / Profiler / Bench based on system

In Nanocoder we offer LLM recommendations via a /recommendations command. This has proven useful for users to see how well a particular LLM will run on a system.

We're taking this idea and building an isolated package that takes in a list of models and system information, returning a series of metrics for each model and an overall performance score.

How it might work...

This package produces metrics for each model, based on...

• User's system information (memory, CPU, GPU, TPU, Swap, Arch, ...)
• LLM's model card information (arch, quant, ...)
• LLM's skills (vision, tools, ...)

... then rate how well the LLM operates on the user's system.

Primarily this package will be used to rate local LLMs on the runtime's system, but we also see merits in scoring skills and allowing you to override system information too. We also need to consider when the list of models features entries that are cloud-side, meaning system information doesn't weigh in nearly as much.

Usage may look something like this:

const result = llmScorer({
    models: [
        {
            model: 'meta-llama/Llama-3.1-8B',
            quantization: ...,
            arch: 'llama',
            // Many other optional parameters for this model...
        },
        {
            model: 'meta-llama/Llama-3.2-1B',
            quantization: ...,
            arch: 'llama',
        },
    ],
    system: {
        // optionally override default behaviour of pulling system information
        memoryGB: 24,
        // ...
    }
})

And the results will look something like this, ordered by best rated model first:

[
    {
        model: "meta-llama/Llama-3.1-8B",
        score: 0.8,
        metrics: {
            ...
        }
    },
    {
        ...
    },
]

What Next

We're in the process of building an initial solution and plan to release this on a public repository to gather feedback from our community and evolve it.

We recognise that there's many angles and opinions to consider for this, so we welcome contributors and our community to help form how this package works.

We'll announce the alpha version of this package once it is prepared and released.

Community

We're a small community-led team building local and privacy-first AI solutions under the Nano Collective and would love your help! Whether you're interested in contributing code, documentation, or just being part of our community, there are several ways to get involved.

If you want to contribute to this new concept:

• Drop comments and thoughts below, or join our Discord server

If you want to be part of our community or help with other aspects like code, design or marketing:

• Join our Discord server to connect with other users, ask questions, share ideas, and get help: Join our Discord server

• Head to our GitHub issues or discussions to open and join current conversations with others in the community.