This week is the last week of Google Summer of Code for this year edition, and I'll do a summary of what we have been doing and future plans for the Log Detective integration in openSUSE.

I wrote a blog post about this project when it starts, so if you didn't read, you can take a look.

All the work and code was done in the logdetective-obs github repository.

Aazam, aka the intern, also wrote some blog post about his work.

Initial Research

The idea of the project was to explore ways of integration of LogDetective with the openSUSE dev workflow. So we started collecting some build failures in the openSUSE build service and testing with log detective to check if the output is smart or even relevant.

The intern creates a script to collect log from build failures and we store the LLM answer.

Doing this we detected that the log-detective.com explain is very slow and the local tool is less accurate.

The results that we got weren't too good, but at this point of the project is something expected. The model is being trained and will improve with every new log that users send.

Local vs Remote, model comparison

The logdetective command line tool is nice, but LLM requires a lot of resources to run locally. This tool also uses the models published by fedora in huggingface.co, so it's not as accurate as the remote instance that has a better trained model and is up to date.

In any case, the local logdetective tool is interesting and, at this moment, it's faster than the deployed log-detective.com.

Using this tool, Aazam did some research, comparing the output with different models.

logdetective apache-arrow_standard_x86_64.log\
  --model unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF\
  -F Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf\
  --prompt ~/prompt.yaml

So using other specific models, the logdetective tool can return better results.

The plugin

openSUSE packagers uses the osc tool, to build packages in build.opensuse.org. This tool can be extended with plugins and we created a new plugin to use log-detective.

So a packager can get some AI explanation about a build failure with a simple command:

$ osc ld -r -p devel:languages:python --package python-pygame --repo openSUSE_Tumbleweed
🔍 Running logdetective for python-pygame (openSUSE_Tumbleweed/x86_64)...
Log url: https://api.opensuse.org/public/build/devel:languages:python/openSUSE_Tumbleweed/x86_64/python-pygame/_log
🌐 Sending log to LogDetective API...
 The RPM build process for the `python-pygame` package failed during the
`%check` phase, as indicated by the "Bad exit status from /var/tmp/rpm-
tmp.hiddzT (%check)" log snippet. This failure occurred due to seven test
failures, one error, and six skipped steps in the test suite of the package.



The primary issue causing the build failure seems to be the traceback error in
the `surface_test.py` file, which can be seen in the following log snippets:



[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 284, in test_copy_rle
]
[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 274, in
test_mustlock_surf_alpha_rle
]
[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 342, in test_solarwolf_rle_usage
]



These traceback errors suggest that there are issues with the test functions
`test_copy_rle`, `test_mustlock_surf_alpha_rle`, and `test_solarwolf_rle_usage`
in the `surface_test.py` file. To resolve this issue, you should:



1. Identify the root cause of the traceback errors in each of the functions.
2. Fix the issues found, either by modifying the test functions or correcting
the underlying problem they are testing for.
3. Re-build the RPM package with the updated `surface_test.py` file.



It's also recommended to review the other test failures and skipped steps, as
they might indicate other issues with the package or its dependencies that
should be addressed.

Or if we are building locally, it's possible to run using the installed logdetective command line. But this method is less accurate, because the model used is not smart enough, yet:

$ osc build
[...]
$ osc ld --local-log --repo openSUSE_Tumbleweed
Found local build log: /var/tmp/build-root/openSUSE_Tumbleweed-x86_64/.build.log
🚀 Analyzing local build log: /tmp/tmpuu1sg1q0
INFO:logdetective:Loading model from fedora-copr/Mistral-7B-Instruct-v0.3-GGUF
...

Future plans

1. Adding submit-log functionality to osc-ld-plugin. In-progress. This will make it easier to collaborate with log-detective.com, allowing openSUSE packagers to send new data for model training.
2. Gitea bot. openSUSE development workflow is moving to git, so we can create a bot that comment on Pull Request with packages that fails to build. Something similar to what fedora people have for centos gitlab
3. Add a new tab to log-detective.com website to submit logs directly from OBS

This was an interesting Summer of Code project. I learned a bit about LLM. I think that this project has a lot of potential, this can be integrated in different workflows and an expert LLM can be a great tool to help packagers, summarizing big logs, tagging similar failures, etc.

We have a lot of packages and a lot of data in OBS, so we should start to feed the LLM with this data, and in combination with the fedora project, the log-detective could be a real expert in open source RPM packaging.