Use the Local LLM feature
This page outlines the steps involved in using the Local LLM integration.
Last updated
Was this helpful?
This page outlines the steps involved in using the Local LLM integration.
Last updated
Was this helpful?
This feature is in the experimental phase and might contain bugs. Your bug reports, sent to , are essential for enhancing its stability.
Prior to submitting bug reports, please ensure that your system meets the requirements outlined in the section.
Go to the Server tab.
Start the server by clicking the Start server button. The initial launch may take some time, so please wait until the message Server is running on port <PORT> appears. You can view the server status, including the PID of the running process, at the bottom of the view.
In scenarios with restricted system PATH access, manually providing the Python executable's absolute path in the designated Python path field ensures the local server's initiation. If left blank, the system PATH will be used for automatic Python binary detection.
Switch to the Local LLM tab and select one of the pre-built models from the Model dropdown field. The associated number of datapoints used to train the model is displayed under the Model size field.
When selecting certain models from the Hugging Face hub, such as meta-llama/Meta-Llama-3.1-8B, you might encounter the following error:
Failed to load model and tokenizer: You are trying to access a gated repository. Make sure you have access to it at https://huggingface.co/<MODEL>.
If this happens, follow these steps:
Request Access: Complete the COMMUNITY LICENSE AGREEMENT form located on the model's repository page. You may need to agree to share your contact information.
Authenticate: Log in with your Hugging Face account by following the instructions at .
Retry: After completing the above steps, attempt to load the model again.
To optimise the performance of your local model, set the Max prompt length and Max token length parameters appropriately. By adjusting these parameters, you can optimise the amount of information you can provide to the model and achieve the desired length of the response.
Max prompt length: determines the maximum size of your prompt once the placeholders have been replaced.
Max token length: specifies the maximum length allowed for both the prompt and the model response. This variable depends on the model type and technology. For instance, GPT-2-based models usually have a max token length of 1,024, while GPT-3-based models have a larger value of 2,048.
Please note that the models you download will be stored in the cache folder on your system at the following locations:
Linux and Mac: ~/.cache/huggingface/transformers/
Windows: %userprofile%/.cache/huggingface/transformers/
These folders, containing all locally cached models, can serve as a base to train your own custom models. This topic will be covered in a future post.
Once the model has finished downloading and processing your request, the results of your query will be presented in a dialog box, as follows:
Finally, to scan your HTTP traffic against your model and prompt, you can either:
Instruct BurpGPT to use the local model when performing passive scans with Burp Suite by clicking on the Passive Scan: Local LLM button.
Use the custom context menu actions to send relevant requests for analysis, by simply right-clicking in request/response and selecting Extensions -> BurpGPT Pro -> Send to local LLM.
A new Information-level severity issue, named GPT-generated insights, will appear under Target -> Site map and Dashboard -> Issue Activity.
This issue will provide detailed information about the query made to the selected model and will also include the model response as illustrated in the following screenshot:
After , you can test and cache the selected model by clicking on the Test / Cache model button. This will send a test query to the local server and prompt BurpGPT to download and cache the selected model locally as illustrated in the screenshot below. Subsequent queries will then use the cached model instead of requiring a new download.
