Use the Local LLM feature

This page outlines the steps involved in using the Local LLM integration.

• This feature is in the experimental phase and might contain bugs. Your bug reports, sent to [e-mail], are essential for enhancing its stability.

• Prior to submitting bug reports, please ensure that your system meets the requirements outlined in the Installation section.

Configuration

1. Go to the Server tab.

2. Start the server by clicking the Start server button. The initial launch may take some time, so please wait until the message Server is running on port <PORT> appears. You can view the server status, including the PID of the running process, at the bottom of the view.

The local server powers the local LLM capabilities of BurpGPT Pro, and all computations are made locally, ensuring complete data privacy of your prompts and HTTP traffic.

1. In scenarios with restricted system PATH access, manually providing the Python executable's absolute path in the designated Python path field ensures the local server's initiation. If left blank, the system PATH will be used for automatic Python binary detection.

2. Switch to the Local LLM tab and select one of the pre-built models from the Model dropdown field. The associated number of datapoints used to train the model is displayed under the Model size field.

When selecting certain models from the Hugging Face hub, such as meta-llama/Meta-Llama-3.1-8B, you might encounter the following error:

Failed to load model and tokenizer: You are trying to access a gated repository. Make sure you have access to it at https://huggingface.co/<MODEL>.

If this happens, follow these steps:

1. Request Access: Complete the COMMUNITY LICENSE AGREEMENT form located on the model's repository page. You may need to agree to share your contact information.

2. Authenticate: Log in with your Hugging Face account by following the instructions at Hugging Face CLI Login.

3. Retry: After completing the above steps, attempt to load the model again.

Keep in mind that the larger the number of datapoints used to train a model, the larger the resulting model size will be. In some cases, the model size can be in the gigabytes range, which may impact processing time for your queries.

When selecting models on the Hugging Face hub, it is recommended to choose instruct models, typically suffixed with it or instruct. These models work best with BurpGPT Pro. The built-in list includes examples from models provided by Google, Meta, Microsoft, and the OpenAI Community.

1. To optimise the performance of your local model, set the Max prompt length and Max token length parameters appropriately. By adjusting these parameters, you can optimise the amount of information you can provide to the model and achieve the desired length of the response.

   • Max prompt length: determines the maximum size of your prompt once the placeholders have been replaced.

   • Max token length: specifies the maximum length allowed for both the prompt and the model response. This variable depends on the model type and technology. For instance, GPT-2-based models usually have a max token length of 1,024, while GPT-3-based models have a larger value of 2,048.

Test and locally cache the model

After configuring the model settings, you can test and cache the selected model by clicking on the Test / Cache model button. This will send a test query to the local server and prompt BurpGPT to download and cache the selected model locally as illustrated in the screenshot below. Subsequent queries will then use the cached model instead of requiring a new download.

Local LLM: Model download and caching.

Please keep in mind that the caching process can take a considerable amount of time, depending on the model size and the speed of your internet connection. Additionally, ensure that your file system has sufficient free space to accommodate the downloaded models.

Please note that the models you download will be stored in the cache folder on your system at the following locations:

• Linux and Mac: ~/.cache/huggingface/transformers/

• Windows: %userprofile%/.cache/huggingface/transformers/

These folders, containing all locally cached models, can serve as a base to train your own custom models. This topic will be covered in a future post.

Once the model has finished downloading and processing your request, the results of your query will be presented in a dialog box, as follows:

Local LLM: Test results.

Analyse HTTP traffic

Finally, to scan your HTTP traffic against your model and prompt, you can either:

• Instruct BurpGPT to use the local model when performing passive scans with Burp Suite by clicking on the Passive Scan: Local LLM button.

• Use the custom context menu actions to send relevant requests for analysis, by simply right-clicking in request/response and selecting Extensions -> BurpGPT Pro -> Send to local LLM.

View GPT-generated insights

A new Information-level severity issue, named GPT-generated insights, will appear under Target -> Site map and Dashboard -> Issue Activity.

This issue will provide detailed information about the query made to the selected model and will also include the model response as illustrated in the following screenshot:

Local LLM: GPT-generated insights.