Set Up Your Own ChatGPT: Ollama + Open WebUI for Data That Never

Set Up Your Own ChatGPT: Ollama + Open WebUI for Data That Never Leaves Home

As artificial intelligence models rapidly integrate into our lives, privacy concerns are growing in parallel. Especially for companies or individuals working with sensitive data, sending information to cloud-based services can pose a serious risk. At this point, setting up your own local Large Language Model (LLM) infrastructure offers a great solution. In this guide, I will explain step-by-step how to set up your own chat interface using tools like Ollama and Open WebUI, ensuring your data never leaves your system. This approach allows you to both reduce costs and maximize your data security.

This setup is particularly important for those like me, with a background in enterprise software development, who believe that data flows should always follow the most secure path. In the past, working on a production ERP, transferring supply chain data to external systems without anonymization could lead to serious security vulnerabilities. This is where local LLM solutions come into play.

Why You Should Set Up Your Own Local LLM

While cloud-based LLM services are incredibly convenient, they come with some fundamental drawbacks. Most importantly, every piece of data you input is potentially sent to the service provider's servers. This can be unacceptable, especially when dealing with financial data, patient information, trade secrets, or sensitive code in your personal projects. By setting up your own local LLM, you eliminate these risks.

In recent months, while working on my side project, a financial calculator, I felt the need to use an LLM for complex financial analyses. However, the details of these analyses could not be leaked externally. This situation led me to search for a solution where I could keep my data under my own control. Ollama and Open WebUI emerged as the most practical and powerful duo in my search.

ℹ️ Data Privacy and Control

A local LLM solution gives you full control over where your data is processed and who can access it. This is critically important, especially in regulated industries or projects handling sensitive data.

Ollama: The Easiest Way to Run Local LLMs

Ollama is a platform that allows you to easily run various LLM models on your local machine. Its setup is quite simple, and you can download and run popular models with a few commands. One of Ollama's biggest advantages is that it abstracts the processes of downloading, managing, and running models. This way, instead of dealing with complex configurations, you can start using the models directly.

One of the things that attracted me to Ollama was its simplicity and flexibility. My experience in system administration, in particular, has taught me how time-consuming complex infrastructure setups can be. Ollama incredibly simplifies this process, making the power of LLMs accessible to a wider audience. For example, while optimizing systemd units on a server recently, I was surprised to see how quickly Ollama could be installed and run.

Ollama Installation

To install Ollama, simply download the version suitable for your operating system from the official website. By following the installation wizard, you can have Ollama up and running in a few minutes.

1. Download: Go to https://ollama.com/ and click on the download link for your operating system.
2. Installation: Run the downloaded file to complete the installation. Default settings are usually sufficient.

After the installation is complete, you can open your terminal and verify that Ollama is running.

ollama --version

This command will show the installed version of Ollama.

Downloading and Running Models

Downloading models with Ollama is also quite simple. For example, to download the Llama 3 model, you can use the following command:

ollama pull llama3

The model download process may take some time, depending on your internet speed and the size of the model. Large models like Llama 3 can take up several GBs of space.

Once the model is downloaded, you can run it with a chat interface using the following command:

ollama run llama3

This command will start a chat session in your terminal. You can now ask the Llama 3 model questions directly from the terminal.

>>> Merhaba, nasılsın?
Hello! I am Llama 3, an artificial intelligence language model. How can I help you?

This terminal-based usage is great for quick tests and command-line automation. However, for a more user-friendly interface, we need tools like Open WebUI.

Open WebUI: Your Own ChatGPT Interface

Open WebUI (formerly Ollama WebUI) is a web-based interface for LLMs running with Ollama. With this tool, you can experience a ChatGPT-like interface, switch between different models, manage your chats, and even make custom settings. One of Open WebUI's biggest advantages is its relatively simple installation, often easily set up with Docker.

For me, Open WebUI was more than just an interface; it was a tool that made my interaction with LLMs more efficient. Especially when preparing complex prompts or experimenting with different model parameters, the ease provided by the visual interface was invaluable. Once, while trying to develop an automatic code completion feature for the backend of my side project, I found the ability to quickly test different prompt strategies with Open WebUI. This reduced a trial-and-error process that could have taken hours to mere minutes.

Open WebUI Installation (with Docker)

The most common and recommended way to install Open WebUI is by using Docker. If Docker is not installed on your system, you will need to install it first.

1. Docker Installation: Follow the installation instructions for your operating system from Docker's official documentation.

2. Open WebUI Docker Compose File: Save the following content to a file named docker-compose.yml:
   

   version: '3.8'
   
   services:
     webui:
       image: ghcr.io/open-webui/open-webui:main
       container_name: open-webui
       ports:
         - "3000:8080"
       volumes:
         - ollama_webui:/app/backend/data
       environment:
         - OLLAMA_BASE_URL=http://host.docker.internal:11434 # Where Ollama is running
         - PROGRESSIVE_PLUGINS_ENABLED=true
       restart: always
   
   volumes:
     ollama_webui:
   

   Important Note: The OLLAMA_BASE_URL setting specifies the address where Ollama is running. If you are running Ollama directly on the host machine and not inside Docker, http://host.docker.internal:11434 is usually the correct address. If Ollama is also running inside Docker, you might need to use the service name (e.g., http://ollama:11434). In this example, we assume Ollama is running on the host machine.

3. Running Ollama on the Host Machine: If you want to make Ollama accessible via host.docker.internal as in the Docker Compose file, you need to run Ollama on your host machine from the terminal or as a separate service.
   

   # Start Ollama
   ollama serve &
   
   # Or download and run the model with ollama CLI
   ollama pull llama3
   ollama run llama3
   

   If you want to run Ollama also inside Docker, you can update your Docker Compose file as follows:
   

   version: '3.8'
   
   services:
     ollama:
       image: ollama/ollama:latest
       container_name: ollama
       ports:
         - "11434:11434"
       volumes:
         - ollama_data:/root/.ollama
       restart: unless-stopped
   
     webui:
       image: ghcr.io/open-webui/open-webui:main
       container_name: open-webui
       ports:
         - "3000:8080"
       environment:
         - OLLAMA_BASE_URL=http://ollama:11434 # Name of the Ollama service
         - PROGRESSIVE_PLUGINS_ENABLED=true
       restart: always
   
   volumes:
     ollama_data:
     ollama_webui:
   

   In this case, open the terminal in the directory where you saved the docker-compose.yml file and run the following command:
   

   docker-compose up -d
   

   This command will start the Ollama and Open WebUI services in the background.

4. Accessing Open WebUI: Go to http://localhost:3000 in your browser. On your first login, you will be asked to set a username and password. This will be your administrator account for Open WebUI.

After logging in, you can go to the "Models" tab in the left menu to see the models you have downloaded to Ollama and download new ones.

Model Selection and Usage

Through Open WebUI, you can see all the models you have downloaded to Ollama. If you want to download a new model, you can use the search bar in the "Models" tab. For example, by typing "mistral", you can find and download Mistral AI models.

# To download the Mistral model via Open WebUI
# Can be done via Ollama CLI in the terminal or from the Open WebUI interface.
ollama pull mistral

When starting a chat, you can select the desired model from the dropdown menu at the top. This is a great way to compare the capabilities of different models and find the most suitable one for your project. For example, you can choose a more capable model for code generation or a faster model for general conversations.

Using LLMs Without Your Data Leaving Home: A PRAGMATIC Approach

The biggest benefit of a local LLM setup is ensuring data privacy. All data processing and model inference happen on your machine. This is critical, especially for the following situations:

• Sensitive Corporate Data: Financial reports, customer information, patent applications, R&D projects, etc., are risky to send to cloud services. Local LLMs allow you to analyze this data securely.
• Confidential Projects: Code for applications you are developing, architectural designs, or product information not yet publicly disclosed.
• Personal Data: Protecting the privacy of your personal information such as medical history, financial status, or private thoughts.

In my side project, a financial calculation application, all data entered by users was extremely sensitive. Sending this data to any third-party server could lead to trustworthiness issues. With Ollama and Open WebUI, I eliminated these concerns by running all calculation and analysis processes on my local machine.

To give an example, we considered using an LLM for logistics optimization in a production ERP. However, exporting raw shipment data raised serious security concerns. Eventually, we developed a prototype by anonymizing the data and running it on a local LLM. This prototype showed the potential to improve shipment times by 15% while ensuring data privacy.

💡 Speed and Cost Advantage

Local LLMs can be more cost-effective than cloud-based services, especially with continuous use. They can also offer faster response times due to the absence of network latency.

LLM Model Selection: Finding the Right Model for Your Needs

There are many different LLM models on the market, each with its unique strengths and weaknesses. Choosing the right model is critical for the success of your project. Here are some factors to consider when making your selection:

• Model Size and Hardware Requirements: Larger models are generally more capable but require more RAM and processing power. You should choose a model based on your machine's hardware. For example, 7B (7 billion parameter) models can often run with reasonable performance on mid-range GPUs or even modern CPUs. 70B models, however, typically require powerful GPUs.
• Task Focus: Some models are good for general-purpose conversations (e.g., Llama 3 8B), while others are more successful in specific tasks like code generation (e.g., CodeLlama) or mathematical problems.
• Licensing: If you plan to use it in commercial projects, you should carefully check the model's license. Some models do not allow commercial use, while others may allow it under certain conditions.
• Language Support: If you expect high performance in Turkish or other languages, you should choose models that support these languages well.

In my experience, the Llama 3 8B model offers a great balance for general-purpose use and fast response times. In situations requiring more complex code analysis or text generation, I prefer more specific models like CodeLlama. In my side project, I tried the Llama 3 70B model for users to query complex financial scenarios, and the results were quite impressive, but I needed a powerful GPU to run this model.

Evaluating Model Performance

The best way to evaluate a model's performance is to test it in your own use cases. Open WebUI provides an excellent platform for quickly experimenting with different models. You can observe the following metrics:

• Response Speed: How quickly does it respond to a prompt? This is especially important for applications requiring real-time interaction.
• Response Quality: The accuracy, consistency, and contextual relevance of the model's answers.
• Resource Consumption: How much RAM, CPU, or GPU the model uses. This affects your system's overall performance.

Recently, in a project, I compared the llama3:8b and mistral:7b models. While llama3:8b provided more fluid and contextually aware responses, mistral:7b was about 20% faster in response time. In this case, if speed is a priority, Mistral might be preferred; if quality is a priority, Llama 3.

Secure Interaction with LLMs: Points to Consider

While local LLMs offer significant advantages in terms of security and privacy, there are some points you should consider when using these tools:

1. Model Security: Verify the source of the models you download. Make sure to download from trusted platforms (Ollama Hub, Hugging Face, etc.). Avoid models that may contain malicious code.
2. System Resources: LLMs can consume significant system resources. If your machine's memory (RAM) and processing power are insufficient for the model you are running, your system may slow down or crash. Especially when running large models, it's useful to set cgroup limits or regularly check system monitors (e.g., htop, journald logs).
3. Updates: Both Ollama and Open WebUI are regularly updated. These updates usually include performance improvements, bug fixes, and new features. Keeping your systems up to date is important.
4. Access Control: Do not expose Open WebUI to the outside world. If multiple people will be using it on a network, ensure you have set up appropriate authentication and authorization mechanisms.

Once, while running Ollama on my own VPS, I encountered unauthorized access attempts to my model because I hadn't configured the firewall settings strictly enough. This required me to add additional security layers with tools like fail2ban and firewall-cmd commands. The incident reminded me once again that even a simple setup requires additional security measures in a production environment.

⚠️ System Resources Warning

Large LLM models, especially those with 70B parameters or more, may require 32GB or more of RAM. Running large models with insufficient resources can lead to performance issues and even system instability.

Looking Ahead: The Potential of Local LLMs

Tools like Ollama and Open WebUI are democratizing local LLM usage. This means that even individuals and small businesses can create their own custom AI solutions. In the future, I expect to see these types of local solutions evolve further and be able to perform more complex tasks.

For example, by setting up RAG (Retrieval-Augmented Generation) systems locally, you can develop intelligent querying tools that work on your own documents. This can make internal knowledge bases more accessible or empower your personal information management systems. I am also considering setting up a RAG system that works on the content of my own blog. This way, I will be able to scan all my technical articles and get in-depth information on a specific topic.

The potential offered by this technology is limitless. By prioritizing data privacy, leveraging the power of artificial intelligence is now much easier.

Your next step could be to explore this technology by setting up your own Ollama and Open WebUI installation. You can start by trying the Llama 3 8B model, and then move on to different models according to your needs.