badjware
/
fun-with-ai
generated from badjware/jupiter-notebook-template
1
0
Fork 0
Code Releases Activity
fun-with-ai/work/rag.ipynb

338 lines
19 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is an exploration of how a RAG could be created. The idea is to create a chatbot who can answer questions based on the documents it has been trained on.\n",
"\n",
"Requirements are as follow:\n",
"* Source documents are in markdown format\n",
"* Source documents are stored in a git repository\n",
"* Everything needs to be self-hosted\n",
" * An Ollama server is already running locally (https://localhost:11434)\n",
"* The interface is unimportant for now\n",
" * Eventually, we want it to be a bot hosted in Teams and/or Discord\n",
"\n",
"For this notebook, we will ingest the documentation of Bitburner.\n",
"\n",
">Bitburner is a programming-based incremental game that revolves around hacking and cyberpunk themes.\n",
"\n",
"The documentation located in this repository: https://github.com/bitburner-official/bitburner-src.git and the documentation is within the *src/Documentation/* folder of this repository."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Steps\n",
"\n",
"### Step 1: Fetch the documents from the git repository\n",
"\n",
"We will use `gitpython` to clone the repository and fetch the documents.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install --quiet --upgrade langchain-community GitPython"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First thing first, we need to clone the repository. If it already exists locally, we can just pull the latest changes."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from git import Repo\n",
"\n",
"repo_url = \"https://github.com/bitburner-official/bitburner-src.git\"\n",
"local_repo_path = \"./bitburner\"\n",
"\n",
"if not os.path.exists(local_repo_path):\n",
" Repo.clone_from(repo_url, local_repo_path)\n",
"else:\n",
" repo = Repo(local_repo_path)\n",
" repo.remotes.origin.pull()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have a local copy of the repository, we can find the documents we are interested in within and list them."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 63 documents.\n"
]
}
],
"source": [
"doc_root = os.path.join(local_repo_path, \"src/Documentation\")\n",
"doc_files = []\n",
"\n",
"# Walk through the directory and find all markdown files\n",
"for root, dirs, files in os.walk(doc_root):\n",
" for file in files:\n",
" if file.endswith(\".md\"):\n",
" doc_files.append(os.path.join(root, file))\n",
"\n",
"print(f\"Found {len(doc_files)} documents.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Index the documents in a vector database\n",
"\n",
"To build our RAG, we need to store the documents in a vector database. Several options are available:\n",
"* [FAISS](https://faiss.ai/)\n",
"* [ChromaDB](https://www.trychroma.com/)\n",
"* [Qdrant](https://qdrant.tech/)\n",
"* [LangChain](https://langchain.com/)\n",
"* etc.\n",
"\n",
"For this example, we will use LangChain because it's a very convenient an all-in-one framework that is commonly used in LLM applications. As for our backend, we'll use Ollama because it's convenient to be able to run the models locally."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install --quiet --upgrade langchain-community langchain-ollama langgraph"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To index our documents:\n",
"1. We need convert our documents into vectors. For this, we can use a embedding model. *nomic-embed-text* should provide reasonable performance for our purpose."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from langchain_ollama.embeddings import OllamaEmbeddings\n",
"\n",
"embeddings = OllamaEmbeddings(model='nomic-embed-text')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. Once we have our vector, we can store them in a database for future retrieval. LangChain conveniently provides us an `InMemoryVectorStore` which will do the job."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from langchain_core.vectorstores import InMemoryVectorStore\n",
"\n",
"vector_store = InMemoryVectorStore(embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Now that we have our embedding model and our vector database, we can start indexing our documents. LangChain has 100+ `DocumentLoader`s to aid us with this task. The documentation is written in markdown, so we can use [UnstructuredMarkdownLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader.html#langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader) to load them."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install --quiet --upgrade unstructured markdown"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 63 documents\n"
]
}
],
"source": [
"from langchain_community.document_loaders import UnstructuredMarkdownLoader\n",
"\n",
"\n",
"documents =[]\n",
"\n",
"for file in doc_files:\n",
" loader = UnstructuredMarkdownLoader(\n",
" file,\n",
" mode='single',\n",
" strategy='fast',\n",
" )\n",
" documents.append(loader.load())\n",
"\n",
"print(f'Loaded {len(documents)} documents')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. We can now store our documents into the database."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"for document in documents:\n",
" vector_store.add_documents(documents=document)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5. Finally, we can retrieve documents from our database."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(id='93e7fa11-5553-40c0-aef7-7e94299bdd72', metadata={'source': './bitburner/src/Documentation/doc/basic/hacking.md'}, page_content='Hacking\\n\\nIn the year 2077, currency has become digital and decentralized. People and corporations store their money on servers. By hacking these servers, you can steal their money and gain experience.\\n\\nGaining Root Access\\n\\nThe first step to hacking a server is to gain root access to that server. This can be done using the NUKE.exe virus. You start the game with a copy of the NUKE.exe virus on your home computer. The NUKE.exe virus attacks the target server\\'s open ports using buffer overflow exploits. When successful, you are granted root administrative access to the machine.\\n\\nIn order for the NUKE.exe virus to succeed, the target server needs to have enough open ports. Some servers have no security and will not need any ports opened. Some will have very high security and will need many ports opened. In order to open ports on another server, you will need to run programs that attack the server to open specific ports. These programs can be coded once your hacking skill gets high enough, or they can be purchased if you can find a seller.\\n\\nThere are two ways to execute port-opening programs and the NUKE virus:\\n\\nConnect to the target server through the Terminal and use the run command: $ run [programName]\\n\\nUse a function:\\n\\nnuke\\n\\nbrutessh\\n\\nftpcrack\\n\\nrelaysmtp\\n\\nhttpworm\\n\\nsqlinject\\n\\nThere are two ways to determine how many ports need to be opened on a server in order to successfully NUKE it:\\n\\nConnect to that server through the Terminal and use the analyze command.\\n\\nUse the getServerNumPortsRequired function.\\n\\nOnce you have enough ports opened on a server and have ran the NUKE virus to gain root access, you will be able to hack it.\\n\\nFor specific details of how Hacking work \"offline\"\\n\\nSee Offline And Bonus Time.\\n\\nGeneral Hacking Mechanics\\n\\nWhen you execute the hack command, either manually through the Terminal or automatically through a script, you attempt to hack the server. This action takes time. The more advanced a server\\'s security is, the more time it will take. Your hacking skill level also affects the hacking time, with a higher hacking skill leading to shorter hacking times. Also, running the hack command manually through Terminal is faster than hacking from a script.\\n\\nYour attempt to hack a server will not always succeed. The chance you have to successfully hack a server is also determined by the server\\'s security and your hacking skill level. Even if your hacking attempt is unsuccessful, you will still gain experience points.\\n\\nWhen you successfully hack a server. You steal a certain percentage of that server\\'s total money. This percentage is, once again, determined by the server\\'s security and your hacking skill level. The amount of money on a server is not limitless. So, if you constantly hack a server and deplete its money, then you will encounter diminishing returns in your hacking (since you are only hacking a certain percentage). You can increase the amount of money on a server using a script and the grow function.\\n\\nServer Security\\n\\nEach server has a security level, typically between 1 and 100. A higher number means the server has stronger security.\\n\\nAs mentioned above, a server\\'s security level is an important factor to consider when hacking. You can check a server\\'s security level using the analyze Terminal command. You can also check a server\\'s security in a script, using the getServerSecurityLevel function.\\n\\nWhenever a server is hacked manually or through a script, its security level increases by a small amount. Calling the grow function in a script will also increase security level of the target server. These actions will make it harder for you to hack the server, and decrease the amount of money you can steal. You can lower a server\\'s security level in a script using the weaken function.\\n\\nEach server has a minimum security level. The server\\'s security level will not fall below this value if you try to weaken it. You can get this value with the getServerMinSecurityLevel function.\\n\\nBackdoors\\n\\nServers that can be hacked can also have backdoors installed. These backdoors will provide you with a benefit - the services may be cheaper, penalties may be reduced or there may be other results. Honeypots exist and will let factions know when you have succeeded at backdooring their system. Once you have a backdoor installed, you can connect to that server directly.\\n\\nWhen you visit a location in the city and see that the name is partially scrambled, this indicates that you have backdoored the server related to the location.'),\n",
" Document(id='0d1eadf5-0dd7-47c7-b325-bc7ad98d3439', metadata={'source': './bitburner/src/Documentation/doc/basic/hacknet_nodes.md'}, page_content=\"Hacknet nodes\\n\\nThis distributed network of computers allows you to gain passive income. By upgrading a node's level, RAM, and CPU cores you can increase the amount of money it earns. You can also purchase new nodes to expand your Hacknet - The cost for each node increases as your network grows.\\n\\nHacknet nodes won't make as much money as basic hacking scripts, and they are not enough to progress alone.\\n\\nLater in the game, there is a powerful change to the Hacknet system called Hacknet Servers.\"),\n",
" Document(id='296d94bf-a495-4365-b371-260257f0306a', metadata={'source': './bitburner/src/Documentation/doc/basic/stats.md'}, page_content=\"Below are some of the stats that will increase with play and reset during augmentation installs as you progress through the game. Your stats can be found in the Overview panel, the Stats subpage of the side menu, or with API methods like ns.getPlayer().\\n\\nHack Skill\\n\\nFor many aspects of Bitburner, increasing your Hack skill will be an important goal. Primarily affected by the efficiency of your hacking strategies, you will also be offered Augmentations that greatly enhance your Hack Skill level and how effective its results are.\\n\\nAffects:\\n\\nTime needed to execute hack, grow, or weaken and similar methods\\n\\nYour chance to successfully hack a server\\n\\nPercent of a server's money stolen when hacking it\\n\\nSuccess rate of certain crimes\\n\\nTime needed to create a Program\\n\\nFaction Reputation gain when carrying out Hacking Contracts or Field Work\\n\\nCompany Reputation gain for certain jobs\\n\\nGain Hack experience by:\\n\\nManually hacking servers through the Terminal\\n\\nUsing ns.hack(), ns.grow(), or ns.weaken() through scripts\\n\\nCommitting certain crimes\\n\\nCarrying out Hacking Contracts or doing Field work for Factions\\n\\nSome Company jobs and other types of work\\n\\nStudying at a university\\n\\nCombat Skills\\n\\nStrength, Defense, Dexterity, and Agility\\n\\nThese represent your physical skill and attributes, including your ability to sneak, inflict or endure damage, and pull off high precision tasks. Similar to your Hack skill, you will be offered Faction Augmentations to multiplicatively enhance your Combat Skills and exp gain.\\n\\nAffects:\\n\\nHP scales with Defense. Infiltration and some jobs may cause you to take damage.\\n\\nSuccess rate of certain crimes\\n\\nFaction Reputation gain for Security and Field Work\\n\\nCompany Reputation gain for certain jobs\\n\\nGain experience by:\\n\\nWorking out at a gym\\n\\nCommitting certain crimes\\n\\nDoing Security or Field Work for a Faction\\n\\nWorking certain jobs at a Company\\n\\nCharisma\\n\\nRarely as useful as Hacking and Physical skills, Charisma can help get a company job, gain trust, or calm chaos in social situations.\\n\\nCharisma can also be enhanced with Augmentations.\\n\\nAffects:\\n\\nSuccess rate of certain crimes\\n\\nFaction Reputation gain for Field Work\\n\\nCompany Reputation gain for most jobs\\n\\nGain experience by:\\n\\nCommitting certain crimes\\n\\nStudying at a university\\n\\nWorking certain jobs at a Company\\n\\nDoing Field work for a Faction\\n\\nOther Stats and abilities are available in later stages of the game.\"),\n",
" Document(id='cc594be4-3c56-420d-833e-be94684b1ede', metadata={'source': './bitburner/src/Documentation/doc/index.md'}, page_content=\"Documentation\\n\\nGuides\\n\\nBeginner's guide\\n\\nBasic Mechanics\\n\\nStats\\n\\nTerminal\\n\\nHacking\\n\\nScripts\\n\\nServers\\n\\nRAM\\n\\nHacknet nodes\\n\\nAugmentations\\n\\nCompanies\\n\\nFactions\\n\\nCrimes\\n\\nInfiltration\\n\\nPrograms\\n\\nReputation\\n\\nStock market\\n\\nWorld\\n\\nCoding contracts\\n\\nAutocomplete\\n\\nAdvanced Mechanics\\n\\nHacking algorithms\\n\\nList of factions and their requirements\\n\\nOffline scripts and bonus time\\n\\nBitNodes\\n\\nSource-Files\\n\\nGang\\n\\nCorporation\\n\\nIntelligence\\n\\nBladeburner\\n\\nHacknet servers\\n\\nSleeves\\n\\nGrafting\\n\\nStanek's Gift\\n\\nIPvGO\\n\\nResources\\n\\nNS API documentation\\n\\nLearn to program\\n\\nRemote API\\n\\nGame frozen or stuck?\\n\\nReact\\n\\nTools & Resources\\n\\nChangelog\\n\\nChangelog - Legacy v1\\n\\nChangelog - Legacy v0\\n\\nMigration\\n\\nBitburner v1.0.0 script migration guide\\n\\nBitburner v2.0.0 script migration guide\\n\\nNetscript 2 migration guide (.script to .js)\")]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector_store.search(\n",
" query='How to hack',\n",
" search_type='similarity',\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Build the RAG chain\n",
"\n",
"There are a number of frameworks available to build our RAG.\n",
"* [LangChain](https://langchain.com/)\n",
"* [LlamaIndex](https://docs.llamaindex.ai/en/latest/)\n",
"\n",
"In this example we will use LangChain."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "env",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
View Git Blame Copy Permalink