"This notebook is an exploration of how a RAG could be created. The idea is to create a chatbot who can answer questions based on the documents it has been trained on.\n",
"\n",
"Requirements are as follow:\n",
"* Source documents are in markdown format\n",
"* Source documents are stored in a git repository\n",
"* Everything needs to be self-hosted\n",
" * An Ollama server is already running locally (https://localhost:11434)\n",
" * Eventually, we want it to be a bot hosted in Teams and/or Discord\n",
"\n",
"For this notebook, we will ingest the documentation of Bitburner.\n",
"\n",
">Bitburner is a programming-based incremental game that revolves around hacking and cyberpunk themes.\n",
"\n",
"The documentation located in this repository: https://github.com/bitburner-official/bitburner-src.git and the documentation is within the *src/Documentation/* folder of this repository."
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
"For this example, we will use LangChain because it's a very convenient an all-in-one framework that is commonly used in LLM applications. As for our backend, we'll use Ollama because it's convenient to be able to run the models locally."
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
"1. We need convert our documents into vectors. For this, we can use a embedding model. *nomic-embed-text* should provide reasonable performance for our purpose."
"2. Once we have our vector, we can store them in a database for future retrieval. LangChain conveniently provides us an `InMemoryVectorStore` which will do the job."
"3. Now that we have our embedding model and our vector database, we can start indexing our documents. LangChain has 100+ `DocumentLoader`s to aid us with this task. The documentation is written in markdown, so we can use [UnstructuredMarkdownLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader.html#langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader) to load them."
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
"Note: you may need to restart the kernel to use updated packages.\n"
"4. We can now store our documents into the database."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"for document in documents:\n",
" vector_store.add_documents(documents=document)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5. Finally, we can retrieve documents from our database."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(id='7b012d63-34f7-4f20-baa2-fae434ecfbd3', metadata={'source': './bitburner/src/Documentation/doc/basic/hacking.md'}, page_content='Hacking\\n\\nIn the year 2077, currency has become digital and decentralized. People and corporations store their money on servers. By hacking these servers, you can steal their money and gain experience.\\n\\nGaining Root Access\\n\\nThe first step to hacking a server is to gain root access to that server. This can be done using the NUKE.exe virus. You start the game with a copy of the NUKE.exe virus on your home computer. The NUKE.exe virus attacks the target server\\'s open ports using buffer overflow exploits. When successful, you are granted root administrative access to the machine.\\n\\nIn order for the NUKE.exe virus to succeed, the target server needs to have enough open ports. Some servers have no security and will not need any ports opened. Some will have very high security and will need many ports opened. In order to open ports on another server, you will need to run programs that attack the server to open specific ports. These programs can be coded once your hacking skill gets high enough, or they can be purchased if you can find a seller.\\n\\nThere are two ways to execute port-opening programs and the NUKE virus:\\n\\nConnect to the target server through the Terminal and use the run command: $ run [programName]\\n\\nUse a function:\\n\\nnuke\\n\\nbrutessh\\n\\nftpcrack\\n\\nrelaysmtp\\n\\nhttpworm\\n\\nsqlinject\\n\\nThere are two ways to determine how many ports need to be opened on a server in order to successfully NUKE it:\\n\\nConnect to that server through the Terminal and use the analyze command.\\n\\nUse the getServerNumPortsRequired function.\\n\\nOnce you have enough ports opened on a server and have ran the NUKE virus to gain root access, you will be able to hack it.\\n\\nFor specific details of how Hacking work \"offline\"\\n\\nSee Offline And Bonus Time.\\n\\nGeneral Hacking Mechanics\\n\\nWhen you execute the hack command, either manually through the Terminal or automatically through a script, you attempt to hack the server. This action takes time. The more advanced a server\\'s security is, the more time it will take. Your hacking skill level also affects the hacking time, with a higher hacking skill leading to shorter hacking times. Also, running the hack command manually through Terminal is faster than hacking from a script.\\n\\nYour attempt to hack a server will not always succeed. The chance you have to successfully hack a server is also determined by the server\\'s security and your hacking skill level. Even if your hacking attempt is unsuccessful, you will still gain experience points.\\n\\nWhen you successfully hack a server. You steal a certain percentage of that server\\'s total money. This percentage is, once again, determined by the server\\'s security and your hacking skill level. The amount of money on a server is not limitless. So, if you constantly hack a server and deplete its money, then you will encounter diminishing returns in your hacking (since you are only hacking a certain percentage). You can increase the amount of money on a server using a script and the grow function.\\n\\nServer Security\\n\\nEach server has a security level, typically between 1 and 100. A higher number means the server has stronger security.\\n\\nAs mentioned above, a server\\'s security level is an important factor to consider when hacking. You can check a server\\'s security level using the analyze Terminal command. You can also check a server\\'s security in a script, using the getServerSecurityLevel function.\\n\\nWhenever a server is hacked manually or through a script, its security level increases by a small amount. Calling the grow function in a script will also increase security level of the target server. These actions will make it harder for you to hack the server, and decrease the amount of money you can steal. You can lower a server\\'s security level in a script using the weaken function.\\n\\nEach server has a minimum security level. T
" Document(id='d26660d8-debb-4958-a010-66ba53540ae5', metadata={'source': './bitburner/src/Documentation/doc/index.md'}, page_content=\"Documentation\\n\\nGuides\\n\\nBeginner's guide\\n\\nBasic Mechanics\\n\\nStats\\n\\nTerminal\\n\\nHacking\\n\\nScripts\\n\\nServers\\n\\nRAM\\n\\nHacknet nodes\\n\\nAugmentations\\n\\nCompanies\\n\\nFactions\\n\\nCrimes\\n\\nInfiltration\\n\\nPrograms\\n\\nReputation\\n\\nStock market\\n\\nWorld\\n\\nCoding contracts\\n\\nAutocomplete\\n\\nAdvanced Mechanics\\n\\nHacking algorithms\\n\\nList of factions and their requirements\\n\\nOffline scripts and bonus time\\n\\nBitNodes\\n\\nSource-Files\\n\\nGang\\n\\nCorporation\\n\\nIntelligence\\n\\nBladeburner\\n\\nHacknet servers\\n\\nSleeves\\n\\nGrafting\\n\\nStanek's Gift\\n\\nIPvGO\\n\\nResources\\n\\nNS API documentation\\n\\nLearn to program\\n\\nRemote API\\n\\nGame frozen or stuck?\\n\\nReact\\n\\nTools & Resources\\n\\nChangelog\\n\\nChangelog - Legacy v1\\n\\nChangelog - Legacy v0\\n\\nMigration\\n\\nBitburner v1.0.0 script migration guide\\n\\nBitburner v2.0.0 script migration guide\\n\\nNetscript 2 migration guide (.script to .js)\"),\n",
" Document(id='ef442fa9-5c59-435e-8d08-0d5fa0b09ebb', metadata={'source': './bitburner/src/Documentation/doc/basic/infiltration.md'}, page_content='Infiltration\\n\\nInfiltration is a gameplay mechanic that allows you to infiltrate a Company\\'s facility to try and steal the Company\\'s classified secrets. These secrets can be sold for money or for Reputation with a Faction.\\n\\nOverview\\n\\nMany companies have facilities that you can attempt to infiltrate. By infiltrating, you can steal classified Company secrets and then sell these for money or for Faction Reputation. To try and infiltrate a Company, visit a Company through the World menu. There will be an option that says \\'Infiltrate Company\\'.\\n\\nWhen infiltrating a Company, you will be presented with short active challenges.\\n\\nNone of the challenges uses the mouse.\\n\\nMost challenges use spacebar as the action.\\n\\nSome challenges use WASD or arrows interchangeably.\\n\\nA few others use the rest of the keyboard.\\n\\nEach location that can be infiltrated has 3 important values:\\n\\nDifficulty: It affects how difficult the challenges are. This value depends on your \"common\" stats (combat stats and charisma). It\\'s reduced when you improve your stats. It is not recommended to attempt infiltrations when the difficulty is above normal.\\n\\nMax clearance level: It is the number of challenges you need to pass to receive the infiltration reward.\\n\\nStarting security level: It affects the difficulty and rewards.\\n\\nEvery time you successfully complete an infiltration challenge and go to the higher clearance level, the \"effective\" difficulty is increased. The difficulty value in the introduction screen is the initial value for the first clearance level. When you go to higher levels, this value will be increased, so the challenges will become harder.\\n\\nEvery time you fail an infiltration challenge, you will take damage based on the starting security level of the location. If your HP is reduced to 0, you will be hospitalized, and the infiltration will immediately end.\\n\\nInfiltration rewards depend on:\\n\\nMax clearance level. Higher max clearance level = Higher rewards.\\n\\nStarting security level. Higher starting security level = Higher rewards.\\n\\nYour stats\\' multipliers (NOT skill levels). Lower multipliers = Higher rewards.\\n\\n\"SoA - phyzical WKS harmonizer\" augmentation.\\n\\nAn endgame stat [1]. You will know what it is when you reach the endgame.\\n\\nAn endgame multiplier [2]. You will know what it is when you reach the endgame.\\n\\nThe most common misconception of infiltration rewards is that they depend on skill levels. This is wrong. The rewards do NOT depend on skill levels. They depend on stats\\' multipliers. When you install augmentations that improve stats\\' multipliers, the rewards are reduced.\\n\\nIn some special cases, you may observe this behavior: When you install augmentations or soft reset, your skill levels drop to 1 and infiltration rewards are increased. Some players mistakenly think that this is why infiltrations should be done with low stats to optimize the rewards. This is totally wrong. Again, infiltration rewards do NOT depend on skill levels. In these special cases, the rewards are increased because stats\\' multipliers are reduced due to losing bonuses from other mechanics (IPvGO and some endgame mechanics).\\n\\nRaising raw values of stats (skill levels) is one of the proper ways to optimize the infiltration. Having higher stats does not increase the rewards, but it allows you to infiltrate harder locations (higher max clearance level, higher starting security level, easier to complete more infiltrations in the same time frame, etc.).\\n\\nProper ways to optimize the infiltration (with the same [2] and the same stats\\' multipliers) are:\\n\\nRaise stats. Do NOT purposely keep them low. Having higher stats does not increase the rewards, but it allows you to infiltrate harder locations. Training at gyms, committing crimes, and working for companies/factions are good and easy ways to raise stat
" Document(id='6ef32e3d-d6b7-4c42-874a-96c21e4a63a9', metadata={'source': './bitburner/src/Documentation/doc/advanced/grafting.md'}, page_content=\"Grafting\\n\\nGrafting is an experimental process through which you can obtain the benefits of Augmentations, without needing to reboot your body.\\n\\nGrafting can be done at VitaLife in New Tokyo, where you'll find a shady researcher with questionable connections. From there, you can spend a sum of money to begin grafting Augmentations. This will take some time. When done, the Augmentation will be applied to your character without needing to install.\\n\\nBe warned, some who have tested grafting have reported an unidentified malware. Dubbed Entropy, this virus seems to grow in potency as more Augmentations are grafted, causing unpredictable affects to the victim.\\n\\nNote that when grafting an Augmentation, cancelling will not save your progress, and the money spent will not be returned.\")]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector_store.search(\n",
" query='How to hack',\n",
" search_type='similarity',\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Build the RAG chain\n",
"\n",
"There are a number of frameworks available to build our RAG.\n",
