diff --git a/requirements.txt b/requirements.txt index a8709ea..c15d402 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,5 +5,4 @@ python-lsp-server[all] matplotlib pandas -GitPython -chromadb \ No newline at end of file +# chromadb \ No newline at end of file diff --git a/work/rag.ipynb b/work/rag.ipynb index afaf7d9..5b7ab3e 100644 --- a/work/rag.ipynb +++ b/work/rag.ipynb @@ -12,7 +12,13 @@ "* Everything needs to be self-hosted\n", " * An Ollama server is already running locally (https://localhost:11434)\n", "* The interface is unimportant for now\n", - " * Eventually, we want it to be a bot hosted in Teams and/or Discord" + " * Eventually, we want it to be a bot hosted in Teams and/or Discord\n", + "\n", + "For this notebook, we will ingest the documentation of Bitburner.\n", + "\n", + ">Bitburner is a programming-based incremental game that revolves around hacking and cyberpunk themes.\n", + "\n", + "The documentation located in this repository: https://github.com/bitburner-official/bitburner-src.git and the documentation is within the *src/Documentation/* folder of this repository." ] }, { @@ -23,25 +29,48 @@ "\n", "### Step 1: Fetch the documents from the git repository\n", "\n", - "We will use `gitpython` to clone the repository and fetch the documents.\n", - "\n", - "For this notebook, we will ingest the documentation of prometheus-operator located in this repository: https://github.com/prometheus-operator/prometheus-operator.git. The documentation is within the *Documentation/* folder of this repository.\n", - "\n", - "First thing first, we need to clone the repository. If it already exists locally, we can just pull the latest changes." + "We will use `gitpython` to clone the repository and fetch the documents.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install --quiet --upgrade langchain-community GitPython" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First thing first, we need to clone the repository. If it already exists locally, we can just pull the latest changes." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "from git import Repo\n", "\n", - "repo_url = \"https://github.com/prometheus-operator/prometheus-operator.git\"\n", - "local_repo_path = \"./prometheus-operator\"\n", + "repo_url = \"https://github.com/bitburner-official/bitburner-src.git\"\n", + "local_repo_path = \"./bitburner\"\n", "\n", "if not os.path.exists(local_repo_path):\n", " Repo.clone_from(repo_url, local_repo_path)\n", @@ -59,72 +88,128 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Found 40 documents.\n" + "Found 63 documents.\n" ] } ], "source": [ - "documentation_root = os.path.join(local_repo_path, \"Documentation\")\n", - "documentation_files = []\n", + "doc_root = os.path.join(local_repo_path, \"src/Documentation\")\n", + "doc_files = []\n", "\n", "# Walk through the directory and find all markdown files\n", - "for root, dirs, files in os.walk(documentation_root):\n", + "for root, dirs, files in os.walk(doc_root):\n", " for file in files:\n", " if file.endswith(\".md\"):\n", - " documentation_files.append(os.path.join(root, file))\n", + " doc_files.append(os.path.join(root, file))\n", "\n", - "print(f\"Found {len(documentation_files)} documents.\")" + "print(f\"Found {len(doc_files)} documents.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Ingest the documents in a vector database\n", + "## Step 2: Index the documents in a vector database\n", "\n", "To build our RAG, we need to store the documents in a vector database. Several options are available:\n", "* [FAISS](https://faiss.ai/)\n", "* [ChromaDB](https://www.trychroma.com/)\n", "* [Qdrant](https://qdrant.tech/)\n", + "* [LangChain](https://langchain.com/)\n", "* etc.\n", "\n", - "For this example, we will use ChromaDB because it is easy to set up and use. Helpfully, ChromaDB is able to automatically generate embeddings for us. We will store the documents in a collection called `documentation`. The collection will live in-memory, but in a more complete setup, we could setup Chroma in an client-server mode and/or with persistence enabled." + "For this example, we will use LangChain because it's a very convenient an all-in-one framework that is commonly used in LLM applications. As for our backend, we'll use Ollama because it's convenient to be able to run the models locally." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], "source": [ - "import chromadb\n", - "\n", - "chroma_client = chromadb.Client()\n", - "collection = chroma_client.create_collection(name=\"documentation\")\n", - "\n", - "# Read the contents of each document and store them in a list\n", - "documents = []\n", - "for file in documentation_files:\n", - " with open(file, \"r\") as f:\n", - " content = f.read()\n", - " documents.append(content)\n", - "\n", - "# Add the documents to the collection\n", - "collection.add(documents=documents, ids=documentation_files)" + "%pip install --quiet --upgrade langchain-community langchain-ollama langgraph" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Now that we have built our collection, we can try to query it. Let's search for a document about prometheus." + "To index our documents:\n", + "1. We need convert our documents into vectors. For this, we can use a embedding model. *nomic-embed-text* should provide reasonable performance for our purpose." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_ollama.embeddings import OllamaEmbeddings\n", + "\n", + "embeddings = OllamaEmbeddings(model='nomic-embed-text')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "2. Once we have our vector, we can store them in a database for future retrieval. LangChain conveniently provides us an `InMemoryVectorStore` which will do the job." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_core.vectorstores import InMemoryVectorStore\n", + "\n", + "vector_store = InMemoryVectorStore(embeddings)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "3. Now that we have our embedding model and our vector database, we can start indexing our documents. LangChain has 100+ `DocumentLoader`s to aid us with this task. The documentation is written in markdown, so we can use [UnstructuredMarkdownLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader.html#langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader) to load them." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install --quiet --upgrade unstructured markdown" ] }, { @@ -136,56 +221,96 @@ "name": "stdout", "output_type": "stream", "text": [ - "{\n", - " \"ids\": [\n", - " [\n", - " \"./prometheus-operator/Documentation/platform/prometheus-agent.md\",\n", - " \"./prometheus-operator/Documentation/proposals/202201-prometheus-agent.md\",\n", - " \"./prometheus-operator/Documentation/additional-scrape-config.md\"\n", - " ]\n", - " ],\n", - " \"embeddings\": null,\n", - " \"documents\": [\n", - " [\n", - " \"---\\nweight: 204\\ntoc: true\\ntitle: Prometheus Agent\\nmenu:\\n docs:\\n parent: user-guides\\nlead: \\\"\\\"\\nimages: []\\ndraft: false\\ndescription: Guide for running Prometheus in Agent mode\\n---\\n\\n{{< alert icon=\\\"\\ud83d\\udc49\\\" text=\\\"Prometheus Operator >= v0.64.0 is required.\\\"/>}}\\n\\nAs mentioned in [Prometheus's blog](https://prometheus.io/blog/2021/11/16/agent/), Prometheus Agent\\nis a deployment model optimized for environments where all collected data is forwarded to\\na long-term storage solution, e.g. Cortex, Thanos or Prometheus, that do not need storage or rule evaluation.\\n\\nFirst of all, make sure that the PrometheusAgent CRD is installed in the cluster and that the operator has the proper RBAC permissions to reconcile the PrometheusAgent resources.\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRole\\nmetadata:\\n labels:\\n app.kubernetes.io/component: controller\\n app.kubernetes.io/name: prometheus-operator\\n app.kubernetes.io/version: 0.80.0\\n name: prometheus-operator\\nrules:\\n- apiGroups:\\n - monitoring.coreos.com\\n resources:\\n - alertmanagers\\n - alertmanagers/finalizers\\n - alertmanagers/status\\n - alertmanagerconfigs\\n - prometheuses\\n - prometheuses/finalizers\\n - prometheuses/status\\n - prometheusagents\\n - prometheusagents/finalizers\\n - prometheusagents/status\\n - thanosrulers\\n - thanosrulers/finalizers\\n - thanosrulers/status\\n - scrapeconfigs\\n - servicemonitors\\n - podmonitors\\n - probes\\n - prometheusrules\\n verbs:\\n - '*'\\n- apiGroups:\\n - apps\\n resources:\\n - statefulsets\\n verbs:\\n - '*'\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - configmaps\\n - secrets\\n verbs:\\n - '*'\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - pods\\n verbs:\\n - list\\n - delete\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - services\\n - services/finalizers\\n verbs:\\n - get\\n - create\\n - update\\n - delete\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - nodes\\n verbs:\\n - list\\n - watch\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - namespaces\\n verbs:\\n - get\\n - list\\n - watch\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - events\\n verbs:\\n - patch\\n - create\\n- apiGroups:\\n - networking.k8s.io\\n resources:\\n - ingresses\\n verbs:\\n - get\\n - list\\n - watch\\n- apiGroups:\\n - storage.k8s.io\\n resources:\\n - storageclasses\\n verbs:\\n - get\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - endpoints\\n verbs:\\n - get\\n - create\\n - update\\n - delete\\n```\\n\\nSimilarly to Prometheus, Prometheus Agent will also require permission to scrape targets. Because of this, we will create a new service account for the Agent with the necessary permissions to scrape targets.\\n\\nStart with the ServiceAccount, ClusterRole and ClusterRoleBinding:\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-service-account.yaml\\\"\\napiVersion: v1\\nkind: ServiceAccount\\nmetadata:\\n name: prometheus-agent\\n```\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-cluster-role.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRole\\nmetadata:\\n name: prometheus-agent\\nrules:\\n- apiGroups: [\\\"\\\"]\\n resources:\\n - services\\n - endpoints\\n - pods\\n verbs: [\\\"get\\\", \\\"list\\\", \\\"watch\\\"]\\n- apiGroups: [\\\"\\\"]\\n resources:\\n - configmaps\\n verbs: [\\\"get\\\"]\\n- apiGroups:\\n - networking.k8s.io\\n resources:\\n - ingresses\\n verbs: [\\\"get\\\", \\\"list\\\", \\\"watch\\\"]\\n- nonResourceURLs: [\\\"/metrics\\\"]\\n verbs: [\\\"get\\\"]\\n```\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-cluster-role-binding.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRoleBinding\\nmetadata:\\n name: prometheus-agent\\nroleRef:\\n apiGroup: rbac.authorization.k8s.io\\n kind: ClusterRole\\n name: prometheus-agent\\nsubjects:\\n- kind: ServiceAccount\\n name: prometheus-agent\\n namespace: default\\n```\\n\\nLastly, we can deploy the Agent. The `spec` field is very similar to the Prometheus CRD but the features that aren't applicable to the agent mode (like alerting, retention, Thanos, ...) are not available.\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus.yaml\\\"\\napiVersion: monitoring.coreos.com/v1alpha1\\nkind: PrometheusAgent\\nmetadata:\\n name: prometheus-agent\\nspec:\\n replicas: 2\\n serviceAccountName: prometheus-agent\\n serviceMonitorSelector:\\n matchLabels:\\n team: frontend\\n```\\n\\nContinue with the [Getting Started page]({{}}) to learn how to monitor applications running on Kubernetes.\\n\",\n", - " \"# Prometheus Agent support\\n\\n## Summary\\n\\nThe Prometheus 2.32.0 release introduces the Prometheus Agent, a mode optimized for remote-write dominant scenarios. This document proposes extending the Prometheus Operator to allow running a Prometheus Agent with different deployment strategies.\\n\\n## Background\\n\\nThe Prometheus Operator in its current state does not allow a simple way of deploying the Prometheus agent. A potential workaround has been described in a [Github comment](https://github.com/prometheus-operator/prometheus-operator/issues/3989#issuecomment-974137486), where the agent can be deployed through the existing Prometheus CRD by explicitly setting command-line arguments specific to the agent mode.\\n\\nAs described in the comment, one significant problem with this approach is that the Prometheus Operator always generates `alerts` and `rules` sections in the Prometheus config file. These sections are not allowed when running the agent so users need to take additional actions to pause reconciliation of the Prometheus CR, tweak the generated secret and then unpause reconciliation in order to resolve the problem. Alternatively, users can apply a strategic merge patch to the prometheus container as described in the kube-prometheus docs: [https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/prometheus-agent.md](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/prometheus-agent.md)\\n\\nWhile this workaround can be used as a stop-gap solution to unblock users in the short term, it has the drawback of needing additional steps which require understanding implementation details of the operator itself. In addition to this, overriding the value of the argument `--config.file` also requires knowledge of Prometheus Operator internals.\\n\\nA lot of the fields supported by the current PrometheusSpec are not applicable to the agent mode. These fields are documented in the PrometheusAgent CRD section.\\n\\nFinally, the Prometheus agent is significantly different from the Prometheus server in the way that it fits in a monitoring stack. Therefore, running it as a StatefulSet might not be the only possible deployment strategy, users might want to run it as a DaemonSet or a Deployment instead.\\n\\n## Proposal\\n\\nThis document proposes introducing a PrometheusAgent CRD to allow users to run Prometheus in agent mode. Having a separate CRD allows the Prometheus and PrometheusAgent CRDs to evolve independently and expose parameters specific to each Prometheus mode.\\n\\nFor example, the PrometheusAgent CRD could have a `strategy` field indicating the deployment strategy for the agent, but no `alerting` field since alerts are not supported in agent mode. Even though there will be an upfront cost for introducing a new CRD, having separate APIs would simplify long-term maintenance by allowing the use of CRD validation mechanisms provided by Kubernetes.\\n\\nIn addition, dedicated APIs with mode-specific fields are self documenting since they remove the need to explicitly document which fields and field values are allowed or required for each individual mode. Users will also be able to get an easier overview of the different parameters they could set for each mode, which leads to a better user experience when using the operator.\\n\\nFinally, the advantage of using a separate CRD is the possibility of using an alpha API version, which would clearly indicate that the CRD is still under development. The Prometheus CRD, on the other hand, has already been declared as v1 and adding experimental fields to it will be challenging from both documentation and implementation aspects.\\n\\n### Prometheus Agent CRD\\n\\nThe PrometheusAgent CRD would be similar to the Prometheus CRD, with the exception of removing fields which are not applicable to the prometheus agent mode.\\n\\nHere is the list of fields we want to exclude:\\n* `retention`\\n* `retentionSize`\\n* `disableCompaction`\\n* `evaluationInterval`\\n* `rules`\\n* `query`\\n* `ruleSelector`\\n* `ruleNamespaceSelector`\\n* `alerting`\\n* `remoteRead`\\n* `additionalAlertRelabelConfigs`\\n* `additionalAlertManagerConfigs`\\n* `thanos`\\n* `prometheusRulesExcludedFromEnforce`\\n* `queryLogFile`\\n* `allowOverlappingBlocks`\\n\\nThe `enabledFeatures` field can be validated for agent-specific features only, which include: `expand-external-labels`, `extra-scrape-metrics` and `new-service-discovery-manager`.\\n\\nFinally, the `remoteWrite` field should be made required only for the agent since it is a mandatory configuration section in agent mode.\\n\\n### Deployment Strategies\\n\\nWhen using Prometheus in server mode, scraped samples are stored in memory and on disk. These samples need to be preserved during disruptions, such as pod replacements or cluster maintenance operations which cause evictions. Because of this, the Prometheus Operator currently deploys Prometheus instances as Kubernetes StatefulSets.\\n\\nOn the other hand, when running Prometheus in agent mode, samples are sent to a remote write target immediately, and are not kept locally for a long time. The only use-case for storing samples locally is to allow retries when remote write targets are not available. This is achieved by keeping scraped samples in a WAL for 2h at most. Samples which have been successfully sent to remote write targets are immediately removed from local storage.\\n\\nSince the Prometheus agent has slightly different storage requirements, this proposal suggests allowing users to choose different deployment strategies.\\n\\n#### Running the agent with cluster-wide scope\\n\\nEven though the Prometheus agent has very little need for storage, there are still scenarios where sample data can be lost if persistent storage is not used. If a remote write target is unavailable and an agent pod is evicted at the same time, the samples collected during the unavailability window of the remote write target will be completely lost.\\n\\nFor this reason, the cluster-wide strategy would be implemented by deploying a StatefulSet, similarly to how `Prometheus` CRs are currently reconciled. This also allows for reusing existing code from the operator and delivering a working solution faster and with fewer changes. Familiarity with how StatefulSets work, together with the possibility to reuse existing code, were the primary reasons for choosing StatefulSets for this strategy over Deployments.\\n\\nThe following table documents the problems that could occur with a Deployment and StatefulSet strategy in different situations.\\n\\n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n
\\n Pod update\\n Network outage during pod update\\n Network outage during node drain\\n Cloud k8s node rotation\\n Non-graceful pod deletion\\n
Deployment with emptyDir volume\\n No delay in scrapes if the new pod is created before the old one is terminated\\n Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Unsent samples will be lost\\n Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Statefulset with a PVC\\n Potential delay in a subsequent scrape due to recreation of the pod\\n No data loss, the volume will contain all unsent data\\n No data loss, the volume will contain all unsent data\\n No data loss if a new pod scheduled to the same AZ node. May be stuck in pending state otherwise\\n No data loss, the volume will contain all unsent data\\n
Deployment or STS with replicas\\n No delay, mitigated by replicas\\n Unsent data will be lost if last replica terminated before network outage resolves\\n No data loss, as other replicas are running on other nodes\\n No data loss, as other replicas running on other nodes\\n No data loss as other replicas untouched\\n
\\n\\n#### Running the agent with node-specific scope\\n\\nThis strategy has a built-in auto-scaling mechanism since each agent will scrape only a subset of the targets. As the cluster grows and more nodes are added to it, new agent instances will automatically be scheduled to scrape pods on those nodes. Even though the load distribution will not be perfect (targets on certain nodes might produce far more metrics than targets on other nodes), it is a simple way of adding some sort of load management.\\n\\nAnother advantage is that persistent storage can now be handled by mounting a host volume, a strategy commonly used by log collectors. The need for persistent storage is described in the StatefulSet strategy section.\\n\\nThe Grafana Agent config exposes a `host_filter` boolean flag which, when enabled, instructs the agent to only filter targets from the same node, in addition to the scrape config already provided. With this option, the same config can be used for agents running on multiple nodes, and the agents will automatically scrape targets from their own nodes. Such a config option is not yet available in Prometheus. An issue has already been raised [[3]](https://github.com/prometheus/prometheus/issues/9637) and there is an open PR for addressing it [[4]](https://github.com/prometheus/prometheus/pull/10004).\\n\\nUntil the upstream work has been completed, it could be possible to implement this strategy with a few tweaks:\\n* the operator could use the [downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api) to inject the node name in the pods.\\n* the operator's config reloader already supports expansion of environment variables.\\n\\nWith this setup, the unexpanded Prometheus configuration would look as follows\\n\\n```yaml\\nrelabel_configs:\\n- source_labels: [__meta_kubernetes_pod_node_name]\\n action: keep\\n regex: $NODE_NAME\\n\\nin the pod definition:\\nspec:\\n- container: config-reloader\\n env:\\n- name: NODE_NAME\\n valueFrom:\\n fieldRef:\\n fieldPath: spec.nodeName\\n```\\n\\n## Additional implementation details\\n\\nThere has been a suggestion in [a Github comment](https://github.com/prometheus-operator/prometheus-operator/issues/3989#issuecomment-821249404) to introduce a ScrapeConfig CRD in parallel to adding the PrometheusAgent CRD, and \\u201ctranslate\\u201d PrometheusAgent CRs to ScrapeConfig CRs. The main challenge with this approach is that it significantly increases the scope of the work that needs to be done to support deploying Prometheus agents.\\n\\nA leaner alternative would be to focus on implementing the PrometheusAgent CRD by reusing code from the existing Prometheus controller. The ScrapeConfig can then be introduced separately, and the PrometheusAgent can be the first CRD which gets migrated to it.\\n\\n### Implementation steps\\n\\nThe first step in the implementation process would include creating the PrometheusAgent CRD and deploying the agent as a StatefulSet, similar to how the Prometheus CRD is currently reconciled. This will allow for reusing a lot of the existing codebase from the Prometheus controller and the new CRD can be released in a timely manner.\\n\\nSubsequent steps would include iterating on users' feedback and either implementing different deployment strategies, or refining the existing one.\\n\\n## References\\n* [1] [https://github.com/grafana/agent/blob/5bf8cf452fa76c75387e30b6373630923679221c/production/kubernetes/agent-bare.yaml#L43](https://github.com/grafana/agent/blob/5bf8cf452fa76c75387e30b6373630923679221c/production/kubernetes/agent-bare.yaml#L43)\\n* [2] [https://github.com/open-telemetry/opentelemetry-operator#deployment-modes](https://github.com/open-telemetry/opentelemetry-operator#deployment-modes)\\n* [3] [https://github.com/prometheus/prometheus/issues/9637](https://github.com/prometheus/prometheus/issues/9637)\\n* [4] [https://github.com/prometheus/prometheus/pull/10004](https://github.com/prometheus/prometheus/pull/10004)\\n\",\n", - " \"# Additional Scrape Configuration\\n\\nAdditionalScrapeConfigs allows specifying a key of a Secret containing\\nadditional Prometheus scrape configurations. Scrape configurations specified\\nare appended to the configurations generated by the Prometheus Operator.\\n\\nJob configurations specified must have the form as specified in the official\\n[Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).\\nAs scrape configs are appended, the user is responsible to make sure it is\\nvalid. *Note* that using this feature may expose the possibility to break\\nupgrades of Prometheus.\\n\\nIt is advised to review Prometheus release notes to ensure that no incompatible\\nscrape configs are going to break Prometheus after the upgrade.\\n\\n## Creating an additional configuration\\n\\nFirst, you will need to create the additional configuration.\\nBelow we are making a simple \\\"prometheus\\\" config. Name this\\n`prometheus-additional.yaml` or something similar.\\n\\n```yaml\\n- job_name: \\\"prometheus\\\"\\n static_configs:\\n - targets: [\\\"localhost:9090\\\"]\\n```\\n\\nThen you will need to make a secret out of this configuration.\\n\\n```sh\\nkubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml\\n```\\n\\nNext, apply the generated kubernetes manifest\\n\\n```\\nkubectl apply -f additional-scrape-configs.yaml -n monitoring\\n```\\n\\nFinally, reference this additional configuration in your `prometheus.yaml` CRD.\\n\\n```yaml\\napiVersion: monitoring.coreos.com/v1\\nkind: Prometheus\\nmetadata:\\n name: prometheus\\n labels:\\n prometheus: prometheus\\nspec:\\n replicas: 2\\n serviceAccountName: prometheus\\n serviceMonitorSelector:\\n matchLabels:\\n team: frontend\\n additionalScrapeConfigs:\\n name: additional-scrape-configs\\n key: prometheus-additional.yaml\\n```\\n\\nNOTE: Use only one secret for ALL additional scrape configurations.\\n\\n## Additional References\\n\\n* [Prometheus Spec](api.md#monitoring.coreos.com/v1.PrometheusSpec)\\n* [Additional Scrape Configs](../example/additional-scrape-configs)\\n\"\n", - " ]\n", - " ],\n", - " \"uris\": null,\n", - " \"data\": null,\n", - " \"metadatas\": [\n", - " [\n", - " null,\n", - " null,\n", - " null\n", - " ]\n", - " ],\n", - " \"distances\": [\n", - " [\n", - " 0.835404098033905,\n", - " 0.8792126774787903,\n", - " 0.8795778155326843\n", - " ]\n", - " ],\n", - " \"included\": [\n", - " \"distances\",\n", - " \"documents\",\n", - " \"metadatas\"\n", - " ]\n", - "}\n" + "Loaded 63 documents\n" ] } ], "source": [ - "import json # we will use this to pretty-print the result\n", + "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n", "\n", - "result = collection.query(\n", - " query_texts=[\"This is a document about prometheus.\"],\n", - " n_results=3, # how many results to return (10 by default)\n", - ")\n", - "print(json.dumps(result, indent=2))" + "\n", + "documents =[]\n", + "\n", + "for file in doc_files:\n", + " loader = UnstructuredMarkdownLoader(\n", + " file,\n", + " mode='single',\n", + " strategy='fast',\n", + " )\n", + " documents.append(loader.load())\n", + "\n", + "print(f'Loaded {len(documents)} documents')" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "4. We can now store our documents into the database." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "for document in documents:\n", + " vector_store.add_documents(documents=document)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "5. Finally, we can retrieve documents from our database." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(id='93e7fa11-5553-40c0-aef7-7e94299bdd72', metadata={'source': './bitburner/src/Documentation/doc/basic/hacking.md'}, page_content='Hacking\\n\\nIn the year 2077, currency has become digital and decentralized. People and corporations store their money on servers. By hacking these servers, you can steal their money and gain experience.\\n\\nGaining Root Access\\n\\nThe first step to hacking a server is to gain root access to that server. This can be done using the NUKE.exe virus. You start the game with a copy of the NUKE.exe virus on your home computer. The NUKE.exe virus attacks the target server\\'s open ports using buffer overflow exploits. When successful, you are granted root administrative access to the machine.\\n\\nIn order for the NUKE.exe virus to succeed, the target server needs to have enough open ports. Some servers have no security and will not need any ports opened. Some will have very high security and will need many ports opened. In order to open ports on another server, you will need to run programs that attack the server to open specific ports. These programs can be coded once your hacking skill gets high enough, or they can be purchased if you can find a seller.\\n\\nThere are two ways to execute port-opening programs and the NUKE virus:\\n\\nConnect to the target server through the Terminal and use the run command: $ run [programName]\\n\\nUse a function:\\n\\nnuke\\n\\nbrutessh\\n\\nftpcrack\\n\\nrelaysmtp\\n\\nhttpworm\\n\\nsqlinject\\n\\nThere are two ways to determine how many ports need to be opened on a server in order to successfully NUKE it:\\n\\nConnect to that server through the Terminal and use the analyze command.\\n\\nUse the getServerNumPortsRequired function.\\n\\nOnce you have enough ports opened on a server and have ran the NUKE virus to gain root access, you will be able to hack it.\\n\\nFor specific details of how Hacking work \"offline\"\\n\\nSee Offline And Bonus Time.\\n\\nGeneral Hacking Mechanics\\n\\nWhen you execute the hack command, either manually through the Terminal or automatically through a script, you attempt to hack the server. This action takes time. The more advanced a server\\'s security is, the more time it will take. Your hacking skill level also affects the hacking time, with a higher hacking skill leading to shorter hacking times. Also, running the hack command manually through Terminal is faster than hacking from a script.\\n\\nYour attempt to hack a server will not always succeed. The chance you have to successfully hack a server is also determined by the server\\'s security and your hacking skill level. Even if your hacking attempt is unsuccessful, you will still gain experience points.\\n\\nWhen you successfully hack a server. You steal a certain percentage of that server\\'s total money. This percentage is, once again, determined by the server\\'s security and your hacking skill level. The amount of money on a server is not limitless. So, if you constantly hack a server and deplete its money, then you will encounter diminishing returns in your hacking (since you are only hacking a certain percentage). You can increase the amount of money on a server using a script and the grow function.\\n\\nServer Security\\n\\nEach server has a security level, typically between 1 and 100. A higher number means the server has stronger security.\\n\\nAs mentioned above, a server\\'s security level is an important factor to consider when hacking. You can check a server\\'s security level using the analyze Terminal command. You can also check a server\\'s security in a script, using the getServerSecurityLevel function.\\n\\nWhenever a server is hacked manually or through a script, its security level increases by a small amount. Calling the grow function in a script will also increase security level of the target server. These actions will make it harder for you to hack the server, and decrease the amount of money you can steal. You can lower a server\\'s security level in a script using the weaken function.\\n\\nEach server has a minimum security level. The server\\'s security level will not fall below this value if you try to weaken it. You can get this value with the getServerMinSecurityLevel function.\\n\\nBackdoors\\n\\nServers that can be hacked can also have backdoors installed. These backdoors will provide you with a benefit - the services may be cheaper, penalties may be reduced or there may be other results. Honeypots exist and will let factions know when you have succeeded at backdooring their system. Once you have a backdoor installed, you can connect to that server directly.\\n\\nWhen you visit a location in the city and see that the name is partially scrambled, this indicates that you have backdoored the server related to the location.'),\n", + " Document(id='0d1eadf5-0dd7-47c7-b325-bc7ad98d3439', metadata={'source': './bitburner/src/Documentation/doc/basic/hacknet_nodes.md'}, page_content=\"Hacknet nodes\\n\\nThis distributed network of computers allows you to gain passive income. By upgrading a node's level, RAM, and CPU cores you can increase the amount of money it earns. You can also purchase new nodes to expand your Hacknet - The cost for each node increases as your network grows.\\n\\nHacknet nodes won't make as much money as basic hacking scripts, and they are not enough to progress alone.\\n\\nLater in the game, there is a powerful change to the Hacknet system called Hacknet Servers.\"),\n", + " Document(id='296d94bf-a495-4365-b371-260257f0306a', metadata={'source': './bitburner/src/Documentation/doc/basic/stats.md'}, page_content=\"Below are some of the stats that will increase with play and reset during augmentation installs as you progress through the game. Your stats can be found in the Overview panel, the Stats subpage of the side menu, or with API methods like ns.getPlayer().\\n\\nHack Skill\\n\\nFor many aspects of Bitburner, increasing your Hack skill will be an important goal. Primarily affected by the efficiency of your hacking strategies, you will also be offered Augmentations that greatly enhance your Hack Skill level and how effective its results are.\\n\\nAffects:\\n\\nTime needed to execute hack, grow, or weaken and similar methods\\n\\nYour chance to successfully hack a server\\n\\nPercent of a server's money stolen when hacking it\\n\\nSuccess rate of certain crimes\\n\\nTime needed to create a Program\\n\\nFaction Reputation gain when carrying out Hacking Contracts or Field Work\\n\\nCompany Reputation gain for certain jobs\\n\\nGain Hack experience by:\\n\\nManually hacking servers through the Terminal\\n\\nUsing ns.hack(), ns.grow(), or ns.weaken() through scripts\\n\\nCommitting certain crimes\\n\\nCarrying out Hacking Contracts or doing Field work for Factions\\n\\nSome Company jobs and other types of work\\n\\nStudying at a university\\n\\nCombat Skills\\n\\nStrength, Defense, Dexterity, and Agility\\n\\nThese represent your physical skill and attributes, including your ability to sneak, inflict or endure damage, and pull off high precision tasks. Similar to your Hack skill, you will be offered Faction Augmentations to multiplicatively enhance your Combat Skills and exp gain.\\n\\nAffects:\\n\\nHP scales with Defense. Infiltration and some jobs may cause you to take damage.\\n\\nSuccess rate of certain crimes\\n\\nFaction Reputation gain for Security and Field Work\\n\\nCompany Reputation gain for certain jobs\\n\\nGain experience by:\\n\\nWorking out at a gym\\n\\nCommitting certain crimes\\n\\nDoing Security or Field Work for a Faction\\n\\nWorking certain jobs at a Company\\n\\nCharisma\\n\\nRarely as useful as Hacking and Physical skills, Charisma can help get a company job, gain trust, or calm chaos in social situations.\\n\\nCharisma can also be enhanced with Augmentations.\\n\\nAffects:\\n\\nSuccess rate of certain crimes\\n\\nFaction Reputation gain for Field Work\\n\\nCompany Reputation gain for most jobs\\n\\nGain experience by:\\n\\nCommitting certain crimes\\n\\nStudying at a university\\n\\nWorking certain jobs at a Company\\n\\nDoing Field work for a Faction\\n\\nOther Stats and abilities are available in later stages of the game.\"),\n", + " Document(id='cc594be4-3c56-420d-833e-be94684b1ede', metadata={'source': './bitburner/src/Documentation/doc/index.md'}, page_content=\"Documentation\\n\\nGuides\\n\\nBeginner's guide\\n\\nBasic Mechanics\\n\\nStats\\n\\nTerminal\\n\\nHacking\\n\\nScripts\\n\\nServers\\n\\nRAM\\n\\nHacknet nodes\\n\\nAugmentations\\n\\nCompanies\\n\\nFactions\\n\\nCrimes\\n\\nInfiltration\\n\\nPrograms\\n\\nReputation\\n\\nStock market\\n\\nWorld\\n\\nCoding contracts\\n\\nAutocomplete\\n\\nAdvanced Mechanics\\n\\nHacking algorithms\\n\\nList of factions and their requirements\\n\\nOffline scripts and bonus time\\n\\nBitNodes\\n\\nSource-Files\\n\\nGang\\n\\nCorporation\\n\\nIntelligence\\n\\nBladeburner\\n\\nHacknet servers\\n\\nSleeves\\n\\nGrafting\\n\\nStanek's Gift\\n\\nIPvGO\\n\\nResources\\n\\nNS API documentation\\n\\nLearn to program\\n\\nRemote API\\n\\nGame frozen or stuck?\\n\\nReact\\n\\nTools & Resources\\n\\nChangelog\\n\\nChangelog - Legacy v1\\n\\nChangelog - Legacy v0\\n\\nMigration\\n\\nBitburner v1.0.0 script migration guide\\n\\nBitburner v2.0.0 script migration guide\\n\\nNetscript 2 migration guide (.script to .js)\")]" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "vector_store.search(\n", + " query='How to hack',\n", + " search_type='similarity',\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Build the RAG chain\n", + "\n", + "There are a number of frameworks available to build our RAG.\n", + "* [LangChain](https://langchain.com/)\n", + "* [LlamaIndex](https://docs.llamaindex.ai/en/latest/)\n", + "\n", + "In this example we will use LangChain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { @@ -204,7 +329,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.1" + "version": "3.13.3" } }, "nbformat": 4,