diff --git a/requirements.txt b/requirements.txt index a8709ea..c15d402 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,5 +5,4 @@ python-lsp-server[all] matplotlib pandas -GitPython -chromadb \ No newline at end of file +# chromadb \ No newline at end of file diff --git a/work/rag.ipynb b/work/rag.ipynb index afaf7d9..d643222 100644 --- a/work/rag.ipynb +++ b/work/rag.ipynb @@ -12,7 +12,13 @@ "* Everything needs to be self-hosted\n", " * An Ollama server is already running locally (https://localhost:11434)\n", "* The interface is unimportant for now\n", - " * Eventually, we want it to be a bot hosted in Teams and/or Discord" + " * Eventually, we want it to be a bot hosted in Teams and/or Discord\n", + "\n", + "For this notebook, we will ingest the documentation of Bitburner.\n", + "\n", + ">Bitburner is a programming-based incremental game that revolves around hacking and cyberpunk themes.\n", + "\n", + "The documentation located in this repository: https://github.com/bitburner-official/bitburner-src.git and the documentation is within the *src/Documentation/* folder of this repository." ] }, { @@ -23,25 +29,48 @@ "\n", "### Step 1: Fetch the documents from the git repository\n", "\n", - "We will use `gitpython` to clone the repository and fetch the documents.\n", - "\n", - "For this notebook, we will ingest the documentation of prometheus-operator located in this repository: https://github.com/prometheus-operator/prometheus-operator.git. The documentation is within the *Documentation/* folder of this repository.\n", - "\n", - "First thing first, we need to clone the repository. If it already exists locally, we can just pull the latest changes." + "We will use `gitpython` to clone the repository and fetch the documents.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install --quiet --upgrade langchain-community GitPython" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First thing first, we need to clone the repository. If it already exists locally, we can just pull the latest changes." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "from git import Repo\n", "\n", - "repo_url = \"https://github.com/prometheus-operator/prometheus-operator.git\"\n", - "local_repo_path = \"./prometheus-operator\"\n", + "repo_url = \"https://github.com/bitburner-official/bitburner-src.git\"\n", + "local_repo_path = \"./bitburner\"\n", "\n", "if not os.path.exists(local_repo_path):\n", " Repo.clone_from(repo_url, local_repo_path)\n", @@ -59,133 +88,229 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Found 40 documents.\n" + "Found 63 documents.\n" ] } ], "source": [ - "documentation_root = os.path.join(local_repo_path, \"Documentation\")\n", - "documentation_files = []\n", + "doc_root = os.path.join(local_repo_path, \"src/Documentation\")\n", + "doc_files = []\n", "\n", "# Walk through the directory and find all markdown files\n", - "for root, dirs, files in os.walk(documentation_root):\n", + "for root, dirs, files in os.walk(doc_root):\n", " for file in files:\n", " if file.endswith(\".md\"):\n", - " documentation_files.append(os.path.join(root, file))\n", + " doc_files.append(os.path.join(root, file))\n", "\n", - "print(f\"Found {len(documentation_files)} documents.\")" + "print(f\"Found {len(doc_files)} documents.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Ingest the documents in a vector database\n", + "## Step 2: Index the documents in a vector database\n", "\n", "To build our RAG, we need to store the documents in a vector database. Several options are available:\n", "* [FAISS](https://faiss.ai/)\n", "* [ChromaDB](https://www.trychroma.com/)\n", "* [Qdrant](https://qdrant.tech/)\n", + "* [LangChain](https://langchain.com/)\n", "* etc.\n", "\n", - "For this example, we will use ChromaDB because it is easy to set up and use. Helpfully, ChromaDB is able to automatically generate embeddings for us. We will store the documents in a collection called `documentation`. The collection will live in-memory, but in a more complete setup, we could setup Chroma in an client-server mode and/or with persistence enabled." + "For this example, we will use LangChain because it's a very convenient an all-in-one framework that is commonly used in LLM applications. As for our backend, we'll use Ollama because it's convenient to be able to run the models locally." ] }, { "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import chromadb\n", - "\n", - "chroma_client = chromadb.Client()\n", - "collection = chroma_client.create_collection(name=\"documentation\")\n", - "\n", - "# Read the contents of each document and store them in a list\n", - "documents = []\n", - "for file in documentation_files:\n", - " with open(file, \"r\") as f:\n", - " content = f.read()\n", - " documents.append(content)\n", - "\n", - "# Add the documents to the collection\n", - "collection.add(documents=documents, ids=documentation_files)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that we have built our collection, we can try to query it. Let's search for a document about prometheus." - ] - }, - { - "cell_type": "code", - "execution_count": 8, + "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "{\n", - " \"ids\": [\n", - " [\n", - " \"./prometheus-operator/Documentation/platform/prometheus-agent.md\",\n", - " \"./prometheus-operator/Documentation/proposals/202201-prometheus-agent.md\",\n", - " \"./prometheus-operator/Documentation/additional-scrape-config.md\"\n", - " ]\n", - " ],\n", - " \"embeddings\": null,\n", - " \"documents\": [\n", - " [\n", - " \"---\\nweight: 204\\ntoc: true\\ntitle: Prometheus Agent\\nmenu:\\n docs:\\n parent: user-guides\\nlead: \\\"\\\"\\nimages: []\\ndraft: false\\ndescription: Guide for running Prometheus in Agent mode\\n---\\n\\n{{< alert icon=\\\"\\ud83d\\udc49\\\" text=\\\"Prometheus Operator >= v0.64.0 is required.\\\"/>}}\\n\\nAs mentioned in [Prometheus's blog](https://prometheus.io/blog/2021/11/16/agent/), Prometheus Agent\\nis a deployment model optimized for environments where all collected data is forwarded to\\na long-term storage solution, e.g. Cortex, Thanos or Prometheus, that do not need storage or rule evaluation.\\n\\nFirst of all, make sure that the PrometheusAgent CRD is installed in the cluster and that the operator has the proper RBAC permissions to reconcile the PrometheusAgent resources.\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRole\\nmetadata:\\n labels:\\n app.kubernetes.io/component: controller\\n app.kubernetes.io/name: prometheus-operator\\n app.kubernetes.io/version: 0.80.0\\n name: prometheus-operator\\nrules:\\n- apiGroups:\\n - monitoring.coreos.com\\n resources:\\n - alertmanagers\\n - alertmanagers/finalizers\\n - alertmanagers/status\\n - alertmanagerconfigs\\n - prometheuses\\n - prometheuses/finalizers\\n - prometheuses/status\\n - prometheusagents\\n - prometheusagents/finalizers\\n - prometheusagents/status\\n - thanosrulers\\n - thanosrulers/finalizers\\n - thanosrulers/status\\n - scrapeconfigs\\n - servicemonitors\\n - podmonitors\\n - probes\\n - prometheusrules\\n verbs:\\n - '*'\\n- apiGroups:\\n - apps\\n resources:\\n - statefulsets\\n verbs:\\n - '*'\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - configmaps\\n - secrets\\n verbs:\\n - '*'\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - pods\\n verbs:\\n - list\\n - delete\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - services\\n - services/finalizers\\n verbs:\\n - get\\n - create\\n - update\\n - delete\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - nodes\\n verbs:\\n - list\\n - watch\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - namespaces\\n verbs:\\n - get\\n - list\\n - watch\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - events\\n verbs:\\n - patch\\n - create\\n- apiGroups:\\n - networking.k8s.io\\n resources:\\n - ingresses\\n verbs:\\n - get\\n - list\\n - watch\\n- apiGroups:\\n - storage.k8s.io\\n resources:\\n - storageclasses\\n verbs:\\n - get\\n- apiGroups:\\n - \\\"\\\"\\n resources:\\n - endpoints\\n verbs:\\n - get\\n - create\\n - update\\n - delete\\n```\\n\\nSimilarly to Prometheus, Prometheus Agent will also require permission to scrape targets. Because of this, we will create a new service account for the Agent with the necessary permissions to scrape targets.\\n\\nStart with the ServiceAccount, ClusterRole and ClusterRoleBinding:\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-service-account.yaml\\\"\\napiVersion: v1\\nkind: ServiceAccount\\nmetadata:\\n name: prometheus-agent\\n```\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-cluster-role.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRole\\nmetadata:\\n name: prometheus-agent\\nrules:\\n- apiGroups: [\\\"\\\"]\\n resources:\\n - services\\n - endpoints\\n - pods\\n verbs: [\\\"get\\\", \\\"list\\\", \\\"watch\\\"]\\n- apiGroups: [\\\"\\\"]\\n resources:\\n - configmaps\\n verbs: [\\\"get\\\"]\\n- apiGroups:\\n - networking.k8s.io\\n resources:\\n - ingresses\\n verbs: [\\\"get\\\", \\\"list\\\", \\\"watch\\\"]\\n- nonResourceURLs: [\\\"/metrics\\\"]\\n verbs: [\\\"get\\\"]\\n```\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus-cluster-role-binding.yaml\\\"\\napiVersion: rbac.authorization.k8s.io/v1\\nkind: ClusterRoleBinding\\nmetadata:\\n name: prometheus-agent\\nroleRef:\\n apiGroup: rbac.authorization.k8s.io\\n kind: ClusterRole\\n name: prometheus-agent\\nsubjects:\\n- kind: ServiceAccount\\n name: prometheus-agent\\n namespace: default\\n```\\n\\nLastly, we can deploy the Agent. The `spec` field is very similar to the Prometheus CRD but the features that aren't applicable to the agent mode (like alerting, retention, Thanos, ...) are not available.\\n\\n```yaml mdox-exec=\\\"cat example/rbac/prometheus-agent/prometheus.yaml\\\"\\napiVersion: monitoring.coreos.com/v1alpha1\\nkind: PrometheusAgent\\nmetadata:\\n name: prometheus-agent\\nspec:\\n replicas: 2\\n serviceAccountName: prometheus-agent\\n serviceMonitorSelector:\\n matchLabels:\\n team: frontend\\n```\\n\\nContinue with the [Getting Started page]({{}}) to learn how to monitor applications running on Kubernetes.\\n\",\n", - " \"# Prometheus Agent support\\n\\n## Summary\\n\\nThe Prometheus 2.32.0 release introduces the Prometheus Agent, a mode optimized for remote-write dominant scenarios. This document proposes extending the Prometheus Operator to allow running a Prometheus Agent with different deployment strategies.\\n\\n## Background\\n\\nThe Prometheus Operator in its current state does not allow a simple way of deploying the Prometheus agent. A potential workaround has been described in a [Github comment](https://github.com/prometheus-operator/prometheus-operator/issues/3989#issuecomment-974137486), where the agent can be deployed through the existing Prometheus CRD by explicitly setting command-line arguments specific to the agent mode.\\n\\nAs described in the comment, one significant problem with this approach is that the Prometheus Operator always generates `alerts` and `rules` sections in the Prometheus config file. These sections are not allowed when running the agent so users need to take additional actions to pause reconciliation of the Prometheus CR, tweak the generated secret and then unpause reconciliation in order to resolve the problem. Alternatively, users can apply a strategic merge patch to the prometheus container as described in the kube-prometheus docs: [https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/prometheus-agent.md](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/prometheus-agent.md)\\n\\nWhile this workaround can be used as a stop-gap solution to unblock users in the short term, it has the drawback of needing additional steps which require understanding implementation details of the operator itself. In addition to this, overriding the value of the argument `--config.file` also requires knowledge of Prometheus Operator internals.\\n\\nA lot of the fields supported by the current PrometheusSpec are not applicable to the agent mode. These fields are documented in the PrometheusAgent CRD section.\\n\\nFinally, the Prometheus agent is significantly different from the Prometheus server in the way that it fits in a monitoring stack. Therefore, running it as a StatefulSet might not be the only possible deployment strategy, users might want to run it as a DaemonSet or a Deployment instead.\\n\\n## Proposal\\n\\nThis document proposes introducing a PrometheusAgent CRD to allow users to run Prometheus in agent mode. Having a separate CRD allows the Prometheus and PrometheusAgent CRDs to evolve independently and expose parameters specific to each Prometheus mode.\\n\\nFor example, the PrometheusAgent CRD could have a `strategy` field indicating the deployment strategy for the agent, but no `alerting` field since alerts are not supported in agent mode. Even though there will be an upfront cost for introducing a new CRD, having separate APIs would simplify long-term maintenance by allowing the use of CRD validation mechanisms provided by Kubernetes.\\n\\nIn addition, dedicated APIs with mode-specific fields are self documenting since they remove the need to explicitly document which fields and field values are allowed or required for each individual mode. Users will also be able to get an easier overview of the different parameters they could set for each mode, which leads to a better user experience when using the operator.\\n\\nFinally, the advantage of using a separate CRD is the possibility of using an alpha API version, which would clearly indicate that the CRD is still under development. The Prometheus CRD, on the other hand, has already been declared as v1 and adding experimental fields to it will be challenging from both documentation and implementation aspects.\\n\\n### Prometheus Agent CRD\\n\\nThe PrometheusAgent CRD would be similar to the Prometheus CRD, with the exception of removing fields which are not applicable to the prometheus agent mode.\\n\\nHere is the list of fields we want to exclude:\\n* `retention`\\n* `retentionSize`\\n* `disableCompaction`\\n* `evaluationInterval`\\n* `rules`\\n* `query`\\n* `ruleSelector`\\n* `ruleNamespaceSelector`\\n* `alerting`\\n* `remoteRead`\\n* `additionalAlertRelabelConfigs`\\n* `additionalAlertManagerConfigs`\\n* `thanos`\\n* `prometheusRulesExcludedFromEnforce`\\n* `queryLogFile`\\n* `allowOverlappingBlocks`\\n\\nThe `enabledFeatures` field can be validated for agent-specific features only, which include: `expand-external-labels`, `extra-scrape-metrics` and `new-service-discovery-manager`.\\n\\nFinally, the `remoteWrite` field should be made required only for the agent since it is a mandatory configuration section in agent mode.\\n\\n### Deployment Strategies\\n\\nWhen using Prometheus in server mode, scraped samples are stored in memory and on disk. These samples need to be preserved during disruptions, such as pod replacements or cluster maintenance operations which cause evictions. Because of this, the Prometheus Operator currently deploys Prometheus instances as Kubernetes StatefulSets.\\n\\nOn the other hand, when running Prometheus in agent mode, samples are sent to a remote write target immediately, and are not kept locally for a long time. The only use-case for storing samples locally is to allow retries when remote write targets are not available. This is achieved by keeping scraped samples in a WAL for 2h at most. Samples which have been successfully sent to remote write targets are immediately removed from local storage.\\n\\nSince the Prometheus agent has slightly different storage requirements, this proposal suggests allowing users to choose different deployment strategies.\\n\\n#### Running the agent with cluster-wide scope\\n\\nEven though the Prometheus agent has very little need for storage, there are still scenarios where sample data can be lost if persistent storage is not used. If a remote write target is unavailable and an agent pod is evicted at the same time, the samples collected during the unavailability window of the remote write target will be completely lost.\\n\\nFor this reason, the cluster-wide strategy would be implemented by deploying a StatefulSet, similarly to how `Prometheus` CRs are currently reconciled. This also allows for reusing existing code from the operator and delivering a working solution faster and with fewer changes. Familiarity with how StatefulSets work, together with the possibility to reuse existing code, were the primary reasons for choosing StatefulSets for this strategy over Deployments.\\n\\nThe following table documents the problems that could occur with a Deployment and StatefulSet strategy in different situations.\\n\\n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n
\\n Pod update\\n Network outage during pod update\\n Network outage during node drain\\n Cloud k8s node rotation\\n Non-graceful pod deletion\\n
Deployment with emptyDir volume\\n No delay in scrapes if the new pod is created before the old one is terminated\\n Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Unsent samples will be lost\\n Unsent samples will be lost. \\n

\\nEmptyDir is tied to a pod and node, and data from the old pod will not be preserved.\\n

Statefulset with a PVC\\n Potential delay in a subsequent scrape due to recreation of the pod\\n No data loss, the volume will contain all unsent data\\n No data loss, the volume will contain all unsent data\\n No data loss if a new pod scheduled to the same AZ node. May be stuck in pending state otherwise\\n No data loss, the volume will contain all unsent data\\n
Deployment or STS with replicas\\n No delay, mitigated by replicas\\n Unsent data will be lost if last replica terminated before network outage resolves\\n No data loss, as other replicas are running on other nodes\\n No data loss, as other replicas running on other nodes\\n No data loss as other replicas untouched\\n
\\n\\n#### Running the agent with node-specific scope\\n\\nThis strategy has a built-in auto-scaling mechanism since each agent will scrape only a subset of the targets. As the cluster grows and more nodes are added to it, new agent instances will automatically be scheduled to scrape pods on those nodes. Even though the load distribution will not be perfect (targets on certain nodes might produce far more metrics than targets on other nodes), it is a simple way of adding some sort of load management.\\n\\nAnother advantage is that persistent storage can now be handled by mounting a host volume, a strategy commonly used by log collectors. The need for persistent storage is described in the StatefulSet strategy section.\\n\\nThe Grafana Agent config exposes a `host_filter` boolean flag which, when enabled, instructs the agent to only filter targets from the same node, in addition to the scrape config already provided. With this option, the same config can be used for agents running on multiple nodes, and the agents will automatically scrape targets from their own nodes. Such a config option is not yet available in Prometheus. An issue has already been raised [[3]](https://github.com/prometheus/prometheus/issues/9637) and there is an open PR for addressing it [[4]](https://github.com/prometheus/prometheus/pull/10004).\\n\\nUntil the upstream work has been completed, it could be possible to implement this strategy with a few tweaks:\\n* the operator could use the [downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api) to inject the node name in the pods.\\n* the operator's config reloader already supports expansion of environment variables.\\n\\nWith this setup, the unexpanded Prometheus configuration would look as follows\\n\\n```yaml\\nrelabel_configs:\\n- source_labels: [__meta_kubernetes_pod_node_name]\\n action: keep\\n regex: $NODE_NAME\\n\\nin the pod definition:\\nspec:\\n- container: config-reloader\\n env:\\n- name: NODE_NAME\\n valueFrom:\\n fieldRef:\\n fieldPath: spec.nodeName\\n```\\n\\n## Additional implementation details\\n\\nThere has been a suggestion in [a Github comment](https://github.com/prometheus-operator/prometheus-operator/issues/3989#issuecomment-821249404) to introduce a ScrapeConfig CRD in parallel to adding the PrometheusAgent CRD, and \\u201ctranslate\\u201d PrometheusAgent CRs to ScrapeConfig CRs. The main challenge with this approach is that it significantly increases the scope of the work that needs to be done to support deploying Prometheus agents.\\n\\nA leaner alternative would be to focus on implementing the PrometheusAgent CRD by reusing code from the existing Prometheus controller. The ScrapeConfig can then be introduced separately, and the PrometheusAgent can be the first CRD which gets migrated to it.\\n\\n### Implementation steps\\n\\nThe first step in the implementation process would include creating the PrometheusAgent CRD and deploying the agent as a StatefulSet, similar to how the Prometheus CRD is currently reconciled. This will allow for reusing a lot of the existing codebase from the Prometheus controller and the new CRD can be released in a timely manner.\\n\\nSubsequent steps would include iterating on users' feedback and either implementing different deployment strategies, or refining the existing one.\\n\\n## References\\n* [1] [https://github.com/grafana/agent/blob/5bf8cf452fa76c75387e30b6373630923679221c/production/kubernetes/agent-bare.yaml#L43](https://github.com/grafana/agent/blob/5bf8cf452fa76c75387e30b6373630923679221c/production/kubernetes/agent-bare.yaml#L43)\\n* [2] [https://github.com/open-telemetry/opentelemetry-operator#deployment-modes](https://github.com/open-telemetry/opentelemetry-operator#deployment-modes)\\n* [3] [https://github.com/prometheus/prometheus/issues/9637](https://github.com/prometheus/prometheus/issues/9637)\\n* [4] [https://github.com/prometheus/prometheus/pull/10004](https://github.com/prometheus/prometheus/pull/10004)\\n\",\n", - " \"# Additional Scrape Configuration\\n\\nAdditionalScrapeConfigs allows specifying a key of a Secret containing\\nadditional Prometheus scrape configurations. Scrape configurations specified\\nare appended to the configurations generated by the Prometheus Operator.\\n\\nJob configurations specified must have the form as specified in the official\\n[Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).\\nAs scrape configs are appended, the user is responsible to make sure it is\\nvalid. *Note* that using this feature may expose the possibility to break\\nupgrades of Prometheus.\\n\\nIt is advised to review Prometheus release notes to ensure that no incompatible\\nscrape configs are going to break Prometheus after the upgrade.\\n\\n## Creating an additional configuration\\n\\nFirst, you will need to create the additional configuration.\\nBelow we are making a simple \\\"prometheus\\\" config. Name this\\n`prometheus-additional.yaml` or something similar.\\n\\n```yaml\\n- job_name: \\\"prometheus\\\"\\n static_configs:\\n - targets: [\\\"localhost:9090\\\"]\\n```\\n\\nThen you will need to make a secret out of this configuration.\\n\\n```sh\\nkubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml\\n```\\n\\nNext, apply the generated kubernetes manifest\\n\\n```\\nkubectl apply -f additional-scrape-configs.yaml -n monitoring\\n```\\n\\nFinally, reference this additional configuration in your `prometheus.yaml` CRD.\\n\\n```yaml\\napiVersion: monitoring.coreos.com/v1\\nkind: Prometheus\\nmetadata:\\n name: prometheus\\n labels:\\n prometheus: prometheus\\nspec:\\n replicas: 2\\n serviceAccountName: prometheus\\n serviceMonitorSelector:\\n matchLabels:\\n team: frontend\\n additionalScrapeConfigs:\\n name: additional-scrape-configs\\n key: prometheus-additional.yaml\\n```\\n\\nNOTE: Use only one secret for ALL additional scrape configurations.\\n\\n## Additional References\\n\\n* [Prometheus Spec](api.md#monitoring.coreos.com/v1.PrometheusSpec)\\n* [Additional Scrape Configs](../example/additional-scrape-configs)\\n\"\n", - " ]\n", - " ],\n", - " \"uris\": null,\n", - " \"data\": null,\n", - " \"metadatas\": [\n", - " [\n", - " null,\n", - " null,\n", - " null\n", - " ]\n", - " ],\n", - " \"distances\": [\n", - " [\n", - " 0.835404098033905,\n", - " 0.8792126774787903,\n", - " 0.8795778155326843\n", - " ]\n", - " ],\n", - " \"included\": [\n", - " \"distances\",\n", - " \"documents\",\n", - " \"metadatas\"\n", - " ]\n", - "}\n" + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ - "import json # we will use this to pretty-print the result\n", - "\n", - "result = collection.query(\n", - " query_texts=[\"This is a document about prometheus.\"],\n", - " n_results=3, # how many results to return (10 by default)\n", - ")\n", - "print(json.dumps(result, indent=2))" + "%pip install --quiet --upgrade langchain-community langchain-ollama langgraph" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To index our documents:\n", + "1. We need convert our documents into vectors. For this, we can use a embedding model. *nomic-embed-text* should provide reasonable performance for our purpose." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_ollama.embeddings import OllamaEmbeddings\n", + "\n", + "embeddings = OllamaEmbeddings(model='nomic-embed-text')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "2. Once we have our vector, we can store them in a database for future retrieval. LangChain conveniently provides us an `InMemoryVectorStore` which will do the job." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_core.vectorstores import InMemoryVectorStore\n", + "\n", + "vector_store = InMemoryVectorStore(embeddings)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "3. Now that we have our embedding model and our vector database, we can start indexing our documents. LangChain has 100+ `DocumentLoader`s to aid us with this task. The documentation is written in markdown, so we can use [UnstructuredMarkdownLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader.html#langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader) to load them." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install --quiet --upgrade unstructured markdown" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 63 documents\n" + ] + } + ], + "source": [ + "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n", + "\n", + "\n", + "documents =[]\n", + "\n", + "for file in doc_files:\n", + " loader = UnstructuredMarkdownLoader(\n", + " file,\n", + " mode='single',\n", + " strategy='fast',\n", + " )\n", + " documents.append(loader.load())\n", + "\n", + "print(f'Loaded {len(documents)} documents')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "4. We can now store our documents into the database." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [], + "source": [ + "for document in documents:\n", + " vector_store.add_documents(documents=document)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "5. Finally, we can retrieve documents from our database." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(id='7b012d63-34f7-4f20-baa2-fae434ecfbd3', metadata={'source': './bitburner/src/Documentation/doc/basic/hacking.md'}, page_content='Hacking\\n\\nIn the year 2077, currency has become digital and decentralized. People and corporations store their money on servers. By hacking these servers, you can steal their money and gain experience.\\n\\nGaining Root Access\\n\\nThe first step to hacking a server is to gain root access to that server. This can be done using the NUKE.exe virus. You start the game with a copy of the NUKE.exe virus on your home computer. The NUKE.exe virus attacks the target server\\'s open ports using buffer overflow exploits. When successful, you are granted root administrative access to the machine.\\n\\nIn order for the NUKE.exe virus to succeed, the target server needs to have enough open ports. Some servers have no security and will not need any ports opened. Some will have very high security and will need many ports opened. In order to open ports on another server, you will need to run programs that attack the server to open specific ports. These programs can be coded once your hacking skill gets high enough, or they can be purchased if you can find a seller.\\n\\nThere are two ways to execute port-opening programs and the NUKE virus:\\n\\nConnect to the target server through the Terminal and use the run command: $ run [programName]\\n\\nUse a function:\\n\\nnuke\\n\\nbrutessh\\n\\nftpcrack\\n\\nrelaysmtp\\n\\nhttpworm\\n\\nsqlinject\\n\\nThere are two ways to determine how many ports need to be opened on a server in order to successfully NUKE it:\\n\\nConnect to that server through the Terminal and use the analyze command.\\n\\nUse the getServerNumPortsRequired function.\\n\\nOnce you have enough ports opened on a server and have ran the NUKE virus to gain root access, you will be able to hack it.\\n\\nFor specific details of how Hacking work \"offline\"\\n\\nSee Offline And Bonus Time.\\n\\nGeneral Hacking Mechanics\\n\\nWhen you execute the hack command, either manually through the Terminal or automatically through a script, you attempt to hack the server. This action takes time. The more advanced a server\\'s security is, the more time it will take. Your hacking skill level also affects the hacking time, with a higher hacking skill leading to shorter hacking times. Also, running the hack command manually through Terminal is faster than hacking from a script.\\n\\nYour attempt to hack a server will not always succeed. The chance you have to successfully hack a server is also determined by the server\\'s security and your hacking skill level. Even if your hacking attempt is unsuccessful, you will still gain experience points.\\n\\nWhen you successfully hack a server. You steal a certain percentage of that server\\'s total money. This percentage is, once again, determined by the server\\'s security and your hacking skill level. The amount of money on a server is not limitless. So, if you constantly hack a server and deplete its money, then you will encounter diminishing returns in your hacking (since you are only hacking a certain percentage). You can increase the amount of money on a server using a script and the grow function.\\n\\nServer Security\\n\\nEach server has a security level, typically between 1 and 100. A higher number means the server has stronger security.\\n\\nAs mentioned above, a server\\'s security level is an important factor to consider when hacking. You can check a server\\'s security level using the analyze Terminal command. You can also check a server\\'s security in a script, using the getServerSecurityLevel function.\\n\\nWhenever a server is hacked manually or through a script, its security level increases by a small amount. Calling the grow function in a script will also increase security level of the target server. These actions will make it harder for you to hack the server, and decrease the amount of money you can steal. You can lower a server\\'s security level in a script using the weaken function.\\n\\nEach server has a minimum security level. The server\\'s security level will not fall below this value if you try to weaken it. You can get this value with the getServerMinSecurityLevel function.\\n\\nBackdoors\\n\\nServers that can be hacked can also have backdoors installed. These backdoors will provide you with a benefit - the services may be cheaper, penalties may be reduced or there may be other results. Honeypots exist and will let factions know when you have succeeded at backdooring their system. Once you have a backdoor installed, you can connect to that server directly.\\n\\nWhen you visit a location in the city and see that the name is partially scrambled, this indicates that you have backdoored the server related to the location.'),\n", + " Document(id='d26660d8-debb-4958-a010-66ba53540ae5', metadata={'source': './bitburner/src/Documentation/doc/index.md'}, page_content=\"Documentation\\n\\nGuides\\n\\nBeginner's guide\\n\\nBasic Mechanics\\n\\nStats\\n\\nTerminal\\n\\nHacking\\n\\nScripts\\n\\nServers\\n\\nRAM\\n\\nHacknet nodes\\n\\nAugmentations\\n\\nCompanies\\n\\nFactions\\n\\nCrimes\\n\\nInfiltration\\n\\nPrograms\\n\\nReputation\\n\\nStock market\\n\\nWorld\\n\\nCoding contracts\\n\\nAutocomplete\\n\\nAdvanced Mechanics\\n\\nHacking algorithms\\n\\nList of factions and their requirements\\n\\nOffline scripts and bonus time\\n\\nBitNodes\\n\\nSource-Files\\n\\nGang\\n\\nCorporation\\n\\nIntelligence\\n\\nBladeburner\\n\\nHacknet servers\\n\\nSleeves\\n\\nGrafting\\n\\nStanek's Gift\\n\\nIPvGO\\n\\nResources\\n\\nNS API documentation\\n\\nLearn to program\\n\\nRemote API\\n\\nGame frozen or stuck?\\n\\nReact\\n\\nTools & Resources\\n\\nChangelog\\n\\nChangelog - Legacy v1\\n\\nChangelog - Legacy v0\\n\\nMigration\\n\\nBitburner v1.0.0 script migration guide\\n\\nBitburner v2.0.0 script migration guide\\n\\nNetscript 2 migration guide (.script to .js)\"),\n", + " Document(id='ef442fa9-5c59-435e-8d08-0d5fa0b09ebb', metadata={'source': './bitburner/src/Documentation/doc/basic/infiltration.md'}, page_content='Infiltration\\n\\nInfiltration is a gameplay mechanic that allows you to infiltrate a Company\\'s facility to try and steal the Company\\'s classified secrets. These secrets can be sold for money or for Reputation with a Faction.\\n\\nOverview\\n\\nMany companies have facilities that you can attempt to infiltrate. By infiltrating, you can steal classified Company secrets and then sell these for money or for Faction Reputation. To try and infiltrate a Company, visit a Company through the World menu. There will be an option that says \\'Infiltrate Company\\'.\\n\\nWhen infiltrating a Company, you will be presented with short active challenges.\\n\\nNone of the challenges uses the mouse.\\n\\nMost challenges use spacebar as the action.\\n\\nSome challenges use WASD or arrows interchangeably.\\n\\nA few others use the rest of the keyboard.\\n\\nEach location that can be infiltrated has 3 important values:\\n\\nDifficulty: It affects how difficult the challenges are. This value depends on your \"common\" stats (combat stats and charisma). It\\'s reduced when you improve your stats. It is not recommended to attempt infiltrations when the difficulty is above normal.\\n\\nMax clearance level: It is the number of challenges you need to pass to receive the infiltration reward.\\n\\nStarting security level: It affects the difficulty and rewards.\\n\\nEvery time you successfully complete an infiltration challenge and go to the higher clearance level, the \"effective\" difficulty is increased. The difficulty value in the introduction screen is the initial value for the first clearance level. When you go to higher levels, this value will be increased, so the challenges will become harder.\\n\\nEvery time you fail an infiltration challenge, you will take damage based on the starting security level of the location. If your HP is reduced to 0, you will be hospitalized, and the infiltration will immediately end.\\n\\nInfiltration rewards depend on:\\n\\nMax clearance level. Higher max clearance level = Higher rewards.\\n\\nStarting security level. Higher starting security level = Higher rewards.\\n\\nYour stats\\' multipliers (NOT skill levels). Lower multipliers = Higher rewards.\\n\\n\"SoA - phyzical WKS harmonizer\" augmentation.\\n\\nAn endgame stat [1]. You will know what it is when you reach the endgame.\\n\\nAn endgame multiplier [2]. You will know what it is when you reach the endgame.\\n\\nThe most common misconception of infiltration rewards is that they depend on skill levels. This is wrong. The rewards do NOT depend on skill levels. They depend on stats\\' multipliers. When you install augmentations that improve stats\\' multipliers, the rewards are reduced.\\n\\nIn some special cases, you may observe this behavior: When you install augmentations or soft reset, your skill levels drop to 1 and infiltration rewards are increased. Some players mistakenly think that this is why infiltrations should be done with low stats to optimize the rewards. This is totally wrong. Again, infiltration rewards do NOT depend on skill levels. In these special cases, the rewards are increased because stats\\' multipliers are reduced due to losing bonuses from other mechanics (IPvGO and some endgame mechanics).\\n\\nRaising raw values of stats (skill levels) is one of the proper ways to optimize the infiltration. Having higher stats does not increase the rewards, but it allows you to infiltrate harder locations (higher max clearance level, higher starting security level, easier to complete more infiltrations in the same time frame, etc.).\\n\\nProper ways to optimize the infiltration (with the same [2] and the same stats\\' multipliers) are:\\n\\nRaise stats. Do NOT purposely keep them low. Having higher stats does not increase the rewards, but it allows you to infiltrate harder locations. Training at gyms, committing crimes, and working for companies/factions are good and easy ways to raise stats without changing stats\\' multipliers.\\n\\nInfiltrate harder locations after getting higher stats (higher max clearance level, higher starting security level, easier to complete more infiltrations in the same time frame, etc.).\\n\\nIncrease [1].\\n\\nBuy SoA augmentations (especially \"SoA - phyzical WKS harmonizer\").\\n\\nChallenge list\\n\\nAttack the distracted sentinel\\n\\nPress space bar to attack when the sentinel drops his guard and is distracted. Do not alert him!\\n\\nThere are 3 phases:\\n\\nGuarding - The sentinel is guarding. Attacking will result in a failure.\\n\\nDistracted - The sentinel is distracted. Attacking will result in a victory.\\n\\nAlerted - The sentinel is alerted. Attacking will result in a failure.\\n\\nClose the brackets\\n\\nEnter all the matching brackets in reverse order.\\n\\nType it backward\\n\\nType the words that are written backward.\\n\\nSay something nice about the guard.\\n\\nUse the arrows to find a compliment for the guard.\\n\\nEnter the Code!\\n\\nMatch the arrows as they appear.\\n\\nMatch the symbols!\\n\\nMove the cursor to the matching symbol and press space to confirm.\\n\\nRemember all the mines!\\n\\nAt first, the cursor cannot be moved - remember the positions of the mines. Next, move the cursor and press space to mark the mines on the board.\\n\\nCut the wires\\n\\nFollow the instructions and press the numbers 1 through 9 to cut the appropriate wires.'),\n", + " Document(id='6ef32e3d-d6b7-4c42-874a-96c21e4a63a9', metadata={'source': './bitburner/src/Documentation/doc/advanced/grafting.md'}, page_content=\"Grafting\\n\\nGrafting is an experimental process through which you can obtain the benefits of Augmentations, without needing to reboot your body.\\n\\nGrafting can be done at VitaLife in New Tokyo, where you'll find a shady researcher with questionable connections. From there, you can spend a sum of money to begin grafting Augmentations. This will take some time. When done, the Augmentation will be applied to your character without needing to install.\\n\\nBe warned, some who have tested grafting have reported an unidentified malware. Dubbed Entropy, this virus seems to grow in potency as more Augmentations are grafted, causing unpredictable affects to the victim.\\n\\nNote that when grafting an Augmentation, cancelling will not save your progress, and the money spent will not be returned.\")]" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "vector_store.search(\n", + " query='How to hack',\n", + " search_type='similarity',\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Build the RAG chain\n", + "\n", + "There are a number of frameworks available to build our RAG.\n", + "* [LangChain](https://langchain.com/)\n", + "* [LlamaIndex](https://docs.llamaindex.ai/en/latest/)\n", + "\n", + "In this example we will use LangChain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { @@ -204,7 +329,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.1" + "version": "3.13.3" } }, "nbformat": 4,