Fuck You, Show Me The Prompt.

Table Of Contents Background Motivation: Minimize accidental complexity Intercepting LLM API calls Setting Up mitmproxy Environment variables for Python Examples Guardrails Guidance Langchain Instructor DSPy My Personal Experience Background There are many libraries that aim to make the output of your LLMs better by re-writing or constructing the prompt for you. These libraries purport to make the output of your LLMs: safer (ex: guardrails) deterministic (ex: guidance) structured (ex: instructor) resilient (ex: langchain) … or even optimized for an arbitrary metric (ex: DSPy). A common theme among some of these tools is they encourage users to disintermediate themselves from prompting. DSPy: “This is a new paradigm in which LMs and their prompts fade into the background …. you can compile your program again DSPy will create new effective prompts” guidance “guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting …” Even when tools don’t discourage prompting, I’ve often found it difficult to retrieve the final prompt(s) these tools send to the language model. The prompts sent by these tools to the LLM is a natural language description of what these tools are doing, and is the fastest way to understand how they work. Furthermore, some tools have dense terminology to describe internal constructs which can further obfuscate what they are doing. For reasons I’ll explain below, I think most people would benefit from the following mindset: In this blog post, I’ll show you how you can intercept API calls w/prompts for any tool, without having to fumble through docs or read source code. I’ll show you how to setup and operate mitmproxy with examples from the LLM the tools I previously mentioned. Motivation: Minimize accidental complexity Before adopting an abstraction, its important to consider the dangers of taking on accidental complexity. This danger is acute for LLM abstractions relative to programming abstractions. With LLM abstractions, we often force the user to regress towards writing code instead of conversing with the AI in natural language, which can run counter to the purpose of LLMs: Programming abstraction -> a human-like language you can use to translate your task into machine codeLLM abstraction -> an unintelligible framework you can use to translate your task into human language — Hamel Husain (@HamelHusain) February 5, 2024 While this is a cheeky comment, it’s worth keeping this in mind while evaluating tools. There are two primary types of automation that tools provide: Interleaving code and LLMs: Expressing this automation is often best done through code, since code must be run to carry out the task. Examples include routing, executing functions, retries, chaining, etc. Re-Writing and constructing prompts: Expressing your intent is often best done through natural language. However, there are exceptions! For example, it is convenient to express a function definition or schema from code instead of natural language. Many frameworks offer both types of automation. However, going too far with the second type can have negative consequences. Seeing the prompt allows you decide: Is this framework really necessary? Should I just steal the final prompt (a string) and jettison the framework? Can we write a better prompt than this (shorter, aligned with your intent, etc)? Is this the best approach (do the # of API calls seem appropriate)? In my experience, seeing the prompts and API calls are essential to making informed decisions. Intercepting LLM API calls There are many possible ways to intercept LLM API calls, such as monkey patching source code or finding a user-facing option. I’ve found that those approaches take far too much time since the quality of source code and documentation can vary greatly. After all, I just want to see API calls without worrying about how the code works! A framework agnostic way to see API calls is to setup a proxy that logs your outgoing API requests. This is easy to do with mitmproxy, an free, open-source HTTPS proxy. Setting Up mitmproxy This is an opinionated way to setup mitmproxythat’s beginner-friendly for our intended purposes: Follow the installation instructions on the website Start the interactive UI by running mitmweb in the terminal. Pay attention to the url of the interactive UI in the logs which will look something like this: Web server listening at http://127.0.0.1:8081/ Next, you need to configure your device (i.e. your laptop) to route all traffic through mitproxy, which listens on http://localhost:8080. Per the documentation: We recommend to simply search the web on how to configure an HTTP proxy for your system. Some operating system have a global settings, some browser have their own, other applications use environment variables, etc. In my case, A google search for “set proxy for macos” returned these results: choose Apple menu > System Settings, click Network in the sidebar, click a network service on the right, click Details, then click Proxies. I then insert localhost and 8080 in the following places in the UI: Next, navigate to http://mitm.it and it will give you instructions on how to install the mitmproxy Certificate Authority (CA), which you will need for intercepting HTTPS requests. (You can also do this manually here.) Also, take note of the location of the CA file as we will reference it later. You can test that everything works by browsing to a website like https://mitmproxy.org/, and seeing the corresponding output in the mtimweb UI which for me is located at http://127.0.0.1:8081/ (look at the logs in your terminal to get the URL). Now that you set everything up, you can disable the proxy that you previously enabled on your network. I do this on my mac by toggling the proxy buttons in the screenshot I showed above. This is because we want to scope the proxy to only the python program to eliminate unnecessary noise. Tip Networking related software commonly allows you to proxy outgoing requests by setting environment variables. This is the approach we will use to scope our proxy to specific Python programs. However, I encourage you to play with other types of programs to see what you find after you are comfortable! Environment variables for Python We need to set the following environment variables so that the requests and httpx libraries will direct traffic to the proxy and reference the CA file for HTTPS traffic: Important Make sure you set these environment variables before running any of the code snippets in this blog post. import os # The location of my CA File cert_file = '/Users/hamel/Downloads/mitmproxy-ca-cert.pem' os.environ['REQUESTS_CA_BUNDLE'] = cert_file os.environ['SSL_CERT_FILE'] = cert_file os.environ['HTTPS_PROXY'] = 'http://127.0.0.1:8080' You can do a minimal test by running the following code: import requests requests.post('https://httpbin.org/post', data={'key': 'value'}) <Response [200]> This will appear in the UI like so: Examples Now for the fun part, let’s run through some examples of LLM libraries and intercept their API calls! Guardrails Guardrails allows you specify structure and types, which it uses to validate and correct the outputs of large language models. This is a hello world example from the guardrails-ai/guardrails README: from pydantic import BaseModel, Field from guardrails import Guard import openai class Pet(BaseModel): pet_type: str = Field(description="Species of pet") name: str = Field(description="a unique pet name") prompt = """ What kind of pet should I get and what should I name it? ${gr.complete_json_suffix_v2} """ guard = Guard.from_pydantic(output_class=Pet, prompt=prompt) validated_output, *rest = guard( llm_api=openai.completions.create, engine="gpt-3.5-turbo-instruct" ) print(f"{validated_output}") { "pet_type": "dog", "name": "Buddy What is happening here? How is this structured output and validation working? Looking at the mitmproxy UI, I can see that the above code resulted in two LLM API calls, the first one with this prompt: What kind of pet should I get and what should I name it? Given below is XML that describes the information to extract from this document and the tags to extract it into. <output> <string name="pet_type" description="Species of pet"/> <string name="name" description="a unique pet name"/> </output> ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. Here are examples of simple (XML, JSON) pairs that show the expected behavior: - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}` - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` Followed by another call with this prompt: I was given the following response, which was not parseable as JSON. "{\n \"pet_type\": \"dog\",\n \"name\": \"Buddy" Help me correct this by making it valid JSON. Given below is XML that describes the information to extract from this document and the tags to extract it into. <output> <string name="pet_type" description="Species of pet"/> <string name="name" description="a unique pet name"/> </output>