Interface | 🦜🔗 Langchain

num': ['function'],          'type': 'string'},         'name': {'title': 'Name', 'type': 'string'}},        'required': ['content', 'name']},       'ChatPromptValueConcrete': {'title': 'ChatPromptValueConcrete',        'description': 'Chat prompt value which explicitly lists out the message types it accepts.\nFor use in external schemas.',        'type': 'object',        'properties': {'messages': {'title': 'Messages',          'type': 'array',          'items': {'anyOf': [{'$ref': '#/definitions/AIMessage'},            {'$ref': '#/definitions/HumanMessage'},            {'$ref': '#/definitions/ChatMessage'},            {'$ref': '#/definitions/SystemMessage'},            {'$ref': '#/definitions/FunctionMessage'}]}},         'type': {'title': 'Type',          'default': 'ChatPromptValueConcrete',          'enum': ['ChatPromptValueConcrete'],          'type': 'string'}},        'required': ['messages']}}}  ##Output Schema[](#output-schema) A description of the outputs produced by a Runnable. This is a Pydantic model dynamically generated from the structure of any Runnable. You can call .schema() on it to obtain a JSONSchema representation.  # The output schema of the chain is the output schema of its last part, in this case a ChatModel, which outputs a ChatMessage chain.output_schema.schema()      {'title': 'ChatOpenAIOutput',      'anyOf': [{'$ref': '#/definitions/HumanMessage'},       {'$ref': '#/definitions/AIMessage'},       {'$ref': '#/definitions/ChatMessage'},       {'$ref': '#/definitions/FunctionMessage'},       {'$ref': '#/definitions/SystemMessage'}],      'definitions': {'HumanMessage': {'title': 'HumanMessage',        'description': 'A Message from a human.',        'type': 'object',        'properties': {'content': {'title': 'Content', 'type': 'string'},         'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},         'type': {'title': 'Type',          'default': 'human',          'enum': ['human'],          'type': 'string'},         'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},        'required': ['content']},       'AIMessage': {'title': 'AIMessage',        'description': 'A Message from an AI.',        'type': 'object',        'properties': {'content': {'title': 'Content', 'type': 'string'},         'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},         'type': {'title': 'Type',          'default': 'ai',          'enum': ['ai'],          'type': 'string'},         'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},        'required': ['content']},       'ChatMessage': {'title': 'ChatMessage',        'description': 'A Message that can be assigned an arbitrary speaker (i.e. role).',        'type': 'object',        'properties': {'content': {'title': 'Content', 'type': 'string'},         'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},         'type': {'title': 'Type',          'default': 'chat',          'enum': ['chat'],          'type': 'string'},         'role': {'title': 'Role', 'type': 'string'}},        'required': ['content', 'role']},       'FunctionMessage': {'title': 'FunctionMessage',        'description': 'A Message for passing the result of executing a function back to a model.',        'type': 'object',        'properties': {'content': {'title': 'Content', 'type': 'string'},         'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},         'type': {'title': 'Type',          'default': 'function',          'enum': ['function'],          'type': 'string'},         'name': {'title': 'Name', 'type': 'string'}},        'required': ['content', 'name']},       'SystemMessage': {'title': 'SystemMessage',        'description': 'A Message for priming AI behavior, usually passed in as the first of a sequence\nof input messages.',        'type': 'object',        'properties': {'content': {'title': 'Content', 'type': 'string'},         'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},         'type': {'title': 'Type',          'default': 'system',          'enum': ['system'],          'type': 'string'}},        'required': ['content']}}}  ##Stream[](#stream) for s in chain.stream({"topic": "bears"}):     print(s.content, end="", flush=True)      Why don't bears wear shoes?          Because they already have bear feet!  ##Invoke[](#invoke) chain.invoke({"topic": "bears"})      AIMessage(content="Why don't bears wear shoes?\n\nBecause they already have bear feet!")  ##Batch[](#batch) chain.batch([{"topic": "bears"}, {"topic": "cats"}])      [AIMessage(content="Why don't bears wear shoes?\n\nBecause they have bear feet!"),      AIMessage(content="Why don't cats play poker in the wild?\n\nToo many cheetahs!")]  You can set the number of concurrent requests by using the max_concurrency parameter  chain.batch([{"topic": "bears"}, {"topic": "cats"}], config={"max_concurrency": 5})      [AIMessage(content="Why don't bears wear shoes? \n\nBecause they have bear feet!"),      AIMessage(content="Why don't cats play poker in the wild?\n\nToo many cheetahs!")]  ##Async Stream[](#async-stream) async for s in chain.astream({"topic": "bears"}):     print(s.content, end="", flush=True)      Sure, here's a bear-themed joke for you:          Why don't bears wear shoes?          Because they already have bear feet!  ##Async Invoke[](#async-invoke) await chain.ainvoke({"topic": "bears"})      AIMessage(content="Why don't bears wear shoes? \n\nBecause they have bear feet!")  ##Async Batch[](#async-batch) await chain.abatch([{"topic": "bears"}])      [AIMessage(content="Why don't bears wear shoes?\n\nBecause they have bear feet!")]  ##Async Stream Intermediate Steps[](#async-stream-intermediate-steps) All runnables also have a method .astream_log() which is used to stream (as they happen) all or part of the intermediate steps of your chain/sequence.  This is useful to show progress to the user, to use intermediate results, or to debug y


Interface | 🦜🔗 Langchain

our chain.  You can stream all steps (default) or include/exclude steps by name, tags or metadata.  This method yields [JSONPatch](https://jsonpatch.com) ops that when applied in the same order as received build up the RunState.  class LogEntry(TypedDict):     id: str     """ID of the sub-run."""     name: str     """Name of the object being run."""     type: str     """Type of the object being run, eg. prompt, chain, llm, etc."""     tags: List[str]     """List of tags for the run."""     metadata: Dict[str, Any]     """Key-value pairs of metadata for the run."""     start_time: str     """ISO-8601 timestamp of when the run started."""      streamed_output_str: List[str]     """List of LLM tokens streamed by this run, if applicable."""     final_output: Optional[Any]     """Final output of this run.     Only available after the run has finished successfully."""     end_time: Optional[str]     """ISO-8601 timestamp of when the run ended.     Only available after the run has finished."""   class RunState(TypedDict):     id: str     """ID of the run."""     streamed_output: List[Any]     """List of output chunks streamed by Runnable.stream()"""     final_output: Optional[Any]     """Final output of the run, usually the result of aggregating (`+`) streamed_output.     Only available after the run has finished successfully."""      logs: Dict[str, LogEntry]     """Map of run names to sub-runs. If filters were supplied, this list will     contain only the runs that matched the filters."""  ###Streaming JSONPatch chunks[](#streaming-jsonpatch-chunks) This is useful eg. to stream the JSONPatch in an HTTP server, and then apply the ops on the client to rebuild the run state there. See [LangServe](https://github.com/langchain-ai/langserve) for tooling to make it easier to build a webserver from any Runnable.  from langchain.embeddings import OpenAIEmbeddings from langchain.schema.output_parser import StrOutputParser from langchain.schema.runnable import RunnablePassthrough from langchain.vectorstores import FAISS  template = """Answer the question based only on the following context: {context}  Question: {question} """ prompt = ChatPromptTemplate.from_template(template)  vectorstore = FAISS.from_texts(     ["harrison worked at kensho"], embedding=OpenAIEmbeddings() ) retriever = vectorstore.as_retriever()  retrieval_chain = (     {         "context": retriever.with_config(run_name="Docs"),         "question": RunnablePassthrough(),     }     | prompt     | model     | StrOutputParser() )  async for chunk in retrieval_chain.astream_log(     "where did harrison work?", include_names=["Docs"] ):     print("-" * 40)     print(chunk)      ----------------------------------------     RunLogPatch({'op': 'replace',       'path': '',       'value': {'final_output': None,                 'id': 'e2f2cc72-eb63-4d20-8326-237367482efb',                 'logs': {},                 'streamed_output': []}})     ----------------------------------------     RunLogPatch({'op': 'add',       'path': '/logs/Docs',       'value': {'end_time': None,                 'final_output': None,                 'id': '8da492cc-4492-4e74-b8b0-9e60e8693390',                 'metadata': {},                 'name': 'Docs',                 'start_time': '2023-10-19T17:50:13.526',                 'streamed_output_str': [],                 'tags': ['map:key:context', 'FAISS'],                 'type': 'retriever'}})     ----------------------------------------     RunLogPatch({'op': 'add',       'path': '/logs/Docs/final_output',       'value': {'documents': [Document(page_content='harrison worked at kensho')]}},      {'op': 'add',       'path': '/logs/Docs/end_time',       'value': '2023-10-19T17:50:13.713'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'H'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'arrison'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' worked'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' at'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' Kens'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'ho'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': '.'})     ----------------------------------------     RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})     ----------------------------------------     RunLogPatch({'op': 'replace',       'path': '/final_output',       'value': {'output': 'Harrison worked at Kensho.'}})  ###Streaming the incremental RunState[](#streaming-the-incremental-runstate) You can simply pass diff=False to get incremental values of RunState. You get more verbose output with more repetitive parts.  async for chunk in retrieval_chain.astream_log(     "where did harrison work?", include_names=["Docs"], diff=False ):     print("-" * 70)     print(chunk)      ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {},      'streamed_output': []})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': None,                        'final_output': None,                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-1


Interface | 🦜🔗 Langchain

0-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': []})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': []})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison', ' worked']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison', ' worked', ' at']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'str


Interface | 🦜🔗 Langchain

eamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho', '.']})     ----------------------------------------------------------------------     RunLog({'final_output': None,      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['',                          'H',                          'arrison',                          ' worked',                          ' at',                          ' Kens',                          'ho',                          '.',                          '']})     ----------------------------------------------------------------------     RunLog({'final_output': {'output': 'Harrison worked at Kensho.'},      'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',      'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',                        'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},                        'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',                        'metadata': {},                        'name': 'Docs',                        'start_time': '2023-10-19T17:52:15.438',                        'streamed_output_str': [],                        'tags': ['map:key:context', 'FAISS'],                        'type': 'retriever'}},      'streamed_output': ['',                          'H',                          'arrison',                          ' worked',                          ' at',                          ' Kens',                          'ho',                          '.',                          '']})  ##Parallelism[](#parallelism) Let's take a look at how LangChain Expression Language supports parallel requests. For example, when using a RunnableParallel (often written as a dictionary) it executes each element in parallel.  from langchain.schema.runnable import RunnableParallel  chain1 = ChatPromptTemplate.from_template("tell me a joke about {topic}") | model chain2 = (     ChatPromptTemplate.from_template("write a short (2 line) poem about {topic}")     | model ) combined = RunnableParallel(joke=chain1, poem=chain2)  chain1.invoke({"topic": "bears"})      CPU times: user 54.3 ms, sys: 0 ns, total: 54.3 ms     Wall time: 2.29 s          AIMessage(content="Why don't bears wear shoes?\n\nBecause they already have bear feet!")  chain2.invoke({"topic": "bears"})      CPU times: user 7.8 ms, sys: 0 ns, total: 7.8 ms     Wall time: 1.43 s          AIMessage(content="In wild embrace,\nNature's strength roams with grace.")  combined.invoke({"topic": "bears"})      CPU times: user 167 ms, sys: 921 µs, total: 168 ms     Wall time: 1.56 s          {'joke': AIMessage(content="Why don't bears wear shoes?\n\nBecause they already have bear feet!"),      'poem': AIMessage(content="Fierce and wild, nature's might,\nBears roam the woods, shadows of the night.")}  ###Parallelism on batches[](#parallelism-on-batches) Parallelism can be combined with other runnables. Let's try to use parallelism with batches.  chain1.batch([{"topic": "bears"}, {"topic": "cats"}])      CPU times: user 159 ms, sys: 3.66 ms, total: 163 ms     Wall time: 1.34 s          [AIMessage(content="Why don't bears wear shoes?\n\nBecause they already have bear feet!"),      AIMessage(content="Sure, here's a cat joke for you:\n\nWhy don't cats play poker in the wild?\n\nBecause there are too many cheetahs!")]  chain2.batch([{"topic": "bears"}, {"topic": "cats"}])      CPU times: user 165 ms, sys: 0 ns, total: 165 ms     Wall time: 1.73 s          [AIMessage(content="Silent giants roam,\nNature's strength, love's emblem shown."),      AIMessage(content='Whiskers aglow, paws tiptoe,\nGraceful hunters, hearts aglow.')]  combined.batch([{"topic": "bears"}, {"topic": "cats"}])      CPU times: user 507 ms, sys: 125 ms, total: 632 ms     Wall time: 1.49 s          [{'joke': AIMessage(content="Why don't bears wear shoes?\n\nBecause they already have bear feet!"),       'poem': AIMessage(content="Majestic bears roam,\nNature's wild guardians of home.")},      {'joke': AIMessage(content="Sure, here's a cat joke for you:\n\nWhy did the cat sit on the computer?\n\nBecause it wanted to keep an eye on the mouse!"),       'poem': AIMessage(content='Whiskers twitch, eyes gleam,\nGraceful creatures, feline dream.')}]


Why use LCEL | 🦜🔗 Langchain

Main:  On this page WE RECOMMEND READING THE LCEL [GET STARTED](/DOCS/EXPRESSION_LANGUAGE/GET_STARTED) SECTION FIRST. LCEL makes it easy to build complex chains from basic components. It does this by providing:  - A unified interface: Every LCEL object implements the Runnable interface, which defines a common set of invocation methods (invoke, batch, stream, ainvoke, ...). This makes it possible for chains of LCEL objects to also automatically support these invocations. That is, every chain of LCEL objects is itself an LCEL object. - Composition primitives: LCEL provides a number of primitives that make it easy to compose chains, parallelize components, add fallbacks, dynamically configure chain internal, and more. To better understand the value of LCEL, it's helpful to see it in action and think about how we might recreate similar functionality without it. In this walkthrough we'll do just that with our [basic example](/docs/expression_language/get_started#basic_example) from the get started section. We'll take our simple prompt + model chain, which under the hood already defines a lot of functionality, and see what it would take to recreate all of it.  from langchain.chat_models import ChatOpenAI from langchain.prompts import ChatPromptTemplate from langchain.schema.output_parser import StrOutputParser   prompt = ChatPromptTemplate.from_template("Tell me a short joke about {topic}") model = ChatOpenAI(model="gpt-3.5-turbo") output_parser = StrOutputParser()  chain = prompt | model | output_parser  ##Invoke[](#invoke) In the simplest case, we just want to pass in a topic string and get back a joke string:  ####Without LCEL[](#without-lcel) from typing import List  import openai   prompt_template = "Tell me a short joke about {topic}" client = openai.OpenAI()  def call_chat_model(messages: List[dict]) -> str:     response = client.chat.completions.create(         model="gpt-3.5-turbo",          messages=messages,     )     return response.choices[0].message.content  def invoke_chain(topic: str) -> str:     prompt_value = prompt_template.format(topic=topic)     messages = [{"role": "user", "content": prompt_value}]     return call_chat_model(messages)  invoke_chain("ice cream")  ####LCEL[](#lcel) from langchain_core.runnables import RunnablePassthrough   prompt = ChatPromptTemplate.from_template(     "Tell me a short joke about {topic}" ) output_parser = StrOutputParser() model = ChatOpenAI(model="gpt-3.5-turbo") chain = (     {"topic": RunnablePassthrough()}      | prompt     | model     | output_parser )  chain.invoke("ice cream")  ##Stream[](#stream) If we want to stream results instead, we'll need to change our function:  ####Without LCEL[](#without-lcel-1) from typing import Iterator   def stream_chat_model(messages: List[dict]) -> Iterator[str]:     stream = client.chat.completions.create(         model="gpt-3.5-turbo",         messages=messages,         stream=True,     )     for response in stream:         content = response.choices[0].delta.content         if content is not None:             yield content  def stream_chain(topic: str) -> Iterator[str]:     prompt_value = prompt.format(topic=topic)     return stream_chat_model([{"role": "user", "content": prompt_value}])   for chunk in stream_chain("ice cream"):     print(chunk, end="", flush=True)  ####LCEL[](#lcel-1) for chunk in chain.stream("ice cream"):     print(chunk, end="", flush=True)  ##Batch[](#batch) If we want to run on a batch of inputs in parallel, we'll again need a new function:  ####Without LCEL[](#without-lcel-2) from concurrent.futures import ThreadPoolExecutor   def batch_chain(topics: list) -> list:     with ThreadPoolExecutor(max_workers=5) as executor:         return list(executor.map(invoke_chain, topics))  batch_chain(["ice cream", "spaghetti", "dumplings"])  ####LCEL[](#lcel-2) chain.batch(["ice cream", "spaghetti", "dumplings"])  ##Async[](#async) If we need an asynchronous version:  ####Without LCEL[](#without-lcel-3) async_client = openai.AsyncOpenAI()  async def acall_chat_model(messages: List[dict]) -> str:     response = await async_client.chat.completions.create(         model="gpt-3.5-turbo",          messages=messages,     )     return response.choices[0].message.content  async def ainvoke_chain(topic: str) -> str:     prompt_value = prompt_template.format(topic=topic)     messages = [{"role": "user", "content": prompt_value}]     return await acall_chat_model(messages)  await ainvoke_chain("ice cream")  ####LCEL[](#lcel-3) chain.ainvoke("ice cream")  ##LLM instead of chat model[](#llm-instead-of-chat-model) If we want to use a completion endpoint instead of a chat endpoint:  ####Without LCEL[](#without-lcel-4) def call_llm(prompt_value: str) -> str:     response = client.completions.create(         model="gpt-3.5-turbo-instruct",         prompt=prompt_value,     )     return response.choices[0].text  def invoke_llm_chain(topic: str) -> str:     prompt_value = prompt_template.format(topic=topic)     return call_llm(prompt_value)  invoke_llm_chain("ice cream")  ####LCEL[](#lcel-4) from langchain.llms import OpenAI  llm = OpenAI(model="gpt-3.5-turbo-instruct") llm_chain = (     {"topic": RunnablePassthrough()}      | prompt     | llm     | output_parser )  llm_chain.invoke("ice cream")  ##Different model provider[](#different-model-provider) If we want to use Anthropic instead of OpenAI:  ####Without LCEL[](#without-lcel-5) import anthropic  anthropic_template = f"Human:\n\n{prompt_template}\n\nAssistant:" anthropic_client = anthropic.Anthropic()  def call_anthropic(prompt_value: str) -> str:     response = anthropic_client.completions.create(         model="claude-2",         prompt=prompt_value,         max_tokens_to_sample=256,     )     return response.completion      def invoke_anthropic_chain(topic: str) -> str:     prompt_value = anthropic_template.format(topic=topic)     return call_anthropic(prompt_value)  invoke_anthropic_chain("ice cream")  ####LCEL[](#lcel-5) from la


Why use LCEL | 🦜🔗 Langchain

ngchain.chat_models import ChatAnthropic  anthropic = ChatAnthropic(model="claude-2") anthropic_chain = (     {"topic": RunnablePassthrough()}      | prompt      | anthropic     | output_parser )  anthropic_chain.invoke("ice cream")  ##Runtime configurability[](#runtime-configurability) If we wanted to make the choice of chat model or LLM configurable at runtime:  ####Without LCEL[](#without-lcel-6) def invoke_configurable_chain(     topic: str,      *,      model: str = "chat_openai" ) -> str:     if model == "chat_openai":         return invoke_chain(topic)     elif model == "openai":         return invoke_llm_chain(topic)     elif model == "anthropic":         return invoke_anthropic_chain(topic)     else:         raise ValueError(             f"Received invalid model '{model}'."             " Expected one of chat_openai, openai, anthropic"         )  def stream_configurable_chain(     topic: str,      *,      model: str = "chat_openai" ) -> Iterator[str]:     if model == "chat_openai":         return stream_chain(topic)     elif model == "openai":         # Note we haven't implemented this yet.         return stream_llm_chain(topic)     elif model == "anthropic":         # Note we haven't implemented this yet         return stream_anthropic_chain(topic)     else:         raise ValueError(             f"Received invalid model '{model}'."             " Expected one of chat_openai, openai, anthropic"         )  def batch_configurable_chain(     topics: List[str],      *,      model: str = "chat_openai" ) -> List[str]:     # You get the idea     ...  async def abatch_configurable_chain(     topics: List[str],      *,      model: str = "chat_openai" ) -> List[str]:     ...  invoke_configurable_chain("ice cream", model="openai") stream = stream_configurable_chain(     "ice_cream",      model="anthropic" ) for chunk in stream:     print(chunk, end="", flush=True)  # batch_configurable_chain(["ice cream", "spaghetti", "dumplings"]) # await ainvoke_configurable_chain("ice cream")  ####With LCEL[](#with-lcel) from langchain_core.runnables import ConfigurableField   configurable_model = model.configurable_alternatives(     ConfigurableField(id="model"),      default_key="chat_openai",      openai=llm,     anthropic=anthropic, ) configurable_chain = (     {"topic": RunnablePassthrough()}      | prompt      | configurable_model      | output_parser )  configurable_chain.invoke(     "ice cream",      config={"model": "openai"} ) stream = configurable_chain.stream(     "ice cream",      config={"model": "anthropic"} ) for chunk in stream:     print(chunk, end="", flush=True)  configurable_chain.batch(["ice cream", "spaghetti", "dumplings"])  # await configurable_chain.ainvoke("ice cream")  ##Logging[](#logging) If we want to log our intermediate results:  ####Without LCEL[](#without-lcel-7) We'll print intermediate steps for illustrative purposes  def invoke_anthropic_chain_with_logging(topic: str) -> str:     print(f"Input: {topic}")     prompt_value = anthropic_template.format(topic=topic)     print(f"Formatted prompt: {prompt_value}")     output = call_anthropic(prompt_value)     print(f"Output: {output}")     return output  invoke_anthropic_chain_with_logging("ice cream")  ####LCEL[](#lcel-6) Every component has built-in integrations with LangSmith. If we set the following two environment variables, all chain traces are logged to LangSmith.  import os  os.environ["LANGCHAIN_API_KEY"] = "..." os.environ["LANGCHAIN_TRACING_V2"] = "true"  anthropic_chain.invoke("ice cream")  Here's what our LangSmith trace looks like: [https://smith.langchain.com/public/e4de52f8-bcd9-4732-b950-deee4b04e313/r](https://smith.langchain.com/public/e4de52f8-bcd9-4732-b950-deee4b04e313/r)  ##Fallbacks[](#fallbacks) If we wanted to add fallback logic, in case one model API is down:  ####Without LCEL[](#without-lcel-8) def invoke_chain_with_fallback(topic: str) -> str:     try:         return invoke_chain(topic)     except Exception:         return invoke_anthropic_chain(topic)  async def ainvoke_chain_with_fallback(topic: str) -> str:     try:         return await ainvoke_chain(topic)     except Exception:         # Note: we haven't actually implemented this.         return ainvoke_anthropic_chain(topic)  async def batch_chain_with_fallback(topics: List[str]) -> str:     try:         return batch_chain(topics)     except Exception:         # Note: we haven't actually implemented this.         return batch_anthropic_chain(topics)  invoke_chain_with_fallback("ice cream") # await ainvoke_chain_with_fallback("ice cream") batch_chain_with_fallback(["ice cream", "spaghetti", "dumplings"]))  ####LCEL[](#lcel-7) fallback_chain = chain.with_fallbacks([anthropic_chain])  fallback_chain.invoke("ice cream") # await fallback_chain.ainvoke("ice cream") fallback_chain.batch(["ice cream", "spaghetti", "dumplings"])  ##Full code comparison[](#full-code-comparison) Even in this simple case, our LCEL chain succinctly packs in a lot of functionality. As chains become more complex, this becomes especially valuable.  ####Without LCEL[](#without-lcel-9) from concurrent.futures import ThreadPoolExecutor from typing import Iterator, List, Tuple  import anthropic import openai   prompt_template = "Tell me a short joke about {topic}" anthropic_template = f"Human:\n\n{prompt_template}\n\nAssistant:" client = openai.OpenAI() async_client = openai.AsyncOpenAI() anthropic_client = anthropic.Anthropic()  def call_chat_model(messages: List[dict]) -> str:     response = client.chat.completions.create(         model="gpt-3.5-turbo",          messages=messages,     )     return response.choices[0].message.content  def invoke_chain(topic: str) -> str:     print(f"Input: {topic}")     prompt_value = prompt_template.format(topic=topic)     print(f"Formatted prompt: {prompt_value}")     messages = [{"role": "user", "content": prompt_value}]     output = call_chat_model(messages)     print(f"Output: {output}")     return output  def stream_chat_model


Why use LCEL | 🦜🔗 Langchain

(messages: List[dict]) -> Iterator[str]:     stream = client.chat.completions.create(         model="gpt-3.5-turbo",         messages=messages,         stream=True,     )     for response in stream:         content = response.choices[0].delta.content         if content is not None:             yield content  def stream_chain(topic: str) -> Iterator[str]:     print(f"Input: {topic}")     prompt_value = prompt.format(topic=topic)     print(f"Formatted prompt: {prompt_value}")     stream = stream_chat_model([{"role": "user", "content": prompt_value}])     for chunk in stream:         print(f"Token: {chunk}", end="")         yield chunk  def batch_chain(topics: list) -> list:     with ThreadPoolExecutor(max_workers=5) as executor:         return list(executor.map(invoke_chain, topics))  def call_llm(prompt_value: str) -> str:     response = client.completions.create(         model="gpt-3.5-turbo-instruct",         prompt=prompt_value,     )     return response.choices[0].text  def invoke_llm_chain(topic: str) -> str:     print(f"Input: {topic}")     prompt_value = promtp_template.format(topic=topic)     print(f"Formatted prompt: {prompt_value}")     output = call_llm(prompt_value)     print(f"Output: {output}")     return output  def call_anthropic(prompt_value: str) -> str:     response = anthropic_client.completions.create(         model="claude-2",         prompt=prompt_value,         max_tokens_to_sample=256,     )     return response.completion     def invoke_anthropic_chain(topic: str) -> str:     print(f"Input: {topic}")     prompt_value = anthropic_template.format(topic=topic)     print(f"Formatted prompt: {prompt_value}")     output = call_anthropic(prompt_value)     print(f"Output: {output}")     return output  async def ainvoke_anthropic_chain(topic: str) -> str:     ...  def stream_anthropic_chain(topic: str) -> Iterator[str]:     ...  def batch_anthropic_chain(topics: List[str]) -> List[str]:     ...  def invoke_configurable_chain(     topic: str,      *,      model: str = "chat_openai" ) -> str:     if model == "chat_openai":         return invoke_chain(topic)     elif model == "openai":         return invoke_llm_chain(topic)     elif model == "anthropic":         return invoke_anthropic_chain(topic)     else:         raise ValueError(             f"Received invalid model '{model}'."             " Expected one of chat_openai, openai, anthropic"         )  def stream_configurable_chain(     topic: str,      *,      model: str = "chat_openai" ) -> Iterator[str]:     if model == "chat_openai":         return stream_chain(topic)     elif model == "openai":         # Note we haven't implemented this yet.         return stream_llm_chain(topic)     elif model == "anthropic":         # Note we haven't implemented this yet         return stream_anthropic_chain(topic)     else:         raise ValueError(             f"Received invalid model '{model}'."             " Expected one of chat_openai, openai, anthropic"         )  def batch_configurable_chain(     topics: List[str],      *,      model: str = "chat_openai" ) -> List[str]:     ...  async def abatch_configurable_chain(     topics: List[str],      *,      model: str = "chat_openai" ) -> List[str]:     ...  def invoke_chain_with_fallback(topic: str) -> str:     try:         return invoke_chain(topic)     except Exception:         return invoke_anthropic_chain(topic)  async def ainvoke_chain_with_fallback(topic: str) -> str:     try:         return await ainvoke_chain(topic)     except Exception:         return ainvoke_anthropic_chain(topic)  async def batch_chain_with_fallback(topics: List[str]) -> str:     try:         return batch_chain(topics)     except Exception:         return batch_anthropic_chain(topics)  ####LCEL[](#lcel-8) import os  from langchain.chat_models import ChatAnthropic, ChatOpenAI from langchain.llms import OpenAI from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough  os.environ["LANGCHAIN_API_KEY"] = "..." os.environ["LANGCHAIN_TRACING_V2"] = "true"  prompt = ChatPromptTemplate.from_template(     "Tell me a short joke about {topic}" ) chat_openai = ChatOpenAI(model="gpt-3.5-turbo") openai = OpenAI(model="gpt-3.5-turbo-instruct") anthropic = ChatAnthropic(model="claude-2") model = (     chat_openai     .with_fallbacks([anthropic])     .configurable_alternatives(         ConfigurableField(id="model"),         default_key="chat_openai",         openai=openai,         anthropic=anthropic,     ) )  chain = (     {"topic": RunnablePassthrough()}      | prompt      | model      | StrOutputParser() )  ##Next steps[](#next-steps) To continue learning about LCEL, we recommend:  - Reading up on the full LCEL [Interface](/docs/expression_language/interface), which we've only partially covered here. - Exploring the [How-to](/docs/expression_language/how_to) section to learn about additional composition primitives that LCEL provides. - Looking through the [Cookbook](/docs/expression_language/cookbook) section to see LCEL in action for common use cases. A good next use case to look at would be [Retrieval-augmented generation](/docs/expression_language/cookbook/retrieval).


Get started | 🦜🔗 Langchain

Main:  Skip to main content  🦜🔗 LangChain  Search CTRLK Get started Get started Get started with LangChain  📄 Introduction LangChain is a framework for developing applications powered by language models. It enables applications that:  📄 Installation Official release  📄 Quickstart In this quickstart we'll show you how to:  📄 Security LangChain has a large ecosystem of integrations with various external resources like local and remote file systems, APIs and databases. These integrations allow developers to create versatile applications that combine the power of LLMs with the ability to access, interact with and manipulate external resources.  Next Introduction Community Discord Twitter GitHub Python JS/TS More Homepage Blog Copyright © 2023 LangChain, Inc.


Installation | 🦜🔗 Langchain

Main:  On this page #Installation ##Official release[](#official-release) To install LangChain run:  - Pip - Conda pip install langchain  This will install the bare minimum requirements of LangChain. A lot of the value of LangChain comes when integrating it with various model providers, datastores, etc. By default, the dependencies needed to do that are NOT installed. You will need to install the dependencies for specific integrations separately.  ##From source[](#from-source) If you want to install from source, you can do so by cloning the repo and be sure that the directory is PATH/TO/REPO/langchain/libs/langchain running:  pip install -e .  ##LangChain experimental[](#langchain-experimental) The langchain-experimental package holds experimental LangChain code, intended for research and experimental uses. Install with:  pip install langchain-experimental  ##LangServe[](#langserve) LangServe helps developers deploy LangChain runnables and chains as a REST API. LangServe is automatically installed by LangChain CLI. If not using LangChain CLI, install with:  pip install "langserve[all]"  for both client and server dependencies. Or pip install "langserve[client]" for client code, and pip install "langserve[server]" for server code.  ##LangChain CLI[](#langchain-cli) The LangChain CLI is useful for working with LangChain templates and other LangServe projects. Install with:  pip install langchain-cli  ##LangSmith SDK[](#langsmith-sdk) The LangSmith SDK is automatically installed by LangChain. If not using LangChain, install with:  pip install langsmith


Introduction | 🦜🔗 Langchain

Main:  On this page #Introduction LangChain is a framework for developing applications powered by language models. It enables applications that:  - Are context-aware: connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.) - Reason: rely on a language model to reason (about how to answer based on provided context, what actions to take, etc.) This framework consists of several parts.  - LangChain Libraries: The Python and JavaScript libraries. Contains interfaces and integrations for a myriad of components, a basic run time for combining these components into chains and agents, and off-the-shelf implementations of chains and agents. - [LangChain Templates](/docs/templates): A collection of easily deployable reference architectures for a wide variety of tasks. - [LangServe](/docs/langserve): A library for deploying LangChain chains as a REST API. - [LangSmith](/docs/langsmith): A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.   Together, these products simplify the entire application lifecycle:  - Develop: Write your applications in LangChain/LangChain.js. Hit the ground running using Templates for reference. - Productionize: Use LangSmith to inspect, test and monitor your chains, so that you can constantly improve and deploy with confidence. - Deploy: Turn any chain into an API with LangServe. ##LangChain Libraries[](#langchain-libraries) The main value props of the LangChain packages are:  - Components: composable tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not - Off-the-shelf chains: built-in assemblages of components for accomplishing higher-level tasks Off-the-shelf chains make it easy to get started. Components make it easy to customize existing chains and build new ones.  ##Get started[](#get-started) [Here’s](/docs/get_started/installation) how to install LangChain, set up your environment, and start building.  We recommend following our [Quickstart](/docs/get_started/quickstart) guide to familiarize yourself with the framework by building your first LangChain application.  Read up on our [Security](/docs/security) best practices to make sure you're developing safely with LangChain.  NOTE These docs focus on the Python LangChain library. [Head here](https://js.langchain.com) for docs on the JavaScript LangChain library.  ##LangChain Expression Language (LCEL)[](#langchain-expression-language-lcel) LCEL is a declarative way to compose chains. LCEL was designed from day 1 to support putting prototypes in production, with no code changes, from the simplest “prompt + LLM” chain to the most complex chains.  - [Overview](/docs/expression_language/): LCEL and its benefits - [Interface](/docs/expression_language/interface): The standard interface for LCEL objects - [How-to](/docs/expression_language/how_to): Key features of LCEL - [Cookbook](/docs/expression_language/cookbook): Example code for accomplishing common tasks ##Modules[](#modules) LangChain provides standard, extendable interfaces and integrations for the following modules:  ####[Model I/O](/docs/modules/model_io/)[](#model-io) Interface with language models  ####[Retrieval](/docs/modules/data_connection/)[](#retrieval) Interface with application-specific data  ####[Agents](/docs/modules/agents/)[](#agents) Let models choose which tools to use given high-level directives  ##Examples, ecosystem, and resources[](#examples-ecosystem-and-resources) ###[Use cases](/docs/use_cases/question_answering/)[](#use-cases) Walkthroughs and techniques for common end-to-end use cases, like:  - [Document question answering](/docs/use_cases/question_answering/) - [Chatbots](/docs/use_cases/chatbots/) - [Analyzing structured data](/docs/use_cases/qa_structured/sql/) - and much more... ###[Integrations](/docs/integrations/providers/)[](#integrations) LangChain is part of a rich ecosystem of tools that integrate with our framework and build on top of it. Check out our growing list of [integrations](/docs/integrations/providers/).  ###[Guides](/docs/guides/guides/debugging)[](#guides) Best practices for developing with LangChain.  ###[API reference](https://api.python.langchain.com)[](#api-reference) Head to the reference section for full documentation of all classes and methods in the LangChain and LangChain Experimental Python packages.  ###[Developer's guide](/docs/contributing)[](#developers-guide) Check out the developer's guide for guidelines on contributing and help getting your dev environment set up.  ###[Community](/docs/community)[](#community) Head to the [Community navigator](/docs/community) to find places to ask questions, share feedback, meet other developers, and dream about the future of LLM’s.


Quickstart | 🦜🔗 Langchain

Main:  Skip to main content  🦜🔗 LangChain  Search CTRLK Get startedQuickstart On this page Quickstart In this quickstart we'll show you how to:  Get setup with LangChain, LangSmith and LangServe Use the most basic and common components of LangChain: prompt templates, models, and output parsers Use LangChain Expression Language, the protocol that LangChain is built on and which facilitates component chaining Build a simple application with LangChain Trace your application with LangSmith Serve your application with LangServe That's a fair amount to cover! Let's dive in.  Setup Installation To install LangChain run:  Pip Conda pip install langchain  For more details, see our Installation guide.  Environment Using LangChain will usually require integrations with one or more model providers, data stores, APIs, etc. For this example, we'll use OpenAI's model APIs.  First we'll need to install their Python package:  pip install openai  Accessing the API requires an API key, which you can get by creating an account and heading here. Once we have a key we'll want to set it as an environment variable by running:  export OPENAI_API_KEY="..."  If you'd prefer not to set an environment variable you can pass the key in directly via the openai_api_key named parameter when initiating the OpenAI LLM class:  from langchain.chat_models import ChatOpenAI  llm = ChatOpenAI(openai_api_key="...")  LangSmith Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with LangSmith.  Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:  export LANGCHAIN_TRACING_V2="true" export LANGCHAIN_API_KEY=...  LangServe LangServe helps developers deploy LangChain chains as a REST API. You do not need to use LangServe to use LangChain, but in this guide we'll show how you can deploy your app with LangServe.  Install with:  pip install "langserve[all]"  Building with LangChain LangChain provides many modules that can be used to build language model applications. Modules can be used as standalones in simple applications and they can be composed for more complex use cases. Composition is powered by LangChain Expression Language (LCEL), which defines a unified Runnable interface that many modules implement, making it possible to seamlessly chain components.  The simplest and most common chain contains three things:  LLM/Chat Model: The language model is the core reasoning engine here. In order to work with LangChain, you need to understand the different types of language models and how to work with them. Prompt Template: This provides instructions to the language model. This controls what the language model outputs, so understanding how to construct prompts and different prompting strategies is crucial. Output Parser: These translate the raw response from the language model to a more workable format, making it easy to use the output downstream. In this guide we'll cover those three components individually, and then go over how to combine them. Understanding these concepts will set you up well for being able to use and customize LangChain applications. Most LangChain applications allow you to configure the model and/or the prompt, so knowing how to take advantage of this will be a big enabler.  LLM / Chat Model There are two types of language models:  LLM: underlying model takes a string as input and returns a string ChatModel: underlying model takes a list of messages as input and returns a message Strings are simple, but what exactly are messages? The base message interface is defined by BaseMessage, which has two required attributes:  content: The content of the message. Usually a string. role: The entity from which the BaseMessage is coming. LangChain provides several objects to easily distinguish between different roles:  HumanMessage: A BaseMessage coming from a human/user. AIMessage: A BaseMessage coming from an AI/assistant. SystemMessage: A BaseMessage coming from the system. FunctionMessage / ToolMessage: A BaseMessage containing the output of a function or tool call. If none of those roles sound right, there is also a ChatMessage class where you can specify the role manually.  LangChain provides a common interface that's shared by both LLMs and ChatModels. However it's useful to understand the difference in order to most effectively construct prompts for a given language model.  The simplest way to call an LLM or ChatModel is using .invoke(), the universal synchronous call method for all LangChain Expression Language (LCEL) objects:  LLM.invoke: Takes in a string, returns a string. ChatModel.invoke: Takes in a list of BaseMessage, returns a BaseMessage. The input types for these methods are actually more general than this, but for simplicity here we can assume LLMs only take strings and Chat models only takes lists of messages. Check out the "Go deeper" section below to learn more about model invocation.  Let's see how to work with these different types of models and these different types of inputs. First, let's import an LLM and a ChatModel.  from langchain.llms import OpenAI from langchain.chat_models import ChatOpenAI  llm = OpenAI() chat_model = ChatOpenAI()  LLM and ChatModel objects are effectively configuration objects. You can initialize them with parameters like temperature and others, and pass them around.  from langchain.schema import HumanMessage  text = "What would be a good company name for a company that makes colorful socks?" messages = [HumanMessage(content=text)]  llm.invoke(text) # >> Feetful of Fun  chat_model.invoke(messages) # >> AIMessage(content="Socks O'Color")  Go deeper Prompt templates Most LLM applications do not pass


Quickstart | 🦜🔗 Langchain

user input directly into an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.  In the previous example, the text we passed to the model contained instructions to generate a company name. For our application, it would be great if the user only had to provide the description of a company/product, without having to worry about giving the model instructions.  PromptTemplates help with exactly this! They bundle up all the logic for going from user input into a fully formatted prompt. This can start off very simple - for example, a prompt to produce the above string would just be:  from langchain.prompts import PromptTemplate  prompt = PromptTemplate.from_template("What is a good name for a company that makes {product}?") prompt.format(product="colorful socks")  What is a good name for a company that makes colorful socks?  However, the advantages of using these over raw string formatting are several. You can "partial" out variables - e.g. you can format only some of the variables at a time. You can compose them together, easily combining different templates into a single prompt. For explanations of these functionalities, see the section on prompts for more detail.  PromptTemplates can also be used to produce a list of messages. In this case, the prompt not only contains information about the content, but also each message (its role, its position in the list, etc.). Here, what happens most often is a ChatPromptTemplate is a list of ChatMessageTemplates. Each ChatMessageTemplate contains instructions for how to format that ChatMessage - its role, and then also its content. Let's take a look at this below:  from langchain.prompts.chat import ChatPromptTemplate  template = "You are a helpful assistant that translates {input_language} to {output_language}." human_template = "{text}"  chat_prompt = ChatPromptTemplate.from_messages([     ("system", template),     ("human", human_template), ])  chat_prompt.format_messages(input_language="English", output_language="French", text="I love programming.")  [     SystemMessage(content="You are a helpful assistant that translates English to French.", additional_kwargs={}),     HumanMessage(content="I love programming.") ]  ChatPromptTemplates can also be constructed in other ways - see the section on prompts for more detail.  Output parsers OutputParsers convert the raw output of a language model into a format that can be used downstream. There are few main types of OutputParsers, including:  Convert text from LLM into structured information (e.g. JSON) Convert a ChatMessage into just a string Convert the extra information returned from a call besides the message (like OpenAI function invocation) into a string. For full information on this, see the section on output parsers.  In this getting started guide, we will write our own output parser - one that converts a comma separated list into a list.  from langchain.schema import BaseOutputParser  class CommaSeparatedListOutputParser(BaseOutputParser):     """Parse the output of an LLM call to a comma-separated list."""       def parse(self, text: str):         """Parse the output of an LLM call."""         return text.strip().split(", ")  CommaSeparatedListOutputParser().parse("hi, bye") # >> ['hi', 'bye']  Composing with LCEL We can now combine all these into one chain. This chain will take input variables, pass those to a prompt template to create a prompt, pass the prompt to a language model, and then pass the output through an (optional) output parser. This is a convenient way to bundle up a modular piece of logic. Let's see it in action!  from typing import List  from langchain.chat_models import ChatOpenAI from langchain.prompts import ChatPromptTemplate from langchain.schema import BaseOutputParser  class CommaSeparatedListOutputParser(BaseOutputParser[List[str]]):     """Parse the output of an LLM call to a comma-separated list."""       def parse(self, text: str) -> List[str]:         """Parse the output of an LLM call."""         return text.strip().split(", ")  template = """You are a helpful assistant who generates comma separated lists. A user will pass in a category, and you should generate 5 objects in that category in a comma separated list. ONLY return a comma separated list, and nothing more.""" human_template = "{text}"  chat_prompt = ChatPromptTemplate.from_messages([     ("system", template),     ("human", human_template), ]) chain = chat_prompt | ChatOpenAI() | CommaSeparatedListOutputParser() chain.invoke({"text": "colors"}) # >> ['red', 'blue', 'green', 'yellow', 'orange']  Note that we are using the | syntax to join these components together. This | syntax is powered by the LangChain Expression Language (LCEL) and relies on the universal Runnable interface that all of these objects implement. To learn more about LCEL, read the documentation here.  Tracing with LangSmith Assuming we've set our environment variables as shown in the beginning, all of the model and chain calls we've been making will have been automatically logged to LangSmith. Once there, we can use LangSmith to debug and annotate our application traces, then turn them into datasets for evaluating future iterations of the application.  Check out what the trace for the above chain would look like: https://smith.langchain.com/public/09370280-4330-4eb4-a7e8-c91817f6aa13/r  For more on LangSmith head here.  Serving with LangServe Now that we've built an application, we need to serve it. That's where LangServe comes in. LangServe helps developers deploy LCEL chains as a REST API. The library is integrated with FastAPI and uses pydantic for data validation.  Server To create a server for our application we'll make a serve.py file with three things:  The definition of our chain (same as above) Our FastAPI app A definition of a route from which to serve the chain, which is done with langserve.add_routes #!/usr/bin/env p


Quickstart | 🦜🔗 Langchain

ython from typing import List  from fastapi import FastAPI from langchain.prompts import ChatPromptTemplate from langchain.chat_models import ChatOpenAI from langchain.schema import BaseOutputParser from langserve import add_routes  # 1. Chain definition  class CommaSeparatedListOutputParser(BaseOutputParser[List[str]]):     """Parse the output of an LLM call to a comma-separated list."""       def parse(self, text: str) -> List[str]:         """Parse the output of an LLM call."""         return text.strip().split(", ")  template = """You are a helpful assistant who generates comma separated lists. A user will pass in a category, and you should generate 5 objects in that category in a comma separated list. ONLY return a comma separated list, and nothing more.""" human_template = "{text}"  chat_prompt = ChatPromptTemplate.from_messages([     ("system", template),     ("human", human_template), ]) category_chain = chat_prompt | ChatOpenAI() | CommaSeparatedListOutputParser()  # 2. App definition app = FastAPI(   title="LangChain Server",   version="1.0",   description="A simple API server using LangChain's Runnable interfaces", )  # 3. Adding chain route add_routes(     app,     category_chain,     path="/category_chain", )  if __name__ == "__main__":     import uvicorn      uvicorn.run(app, host="localhost", port=8000)  And that's it! If we execute this file:  python serve.py  we should see our chain being served at localhost:8000.  Playground Every LangServe service comes with a simple built-in UI for configuring and invoking the application with streaming output and visibility into intermediate steps. Head to http://localhost:8000/category_chain/playground/ to try it out!  Client Now let's set up a client for programmatically interacting with our service. We can easily do this with the langserve.RemoteRunnable. Using this, we can interact with the served chain as if it were running client-side.  from langserve import RemoteRunnable  remote_chain = RemoteRunnable("http://localhost:8000/category_chain/") remote_chain.invoke({"text": "colors"}) # >> ['red', 'blue', 'green', 'yellow', 'orange']  To learn more about the many other features of LangServe head here.  Next steps We've touched on how to build an application with LangChain, how to trace it with LangSmith, and how to serve it with LangServe. There are a lot more features in all three of these than we can cover here. To continue on your journey:  Read up on LangChain Expression Language (LCEL) to learn how to chain these components together Dive deeper into LLMs, prompts, and output parsers and learn the other key components Explore common end-to-end use cases and template applications Read up on LangSmith, the platform for debugging, testing, monitoring and more Learn more about serving your applications with LangServe Previous Installation Next Security Community Discord Twitter GitHub Python JS/TS More Homepage Blog Copyright © 2023 LangChain, Inc.


Debugging | 🦜🔗 Langchain

Main:  On this page #Debugging If you're building with LLMs, at some point something will break, and you'll need to debug. A model call will fail, or the model output will be misformatted, or there will be some nested model calls and it won't be clear where along the way an incorrect output was created.  Here are a few different tools and functionalities to aid in debugging.  ##Tracing[](#tracing) Platforms with tracing capabilities like [LangSmith](/docs/langsmith/) and [WandB](/docs/integrations/providers/wandb_tracing) are the most comprehensive solutions for debugging. These platforms make it easy to not only log and visualize LLM apps, but also to actively debug, test and refine them.  For anyone building production-grade LLM applications, we highly recommend using a platform like this.    ##set_debug and set_verbose[](#set_debug-and-set_verbose) If you're prototyping in Jupyter Notebooks or running Python scripts, it can be helpful to print out the intermediate steps of a Chain run.  There are a number of ways to enable printing at varying degrees of verbosity.  Let's suppose we have a simple agent, and want to visualize the actions it takes and tool outputs it receives. Without any debugging, here's what we see:  from langchain.agents import AgentType, initialize_agent, load_tools from langchain.chat_models import ChatOpenAI  llm = ChatOpenAI(model_name="gpt-4", temperature=0) tools = load_tools(["ddg-search", "llm-math"], llm=llm) agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)  agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")      'The director of the 2023 film Oppenheimer is Christopher Nolan and he is approximately 19345 days old in 2023.'  ###set_debug(True)[](#set_debugtrue) Setting the global debug flag will cause all LangChain components with callback support (chains, models, agents, tools, retrievers) to print the inputs they receive and outputs they generate. This is the most verbose setting and will fully log raw inputs and outputs.  from langchain.globals import set_debug  set_debug(True)  agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")  Console output ###set_verbose(True)[](#set_verbosetrue) Setting the verbose flag will print out inputs and outputs in a slightly more readable format and will skip logging certain raw outputs (like the token usage stats for an LLM call) so that you can focus on application logic.  from langchain.globals import set_verbose  set_verbose(True)  agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")  Console output ###Chain(..., verbose=True)[](#chain-verbosetrue) You can also scope verbosity down to a single object, in which case only the inputs and outputs to that object are printed (along with any additional callbacks calls made specifically by that object).  # Passing verbose=True to initialize_agent will pass that along to the AgentExecutor (which is a Chain). agent = initialize_agent(     tools,      llm,      agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,     verbose=True, )  agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")  Console output ##Other callbacks[](#other-callbacks) Callbacks are what we use to execute any functionality within a component outside the primary component logic. All of the above solutions use Callbacks under the hood to log intermediate steps of components. There are a number of Callbacks relevant for debugging that come with LangChain out of the box, like the [FileCallbackHandler](/docs/modules/callbacks/how_to/filecallbackhandler). You can also implement your own callbacks to execute custom functionality.  See here for more info on [Callbacks](/docs/modules/callbacks/), how to use them, and customize them.


Deployment | 🦜🔗 Langchain

Main:  On this page #Deployment In today's fast-paced technological landscape, the use of Large Language Models (LLMs) is rapidly expanding. As a result, it is crucial for developers to understand how to effectively deploy these models in production environments. LLM interfaces typically fall into two categories:  - Case 1: Utilizing External LLM Providers (OpenAI, Anthropic, etc.) In this scenario, most of the computational burden is handled by the LLM providers, while LangChain simplifies the implementation of business logic around these services. This approach includes features such as prompt templating, chat message generation, caching, vector embedding database creation, preprocessing, etc.  - Case 2: Self-hosted Open-Source Models Alternatively, developers can opt to use smaller, yet comparably capable, self-hosted open-source LLM models. This approach can significantly decrease costs, latency, and privacy concerns associated with transferring data to external LLM providers.  Regardless of the framework that forms the backbone of your product, deploying LLM applications comes with its own set of challenges. It's vital to understand the trade-offs and key considerations when evaluating serving frameworks.  ##Outline[](#outline) This guide aims to provide a comprehensive overview of the requirements for deploying LLMs in a production setting, focusing on:  - Designing a Robust LLM Application Service - Maintaining Cost-Efficiency - Ensuring Rapid Iteration Understanding these components is crucial when assessing serving systems. LangChain integrates with several open-source projects designed to tackle these issues, providing a robust framework for productionizing your LLM applications. Some notable frameworks include:  - [Ray Serve](/docs/ecosystem/integrations/ray_serve) - [BentoML](https://github.com/bentoml/BentoML) - [OpenLLM](/docs/ecosystem/integrations/openllm) - [Modal](/docs/ecosystem/integrations/modal) - [Jina](/docs/ecosystem/integrations/jina#deployment) These links will provide further information on each ecosystem, assisting you in finding the best fit for your LLM deployment needs.  ##Designing a Robust LLM Application Service[](#designing-a-robust-llm-application-service) When deploying an LLM service in production, it's imperative to provide a seamless user experience free from outages. Achieving 24/7 service availability involves creating and maintaining several sub-systems surrounding your application.  ###Monitoring[](#monitoring) Monitoring forms an integral part of any system running in a production environment. In the context of LLMs, it is essential to monitor both performance and quality metrics.  Performance Metrics: These metrics provide insights into the efficiency and capacity of your model. Here are some key examples:  - Query per second (QPS): This measures the number of queries your model processes in a second, offering insights into its utilization. - Latency: This metric quantifies the delay from when your client sends a request to when they receive a response. - Tokens Per Second (TPS): This represents the number of tokens your model can generate in a second. Quality Metrics: These metrics are typically customized according to the business use-case. For instance, how does the output of your system compare to a baseline, such as a previous version? Although these metrics can be calculated offline, you need to log the necessary data to use them later.  ###Fault tolerance[](#fault-tolerance) Your application may encounter errors such as exceptions in your model inference or business logic code, causing failures and disrupting traffic. Other potential issues could arise from the machine running your application, such as unexpected hardware breakdowns or loss of spot-instances during high-demand periods. One way to mitigate these risks is by increasing redundancy through replica scaling and implementing recovery mechanisms for failed replicas. However, model replicas aren't the only potential points of failure. It's essential to build resilience against various failures that could occur at any point in your stack.  ###Zero down time upgrade[](#zero-down-time-upgrade) System upgrades are often necessary but can result in service disruptions if not handled correctly. One way to prevent downtime during upgrades is by implementing a smooth transition process from the old version to the new one. Ideally, the new version of your LLM service is deployed, and traffic gradually shifts from the old to the new version, maintaining a constant QPS throughout the process.  ###Load balancing[](#load-balancing) Load balancing, in simple terms, is a technique to distribute work evenly across multiple computers, servers, or other resources to optimize the utilization of the system, maximize throughput, minimize response time, and avoid overload of any single resource. Think of it as a traffic officer directing cars (requests) to different roads (servers) so that no single road becomes too congested.  There are several strategies for load balancing. For example, one common method is the Round Robin strategy, where each request is sent to the next server in line, cycling back to the first when all servers have received a request. This works well when all servers are equally capable. However, if some servers are more powerful than others, you might use a Weighted Round Robin or Least Connections strategy, where more requests are sent to the more powerful servers, or to those currently handling the fewest active requests. Let's imagine you're running a LLM chain. If your application becomes popular, you could have hundreds or even thousands of users asking questions at the same time. If one server gets too busy (high load), the load balancer would direct new requests to another server that is less busy. This way, all your users get a timely response and the system remains stable.  ##Maintaining Cost-Efficiency and Scalability[](#maintaining-cost-efficiency-and-scalabi


Deployment | 🦜🔗 Langchain

lity) Deploying LLM services can be costly, especially when you're handling a large volume of user interactions. Charges by LLM providers are usually based on tokens used, making a chat system inference on these models potentially expensive. However, several strategies can help manage these costs without compromising the quality of the service.  ###Self-hosting models[](#self-hosting-models) Several smaller and open-source LLMs are emerging to tackle the issue of reliance on LLM providers. Self-hosting allows you to maintain similar quality to LLM provider models while managing costs. The challenge lies in building a reliable, high-performing LLM serving system on your own machines.  ###Resource Management and Auto-Scaling[](#resource-management-and-auto-scaling) Computational logic within your application requires precise resource allocation. For instance, if part of your traffic is served by an OpenAI endpoint and another part by a self-hosted model, it's crucial to allocate suitable resources for each. Auto-scaling—adjusting resource allocation based on traffic—can significantly impact the cost of running your application. This strategy requires a balance between cost and responsiveness, ensuring neither resource over-provisioning nor compromised application responsiveness.  ###Utilizing Spot Instances[](#utilizing-spot-instances) On platforms like AWS, spot instances offer substantial cost savings, typically priced at about a third of on-demand instances. The trade-off is a higher crash rate, necessitating a robust fault-tolerance mechanism for effective use.  ###Independent Scaling[](#independent-scaling) When self-hosting your models, you should consider independent scaling. For example, if you have two translation models, one fine-tuned for French and another for Spanish, incoming requests might necessitate different scaling requirements for each.  ###Batching requests[](#batching-requests) In the context of Large Language Models, batching requests can enhance efficiency by better utilizing your GPU resources. GPUs are inherently parallel processors, designed to handle multiple tasks simultaneously. If you send individual requests to the model, the GPU might not be fully utilized as it's only working on a single task at a time. On the other hand, by batching requests together, you're allowing the GPU to work on multiple tasks at once, maximizing its utilization and improving inference speed. This not only leads to cost savings but can also improve the overall latency of your LLM service.  In summary, managing costs while scaling your LLM services requires a strategic approach. Utilizing self-hosting models, managing resources effectively, employing auto-scaling, using spot instances, independently scaling models, and batching requests are key strategies to consider. Open-source libraries such as Ray Serve and BentoML are designed to deal with these complexities.  ##Ensuring Rapid Iteration[](#ensuring-rapid-iteration) The LLM landscape is evolving at an unprecedented pace, with new libraries and model architectures being introduced constantly. Consequently, it's crucial to avoid tying yourself to a solution specific to one particular framework. This is especially relevant in serving, where changes to your infrastructure can be time-consuming, expensive, and risky. Strive for infrastructure that is not locked into any specific machine learning library or framework, but instead offers a general-purpose, scalable serving layer. Here are some aspects where flexibility plays a key role:  ###Model composition[](#model-composition) Deploying systems like LangChain demands the ability to piece together different models and connect them via logic. Take the example of building a natural language input SQL query engine. Querying an LLM and obtaining the SQL command is only part of the system. You need to extract metadata from the connected database, construct a prompt for the LLM, run the SQL query on an engine, collect and feed back the response to the LLM as the query runs, and present the results to the user. This demonstrates the need to seamlessly integrate various complex components built in Python into a dynamic chain of logical blocks that can be served together.  ##Cloud providers[](#cloud-providers) Many hosted solutions are restricted to a single cloud provider, which can limit your options in today's multi-cloud world. Depending on where your other infrastructure components are built, you might prefer to stick with your chosen cloud provider.  ##Infrastructure as Code (IaC)[](#infrastructure-as-code-iac) Rapid iteration also involves the ability to recreate your infrastructure quickly and reliably. This is where Infrastructure as Code (IaC) tools like Terraform, CloudFormation, or Kubernetes YAML files come into play. They allow you to define your infrastructure in code files, which can be version controlled and quickly deployed, enabling faster and more reliable iterations.  ##CI/CD[](#cicd) In a fast-paced environment, implementing CI/CD pipelines can significantly speed up the iteration process. They help automate the testing and deployment of your LLM applications, reducing the risk of errors and enabling faster feedback and iteration.


LangChain Templates | 🦜🔗 Langchain

Main:  #LangChain Templates For more information on LangChain Templates, visit  - [LangChain Templates Quickstart](https://github.com/langchain-ai/langchain/blob/master/templates/README.md) - [LangChain Templates Index](https://github.com/langchain-ai/langchain/blob/master/templates/docs/INDEX.md) - [Full List of Templates](https://github.com/langchain-ai/langchain/blob/master/templates/)


Evaluation | 🦜🔗 Langchain

Main:  On this page #Evaluation Building applications with language models involves many moving parts. One of the most critical components is ensuring that the outcomes produced by your models are reliable and useful across a broad array of inputs, and that they work well with your application's other software components. Ensuring reliability usually boils down to some combination of application design, testing & evaluation, and runtime checks.  The guides in this section review the APIs and functionality LangChain provides to help you better evaluate your applications. Evaluation and testing are both critical when thinking about deploying LLM applications, since production environments require repeatable and useful outcomes.  LangChain offers various types of evaluators to help you measure performance and integrity on diverse data, and we hope to encourage the community to create and share other useful evaluators so everyone can improve. These docs will introduce the evaluator types, how to use them, and provide some examples of their use in real-world scenarios.  Each evaluator type in LangChain comes with ready-to-use implementations and an extensible API that allows for customization according to your unique requirements. Here are some of the types of evaluators we offer:  - [String Evaluators](/docs/guides/evaluation/string/): These evaluators assess the predicted string for a given input, usually comparing it against a reference string. - [Trajectory Evaluators](/docs/guides/evaluation/trajectory/): These are used to evaluate the entire trajectory of agent actions. - [Comparison Evaluators](/docs/guides/evaluation/comparison/): These evaluators are designed to compare predictions from two runs on a common input. These evaluators can be used across various scenarios and can be applied to different chain and LLM implementations in the LangChain library.  We also are working to share guides and cookbooks that demonstrate how to use these evaluators in real-world scenarios, such as:  - [Chain Comparisons](/docs/guides/evaluation/examples/comparisons): This example uses a comparison evaluator to predict the preferred output. It reviews ways to measure confidence intervals to select statistically significant differences in aggregate preference scores across different models or prompts. ##Reference Docs[](#reference-docs) For detailed information on the available evaluators, including how to instantiate, configure, and customize them, check out the [reference documentation](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.evaluation) directly.  [ ##🗃 String Evaluators 8 items  ](/docs/guides/evaluation/string/) [ ##🗃 Comparison Evaluators 3 items  ](/docs/guides/evaluation/comparison/) [ ##🗃 Trajectory Evaluators 2 items  ](/docs/guides/evaluation/trajectory/) [ ##🗃 Examples 1 items  ](/docs/guides/evaluation/examples/)


Comparison Evaluators | 🦜🔗 Langchain

Main:  #Comparison Evaluators Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.  These evaluators inherit from the PairwiseStringEvaluator class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.  To create a custom comparison evaluator, inherit from the PairwiseStringEvaluator class and overwrite the _evaluate_string_pairs method. If you require asynchronous evaluation, also overwrite the _aevaluate_string_pairs method.  Here's a summary of the key methods and properties of a comparison evaluator:  - evaluate_string_pairs: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators. - aevaluate_string_pairs: Asynchronously evaluate the output string pairs. This function should be overwritten for asynchronous evaluation. - requires_input: This property indicates whether this evaluator requires an input string. - requires_reference: This property specifies whether this evaluator requires a reference label. LANGSMITH SUPPORT The [run_on_dataset](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.smith) evaluation method is designed to evaluate only a single model at a time, and thus, doesn't support these evaluators.  Detailed information about creating custom evaluators and the available built-in comparison evaluators is provided in the following sections.  [ ##📄 Pairwise string comparison Open In Colab  ](/docs/guides/evaluation/comparison/pairwise_string) [ ##📄 Pairwise embedding distance Open In Colab  ](/docs/guides/evaluation/comparison/pairwise_embedding_distance) [ ##📄 Custom pairwise evaluator Open In Colab  ](/docs/guides/evaluation/comparison/custom)


Custom pairwise evaluator | 🦜🔗 Langchain

Main:  Skip to main content  🦜🔗 LangChain  Search CTRLK EvaluationComparison EvaluatorsCustom pairwise evaluator On this page Custom pairwise evaluator   You can make your own pairwise string evaluators by inheriting from PairwiseStringEvaluator class and overwriting the _evaluate_string_pairs method (and the _aevaluate_string_pairs method if you want to use the evaluator asynchronously).  In this example, you will make a simple custom evaluator that just returns whether the first prediction has more whitespace tokenized 'words' than the second.  You can check out the reference docs for the PairwiseStringEvaluator interface for more info.  from typing import Any, Optional  from langchain.evaluation import PairwiseStringEvaluator   class LengthComparisonPairwiseEvaluator(PairwiseStringEvaluator):     """     Custom evaluator to compare two strings.     """      def _evaluate_string_pairs(         self,         *,         prediction: str,         prediction_b: str,         reference: Optional[str] = None,         input: Optional[str] = None,         **kwargs: Any,     ) -> dict:         score = int(len(prediction.split()) > len(prediction_b.split()))         return {"score": score}  evaluator = LengthComparisonPairwiseEvaluator()  evaluator.evaluate_string_pairs(     prediction="The quick brown fox jumped over the lazy dog.",     prediction_b="The quick brown fox jumped over the dog.", )      {'score': 1}  LLM-Based Example That example was simple to illustrate the API, but it wasn't very useful in practice. Below, use an LLM with some custom instructions to form a simple preference scorer similar to the built-in PairwiseStringEvalChain. We will use ChatAnthropic for the evaluator chain.  # %pip install anthropic # %env ANTHROPIC_API_KEY=YOUR_API_KEY  from typing import Any, Optional  from langchain.chains import LLMChain from langchain.chat_models import ChatAnthropic from langchain.evaluation import PairwiseStringEvaluator   class CustomPreferenceEvaluator(PairwiseStringEvaluator):     """     Custom evaluator to compare two strings using a custom LLMChain.     """      def __init__(self) -> None:         llm = ChatAnthropic(model="claude-2", temperature=0)         self.eval_chain = LLMChain.from_string(             llm,             """Which option is preferred? Do not take order into account. Evaluate based on accuracy and helpfulness. If neither is preferred, respond with C. Provide your reasoning, then finish with Preference: A/B/C  Input: How do I get the path of the parent directory in python 3.8? Option A: You can use the following code: ```python import os  os.path.dirname(os.path.dirname(os.path.abspath(__file__)))  Option B: You can use the following code:  from pathlib import Path Path(__file__).absolute().parent  Reasoning: Both options return the same result. However, since option B is more concise and easily understand, it is preferred. Preference: B  Which option is preferred? Do not take order into account. Evaluate based on accuracy and helpfulness. If neither is preferred, respond with C. Provide your reasoning, then finish with Preference: A/B/C Input: {input} Option A: {prediction} Option B: {prediction_b} Reasoning:""", )  @property def requires_input(self) -> bool:     return True  @property def requires_reference(self) -> bool:     return False  def _evaluate_string_pairs(     self,     *,     prediction: str,     prediction_b: str,     reference: Optional[str] = None,     input: Optional[str] = None,     **kwargs: Any, ) -> dict:     result = self.eval_chain(         {             "input": input,             "prediction": prediction,             "prediction_b": prediction_b,             "stop": ["Which option is preferred?"],         },         **kwargs,     )      response_text = result["text"]     reasoning, preference = response_text.split("Preference:", maxsplit=1)     preference = preference.strip()     score = 1.0 if preference == "A" else (0.0 if preference == "B" else None)     return {"reasoning": reasoning.strip(), "value": preference, "score": score}    ```python evaluator = CustomPreferenceEvaluator()  evaluator.evaluate_string_pairs(     input="How do I import from a relative directory?",     prediction="use importlib! importlib.import_module('.my_package', '.')",     prediction_b="from .sibling import foo", )      {'reasoning': 'Option B is preferred over option A for importing from a relative directory, because it is more straightforward and concise.\n\nOption A uses the importlib module, which allows importing a module by specifying the full name as a string. While this works, it is less clear compared to option B.\n\nOption B directly imports from the relative path using dot notation, which clearly shows that it is a relative import. This is the recommended way to do relative imports in Python.\n\nIn summary, option B is more accurate and helpful as it uses the standard Python relative import syntax.',      'value': 'B',      'score': 0.0}  # Setting requires_input to return True adds additional validation to avoid returning a grade when insufficient data is provided to the chain.  try:     evaluator.evaluate_string_pairs(         prediction="use importlib! importlib.import_module('.my_package', '.')",         prediction_b="from .sibling import foo",     ) except ValueError as e:     print(e)      CustomPreferenceEvaluator requires an input string.  Previous Pairwise embedding distance Next Trajectory Evaluators Community Discord Twitter GitHub Python JS/TS More Homepage Blog Copyright © 2023 LangChain, Inc.


Pairwise embedding distance | 🦜🔗 Langchain

Main:  On this page [](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/comparison/pairwise_embedding_distance.ipynb)  One way to measure the similarity (or dissimilarity) between two predictions on a shared or similar input is to embed the predictions and compute a vector distance between the two embeddings.[[1]](#cite_note-1)  You can load the pairwise_embedding_distance evaluator to do this.  Note: This returns a distance score, meaning that the lower the number, the more similar the outputs are, according to their embedded representation.  Check out the reference docs for the [PairwiseEmbeddingDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.embedding_distance.base.PairwiseEmbeddingDistanceEvalChain.html#langchain.evaluation.embedding_distance.base.PairwiseEmbeddingDistanceEvalChain) for more info.  from langchain.evaluation import load_evaluator  evaluator = load_evaluator("pairwise_embedding_distance")  evaluator.evaluate_string_pairs(     prediction="Seattle is hot in June", prediction_b="Seattle is cool in June." )      {'score': 0.0966466944859925}  evaluator.evaluate_string_pairs(     prediction="Seattle is warm in June", prediction_b="Seattle is cool in June." )      {'score': 0.03761174337464557}  ##Select the Distance Metric[](#select-the-distance-metric) By default, the evaluator uses cosine distance. You can choose a different distance metric if you'd like.  from langchain.evaluation import EmbeddingDistance  list(EmbeddingDistance)      [<EmbeddingDistance.COSINE: 'cosine'>,      <EmbeddingDistance.EUCLIDEAN: 'euclidean'>,      <EmbeddingDistance.MANHATTAN: 'manhattan'>,      <EmbeddingDistance.CHEBYSHEV: 'chebyshev'>,      <EmbeddingDistance.HAMMING: 'hamming'>]  evaluator = load_evaluator(     "pairwise_embedding_distance", distance_metric=EmbeddingDistance.EUCLIDEAN )  ##Select Embeddings to Use[](#select-embeddings-to-use) The constructor uses OpenAI embeddings by default, but you can configure this however you want. Below, use huggingface local embeddings  from langchain.embeddings import HuggingFaceEmbeddings  embedding_model = HuggingFaceEmbeddings() hf_evaluator = load_evaluator("pairwise_embedding_distance", embeddings=embedding_model)  hf_evaluator.evaluate_string_pairs(     prediction="Seattle is hot in June", prediction_b="Seattle is cool in June." )      {'score': 0.5486443280477362}  hf_evaluator.evaluate_string_pairs(     prediction="Seattle is warm in June", prediction_b="Seattle is cool in June." )      {'score': 0.21018880025138598}  1. Note: When it comes to semantic similarity, this often gives better results than older string distance metrics (such as those in the `PairwiseStringDistanceEvalChain`), though it tends to be less reliable than evaluators that use the LLM directly (such as the `PairwiseStringEvalChain`)


Pairwise string comparison | 🦜🔗 Langchain

Main:  On this page [](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/comparison/pairwise_string.ipynb)  Often you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:  - Which LLM or prompt produces a preferred output for a given question? - Which examples should I include for few-shot example selection? - Which output is better to include for fine-tuning? The simplest and often most reliable automated way to choose a preferred prediction for a given input is to use the pairwise_string evaluator.  Check out the reference docs for the [PairwiseStringEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain) for more info.  from langchain.evaluation import load_evaluator  evaluator = load_evaluator("labeled_pairwise_string")  evaluator.evaluate_string_pairs(     prediction="there are three dogs",     prediction_b="4",     input="how many dogs are in the park?",     reference="four", )      {'reasoning': 'Both responses are relevant to the question asked, as they both provide a numerical answer to the question about the number of dogs in the park. However, Response A is incorrect according to the reference answer, which states that there are four dogs. Response B, on the other hand, is correct as it matches the reference answer. Neither response demonstrates depth of thought, as they both simply provide a numerical answer without any additional information or context. \n\nBased on these criteria, Response B is the better response.\n',      'value': 'B',      'score': 0}  ##Methods[](#methods) The pairwise string evaluator can be called using [evaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.evaluate_string_pairs) (or async [aevaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.aevaluate_string_pairs)) methods, which accept:  - prediction (str) – The predicted response of the first model, chain, or prompt. - prediction_b (str) – The predicted response of the second model, chain, or prompt. - input (str) – The input question, prompt, or other text. - reference (str) – (Only for the labeled_pairwise_string variant) The reference response. They return a dictionary with the following values:  - value: 'A' or 'B', indicating whether prediction or prediction_b is preferred, respectively - score: Integer 0 or 1 mapped from the 'value', where a score of 1 would mean that the first prediction is preferred, and a score of 0 would mean prediction_b is preferred. - reasoning: String "chain of thought reasoning" from the LLM generated prior to creating the score ##Without References[](#without-references) When references aren't available, you can still predict the preferred response. The results will reflect the evaluation model's preference, which is less reliable and may result in preferences that are factually incorrect.  from langchain.evaluation import load_evaluator  evaluator = load_evaluator("pairwise_string")  evaluator.evaluate_string_pairs(     prediction="Addition is a mathematical operation.",     prediction_b="Addition is a mathematical operation that adds two numbers to create a third number, the 'sum'.",     input="What is addition?", )      {'reasoning': 'Both responses are correct and relevant to the question. However, Response B is more helpful and insightful as it provides a more detailed explanation of what addition is. Response A is correct but lacks depth as it does not explain what the operation of addition entails. \n\nFinal Decision: [[B]]',      'value': 'B',      'score': 0}  ##Defining the Criteria[](#defining-the-criteria) By default, the LLM is instructed to select the 'preferred' response based on helpfulness, relevance, correctness, and depth of thought. You can customize the criteria by passing in a criteria argument, where the criteria could take any of the following forms:  - [Criteria](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.Criteria.html#langchain.evaluation.criteria.eval_chain.Criteria) enum or its string value - to use one of the default criteria and their descriptions - [Constitutional principal](https://api.python.langchain.com/en/latest/chains/langchain.chains.constitutional_ai.models.ConstitutionalPrinciple.html#langchain.chains.constitutional_ai.models.ConstitutionalPrinciple) - use one any of the constitutional principles defined in langchain - Dictionary: a list of custom criteria, where the key is the name of the criteria, and the value is the description. - A list of criteria or constitutional principles - to combine multiple criteria in one. Below is an example for determining preferred writing responses based on a custom style.  custom_criteria = {     "simplicity": "Is the language straightforward and unpretentious?",     "clarity": "Are the sentences clear and easy to understand?",     "precision": "Is the writing precise, with no unnecessary words or details?",     "truthfulness": "Does the writing feel honest and sincere?",     "subtext": "Does the writing suggest deeper meanings or themes?", } evaluator = load_evaluator("pairwise_string", criteria=custom_criteria)  evaluator.evaluate_string_pairs(     prediction="Every cheerful household shares a similar rhythm of joy; but sorrow, in each household, plays a unique, haunting melody.",     prediction_b="Where one finds a symphony of joy, every domicile of happiness resounds in harmonious,"     " identical notes; ye