Document loaders | 🦜🔗 Langchain

Main:  [ ##📄 acreom acreom is a dev-first knowledge base with tasks running on local markdown files.  ](/docs/integrations/document_loaders/acreom) [ ##📄 Airbyte CDK Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_cdk) [ ##📄 Airbyte Gong Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_gong) [ ##📄 Airbyte Hubspot Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_hubspot) [ ##📄 Airbyte JSON Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_json) [ ##📄 Airbyte Salesforce Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_salesforce) [ ##📄 Airbyte Shopify Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_shopify) [ ##📄 Airbyte Stripe Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_stripe) [ ##📄 Airbyte Typeform Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_typeform) [ ##📄 Airbyte Zendesk Support Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  ](/docs/integrations/document_loaders/airbyte_zendesk_support) [ ##📄 Airtable * Get your API key here.  ](/docs/integrations/document_loaders/airtable) [ ##📄 Alibaba Cloud MaxCompute Alibaba Cloud MaxCompute (previously known as ODPS) is a general purpose, fully managed, multi-tenancy data processing platform for large-scale data warehousing. MaxCompute supports various data importing solutions and distributed computing models, enabling users to effectively query massive datasets, reduce production costs, and ensure data security.  ](/docs/integrations/document_loaders/alibaba_cloud_maxcompute) [ ##📄 Apify Dataset Apify Dataset is a scalable append-only storage with sequential access built for storing structured web scraping results, such as a list of products or Google SERPs, and then export them to various formats like JSON, CSV, or Excel. Datasets are mainly used to save results of Apify Actors—serverless cloud programs for various web scraping, crawling, and data extraction use cases.  ](/docs/integrations/document_loaders/apify_dataset) [ ##📄 ArcGIS This notebook demonstrates the use of the langchain.document_loaders.ArcGISLoader class.  ](/docs/integrations/document_loaders/arcgis) [ ##📄 Arxiv arXiv is an open-access archive for 2 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.  ](/docs/integrations/document_loaders/arxiv) [ ##📄 AssemblyAI Audio Transcripts The AssemblyAIAudioTranscriptLoader allows to transcribe audio files with the AssemblyAI API and loads the transcribed text into documents.  ](/docs/integrations/document_loaders/assemblyai) [ ##📄 Async Chromium Chromium is one of the browsers supported by Playwright, a library used to control browser automation.  ](/docs/integrations/document_loaders/async_chromium) [ ##📄 AsyncHtml AsyncHtmlLoader loads raw HTML from a list of URLs concurrently.  ](/docs/integrations/document_loaders/async_html) [ ##📄 AWS S3 Directory Amazon Simple Storage Service (Amazon S3) is an object storage service  ](/docs/integrations/document_loaders/aws_s3_directory) [ ##📄 AWS S3 File Amazon Simple Storage Service (Amazon S3) is an object storage service.  ](/docs/integrations/document_loaders/aws_s3_file) [ ##📄 AZLyrics AZLyrics is a large, legal, every day growing collection of lyrics.  ](/docs/integrations/document_loaders/azlyrics) [ ##📄 Azure AI Data Azure AI Studio provides the capability to upload data assets to cloud storage and register existing data assets from the following sources:  ](/docs/integrations/document_loaders/azure_ai_data) [ ##📄 Azure Blob Storage Container Azure Blob Storage is Microsoft's object storage solution for the cloud. Blob Storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn't adhere to a particular data model or definition, such as text or binary data.  ](/docs/integrations/document_loaders/azure_blob_storage_container) [ ##📄 Azure Blob Storage File Azure Files offers fully managed file shares in the cloud that are accessible via the industry standard Server Message Block (SMB) protocol, Network File System (NFS) protocol, and Azure Files REST API.  ](/docs/integrations/document_loaders/azure_blob_storage_file) [ ##📄 Azure Document Intelligence Azure Document Intelligence (formerly known as Azure Forms Recognizer) is machine-learning  ](/docs/integrations/d


Document loaders | 🦜🔗 Langchain

ocument_loaders/azure_document_intelligence) [ ##📄 BibTeX BibTeX is a file format and reference management system commonly used in conjunction with LaTeX typesetting. It serves as a way to organize and store bibliographic information for academic and research documents.  ](/docs/integrations/document_loaders/bibtex) [ ##📄 BiliBili Bilibili is one of the most beloved long-form video sites in China.  ](/docs/integrations/document_loaders/bilibili) [ ##📄 Blackboard Blackboard Learn (previously the Blackboard Learning Management System) is a web-based virtual learning environment and learning management system developed by Blackboard Inc. The software features course management, customizable open architecture, and scalable design that allows integration with student information systems and authentication protocols. It may be installed on local servers, hosted by Blackboard ASP Solutions, or provided as Software as a Service hosted on Amazon Web Services. Its main purposes are stated to include the addition of online elements to courses traditionally delivered face-to-face and development of completely online courses with few or no face-to-face meetings  ](/docs/integrations/document_loaders/blackboard) [ ##📄 Blockchain Overview  ](/docs/integrations/document_loaders/blockchain) [ ##📄 Brave Search Brave Search is a search engine developed by Brave Software.  ](/docs/integrations/document_loaders/brave_search) [ ##📄 Browserless Browserless is a service that allows you to run headless Chrome instances in the cloud. It's a great way to run browser-based automation at scale without having to worry about managing your own infrastructure.  ](/docs/integrations/document_loaders/browserless) [ ##📄 ChatGPT Data ChatGPT is an artificial intelligence (AI) chatbot developed by OpenAI.  ](/docs/integrations/document_loaders/chatgpt_loader) [ ##📄 College Confidential College Confidential gives information on 3,800+ colleges and universities.  ](/docs/integrations/document_loaders/college_confidential) [ ##📄 Concurrent Loader Works just like the GenericLoader but concurrently for those who choose to optimize their workflow.  ](/docs/integrations/document_loaders/concurrent) [ ##📄 Confluence Confluence is a wiki collaboration platform that saves and organizes all of the project-related material. Confluence is a knowledge base that primarily handles content management activities.  ](/docs/integrations/document_loaders/confluence) [ ##📄 CoNLL-U CoNLL-U is revised version of the CoNLL-X format. Annotations are encoded in plain text files (UTF-8, normalized to NFC, using only the LF character as line break, including an LF character at the end of file) with three types of lines:  ](/docs/integrations/document_loaders/conll-u) [ ##📄 Copy Paste This notebook covers how to load a document object from something you just want to copy and paste. In this case, you don't even need to use a DocumentLoader, but rather can just construct the Document directly.  ](/docs/integrations/document_loaders/copypaste) [ ##📄 CSV A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.  ](/docs/integrations/document_loaders/csv) [ ##📄 Cube Semantic Layer This notebook demonstrates the process of retrieving Cube's data model metadata in a format suitable for passing to LLMs as embeddings, thereby enhancing contextual information.  ](/docs/integrations/document_loaders/cube_semantic) [ ##📄 Datadog Logs Datadog is a monitoring and analytics platform for cloud-scale applications.  ](/docs/integrations/document_loaders/datadog_logs) [ ##📄 Diffbot Unlike traditional web scraping tools, Diffbot doesn't require any rules to read the content on a page.  ](/docs/integrations/document_loaders/diffbot) [ ##📄 Discord Discord is a VoIP and instant messaging social platform. Users have the ability to communicate with voice calls, video calls, text messaging, media and files in private chats or as part of communities called "servers". A server is a collection of persistent chat rooms and voice channels which can be accessed via invite links.  ](/docs/integrations/document_loaders/discord) [ ##📄 Docugami This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders.  ](/docs/integrations/document_loaders/docugami) [ ##📄 Docusaurus Docusaurus is a static-site generator which provides out-of-the-box documentation features.  ](/docs/integrations/document_loaders/docusaurus) [ ##📄 Dropbox Dropbox is a file hosting service that brings everything-traditional files, cloud content, and web shortcuts together in one place.  ](/docs/integrations/document_loaders/dropbox) [ ##📄 DuckDB DuckDB is an in-process SQL OLAP database management system.  ](/docs/integrations/document_loaders/duckdb) [ ##📄 Email This notebook shows how to load email (.eml) or Microsoft Outlook (.msg) files.  ](/docs/integrations/document_loaders/email) [ ##📄 Embaas embaas is a fully managed NLP API service that offers features like embedding generation, document text extraction, document to embeddings and more. You can choose a variety of pre-trained models.  ](/docs/integrations/document_loaders/embaas) [ ##📄 EPub EPUB is an e-book file format that uses the ".epub" file extension. The term is short for electronic publication and is sometimes styled ePub. EPUB is supported by many e-readers, and compatible software is available for most smartphones, tablets, and computers.  ](/docs/integrations/document_loaders/epub) [ ##📄 Etherscan Etherscan is the leading blockchain explorer, search, API and analytics platform for Ethereum,  ](/docs/integrations/document_loaders/etherscan) [ ##📄 EverNote EverNote is intended for archiving and creating notes in which photos, audio and saved web content can be embedded. Notes are stored in virtual "notebooks" an


Document loaders | 🦜🔗 Langchain

d can be tagged, annotated, edited, searched, and exported.  ](/docs/integrations/document_loaders/evernote) [ ##📄 Microsoft Excel The UnstructuredExcelLoader is used to load Microsoft Excel files. The loader works with both .xlsx and .xls files. The page content will be the raw text of the Excel file. If you use the loader in "elements" mode, an HTML representation of the Excel file will be available in the document metadata under the textashtml key.  ](/docs/integrations/document_loaders/excel) [ ##📄 Facebook Chat Messenger) is an American proprietary instant messaging app and platform developed by Meta Platforms. Originally developed as Facebook Chat in 2008, the company revamped its messaging service in 2010.  ](/docs/integrations/document_loaders/facebook_chat) [ ##📄 Fauna Fauna is a Document Database.  ](/docs/integrations/document_loaders/fauna) [ ##📄 Figma Figma is a collaborative web application for interface design.  ](/docs/integrations/document_loaders/figma) [ ##📄 Geopandas Geopandas is an open-source project to make working with geospatial data in python easier.  ](/docs/integrations/document_loaders/geopandas) [ ##📄 Git Git is a distributed version control system that tracks changes in any set of computer files, usually used for coordinating work among programmers collaboratively developing source code during software development.  ](/docs/integrations/document_loaders/git) [ ##📄 GitBook GitBook is a modern documentation platform where teams can document everything from products to internal knowledge bases and APIs.  ](/docs/integrations/document_loaders/gitbook) [ ##📄 GitHub This notebooks shows how you can load issues and pull requests (PRs) for a given repository on GitHub. We will use the LangChain Python repository as an example.  ](/docs/integrations/document_loaders/github) [ ##📄 Google BigQuery Google BigQuery is a serverless and cost-effective enterprise data warehouse that works across clouds and scales with your data.  ](/docs/integrations/document_loaders/google_bigquery) [ ##📄 Google Cloud Storage Directory Google Cloud Storage is a managed service for storing unstructured data.  ](/docs/integrations/document_loaders/google_cloud_storage_directory) [ ##📄 Google Cloud Storage File Google Cloud Storage is a managed service for storing unstructured data.  ](/docs/integrations/document_loaders/google_cloud_storage_file) [ ##📄 Google Drive Google Drive is a file storage and synchronization service developed by Google.  ](/docs/integrations/document_loaders/google_drive) [ ##📄 Google Speech-to-Text Audio Transcripts The GoogleSpeechToTextLoader allows to transcribe audio files with the Google Cloud Speech-to-Text API and loads the transcribed text into documents.  ](/docs/integrations/document_loaders/google_speech_to_text) [ ##📄 Grobid GROBID is a machine learning library for extracting, parsing, and re-structuring raw documents.  ](/docs/integrations/document_loaders/grobid) [ ##📄 Gutenberg Project Gutenberg is an online library of free eBooks.  ](/docs/integrations/document_loaders/gutenberg) [ ##📄 Hacker News Hacker News (sometimes abbreviated as HN) is a social news website focusing on computer science and entrepreneurship. It is run by the investment fund and startup incubator Y Combinator. In general, content that can be submitted is defined as "anything that gratifies one's intellectual curiosity."  ](/docs/integrations/document_loaders/hacker_news) [ ##📄 Huawei OBS Directory The following code demonstrates how to load objects from the Huawei OBS (Object Storage Service) as documents.  ](/docs/integrations/document_loaders/huawei_obs_directory) [ ##📄 Huawei OBS File The following code demonstrates how to load an object from the Huawei OBS (Object Storage Service) as document.  ](/docs/integrations/document_loaders/huawei_obs_file) [ ##📄 HuggingFace dataset The Hugging Face Hub is home to over 5,000 datasets in more than 100 languages that can be used for a broad range of tasks across NLP, Computer Vision, and Audio. They used for a diverse range of tasks such as translation,  ](/docs/integrations/document_loaders/hugging_face_dataset) [ ##📄 iFixit iFixit is the largest, open repair community on the web. The site contains nearly 100k repair manuals, 200k Questions & Answers on 42k devices, and all the data is licensed under CC-BY-NC-SA 3.0.  ](/docs/integrations/document_loaders/ifixit) [ ##📄 Images This covers how to load images such as JPG or PNG into a document format that we can use downstream.  ](/docs/integrations/document_loaders/image) [ ##📄 Image captions By default, the loader utilizes the pre-trained Salesforce BLIP image captioning model.  ](/docs/integrations/document_loaders/image_captions) [ ##📄 IMSDb IMSDb is the Internet Movie Script Database.  ](/docs/integrations/document_loaders/imsdb) [ ##📄 Iugu Iugu is a Brazilian services and software as a service (SaaS) company. It offers payment-processing software and application programming interfaces for e-commerce websites and mobile applications.  ](/docs/integrations/document_loaders/iugu) [ ##📄 Joplin Joplin is an open-source note-taking app. Capture your thoughts and securely access them from any device.  ](/docs/integrations/document_loaders/joplin) [ ##📄 Jupyter Notebook Jupyter Notebook (formerly IPython Notebook) is a web-based interactive computational environment for creating notebook documents.  ](/docs/integrations/document_loaders/jupyter_notebook) [ ##📄 lakeFS lakeFS provides scalable version control over the data lake, and uses Git-like semantics to create and access those versions.  ](/docs/integrations/document_loaders/lakefs) [ ##📄 LarkSuite (FeiShu) LarkSuite is an enterprise collaboration platform developed by ByteDance.  ](/docs/integrations/document_loaders/larksuite) [ ##📄 Mastodon Mastodon is a federated social media and social networking service.  ](/docs/integrations/document_loaders/mastodon) [ ##📄 MediaWiki Dump MediaWiki XML Dumps contain


Document loaders | 🦜🔗 Langchain

the content of a wiki (wiki pages with all their revisions), without the site-related data. A XML dump does not create a full backup of the wiki database, the dump does not contain user accounts, images, edit logs, etc.  ](/docs/integrations/document_loaders/mediawikidump) [ ##📄 Merge Documents Loader Merge the documents returned from a set of specified data loaders.  ](/docs/integrations/document_loaders/merge_doc) [ ##📄 mhtml MHTML is a is used both for emails but also for archived webpages. MHTML, sometimes referred as MHT, stands for MIME HTML is a single file in which entire webpage is archived. When one saves a webpage as MHTML format, this file extension will contain HTML code, images, audio files, flash animation etc.  ](/docs/integrations/document_loaders/mhtml) [ ##📄 Microsoft OneDrive Microsoft OneDrive (formerly SkyDrive) is a file hosting service operated by Microsoft.  ](/docs/integrations/document_loaders/microsoft_onedrive) [ ##📄 Microsoft PowerPoint Microsoft PowerPoint is a presentation program by Microsoft.  ](/docs/integrations/document_loaders/microsoft_powerpoint) [ ##📄 Microsoft SharePoint Microsoft SharePoint is a website-based collaboration system that uses workflow applications, “list” databases, and other web parts and security features to empower business teams to work together developed by Microsoft.  ](/docs/integrations/document_loaders/microsoft_sharepoint) [ ##📄 Microsoft Word Microsoft Word is a word processor developed by Microsoft.  ](/docs/integrations/document_loaders/microsoft_word) [ ##📄 Modern Treasury Modern Treasury simplifies complex payment operations. It is a unified platform to power products and processes that move money.  ](/docs/integrations/document_loaders/modern_treasury) [ ##📄 MongoDB MongoDB is a NoSQL , document-oriented database that supports JSON-like documents with a dynamic schema.  ](/docs/integrations/document_loaders/mongodb) [ ##📄 News URL This covers how to load HTML news articles from a list of URLs into a document format that we can use downstream.  ](/docs/integrations/document_loaders/news) [ ##📄 Notion DB 1/2 Notion is a collaboration platform with modified Markdown support that integrates kanban boards, tasks, wikis and databases. It is an all-in-one workspace for notetaking, knowledge and data management, and project and task management.  ](/docs/integrations/document_loaders/notion) [ ##📄 Notion DB 2/2 Notion is a collaboration platform with modified Markdown support that integrates kanban boards, tasks, wikis and databases. It is an all-in-one workspace for notetaking, knowledge and data management, and project and task management.  ](/docs/integrations/document_loaders/notiondb) [ ##📄 Nuclia Nuclia automatically indexes your unstructured data from any internal and external source, providing optimized search results and generative answers. It can handle video and audio transcription, image content extraction, and document parsing.  ](/docs/integrations/document_loaders/nuclia) [ ##📄 Obsidian Obsidian is a powerful and extensible knowledge base  ](/docs/integrations/document_loaders/obsidian) [ ##📄 Open Document Format (ODT) The Open Document Format for Office Applications (ODF), also known as OpenDocument, is an open file format for word processing documents, spreadsheets, presentations and graphics and using ZIP-compressed XML files. It was developed with the aim of providing an open, XML-based file format specification for office applications.  ](/docs/integrations/document_loaders/odt) [ ##📄 Microsoft OneNote This notebook covers how to load documents from OneNote.  ](/docs/integrations/document_loaders/onenote) [ ##📄 Open City Data Socrata provides an API for city open data.  ](/docs/integrations/document_loaders/open_city_data) [ ##📄 Org-mode A Org Mode document is a document editing, formatting, and organizing mode, designed for notes, planning, and authoring within the free software text editor Emacs.  ](/docs/integrations/document_loaders/org_mode) [ ##📄 Pandas DataFrame This notebook goes over how to load data from a pandas DataFrame.  ](/docs/integrations/document_loaders/pandas_dataframe) [ ##📄 Amazon Textract Amazon Textract is a machine learning (ML) service that automatically extracts text, handwriting, and data from scanned documents. It goes beyond simple optical character recognition (OCR) to identify, understand, and extract data from forms and tables. Today, many companies manually extract data from scanned documents such as PDFs, images, tables, and forms, or through simple OCR software that requires manual configuration (which often must be updated when the form changes). To overcome these manual and expensive processes, Textract uses ML to read and process any type of document, accurately extracting text, handwriting, tables, and other data with no manual effort. You can quickly automate document processing and act on the information extracted, whether you’re automating loans processing or extracting information from invoices and receipts. Textract can extract the data in minutes instead of hours or days.  ](/docs/integrations/document_loaders/pdf-amazonTextractPDFLoader) [ ##📄 Polars DataFrame This notebook goes over how to load data from a polars DataFrame.  ](/docs/integrations/document_loaders/polars_dataframe) [ ##📄 Psychic This notebook covers how to load documents from Psychic. See here for more details.  ](/docs/integrations/document_loaders/psychic) [ ##📄 PubMed PubMed® by The National Center for Biotechnology Information, National Library of Medicine comprises more than 35 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full text content from PubMed Central and publisher web sites.  ](/docs/integrations/document_loaders/pubmed) [ ##📄 PySpark This notebook goes over how to load data from a PySpark DataFrame.  ](/docs/integrations/document_loaders/pyspark_dataframe) [ ##📄 Quip Quip


Document loaders | 🦜🔗 Langchain

is a collaborative productivity software suite for mobile and Web. It allows groups of people to create and edit documents and spreadsheets as a group, typically for business purposes.  ](/docs/integrations/document_loaders/quip) [ ##📄 ReadTheDocs Documentation Read the Docs is an open-sourced free software documentation hosting platform. It generates documentation written with the Sphinx documentation generator.  ](/docs/integrations/document_loaders/readthedocs_documentation) [ ##📄 Recursive URL We may want to process load all URLs under a root directory.  ](/docs/integrations/document_loaders/recursive_url) [ ##📄 Reddit Reddit is an American social news aggregation, content rating, and discussion website.  ](/docs/integrations/document_loaders/reddit) [ ##📄 Roam ROAM is a note-taking tool for networked thought, designed to create a personal knowledge base.  ](/docs/integrations/document_loaders/roam) [ ##📄 Rockset Rockset is a real-time analytics database which enables queries on massive, semi-structured data without operational burden. With Rockset, ingested data is queryable within one second and analytical queries against that data typically execute in milliseconds. Rockset is compute optimized, making it suitable for serving high concurrency applications in the sub-100TB range (or larger than 100s of TBs with rollups).  ](/docs/integrations/document_loaders/rockset) [ ##📄 rspace This notebook shows how to use the RSpace document loader to import research notes and documents from RSpace Electronic  ](/docs/integrations/document_loaders/rspace) [ ##📄 RSS Feeds This covers how to load HTML news articles from a list of RSS feed URLs into a document format that we can use downstream.  ](/docs/integrations/document_loaders/rss) [ ##📄 RST A reStructured Text (RST) file is a file format for textual data used primarily in the Python programming language community for technical documentation.  ](/docs/integrations/document_loaders/rst) [ ##📄 Sitemap Extends from the WebBaseLoader, SitemapLoader loads a sitemap from a given URL, and then scrape and load all pages in the sitemap, returning each page as a Document.  ](/docs/integrations/document_loaders/sitemap) [ ##📄 Slack Slack is an instant messaging program.  ](/docs/integrations/document_loaders/slack) [ ##📄 Snowflake This notebooks goes over how to load documents from Snowflake  ](/docs/integrations/document_loaders/snowflake) [ ##📄 Source Code This notebook covers how to load source code files using a special approach with language parsing: each top-level function and class in the code is loaded into separate documents. Any remaining code top-level code outside the already loaded functions and classes will be loaded into a separate document.  ](/docs/integrations/document_loaders/source_code) [ ##📄 Spreedly Spreedly is a service that allows you to securely store credit cards and use them to transact against any number of payment gateways and third party APIs. It does this by simultaneously providing a card tokenization/vault service as well as a gateway and receiver integration service. Payment methods tokenized by Spreedly are stored at Spreedly, allowing you to independently store a card and then pass that card to different end points based on your business requirements.  ](/docs/integrations/document_loaders/spreedly) [ ##📄 Stripe Stripe is an Irish-American financial services and software as a service (SaaS) company. It offers payment-processing software and application programming interfaces for e-commerce websites and mobile applications.  ](/docs/integrations/document_loaders/stripe) [ ##📄 Subtitle The SubRip file format is described on the Matroska multimedia container format website as "perhaps the most basic of all subtitle formats." SubRip (SubRip Text) files are named with the extension .srt, and contain formatted lines of plain text in groups separated by a blank line. Subtitles are numbered sequentially, starting at 1. The timecode format used is hoursseconds,milliseconds with time units fixed to two zero-padded digits and fractions fixed to three zero-padded digits (0000,000). The fractional separator used is the comma, since the program was written in France.  ](/docs/integrations/document_loaders/subtitle) [ ##📄 Telegram Telegram Messenger is a globally accessible freemium, cross-platform, encrypted, cloud-based and centralized instant messaging service. The application also provides optional end-to-end encrypted chats and video calling, VoIP, file sharing and several other features.  ](/docs/integrations/document_loaders/telegram) [ ##📄 Tencent COS Directory This covers how to load document objects from a Tencent COS Directory.  ](/docs/integrations/document_loaders/tencent_cos_directory) [ ##📄 Tencent COS File This covers how to load document object from a Tencent COS File.  ](/docs/integrations/document_loaders/tencent_cos_file) [ ##📄 TensorFlow Datasets TensorFlow Datasets is a collection of datasets ready to use, with TensorFlow or other Python ML frameworks, such as Jax. All datasets are exposed as tf.data.Datasets, enabling easy-to-use and high-performance input pipelines. To get started see the guide and the list of datasets.  ](/docs/integrations/document_loaders/tensorflow_datasets) [ ##📄 2Markdown 2markdown service transforms website content into structured markdown files.  ](/docs/integrations/document_loaders/tomarkdown) [ ##📄 TOML TOML is a file format for configuration files. It is intended to be easy to read and write, and is designed to map unambiguously to a dictionary. Its specification is open-source. TOML is implemented in many programming languages. The name TOML is an acronym for "Tom's Obvious, Minimal Language" referring to its creator, Tom Preston-Werner.  ](/docs/integrations/document_loaders/toml) [ ##📄 Trello Trello is a web-based project management and collaboration tool that allows individuals and teams to organize and track their tasks and projects. It provides a vi


Document loaders | 🦜🔗 Langchain

sual interface known as a "board" where users can create lists and cards to represent their tasks and activities.  ](/docs/integrations/document_loaders/trello) [ ##📄 TSV A tab-separated values (TSV) file is a simple, text-based file format for storing tabular data.[3] Records are separated by newlines, and values within a record are separated by tab characters.  ](/docs/integrations/document_loaders/tsv) [ ##📄 Twitter Twitter is an online social media and social networking service.  ](/docs/integrations/document_loaders/twitter) [ ##📄 Unstructured File This notebook covers how to use Unstructured package to load files of many types. Unstructured currently supports loading of text files, powerpoints, html, pdfs, images, and more.  ](/docs/integrations/document_loaders/unstructured_file) [ ##📄 URL This covers how to load HTML documents from a list of URLs into a document format that we can use downstream.  ](/docs/integrations/document_loaders/url) [ ##📄 Weather OpenWeatherMap is an open-source weather service provider  ](/docs/integrations/document_loaders/weather) [ ##📄 WebBaseLoader This covers how to use WebBaseLoader to load all text from HTML webpages into a document format that we can use downstream. For more custom logic for loading webpages look at some child class examples such as IMSDbLoader, AZLyricsLoader, and CollegeConfidentialLoader  ](/docs/integrations/document_loaders/web_base) [ ##📄 WhatsApp Chat WhatsApp (also called WhatsApp Messenger) is a freeware, cross-platform, centralized instant messaging (IM) and voice-over-IP (VoIP) service. It allows users to send text and voice messages, make voice and video calls, and share images, documents, user locations, and other content.  ](/docs/integrations/document_loaders/whatsapp_chat) [ ##📄 Wikipedia Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and using a wiki-based editing system called MediaWiki. Wikipedia is the largest and most-read reference work in history.  ](/docs/integrations/document_loaders/wikipedia) [ ##📄 XML The UnstructuredXMLLoader is used to load XML files. The loader works with .xml files. The page content will be the text extracted from the XML tags.  ](/docs/integrations/document_loaders/xml) [ ##📄 Xorbits Pandas DataFrame This notebook goes over how to load data from a xorbits.pandas DataFrame.  ](/docs/integrations/document_loaders/xorbits) [ ##📄 YouTube audio Building chat or QA applications on YouTube videos is a topic of high interest.  ](/docs/integrations/document_loaders/youtube_audio) [ ##📄 YouTube transcripts YouTube is an online video sharing and social media platform created by Google.  ](/docs/integrations/document_loaders/youtube_transcript)


acreom | 🦜🔗 Langchain

Main:  #acreom [acreom](https://acreom.com) is a dev-first knowledge base with tasks running on local markdown files.  Below is an example on how to load a local acreom vault into Langchain. As the local vault in acreom is a folder of plain text .md files, the loader requires the path to the directory.  Vault files may contain some metadata which is stored as a YAML header. These values will be added to the document’s metadata if collect_metadata is set to true.  from langchain.document_loaders import AcreomLoader  loader = AcreomLoader("<path-to-acreom-vault>", collect_metadata=False)  docs = loader.load()


Airbyte CDK | 🦜🔗 Langchain

Main:  On this page #Airbyte CDK [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  A lot of source connectors are implemented using the [Airbyte CDK](https://docs.airbyte.com/connector-development/cdk-python/). This loader allows to run any of these connectors and return the data as documents.  ##Installation[](#installation) First, you need to install the airbyte-cdk python package.  #!pip install airbyte-cdk  Then, either install an existing connector from the [Airbyte Github repository](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors) or create your own connector using the [Airbyte CDK](https://docs.airbyte.io/connector-development/connector-development).  For example, to install the Github connector, run  #!pip install "source_github@git+https://github.com/airbytehq/airbyte.git@master#subdirectory=airbyte-integrations/connectors/source-github"  Some sources are also published as regular packages on PyPI  ##Example[](#example) Now you can create an AirbyteCDKLoader based on the imported source. It takes a config object that's passed to the connector. You also have to pick the stream you want to retrieve records from by name (stream_name). Check the connectors documentation page and spec definition for more information on the config object and available streams. For the Github connectors these are:  - [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json). - [https://docs.airbyte.com/integrations/sources/github/](https://docs.airbyte.com/integrations/sources/github/) from langchain.document_loaders.airbyte import AirbyteCDKLoader from source_github.source import SourceGithub  # plug in your own source here  config = {     # your github configuration     "credentials": {"api_url": "api.github.com", "personal_access_token": "<token>"},     "repository": "<repo>",     "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>", }  issues_loader = AirbyteCDKLoader(     source_class=SourceGithub, config=config, stream_name="issues" )  Now you can load documents the usual way  docs = issues_loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = issues_loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(         page_content=record.data["title"] + "\n" + (record.data["body"] or ""),         metadata=record.data,     )   issues_loader = AirbyteCDKLoader(     source_class=SourceGithub,     config=config,     stream_name="issues",     record_handler=handle_record, )  docs = issues_loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = issues_loader.last_state  # store safely  incremental_issue_loader = AirbyteCDKLoader(     source_class=SourceGithub, config=config, stream_name="issues", state=last_state )  new_docs = incremental_issue_loader.load()


Airbyte Gong | 🦜🔗 Langchain

Main:  On this page #Airbyte Gong [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Gong connector as a document loader, allowing you to load various Gong objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-gong python package.  #!pip install airbyte-source-gong  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/gong/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-gong/source_gong/spec.yaml](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-gong/source_gong/spec.yaml).  The general shape looks like this:  {   "access_key": "<access key name>",   "access_key_secret": "<access key secret>",   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>", }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteGongLoader  config = {     # your gong configuration }  loader = AirbyteGongLoader(     config=config, stream_name="calls" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To process documents, create a class inheriting from the base loader and implement the _handle_records method yourself:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteGongLoader(     config=config, record_handler=handle_record, stream_name="calls" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteGongLoader(     config=config, stream_name="calls", state=last_state )  new_docs = incremental_loader.load()


Airbyte Hubspot | 🦜🔗 Langchain

Main:  Skip to main content  🦜🔗 LangChain  Search CTRLK ComponentsDocument loadersAirbyte Hubspot On this page Airbyte Hubspot Airbyte is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Hubspot connector as a document loader, allowing you to load various Hubspot objects as documents.  Installation First, you need to install the airbyte-source-hubspot python package.  #!pip install airbyte-source-hubspot  Example Check out the Airbyte documentation page for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml.  The general shape looks like this:  {   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",   "credentials": {     "credentials_title": "Private App Credentials",     "access_token": "<access token of your private app>"   } }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteHubspotLoader  config = {     # your hubspot configuration }  loader = AirbyteHubspotLoader(     config=config, stream_name="products" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To process documents, create a class inheriting from the base loader and implement the _handle_records method yourself:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteHubspotLoader(     config=config, record_handler=handle_record, stream_name="products" ) docs = loader.load()  Incremental loads Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteHubspotLoader(     config=config, stream_name="products", state=last_state )  new_docs = incremental_loader.load()  Previous Airbyte Gong Next Airbyte JSON Community Discord Twitter GitHub Python JS/TS More Homepage Blog Copyright © 2023 LangChain, Inc.


Airbyte JSON | 🦜🔗 Langchain

Main:  #Airbyte JSON [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This covers how to load any source from Airbyte into a local JSON file that can be read in as a document  Prereqs: Have docker desktop installed  Steps:  1) Clone Airbyte from GitHub - git clone https://github.com/airbytehq/airbyte.git  2) Switch into Airbyte directory - cd airbyte  3) Start Airbyte - docker compose up  4) In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that's username airbyte and password password.  5) Setup any source you wish.  6) Set destination as Local JSON, with specified destination path - lets say /json_data. Set up manual sync.  7) Run the connection.  7) To see what files are create, you can navigate to: file:///tmp/airbyte_local  8) Find your data and copy path. That path should be saved in the file variable below. It should start with /tmp/airbyte_local  from langchain.document_loaders import AirbyteJSONLoader  ls /tmp/airbyte_local/json_data/      _airbyte_raw_pokemon.jsonl  loader = AirbyteJSONLoader("/tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl")  data = loader.load()  print(data[0].page_content[:500])      abilities:      ability:      name: blaze     url: https://pokeapi.co/api/v2/ability/66/          is_hidden: False     slot: 1               ability:      name: solar-power     url: https://pokeapi.co/api/v2/ability/94/          is_hidden: True     slot: 3          base_experience: 267     forms:      name: charizard     url: https://pokeapi.co/api/v2/pokemon-form/6/          game_indices:      game_index: 180     version:      name: red     url: https://pokeapi.co/api/v2/version/1/                    game_index: 180     version:      name: blue     url: https://pokeapi.co/api/v2/version/2/                    game_index: 180     version:      n


Airbyte Salesforce | 🦜🔗 Langchain

Main:  On this page #Airbyte Salesforce [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Salesforce connector as a document loader, allowing you to load various Salesforce objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-salesforce python package.  #!pip install airbyte-source-salesforce  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/salesforce/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml).  The general shape looks like this:  {   "client_id": "<oauth client id>",   "client_secret": "<oauth client secret>",   "refresh_token": "<oauth refresh token>",   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",   "is_sandbox": False, # set to True if you're using a sandbox environment   "streams_criteria": [ # Array of filters for salesforce objects that should be loadable     {"criteria": "exacts", "value": "Account"}, # Exact name of salesforce object     {"criteria": "starts with", "value": "Asset"}, # Prefix of the name     # Other allowed criteria: ends with, contains, starts not with, ends not with, not contains, not exacts   ], }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteSalesforceLoader  config = {     # your salesforce configuration }  loader = AirbyteSalesforceLoader(     config=config, stream_name="asset" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteSalesforceLoader(     config=config, record_handler=handle_record, stream_name="asset" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteSalesforceLoader(     config=config, stream_name="asset", state=last_state )  new_docs = incremental_loader.load()


Airbyte Shopify | 🦜🔗 Langchain

Main:  On this page #Airbyte Shopify [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Shopify connector as a document loader, allowing you to load various Shopify objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-shopify python package.  #!pip install airbyte-source-shopify  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/shopify/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-shopify/source_shopify/spec.json](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-shopify/source_shopify/spec.json).  The general shape looks like this:  {     "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",     "shop": "<name of the shop you want to retrieve documents from>",     "credentials": {         "auth_method": "api_password",         "api_password": "<your api password>"     } }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteShopifyLoader  config = {     # your shopify configuration }  loader = AirbyteShopifyLoader(     config=config, stream_name="orders" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteShopifyLoader(     config=config, record_handler=handle_record, stream_name="orders" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteShopifyLoader(     config=config, stream_name="orders", state=last_state )  new_docs = incremental_loader.load()


Airbyte Stripe | 🦜🔗 Langchain

Main:  On this page #Airbyte Stripe [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Stripe connector as a document loader, allowing you to load various Stripe objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-stripe python package.  #!pip install airbyte-source-stripe  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/stripe/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml).  The general shape looks like this:  {   "client_secret": "<secret key>",   "account_id": "<account id>",   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>", }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteStripeLoader  config = {     # your stripe configuration }  loader = AirbyteStripeLoader(     config=config, stream_name="invoices" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteStripeLoader(     config=config, record_handler=handle_record, stream_name="invoices" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteStripeLoader(     config=config,     record_handler=handle_record,     stream_name="invoices",     state=last_state, )  new_docs = incremental_loader.load()


Airbyte Typeform | 🦜🔗 Langchain

Main:  On this page #Airbyte Typeform [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Typeform connector as a document loader, allowing you to load various Typeform objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-typeform python package.  #!pip install airbyte-source-typeform  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/typeform/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json).  The general shape looks like this:  {   "credentials": {     "auth_type": "Private Token",     "access_token": "<your auth token>"   },   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",   "form_ids": ["<id of form to load records for>"] # if omitted, records from all forms will be loaded }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteTypeformLoader  config = {     # your typeform configuration }  loader = AirbyteTypeformLoader(     config=config, stream_name="forms" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteTypeformLoader(     config=config, record_handler=handle_record, stream_name="forms" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteTypeformLoader(     config=config, record_handler=handle_record, stream_name="forms", state=last_state )  new_docs = incremental_loader.load()


Airbyte Zendesk Support | 🦜🔗 Langchain

Main:  On this page #Airbyte Zendesk Support [Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.  This loader exposes the Zendesk Support connector as a document loader, allowing you to load various objects as documents.  ##Installation[](#installation) First, you need to install the airbyte-source-zendesk-support python package.  #!pip install airbyte-source-zendesk-support  ##Example[](#example) Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/zendesk-support/) for details about how to configure the reader. The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/spec.json](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/spec.json).  The general shape looks like this:  {   "subdomain": "<your zendesk subdomain>",   "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",   "credentials": {     "credentials": "api_token",     "email": "<your email>",     "api_token": "<your api token>"   } }  By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader.  from langchain.document_loaders.airbyte import AirbyteZendeskSupportLoader  config = {     # your zendesk-support configuration }  loader = AirbyteZendeskSupportLoader(     config=config, stream_name="tickets" )  # check the documentation linked above for a list of all streams  Now you can load documents the usual way  docs = loader.load()  As load returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the lazy_load method which returns an iterator instead:  docs_iterator = loader.lazy_load()  Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:  from langchain.docstore.document import Document   def handle_record(record, id):     return Document(page_content=record.data["title"], metadata=record.data)   loader = AirbyteZendeskSupportLoader(     config=config, record_handler=handle_record, stream_name="tickets" ) docs = loader.load()  ##Incremental loads[](#incremental-loads) Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.  To take advantage of this, store the last_state property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded.  last_state = loader.last_state  # store safely  incremental_loader = AirbyteZendeskSupportLoader(     config=config, stream_name="tickets", state=last_state )  new_docs = incremental_loader.load()


Airtable | 🦜🔗 Langchain

Main:  #Airtable pip install pyairtable  from langchain.document_loaders import AirtableLoader  - Get your API key [here](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens). - Get ID of your base [here](https://airtable.com/developers/web/api/introduction). - Get your table ID from the table url as shown [here](https://www.highviewapps.com/kb/where-can-i-find-the-airtable-base-id-and-table-id/#:~:text=Both%20the%20Airtable%20Base%20ID,URL%20that%20begins%20with%20tbl). api_key = "xxx" base_id = "xxx" table_id = "xxx"  loader = AirtableLoader(api_key, table_id, base_id) docs = loader.load()  Returns each table row as dict.  len(docs)      3  eval(docs[0].page_content)      {'id': 'recF3GbGZCuh9sXIQ',      'createdTime': '2023-06-09T04:47:21.000Z',      'fields': {'Priority': 'High',       'Status': 'In progress',       'Name': 'Document Splitters'}}


Alibaba Cloud MaxCompute | 🦜🔗 Langchain

Main:  On this page #Alibaba Cloud MaxCompute [Alibaba Cloud MaxCompute](https://www.alibabacloud.com/product/maxcompute) (previously known as ODPS) is a general purpose, fully managed, multi-tenancy data processing platform for large-scale data warehousing. MaxCompute supports various data importing solutions and distributed computing models, enabling users to effectively query massive datasets, reduce production costs, and ensure data security.  The MaxComputeLoader lets you execute a MaxCompute SQL query and loads the results as one document per row.  pip install pyodps      Collecting pyodps       Downloading pyodps-0.11.4.post0-cp39-cp39-macosx_10_9_universal2.whl (2.0 MB)          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 1.7 MB/s eta 0:00:0000:0100:010m     Requirement already satisfied: charset-normalizer>=2 in /Users/newboy/anaconda3/envs/langchain/lib/python3.9/site-packages (from pyodps) (3.1.0)     Requirement already satisfied: urllib3<2.0,>=1.26.0 in /Users/newboy/anaconda3/envs/langchain/lib/python3.9/site-packages (from pyodps) (1.26.15)     Requirement already satisfied: idna>=2.5 in /Users/newboy/anaconda3/envs/langchain/lib/python3.9/site-packages (from pyodps) (3.4)     Requirement already satisfied: certifi>=2017.4.17 in /Users/newboy/anaconda3/envs/langchain/lib/python3.9/site-packages (from pyodps) (2023.5.7)     Installing collected packages: pyodps     Successfully installed pyodps-0.11.4.post0  ##Basic Usage[](#basic-usage) To instantiate the loader you'll need a SQL query to execute, your MaxCompute endpoint and project name, and you access ID and secret access key. The access ID and secret access key can either be passed in direct via the access_id and secret_access_key parameters or they can be set as environment variables MAX_COMPUTE_ACCESS_ID and MAX_COMPUTE_SECRET_ACCESS_KEY.  from langchain.document_loaders import MaxComputeLoader  base_query = """ SELECT * FROM (     SELECT 1 AS id, 'content1' AS content, 'meta_info1' AS meta_info     UNION ALL     SELECT 2 AS id, 'content2' AS content, 'meta_info2' AS meta_info     UNION ALL     SELECT 3 AS id, 'content3' AS content, 'meta_info3' AS meta_info ) mydata; """  endpoint = "<ENDPOINT>" project = "<PROJECT>" ACCESS_ID = "<ACCESS ID>" SECRET_ACCESS_KEY = "<SECRET ACCESS KEY>"  loader = MaxComputeLoader.from_params(     base_query,     endpoint,     project,     access_id=ACCESS_ID,     secret_access_key=SECRET_ACCESS_KEY, ) data = loader.load()  print(data)      [Document(page_content='id: 1\ncontent: content1\nmeta_info: meta_info1', metadata={}), Document(page_content='id: 2\ncontent: content2\nmeta_info: meta_info2', metadata={}), Document(page_content='id: 3\ncontent: content3\nmeta_info: meta_info3', metadata={})]  print(data[0].page_content)      id: 1     content: content1     meta_info: meta_info1  print(data[0].metadata)      {}  ##Specifying Which Columns are Content vs Metadata[](#specifying-which-columns-are-content-vs-metadata) You can configure which subset of columns should be loaded as the contents of the Document and which as the metadata using the page_content_columns and metadata_columns parameters.  loader = MaxComputeLoader.from_params(     base_query,     endpoint,     project,     page_content_columns=["content"],  # Specify Document page content     metadata_columns=["id", "meta_info"],  # Specify Document metadata     access_id=ACCESS_ID,     secret_access_key=SECRET_ACCESS_KEY, ) data = loader.load()  print(data[0].page_content)      content: content1  print(data[0].metadata)      {'id': 1, 'meta_info': 'meta_info1'}


Apify Dataset | 🦜🔗 Langchain

Main:  On this page #Apify Dataset [Apify Dataset](https://docs.apify.com/platform/storage/dataset) is a scalable append-only storage with sequential access built for storing structured web scraping results, such as a list of products or Google SERPs, and then export them to various formats like JSON, CSV, or Excel. Datasets are mainly used to save results of [Apify Actors](https://apify.com/store)—serverless cloud programs for various web scraping, crawling, and data extraction use cases.  This notebook shows how to load Apify datasets to LangChain.  ##Prerequisites[](#prerequisites) You need to have an existing dataset on the Apify platform. If you don't have one, please first check out [this notebook](/docs/integrations/tools/apify) on how to use Apify to extract content from documentation, knowledge bases, help centers, or blogs.  #!pip install apify-client  First, import ApifyDatasetLoader into your source code:  from langchain.document_loaders import ApifyDatasetLoader from langchain.document_loaders.base import Document  Then provide a function that maps Apify dataset record fields to LangChain Document format.  For example, if your dataset items are structured like this:  {     "url": "https://apify.com",     "text": "Apify is the best web scraping and automation platform." }  The mapping function in the code below will convert them to LangChain Document format, so that you can use them further with any LLM model (e.g. for question answering).  loader = ApifyDatasetLoader(     dataset_id="your-dataset-id",     dataset_mapping_function=lambda dataset_item: Document(         page_content=dataset_item["text"], metadata={"source": dataset_item["url"]}     ), )  data = loader.load()  ##An example with question answering[](#an-example-with-question-answering) In this example, we use data from a dataset to answer a question.  from langchain.docstore.document import Document from langchain.document_loaders import ApifyDatasetLoader from langchain.indexes import VectorstoreIndexCreator  loader = ApifyDatasetLoader(     dataset_id="your-dataset-id",     dataset_mapping_function=lambda item: Document(         page_content=item["text"] or "", metadata={"source": item["url"]}     ), )  index = VectorstoreIndexCreator().from_loaders([loader])  query = "What is Apify?" result = index.query_with_sources(query)  print(result["answer"]) print(result["sources"])       Apify is a platform for developing, running, and sharing serverless cloud programs. It enables users to create web scraping and automation tools and publish them on the Apify platform.          https://docs.apify.com/platform/actors, https://docs.apify.com/platform/actors/running/actors-in-store, https://docs.apify.com/platform/security, https://docs.apify.com/platform/actors/examples


ArcGIS | 🦜🔗 Langchain

Main:  On this page #ArcGIS This notebook demonstrates the use of the langchain.document_loaders.ArcGISLoader class.  You will need to install the ArcGIS API for Python arcgis and, optionally, bs4.BeautifulSoup.  You can use an arcgis.gis.GIS object for authenticated data loading, or leave it blank to access public data.  from langchain.document_loaders import ArcGISLoader  url = "https://maps1.vcgov.org/arcgis/rest/services/Beaches/MapServer/7" loader = ArcGISLoader(url)  docs = loader.load()      CPU times: user 2.37 ms, sys: 5.83 ms, total: 8.19 ms     Wall time: 1.05 s  docs[0].metadata      {'accessed': '2023-09-13T19:58:32.546576+00:00Z',      'name': 'Beach Ramps',      'url': 'https://maps1.vcgov.org/arcgis/rest/services/Beaches/MapServer/7',      'layer_description': '(Not Provided)',      'item_description': '(Not Provided)',      'layer_properties': {        "currentVersion": 10.81,        "id": 7,        "name": "Beach Ramps",        "type": "Feature Layer",        "description": "",        "geometryType": "esriGeometryPoint",        "sourceSpatialReference": {          "wkid": 2881,          "latestWkid": 2881        },        "copyrightText": "",        "parentLayer": null,        "subLayers": [],        "minScale": 750000,        "maxScale": 0,        "drawingInfo": {          "renderer": {            "type": "simple",            "symbol": {              "type": "esriPMS",              "url": "9bb2e5ca499bb68aa3ee0d4e1ecc3849",              "imageData": "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IB2cksfwAAAAlwSFlzAAAOxAAADsQBlSsOGwAAAJJJREFUOI3NkDEKg0AQRZ9kkSnSGBshR7DJqdJYeg7BMpcS0uQWQsqoCLExkcUJzGqT38zw2fcY1rEzbp7vjXz0EXC7gBxs1ABcG/8CYkCcDqwyLqsV+RlV0I/w7PzuJBArr1VB20H58Ls6h+xoFITkTwWpQJX7XSIBAnFwVj7MLAjJV/AC6G3QoAmK+74Lom04THTBEp/HCSc6AAAAAElFTkSuQmCC",              "contentType": "image/png",              "width": 12,              "height": 12,              "angle": 0,              "xoffset": 0,              "yoffset": 0            },            "label": "",            "description": ""          },          "transparency": 0,          "labelingInfo": null        },        "defaultVisibility": true,        "extent": {          "xmin": -81.09480168806815,          "ymin": 28.858349245353473,          "xmax": -80.77512908572814,          "ymax": 29.41078388840041,          "spatialReference": {            "wkid": 4326,            "latestWkid": 4326          }        },        "hasAttachments": false,        "htmlPopupType": "esriServerHTMLPopupTypeNone",        "displayField": "AccessName",        "typeIdField": null,        "subtypeFieldName": null,        "subtypeField": null,        "defaultSubtypeCode": null,        "fields": [          {            "name": "OBJECTID",            "type": "esriFieldTypeOID",            "alias": "OBJECTID",            "domain": null          },          {            "name": "Shape",            "type": "esriFieldTypeGeometry",            "alias": "Shape",            "domain": null          },          {            "name": "AccessName",            "type": "esriFieldTypeString",            "alias": "AccessName",            "length": 40,            "domain": null          },          {            "name": "AccessID",            "type": "esriFieldTypeString",            "alias": "AccessID",            "length": 50,            "domain": null          },          {            "name": "AccessType",            "type": "esriFieldTypeString",            "alias": "AccessType",            "length": 25,            "domain": null          },          {            "name": "GeneralLoc",            "type": "esriFieldTypeString",            "alias": "GeneralLoc",            "length": 100,            "domain": null          },          {            "name": "MilePost",            "type": "esriFieldTypeDouble",            "alias": "MilePost",            "domain": null          },          {            "name": "City",            "type": "esriFieldTypeString",            "alias": "City",            "length": 50,            "domain": null          },          {            "name": "AccessStatus",            "type": "esriFieldTypeString",            "alias": "AccessStatus",            "length": 50,            "domain": null          },          {            "name": "Entry_Date_Time",            "type": "esriFieldTypeDate",            "alias": "Entry_Date_Time",            "length": 8,            "domain": null          },          {            "name": "DrivingZone",            "type": "esriFieldTypeString",            "alias": "DrivingZone",            "length": 50,            "domain": null          }        ],        "geometryField": {          "name": "Shape",          "type": "esriFieldTypeGeometry",          "alias": "Shape"        },        "indexes": null,        "subtypes": [],        "relationships": [],        "canModifyLayer": true,        "canScaleSymbols": false,        "hasLabels": false,        "capabilities": "Map,Query,Data",        "maxRecordCount": 1000,        "supportsStatistics": true,        "supportsAdvancedQueries": true,        "supportedQueryFormats": "JSON, geoJSON",        "isDataVersioned": false,        "ownershipBasedAccessControlForFeatures": {          "allowOthersToQuery": true        },        "useStandardizedQueries": true,        "advancedQueryCapabilities": {          "useStandardizedQueries": true,          "supportsStatistics": true,          "supportsHavingClause": true,          "supportsCountDistinct": true,          "supportsOrderBy": true,          "supportsDistinct": true,          "supportsPagination": true,          "supportsTrueCurve": true,          "supportsReturningQueryExtent": true,          "supportsQueryWithDistance": true,          "supportsSqlExpression": true        },        "supportsDatumTransformation": true,        "dateFieldsTimeReference": null,        "supportsCoordinatesQuantization": true      }}  ###Retrieving Geometries[](#retrieving-geometries) If you want to retrieve feature geome


ArcGIS | 🦜🔗 Langchain

tries, you may do so with the return_geometry keyword.  Each document's geometry will be stored in its metadata dictionary.  loader_geom = ArcGISLoader(url, return_geometry=True)  docs = loader_geom.load()      CPU times: user 9.6 ms, sys: 5.84 ms, total: 15.4 ms     Wall time: 1.06 s  docs[0].metadata["geometry"]      {'x': -81.01508803280349,      'y': 29.24246579525828,      'spatialReference': {'wkid': 4326, 'latestWkid': 4326}}  for doc in docs:     print(doc.page_content)      {"OBJECTID": 4, "AccessName": "UNIVERSITY BLVD", "AccessID": "DB-048", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "900 BLK N ATLANTIC AV", "MilePost": 13.74, "City": "DAYTONA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694597536000, "DrivingZone": "BOTH"}     {"OBJECTID": 18, "AccessName": "BEACHWAY AV", "AccessID": "NS-106", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "1400 N ATLANTIC AV", "MilePost": 1.57, "City": "NEW SMYRNA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694600478000, "DrivingZone": "YES"}     {"OBJECTID": 24, "AccessName": "27TH AV", "AccessID": "NS-141", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "3600 BLK S ATLANTIC AV", "MilePost": 4.83, "City": "NEW SMYRNA BEACH", "AccessStatus": "CLOSED FOR HIGH TIDE", "Entry_Date_Time": 1694619363000, "DrivingZone": "BOTH"}     {"OBJECTID": 26, "AccessName": "SEABREEZE BLVD", "AccessID": "DB-051", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "500 BLK N ATLANTIC AV", "MilePost": 14.24, "City": "DAYTONA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694597536000, "DrivingZone": "BOTH"}     {"OBJECTID": 30, "AccessName": "INTERNATIONAL SPEEDWAY BLVD", "AccessID": "DB-059", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "300 BLK S ATLANTIC AV", "MilePost": 15.27, "City": "DAYTONA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694598638000, "DrivingZone": "BOTH"}     {"OBJECTID": 33, "AccessName": "GRANADA BLVD", "AccessID": "OB-030", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "20 BLK OCEAN SHORE BLVD", "MilePost": 10.02, "City": "ORMOND BEACH", "AccessStatus": "4X4 ONLY", "Entry_Date_Time": 1694595424000, "DrivingZone": "BOTH"}     {"OBJECTID": 39, "AccessName": "BEACH ST", "AccessID": "PI-097", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "4890 BLK S ATLANTIC AV", "MilePost": 25.85, "City": "PONCE INLET", "AccessStatus": "4X4 ONLY", "Entry_Date_Time": 1694596294000, "DrivingZone": "BOTH"}     {"OBJECTID": 44, "AccessName": "SILVER BEACH AV", "AccessID": "DB-064", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "1000 BLK S ATLANTIC AV", "MilePost": 15.98, "City": "DAYTONA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694598638000, "DrivingZone": "YES"}     {"OBJECTID": 45, "AccessName": "BOTEFUHR AV", "AccessID": "DBS-067", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "1900 BLK S ATLANTIC AV", "MilePost": 16.68, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694598638000, "DrivingZone": "YES"}     {"OBJECTID": 46, "AccessName": "MINERVA RD", "AccessID": "DBS-069", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "2300 BLK S ATLANTIC AV", "MilePost": 17.52, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694598638000, "DrivingZone": "YES"}     {"OBJECTID": 56, "AccessName": "3RD AV", "AccessID": "NS-118", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "1200 BLK HILL ST", "MilePost": 3.25, "City": "NEW SMYRNA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694600478000, "DrivingZone": "YES"}     {"OBJECTID": 65, "AccessName": "MILSAP RD", "AccessID": "OB-037", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "700 BLK S ATLANTIC AV", "MilePost": 11.52, "City": "ORMOND BEACH", "AccessStatus": "4X4 ONLY", "Entry_Date_Time": 1694595749000, "DrivingZone": "YES"}     {"OBJECTID": 72, "AccessName": "ROCKEFELLER DR", "AccessID": "OB-034", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "400 BLK S ATLANTIC AV", "MilePost": 10.9, "City": "ORMOND BEACH", "AccessStatus": "CLOSED - SEASONAL", "Entry_Date_Time": 1694591351000, "DrivingZone": "YES"}     {"OBJECTID": 74, "AccessName": "DUNLAWTON BLVD", "AccessID": "DBS-078", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "3400 BLK S ATLANTIC AV", "MilePost": 20.61, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694601124000, "DrivingZone": "YES"}     {"OBJECTID": 77, "AccessName": "EMILIA AV", "AccessID": "DBS-082", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "3790 BLK S ATLANTIC AV", "MilePost": 21.38, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694601124000, "DrivingZone": "BOTH"}     {"OBJECTID": 84, "AccessName": "VAN AV", "AccessID": "DBS-075", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "3100 BLK S ATLANTIC AV", "MilePost": 19.6, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694601124000, "DrivingZone": "YES"}     {"OBJECTID": 104, "AccessName": "HARVARD DR", "AccessID": "OB-038", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "900 BLK S ATLANTIC AV", "MilePost": 11.72, "City": "ORMOND BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694597536000, "DrivingZone": "YES"}     {"OBJECTID": 106, "AccessName": "WILLIAMS AV", "AccessID": "DB-042", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "2200 BLK N ATLANTIC AV", "MilePost": 12.5, "City": "DAYTONA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694597536000, "DrivingZone": "YES"}     {"OBJECTID": 109, "AccessName": "HARTFORD AV", "AccessID": "DB-043", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "1890 BLK N ATLANTIC AV", "MilePost": 12.76, "City": "DAYTONA BEACH", "AccessStatus": "CLOSED - SEASONAL", "Entry_Date_Time": 1694591351000, "DrivingZone": "YES"}     {"OBJECTID": 138, "AccessName": "CRAWFORD RD", "AccessID": "NS-108", "AccessType": "OPEN VEHICLE RAMP - PASS", "GeneralLoc": "800 BLK N ATLANTIC AV", "MilePost": 2.19, "City": "NEW SMYRNA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694600478000, "DrivingZone"


ArcGIS | 🦜🔗 Langchain

: "YES"}     {"OBJECTID": 140, "AccessName": "FLAGLER AV", "AccessID": "NS-110", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "500 BLK FLAGLER AV", "MilePost": 2.57, "City": "NEW SMYRNA BEACH", "AccessStatus": "OPEN", "Entry_Date_Time": 1694600478000, "DrivingZone": "YES"}     {"OBJECTID": 144, "AccessName": "CARDINAL DR", "AccessID": "OB-036", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "600 BLK S ATLANTIC AV", "MilePost": 11.27, "City": "ORMOND BEACH", "AccessStatus": "4X4 ONLY", "Entry_Date_Time": 1694595749000, "DrivingZone": "YES"}     {"OBJECTID": 174, "AccessName": "EL PORTAL ST", "AccessID": "DBS-076", "AccessType": "OPEN VEHICLE RAMP", "GeneralLoc": "3200 BLK S ATLANTIC AV", "MilePost": 20.04, "City": "DAYTONA BEACH SHORES", "AccessStatus": "OPEN", "Entry_Date_Time": 1694601124000, "DrivingZone": "YES"}


Arxiv | 🦜🔗 Langchain

Main:  On this page #Arxiv [arXiv](https://arxiv.org/) is an open-access archive for 2 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.  This notebook shows how to load scientific articles from Arxiv.org into a document format that we can use downstream.  ##Installation[](#installation) First, you need to install arxiv python package.  #!pip install arxiv  Second, you need to install PyMuPDF python package which transforms PDF files downloaded from the arxiv.org site into the text format.  #!pip install pymupdf  ##Examples[](#examples) ArxivLoader has these arguments:  - query: free text which used to find documents in the Arxiv - optional load_max_docs: default=100. Use it to limit number of downloaded documents. It takes time to download all 100 documents, so use a small number for experiments. - optional load_all_available_meta: default=False. By default only the most important fields downloaded: Published (date when document was published/last updated), Title, Authors, Summary. If True, other fields also downloaded. from langchain.document_loaders import ArxivLoader  docs = ArxivLoader(query="1605.08386", load_max_docs=2).load() len(docs)  docs[0].metadata  # meta-information of the Document      {'Published': '2016-05-26',      'Title': 'Heat-bath random walks with Markov bases',      'Authors': 'Caprice Stanley, Tobias Windisch',      'Summary': 'Graphs on lattice points are studied whose edges come from a finite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on\nfibers of a fixed integer matrix can be bounded from above by a constant. We\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\nalso state explicit conditions on the set of moves so that the heat-bath random\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\ndimension.'}  docs[0].page_content[:400]  # all pages of the Document content      'arXiv:1605.08386v1  [math.CO]  26 May 2016\nHEAT-BATH RANDOM WALKS WITH MARKOV BASES\nCAPRICE STANLEY AND TOBIAS WINDISCH\nAbstract. Graphs on lattice points are studied whose edges come from a ﬁnite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on ﬁbers of a\nﬁxed integer matrix can be bounded from above by a constant. We then study the mixing\nbehaviour of heat-b'


AssemblyAI Audio Transcripts | 🦜🔗 Langchain

Main:  On this page #AssemblyAI Audio Transcripts The AssemblyAIAudioTranscriptLoader allows to transcribe audio files with the [AssemblyAI API](https://www.assemblyai.com) and loads the transcribed text into documents.  To use it, you should have the assemblyai python package installed, and the environment variable ASSEMBLYAI_API_KEY set with your API key. Alternatively, the API key can also be passed as an argument.  More info about AssemblyAI:  - [Website](https://www.assemblyai.com/) - [Get a Free API key](https://www.assemblyai.com/dashboard/signup) - [AssemblyAI API Docs](https://www.assemblyai.com/docs) ##Installation[](#installation) First, you need to install the assemblyai python package.  You can find more info about it inside the [assemblyai-python-sdk GitHub repo](https://github.com/AssemblyAI/assemblyai-python-sdk).  #!pip install assemblyai  ##Example[](#example) The AssemblyAIAudioTranscriptLoader needs at least the file_path argument. Audio files can be specified as an URL or a local file path.  from langchain.document_loaders import AssemblyAIAudioTranscriptLoader  audio_file = "https://storage.googleapis.com/aai-docs-samples/nbc.mp3" # or a local file path: audio_file = "./nbc.mp3"  loader = AssemblyAIAudioTranscriptLoader(file_path=audio_file)  docs = loader.load()  Note: Calling loader.load() blocks until the transcription is finished.  The transcribed text is available in the page_content:  docs[0].page_content  "Load time, a new president and new congressional makeup. Same old ..."  The metadata contains the full JSON response with more meta information:  docs[0].metadata  {'language_code': <LanguageCode.en_us: 'en_us'>,  'audio_url': 'https://storage.googleapis.com/aai-docs-samples/nbc.mp3',  'punctuate': True,  'format_text': True,   ... }  ##Transcript Formats[](#transcript-formats) You can specify the transcript_format argument for different formats.  Depending on the format, one or more documents are returned. These are the different TranscriptFormat options:  - TEXT: One document with the transcription text - SENTENCES: Multiple documents, splits the transcription by each sentence - PARAGRAPHS: Multiple documents, splits the transcription by each paragraph - SUBTITLES_SRT: One document with the transcript exported in SRT subtitles format - SUBTITLES_VTT: One document with the transcript exported in VTT subtitles format from langchain.document_loaders.assemblyai import TranscriptFormat  loader = AssemblyAIAudioTranscriptLoader(     file_path="./your_file.mp3",     transcript_format=TranscriptFormat.SENTENCES, )  docs = loader.load()  ##Transcription Config[](#transcription-config) You can also specify the config argument to use different audio intelligence models.  Visit the [AssemblyAI API Documentation](https://www.assemblyai.com/docs) to get an overview of all available models!  import assemblyai as aai  config = aai.TranscriptionConfig(     speaker_labels=True, auto_chapters=True, entity_detection=True )  loader = AssemblyAIAudioTranscriptLoader(file_path="./your_file.mp3", config=config)  ##Pass the API Key as argument[](#pass-the-api-key-as-argument) Next to setting the API key as environment variable ASSEMBLYAI_API_KEY, it is also possible to pass it as argument.  loader = AssemblyAIAudioTranscriptLoader(     file_path="./your_file.mp3", api_key="YOUR_KEY" )


Async Chromium | 🦜🔗 Langchain

Main:  #Async Chromium Chromium is one of the browsers supported by Playwright, a library used to control browser automation.  By running p.chromium.launch(headless=True), we are launching a headless instance of Chromium.  Headless mode means that the browser is running without a graphical user interface.  AsyncChromiumLoader load the page, and then we use Html2TextTransformer to trasnform to text.  pip install -q playwright beautifulsoup4  playwright install  from langchain.document_loaders import AsyncChromiumLoader  urls = ["https://www.wsj.com"] loader = AsyncChromiumLoader(urls) docs = loader.load() docs[0].page_content[0:100]      '<!DOCTYPE html><html lang="en"><head><script src="https://s0.2mdn.net/instream/video/client.js" asyn'  from langchain.document_transformers import Html2TextTransformer  html2text = Html2TextTransformer() docs_transformed = html2text.transform_documents(docs) docs_transformed[0].page_content[0:500]      "Skip to Main ContentSkip to SearchSkip to... Select * Top News * What's News *\nFeatured Stories * Retirement * Life & Arts * Hip-Hop * Sports * Video *\nEconomy * Real Estate * Sports * CMO * CIO * CFO * Risk & Compliance *\nLogistics Report * Sustainable Business * Heard on the Street * Barron’s *\nMarketWatch * Mansion Global * Penta * Opinion * Journal Reports * Sponsored\nOffers Explore Our Brands * WSJ * * * * * Barron's * * * * * MarketWatch * * *\n* * IBD # The Wall Street Journal SubscribeSig"


AsyncHtml | 🦜🔗 Langchain

Main:  #AsyncHtml AsyncHtmlLoader loads raw HTML from a list of URLs concurrently.  from langchain.document_loaders import AsyncHtmlLoader  urls = ["https://www.espn.com", "https://lilianweng.github.io/posts/2023-06-23-agent/"] loader = AsyncHtmlLoader(urls) docs = loader.load()      Fetching pages: 100%|############| 2/2 [00:00<00:00,  9.96it/s]  docs[0].page_content[1000:2000]      ' news. Stream exclusive games on ESPN+ and play fantasy sports." />\n<meta property="og:image" content="https://a1.espncdn.com/combiner/i?img=%2Fi%2Fespn%2Fespn_logos%2Fespn_red.png"/>\n<meta property="og:image:width" content="1200" />\n<meta property="og:image:height" content="630" />\n<meta property="og:type" content="website" />\n<meta name="twitter:site" content="espn" />\n<meta name="twitter:url" content="https://www.espn.com" />\n<meta name="twitter:title" content="ESPN - Serving Sports Fans. Anytime. Anywhere."/>\n<meta name="twitter:description" content="Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports." />\n<meta name="twitter:card" content="summary">\n<meta name="twitter:app:name:iphone" content="ESPN"/>\n<meta name="twitter:app:id:iphone" content="317469184"/>\n<meta name="twitter:app:name:googleplay" content="ESPN"/>\n<meta name="twitter:app:id:googleplay" content="com.espn.score_center"/>\n<meta name="title" content="ESPN - '  docs[1].page_content[1000:2000]      'al" href="https://lilianweng.github.io/posts/2023-06-23-agent/" />\n<link crossorigin="anonymous" href="/assets/css/stylesheet.min.67a6fb6e33089cb29e856bcc95d7aa39f70049a42b123105531265a0d9f1258b.css" integrity="sha256-Z6b7bjMInLKehWvMldeqOfcASaQrEjEFUxJloNnxJYs=" rel="preload stylesheet" as="style">\n<script defer crossorigin="anonymous" src="/assets/js/highlight.min.7680afc38aa6b15ddf158a4f3780b7b1f7dde7e91d26f073e6229bb7a0793c92.js" integrity="sha256-doCvw4qmsV3fFYpPN4C3sffd5+kdJvBz5iKbt6B5PJI="\n    onload="hljs.initHighlightingOnLoad();"></script>\n<link rel="icon" href="https://lilianweng.github.io/favicon_peach.ico">\n<link rel="icon" type="image/png" sizes="16x16" href="https://lilianweng.github.io/favicon-16x16.png">\n<link rel="icon" type="image/png" sizes="32x32" href="https://lilianweng.github.io/favicon-32x32.png">\n<link rel="apple-touch-icon" href="https://lilianweng.github.io/apple-touch-icon.png">\n<link rel="mask-icon" href="https://lilianweng.github.io/safari-pinned-tab.'


AWS S3 Directory | 🦜🔗 Langchain

Main:  On this page #AWS S3 Directory [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) is an object storage service  [AWS S3 Directory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html)  This covers how to load document objects from an AWS S3 Directory object.  #!pip install boto3  from langchain.document_loaders import S3DirectoryLoader  loader = S3DirectoryLoader("testing-hwc")  loader.load()  ##Specifying a prefix[](#specifying-a-prefix) You can also specify a prefix for more finegrained control over what files to load.  loader = S3DirectoryLoader("testing-hwc", prefix="fake")  loader.load()      [Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 's3://testing-hwc/fake.docx'}, lookup_index=0)]  ##Configuring the AWS Boto3 client[](#configuring-the-aws-boto3-client) You can configure the AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) client by passing named arguments when creating the S3DirectoryLoader. This is useful for instance when AWS credentials can't be set as environment variables. See the [list of parameters](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session) that can be configured.  loader = S3DirectoryLoader(     "testing-hwc", aws_access_key_id="xxxx", aws_secret_access_key="yyyy" )  loader.load()


AWS S3 File | 🦜🔗 Langchain

Main:  On this page #AWS S3 File [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) is an object storage service.  [AWS S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html)  This covers how to load document objects from an AWS S3 File object.  from langchain.document_loaders import S3FileLoader  #!pip install boto3  loader = S3FileLoader("testing-hwc", "fake.docx")  loader.load()      [Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 's3://testing-hwc/fake.docx'}, lookup_index=0)]  ##Configuring the AWS Boto3 client[](#configuring-the-aws-boto3-client) You can configure the AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) client by passing named arguments when creating the S3DirectoryLoader. This is useful for instance when AWS credentials can't be set as environment variables. See the [list of parameters](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session) that can be configured.  loader = S3FileLoader(     "testing-hwc", "fake.docx", aws_access_key_id="xxxx", aws_secret_access_key="yyyy" )  loader.load()


AZLyrics | 🦜🔗 Langchain

Main:  #AZLyrics [AZLyrics](https://www.azlyrics.com/) is a large, legal, every day growing collection of lyrics.  This covers how to load AZLyrics webpages into a document format that we can use downstream.  from langchain.document_loaders import AZLyricsLoader  loader = AZLyricsLoader("https://www.azlyrics.com/lyrics/mileycyrus/flowers.html")  data = loader.load()  data      [Document(page_content="Miley Cyrus - Flowers Lyrics | AZLyrics.com\n\r\nWe were good, we were gold\nKinda dream that can't be sold\nWe were right till we weren't\nBuilt a home and watched it burn\n\nI didn't wanna leave you\nI didn't wanna lie\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\n\nPaint my nails, cherry red\nMatch the roses that you left\nNo remorse, no regret\nI forgive every word you said\n\nI didn't wanna leave you, baby\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours, yeah\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI\n\nI didn't wanna wanna leave you\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours (Yeah)\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than\nYeah, I can love me better than you can, uh\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby (Than you can)\nCan love me better\nI can love me better, baby\nCan love me better\nI\n", lookup_str='', metadata={'source': 'https://www.azlyrics.com/lyrics/mileycyrus/flowers.html'}, lookup_index=0)]


Azure AI Data | 🦜🔗 Langchain

Main:  On this page #Azure AI Data [Azure AI Studio](https://ai.azure.com/) provides the capability to upload data assets to cloud storage and register existing data assets from the following sources:  - Microsoft OneLake - Azure Blob Storage - Azure Data Lake gen 2 The benefit of this approach over AzureBlobStorageContainerLoader and AzureBlobStorageFileLoader is that authentication is handled seamlessly to cloud storage. You can use either identity-based data access control to the data or credential-based (e.g. SAS token, account key). In the case of credential-based data access you do not need to specify secrets in your code or set up key vaults - the system handles that for you.  This notebook covers how to load document objects from a data asset in AI Studio.  #!pip install azureml-fsspec, azure-ai-generative  from azure.ai.resources.client import AIClient from azure.identity import DefaultAzureCredential from langchain.document_loaders import AzureAIDataLoader  # Create a connection to your project client = AIClient(     credential=DefaultAzureCredential(),     subscription_id="<subscription_id>",     resource_group_name="<resource_group_name>",     project_name="<project_name>", )  # get the latest version of your data asset data_asset = client.data.get(name="<data_asset_name>", label="latest")  # load the data asset loader = AzureAIDataLoader(url=data_asset.path)  loader.load()      [Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpaa9xl6ch/fake.docx'}, lookup_index=0)]  ##Specifying a glob pattern[](#specifying-a-glob-pattern) You can also specify a glob pattern for more finegrained control over what files to load. In the example below, only files with a pdf extension will be loaded.  loader = AzureAIDataLoader(url=data_asset.path, glob="*.pdf")  loader.load()      [Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpujbkzf_l/fake.docx'}, lookup_index=0)]