Researching a Multilingual FEMA Disaster Bot Using LangChain and GPT-4
Author:Murphy | View: 23353 | Time: 2025-03-23 13:01:50
TL;DR
In this article, we explore how to build a multilingual US Federal Emergency Management Agency (FEMA) disaster chatbot to help people prepare for and survive disasters such as floods, tornados, wildfires, earthquakes, and winter storms. A chat interface was created from 34 FEMA PDF documents using LangChain and GPT-4. As amazing as this popular pattern is, care is needed with high-risk applications such as Disaster Response bots. Though Large Language Model (LLM) hallucinations are minimized, common issues with semantic search of documents can lead to critical information being excluded in chat responses. We tested a few simple techniques to improve performance for this specific analysis, such as incorporating document metadata into embeddings, enriching user questions with LLM zero-shot context classification, and automatic language detection and translation using Google Translate to better support languages like Swahili. The techniques applied were pretty basic, but the ability of the prototype bot to surface information efficiently from FEMA PDF documents shows promise. Obviously, testing and validation would be needed if using this technique for high-risk situations, ideally using automated LLM techniques to create question-and-answer validation data.
A few weeks ago we had a major flooding event in Vermont. The small brook that gurgles cheerfully past our house turned into a raging monster hellbent on destruction. Luckily we weren't damaged badly, but sadly, many lost property and livelihoods. At one point during the flood, it looked like we might have to evacuate so I started checking the US Federal Emergency Management Agency (FEMA) website for advice. I had already prepared somewhat, but when trouble arrived I wanted to reconfirm a few things. FEMA has really excellent, concise resources in PDF documents and web pages which I started searching, but I wondered …
- In an emergency, is there a faster way to get helpful information that doesn't require searching and reading multiple documents?
One obvious solution might be to ask a chatbot. I think chatbots can get a bit overused at times, but this does seem like a solid use case as a conversational interface can be more efficient when time is of the essence.
It is not a new idea, organisations such as the American Red Cross have developed bots like Clara for disaster response. However, a promising new pattern has recently emerged to use Generative AI Large Language Models (LLMs) such as OpenAI's GPT-4, Meta's LLAMA 2, and a growing plethora of models on HuggingFace. These models can be used to index a specified set of documents and interact with them conversationally. Called Retrieval-Augmented Generation (RAG), there are hundreds of tutorials on the web demonstrating it with the amazing LangChain Python package and I fully expect this technique to appear soon in software we use every day. What's interesting is that it restricts responses to the provided content and so is less prone to hallucinations which can prevent using LLMs in critical situations such as disaster response.
But how safe is it in cases where the information retrieved might save lives?
In this article, I briefly explore creating a LangChain GPT-4 chat interface for asking questions about disaster safety based on a set of documents from the US Federal Emergency Management Agency (FEMA). We will encounter some of the limitations that need to be considered if using this technique in high-risk situations.
FEMA Disaster Preparation and Safety PDF Documents
For this study, I downloaded 34 PDFs from Fema (listed here) which cover a wide range of disaster-related topics for preparing and reacting to emergencies such as wildfires, tornadoes, floods, earthquakes, and winter storms. This isn't the full set of amazing resources FEMA offer, but should suffice to test our chat interface.
Indexing Documents For Information Retrieval
Once downloaded, we can use Langchain to read the PDF documents. Documents are split into chunks of text, and each is given a fingerprint (embeddings) using an embedding model. In this analysis, we will use OpenAI's embeddings, but LangChain supports many others.
import os
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.document_loaders import PyPDFLoader
files = os.listdir(pdf_folder_path)
files.sort()
all_docs_list =[]
for file in files:
if file.endswith('.pdf'):
print(file)
all_docs_list.append(file)
loader = PyPDFDirectoryLoader(pdf_folder_path)
all_docs = loader.load()
print(pdf_folder_path)cfpb_adult-fin-edyour-disaster-checklist.pdf
fema_protect-your-home_flooding.pdf
fema_protect-your-property-storm-surge.pdf
fema_protect-your-property_coastal-erosion.pdf
fema_protect-your-property_earthquakes.pdf
fema_protect-your-property_severe-wind.pdf
fema_protect-your-property_wildfire.pdf
fema_safeguard-critical-documents-and-valuables.pdf
fema_scenario_1-active_shooter-01102020.pdf
fema_scenario_10_power_outage_01102020.pdf
fema_scenario_10_power_outage_answer_key_01102020.pdf
fema_scenario_11_winter_storm_01102020.pdf
fema_scenario_11_winter_storm_answer_key_01102020.pdf
fema_scenario_12_small_business_01102020.pdf
fema_scenario_12_small_business_answer_key_01102020.pdf
fema_scenario_1_active_shooter_TTX_answer_key-01102020.pdf
fema_scenario_2-tornado_TTX_answer_key-01102020.pdf
fema_scenario_2_tornado-01102020.pdf
fema_scenario_3-wildfire_TTX_answer_key-01102020.pdf
fema_scenario_3_wildfire-01102020.pdf
fema_scenario_4-hurricane-01102020.pdf
fema_scenario_4_hurricane_flood_TTX_answer_key-01102020.pdf
fema_scenario_5_extreme_heat-01102020.pdf
fema_scenario_5_extreme_heat_TTX_answer_key_01102020.pdf
fema_scenario_6-pet_preparedness_01102020.pdf
fema_scenario_6-pet_preparedness_TTX_answer_key_01102020.pdf
fema_scenario_7-shelter_in_place_TTX_answer_key_01102020.pdf
fema_scenario_7_shelter_in_place_01102020.pdf
fema_scenario_8_earthquake_01102020.pdf
fema_scenario_8_earthquake_answer_key_01102020.pdf
fema_scenario_9_pandemic_Influenza_01102020.pdf
fema_scenario_9_pandemic_answer_key_01102020.pdf
ready_12-ways-to-prepare_postcard.pdf
ready_document-and-insure-your-property.pdfLooking at data extracted for one document https://www.fema.gov/sites/default/files/2020-11/fema_protect-your-home_flooding.pdf …
import json
for d in all_docs:
if 'fema_protect-your-home_flooding.pdf' in d.metadata['source']:
print('n')
print(json.dumps(vars(d), indent=4)){
"page_content": " n n nPROTECT YOUR nPROPERTY FROM FLOODINGn n",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 0
}
}
{
"page_content": " n n n n n n n n n n n n n n n -nu2014n- - -nOwning a property is one of the most important investments most people make nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? nFlooding is the most common and costly natural disaster in the United States nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. nWhile you canu2019t prevent a natural disaster from happening, there are ways to secure nyour property to minimize damage and keep your home and your future safe. nFirst, determine the Base Flood Elevation (BFE) for your home. The nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMAu2019s Flood Mapping and Insurance eXchangen ant FEMAMapSpecialist@ nrniskmapcds.com or (877) FEMA MAP (1 877 336 2627). nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 1
}
}
{
"page_content": " n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nINSIDE THE HOME nPREPARE OR nUPDAnTE A LIST OF nBELONGINGS Documenting all of your belongings will help with the ninsurance claims process. Consider taking photos of nhigh-value items or doing a video walkthrough of your nhonme to document its contents. nGET FLOOD nINSURANCE Most homeowners insurance npolicies donu2019t cover food damage. nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in na hingh-risk food zone. nSTORE nVALUABLES Store valuables and important documents in waterproof or wanter-resistant containers above the BFE n(preferably on an upper foor). Make copies and store them online or offsite. For more ninformation on purchasing food insurance through the National Flood nInsurance Program, visit nFlonodSmart.gov or contact nyour agent to get coverage. ELnEVATE UTILITIES nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. nREPLACE CARPETING nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. nFLOODPROOF nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. nINSTALL FLOOD nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water nto fow through, drain out, annd lower the risk of nstructural damage. nUSE FLOOD-RESISTnANT nINSULATION & DRYWALL Flnood-resistant insulation and drywall will help minimize ndamage and can be easily cleaned and sanitized. nPREVENT SEWnAGE nBACK-UP Inn some areas, fooding can cause sewage to back nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? nWhen following National Flood nInsurance Program regulations, nvnents can also help lower ninsunrance rates. n",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 2
}
}
{
"page_content": " n n n n n n n n n n n n n n n n n n n nOUTSIDE THE HOME nELEVATE YOUR nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? nElevating your home nmay reduce your food insurance premium. nSECURE YnARD nITEMS Unnsecure items can be swept away or damaged by foodwaters. nThey can also be swept into your home, causing damage. nSecure items in your yard by anchoring them or attaching them to more substantial structures. nSEAL CRAnCKS nAND GAPS Cnheck caulking around windows and doors to make it is not ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. nSET YOUR HOME OR nBUILDINGS BACK, nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 3
}
}
{
"page_content": " n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nDIRECT WATER AWAY nFROM STRUCTURES If you have a single-family home, make sure your yard nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. nANCHOR FUEL nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. nSECURE nMANUFACTURED nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: nSome of these tips may work better together than nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. nAlways consult professionals such as your insurance nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. nFinally, be kind to your neighbors! Talk to adjacent nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 4
}
}
{
"page_content": " n n n n n n n n n n n n ADDITIONAL RESOURCES nFEMA, PROTECT YOUR PROPERTY nLearn how to protect your home or business from natural disasters. fema.gov/protect-your-property nHOMEOWNERu2019S GUIDE TO RETROFITTING nfema.gov/media-library/assets/documents/480 nREDUCING FLOOD RISK TO RESIDENTIAL BUILDINGS nTHAT CANNOT BE ELEVATED nfema.gov/media-library/assets/documents/109669 nPROTECTING YOUR HOME AND PROPERTY FROM nFLOOD DAMAGE fema.gov/media-library/assets/documents/21471 nPROTECTING BUILDING UTILITY SYSTEMS FROM nFLOOD DAMAGE nfema.gov/media-library/assets/documents/3729 nPROTECT YOUR PROPERTY FROM FLOODING nfema.gov/media-library/assets/documents/13261 ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 5
}
}
{
"page_content": "",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 6
}
}We can see that it has split the document by page. As it happens this is not an unreasonable approach for the FEMA documents we are processing, which are very concise guides where each page is a discrete topic, but for other applications, it is usually better to split text at a more granular level using LangChain's text_splitter.
We can now use our text excerpts to create a database of embeddings …
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
embedding_model = OpenAIEmbeddings()
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(all_docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()We have chosen a simple option to persist the embeddings to the file system, but it's worth noting that Chroma supports more options should you have a large number of documents, where performance may be an issue.
Setting up our conversational interface
We will use just one PDF to start with so it's easier to verify results …
import shutil
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain import PromptTemplate, LLMChain
# Load OPENAI_API_KEY from a .env file
from dotenv import load_dotenv
def setup_model(
vecs_dir: str,
docs_sublist: list,
all_docs: list
) -> ConversationalRetrievalChain:
# Subset for docs we are interested in
docs = []
for d in all_docs:
d_dict = vars(d)
# Create vector DB directory
if os.path.exists(vecs_dir):
shutil.rmtree(vecs_dir)
os.makedirs(vecs_dir)
# Choose our models
embedding_model = OpenAIEmbeddings()
chat_model = ChatOpenAI(temperature=temperature,model_name="gpt-4")
# Calculate embeddings
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()
# Set up chat
memory = ConversationBufferMemory(memory_key="chat_history", input_key='question', output_key='answer', return_messages=True)
pdf_qa = ConversationalRetrievalChain.from_llm(chat_model, vectordb.as_retriever(), memory=memory,
return_source_documents=True)
return pdf_qa
vecs_dir = './vector_dbs/one_flood_doc'
docs = all_docs
# Subset to one document
docs_sublist = ['fema_protect-your-home_flooding.pdf']
pdf_qa = setup_model(vecs_dir, docs_sublist, docs)In the above, we have chosen GPT-4 for the chat model where the API key was defined in a .env file with variable OPENAI_API_KEY. LangChain offers support for many other models also.
With just a few lines of code, we have set up a conversational interface onto a set of documents, which includes all the power of LLMs. I have developed chatbots over the years and can say that this concise pattern reduces a LOT of complexity.
Kudos to the amazing LangChain package!
Asking our first question
Since we would also like to see the documents referenced in order to validate chat responses, we need to make a slight adjustment to a LangChain method in order to account for an issue when retrieving matched documents when also using chat memory (The solution was suggested here) …
# A little mod to enable using memory *and* getting docs. See: https://github.com/langchain-ai/langchain/issues/2256
import langchain
from typing import Dict, Any, Tuple
from langchain.memory.utils import get_prompt_input_key
def _get_input_output(
self, inputs: Dict[str, Any], outputs: Dict[str, str]
) -> Tuple[str, str]:
if self.input_key is None:
prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)
else:
prompt_input_key = self.input_key
if self.output_key is None:
output_key = list(outputs.keys())[0]
else:
output_key = self.output_key
return inputs[prompt_input_key], outputs[output_key]
langchain.memory.chat_memory.BaseChatMemory._get_input_output = _get_input_output
OK, now we are ready to ask a question about our one PDF document!
def ask_question(
query: str,
qa: object,
output_docs: bool = True
) -> dict:
print(f"nQuestion: n{query}")
result = qa({"question": query})
print(f"nAnswer:n{result['answer']}")
if output_docs:
for doc in result['source_documents']:
print('n')
print(json.dumps(vars(doc), indent=4))
return result
ask_question("How do I prepare my home for floods?", pdf_qa)
Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Determine the Base Flood Elevation (BFE) for your home. This is how high the water is expected to rise during flooding in high risk areas. Your local floodplain manager can help you find this information.
2. Direct water away from structures. Make sure your yard slopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels.
3. Anchor fuel tanks to prevent them from tipping over or floating in a flood.
4. Floodproof walls by adding water-resistant exterior sheathing and sealing them to prevent shallow flooding from damaging your home.
5. Secure manufactured homes to a permanent foundation to resist flotation, collapse, or side-to-side movement.
6. Inside the home, prepare or update a list of belongings for insurance claims.
7. Get flood insurance as most homeowners insurance policies don't cover flood damage.
8. Store valuables and important documents in waterproof or water-resistant containers above the BFE.
9. Elevate utilities above the BFE.
10. Replace carpeting with tiles as they are more flood-resistant.
11. Floodproof basements by sealing walls with waterproofing compounds and consider installing a sump pump.
12. Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through and drain out.
13. Use flood-resistant insulation and drywall.
14. Prevent sewage back-up by consulting with a plumber and investing in a sewer backflow valve if applicable.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.That seems very reasonable, let's see what content was used to generate this summary …
{
"page_content": " n n nPROTECT YOUR nPROPERTY FROM FLOODINGn n",
"metadata": {
"page": 0,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " n n n n n n n n n n n n n n n -nu2014n- - -nOwning a property is one of the most important investments most people make nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? nFlooding is the most common and costly natural disaster in the United States nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. nWhile you canu2019t prevent a natural disaster from happening, there are ways to secure nyour property to minimize damage and keep your home and your future safe. nFirst, determine the Base Flood Elevation (BFE) for your home. The nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMAu2019s Flood Mapping and Insurance eXchangen ant FEMAMapSpecialist@ nrniskmapcds.com or (877) FEMA MAP (1 877 336 2627). nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"page": 1,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nDIRECT WATER AWAY nFROM STRUCTURES If you have a single-family home, make sure your yard nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. nANCHOR FUEL nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. nSECURE nMANUFACTURED nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: nSome of these tips may work better together than nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. nAlways consult professionals such as your insurance nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. nFinally, be kind to your neighbors! Talk to adjacent nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nINSIDE THE HOME nPREPARE OR nUPDAnTE A LIST OF nBELONGINGS Documenting all of your belongings will help with the ninsurance claims process. Consider taking photos of nhigh-value items or doing a video walkthrough of your nhonme to document its contents. nGET FLOOD nINSURANCE Most homeowners insurance npolicies donu2019t cover food damage. nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in na hingh-risk food zone. nSTORE nVALUABLES Store valuables and important documents in waterproof or wanter-resistant containers above the BFE n(preferably on an upper foor). Make copies and store them online or offsite. For more ninformation on purchasing food insurance through the National Flood nInsurance Program, visit nFlonodSmart.gov or contact nyour agent to get coverage. ELnEVATE UTILITIES nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. nREPLACE CARPETING nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. nFLOODPROOF nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. nINSTALL FLOOD nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water nto fow through, drain out, annd lower the risk of nstructural damage. nUSE FLOOD-RESISTnANT nINSULATION & DRYWALL Flnood-resistant insulation and drywall will help minimize ndamage and can be easily cleaned and sanitized. nPREVENT SEWnAGE nBACK-UP Inn some areas, fooding can cause sewage to back nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? nWhen following National Flood nInsurance Program regulations, nvnents can also help lower ninsunrance rates. n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}The answer looks great and has summarized brilliantly the excerpts that were matched. However, it seems to have missed a page in the source document. This PDF is a short document where everything in it is relevant to flood preparation, so losing pages is actually significant.
It goes to show that blindly accepting LLM patterns on the web may give amazing-looking results, but work is needed to make them truly useful.
Document Context (Metadata) Can Be Important
For this scenario, we have a document that relates to one topic, flooding, even if individual text sections might not mention it explicitly. We might get better results if we provide some context with each text excerpt, ie some document metadata.
For our simple test, let's try prefixing all text excerpts with the filename (fema-protect-your-home-flooding.pdf) with punctuation and suffix removed, plus a text saying "This snippet relates to". The final prefix will be "This snippet relates to fema protect your home flooding:", which provides the LLM a bit more context …
def setup_model(
vecs_dir: str,
docs_sublist: list,
all_docs: list,
prefix_file_name_to_chunks: bool = False,
temperature: float = 0.0,
extra_prefix: str = '',
) -> ConversationalRetrievalChain:
# Subset for docs we are interested in
docs = []
for d in all_docs:
d_dict = vars(d)
if d_dict['metadata']['source'].replace('docs_data/','') in docs_sublist:
if len(d.page_content) > 20:
# Add file name to content for more context
if prefix_file_name_to_chunks:
file_clean = re.sub(r'docs_data/|.pdf', '', d_dict['metadata']['source'])
file_clean = re.sub(r'-|_', ' ', file_clean)
d.page_content = f"{extra_prefix} {file_clean}: {d.page_content}"
docs.append(d)
# Create vector DB directory
if os.path.exists(vecs_dir):
shutil.rmtree(vecs_dir)
os.makedirs(vecs_dir)
# Choose our models
embedding_model = OpenAIEmbeddings()
chat_model = ChatOpenAI(temperature=temperature,model_name="gpt-4")
# Calculate embeddings
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()
# Set up chat
memory = ConversationBufferMemory(memory_key="chat_history", input_key='question', output_key='answer', return_messages=True)
pdf_qa = ConversationalRetrievalChain.from_llm(chat_model, vectordb.as_retriever(), memory=memory,
return_source_documents=True)
return pdf_qa
docs = all_docs
# Note the argument prefix_file_name_to_chunks=True
pdf_qa = setup_model(vecs_dir, docs_sublist, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
ask_question("How do I prepare my home for floods?", pdf_qa)Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Direct Water Away from Structures: Make sure your yard slopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels.
2. Anchor Fuel Tanks: Anchor any fuel tanks to the pad to prevent them from tipping over or floating in a flood. Spilled fuel could become a fire hazard.
3. Floodproof Walls: Add water-resistant exterior sheathing on walls and seal them to prevent shallow flooding from damaging your home. Cover openings below the Base Flood Elevation (BFE) and seal all exterior openings around pumping and equipment.
4. Secure Manufactured Homes: If you have a manufactured home, it must be affixed to a permanent foundation so that the wheels and axles do not support its weight and resist flotation, collapse, or side-to-side movement.
5. Elevate Your Home: Elevating your home prepares your property against floods and lowers flood insurance premiums. The lowest floor should be above the BFE.
6. Secure Yard Items: Unsecure items can be swept away or damaged by floodwaters. Secure items in your yard by anchoring them or attaching them to more substantial structures.
7. Seal Cracks and Gaps: Check caulking around windows and doors to make sure it is not cracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building.
8. Prepare or Update a List of Belongings: Documenting all of your belongings will help with the insurance claims process.
9. Get Flood Insurance: Most homeowners insurance policies don't cover flood damage. Protect your investment by purchasing flood insurance for your home and contents.
10. Store Valuables: Store valuables and important documents in waterproof or water-resistant containers above the BFE.
11. Elevate Utilities Above the BFE: Elevate or floodproof mechanical units, furnaces, water heaters, electrical systems, and other utilities.
12. Replace Carpeting with Tiles: Tiles are more flood-resistant than carpet. Using tile or other flood-resistant materials in areas below the BFE can help reduce water damage.
13. Floodproof Basements: Minimize damage by floodproofing your basement and sealing walls with waterproofing compounds. Consider installing a sump pump.
14. Install Flood Vents: Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through, drain out, and lower the risk of structural damage.
15. Use Flood-Resistant Insulation & Drywall: Flood-resistant insulation and drywall will help minimize damage and can be easily cleaned and sanitized.
16. Prevent Sewage Back-Up: In some areas, flooding can cause sewage to back up through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backflow valve to prevent this potential health hazard.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.
{
"page_content": "This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nDIRECT WATER AWAY nFROM STRUCTURES If you have a single-family home, make sure your yard nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. nANCHOR FUEL nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. nSECURE nMANUFACTURED nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: nSome of these tips may work better together than nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. nAlways consult professionals such as your insurance nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. nFinally, be kind to your neighbors! Talk to adjacent nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n nOUTSIDE THE HOME nELEVATE YOUR nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? nElevating your home nmay reduce your food insurance premium. nSECURE YnARD nITEMS Unnsecure items can be swept away or damaged by foodwaters. nThey can also be swept into your home, causing damage. nSecure items in your yard by anchoring them or attaching them to more substantial structures. nSEAL CRAnCKS nAND GAPS Cnheck caulking around windows and doors to make it is not ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. nSET YOUR HOME OR nBUILDINGS BACK, nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nINSIDE THE HOME nPREPARE OR nUPDAnTE A LIST OF nBELONGINGS Documenting all of your belongings will help with the ninsurance claims process. Consider taking photos of nhigh-value items or doing a video walkthrough of your nhonme to document its contents. nGET FLOOD nINSURANCE Most homeowners insurance npolicies donu2019t cover food damage. nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in na hingh-risk food zone. nSTORE nVALUABLES Store valuables and important documents in waterproof or wanter-resistant containers above the BFE n(preferably on an upper foor). Make copies and store them online or offsite. For more ninformation on purchasing food insurance through the National Flood nInsurance Program, visit nFlonodSmart.gov or contact nyour agent to get coverage. ELnEVATE UTILITIES nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. nREPLACE CARPETING nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. nFLOODPROOF nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. nINSTALL FLOOD nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water nto fow through, drain out, annd lower the risk of nstructural damage. nUSE FLOOD-RESISTnANT nINSULATION & DRYWALL Flnood-resistant insulation and drywall will help minimize ndamage and can be easily cleaned and sanitized. nPREVENT SEWnAGE nBACK-UP Inn some areas, fooding can cause sewage to back nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? nWhen following National Flood nInsurance Program regulations, nvnents can also help lower ninsunrance rates. n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n -nu2014n- - -nOwning a property is one of the most important investments most people make nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? nFlooding is the most common and costly natural disaster in the United States nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. nWhile you canu2019t prevent a natural disaster from happening, there are ways to secure nyour property to minimize damage and keep your home and your future safe. nFirst, determine the Base Flood Elevation (BFE) for your home. The nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMAu2019s Flood Mapping and Insurance eXchangen ant FEMAMapSpecialist@ nrniskmapcds.com or (877) FEMA MAP (1 877 336 2627). nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"page": 1,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}Bingo! That does now seem to have captured the important pages from the PDF and summarized them nicely. Obviously, a very crude approach, a more formal method using metadata instead of just the filename would be better. It might also be more elegant to use a template rather than just prefixing, but it does illustrate how a little context like this can help.
What about if we now use all documents in our set …
vecs_dir = './vector_dbs/all_docs'
docs = all_docs
# Note the argument prefix_file_name_to_chunks=True
pdf_qa = setup_model(vecs_dir, all_docs_list, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
result = ask_question("How do I prepare my home for floods?", pdf_qa)Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Create an emergency plan for your family and practice it regularly. When a storm is approaching, evacuate and move your car to higher ground.
2. Purchase flood insurance for your home and its contents, even if you do not live in a high-risk flood zone.
3. Document your belongings. This will help with the insurance process if you need to file a claim.
4. Store valuables and important documents above the Base Flood Elevation (BFE) in waterproof or water-resistant containers.
5. Elevate appliances and utilities such as water heaters, washers, dryers, and electric panels on higher floors to prevent them from getting damaged by flood water.
6. Use flood-resistant materials for insulation, drywall, and floor coverings like tile to minimize damage.
7. Make sure your yard slopes away from buildings on your property and that water has a place to drain.
8. Anchor any fuel tanks to prevent them from tipping over or floating in a flood.
9. Add water-resistant exterior sheathing on walls and seal them to prevent shallow flooding from damaging your home.
10. If you have a manufactured home, make sure it is affixed to a permanent foundation.
11. Elevate your home above the BFE.
12. Secure items in your yard by anchoring them or attaching them to more substantial structures.
13. Check caulking around windows and doors to make sure it is not cracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building.
14. Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground.
15. Replace carpeting with tiles as they are more flood-resistant.
16. Floodproof your basement and seal walls with waterproofing compounds. Consider installing a sump pump.
17. Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through, drain out, and lower the risk of structural damage.
18. Use flood-resistant insulation and drywall.
19. Prevent sewage back-up by consulting with a plumber and investing in a sewer backflow valve if applicable.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.
{
"page_content": "This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: n n n n n n n n n INSIDE YOUR HOMEnHAVE A nPLANCreate an emergency plan for your family and practice it regularly. nWhen a storm is approaching, evacuate and move your car to higher ground. According to the National Weather Service , just two feet nof water can move a vehicle. Visit Ready.gov/plan to learn more.nGET FLOOD nINSURANCE Most homeowners insurance policies donu2019t cover flood damage. Protect your investment by purchasing flood insurance nfor your home and its contents. nDo so even if you do not live in na high-risk flood zone.For more information on purchasing nflood insurance through the n nNational Flood Insurance Program, visit FloodSmart.gov or contact n nyour agent to nget coverage.n n PREPARE OR nUPDATE A LIST OF n YOUR HOMEu2019S nCONTENTSDocument your belongings. This will give you peace of mind nand help with the insurance process if you need to file a claim. Consider documenting your homeu2019s contents visually. You can either take photos of high-value items or walk through your home and videotape each roomu2019s belongings.nSTORE nVALUABLESStore valuables and important documents above the BFE (preferably on an upper floor). Place them in waterproof or nwater-resistant containers. Also, make copies and store them nonline or offsite.nELEVATE nAPPLIANCES AND nUTILITIES ABOVE nTHE BFEKeep appliances and utilities such as water heaters, washers, dryers, and nelectric panels on higher nfloors. It can prevent them from getting damaged or ruined by flood water.nTalk to your floodplain nmanager about how high to elevate your utilities. Many coastal ncommunities have codes that require utilities to be elevated 12 inches or more above the BFE, called freeboard. nUSE FLOOD- nRESISTANT nMATERIALSnKNOW YOUR nPROPERTY AND nNEIGHBORHOODFlood-resistant insulation, drywall, and floor coverings like tile nwill help minimize damage and are easier to clean and sanitize.nIf you are moving near a large lake or the ocean, talk with nneighbors. Find out about any issues theyu2019ve had, or any nmitigation measures they have taken. Take the time to look nup flood information. You can find flood maps for coastal areas on the National Flood Hazard Layer .ADDITIONAL RESOURCESnFEMA, Homebuilderu2019s Guide to Coastal ConstructionnFEMA, Homeowneru2019s Guide to RetrofittingnFEMA, Recommended Residential Construction for Coastal AreasnNATIONAL HURRICANE CENTER, Storm Surge OverviewnNATIONAL HURRICANE CENTER, Storm Surge Inundation MapnREMEMBER: nSome of these tips may work better than others. Tailor nany mitigation measures to meet your propertyu2019s needs.nAlways consult professionals, such as your insurance agent, architects, nengineers, contractors, or other experts in design and construction, nbefore making changes to your home. Your local planning and zoning noffice or building department is a good place to start for advice.n",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-property-storm-surge.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nDIRECT WATER AWAY nFROM STRUCTURES If you have a single-family home, make sure your yard nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. nANCHOR FUEL nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. nSECURE nMANUFACTURED nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: nSome of these tips may work better together than nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. nAlways consult professionals such as your insurance nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. nFinally, be kind to your neighbors! Talk to adjacent nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n nOUTSIDE THE HOME nELEVATE YOUR nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? nElevating your home nmay reduce your food insurance premium. nSECURE YnARD nITEMS Unnsecure items can be swept away or damaged by foodwaters. nThey can also be swept into your home, causing damage. nSecure items in your yard by anchoring them or attaching them to more substantial structures. nSEAL CRAnCKS nAND GAPS Cnheck caulking around windows and doors to make it is not ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. nSET YOUR HOME OR nBUILDINGS BACK, nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n nINSIDE THE HOME nPREPARE OR nUPDAnTE A LIST OF nBELONGINGS Documenting all of your belongings will help with the ninsurance claims process. Consider taking photos of nhigh-value items or doing a video walkthrough of your nhonme to document its contents. nGET FLOOD nINSURANCE Most homeowners insurance npolicies donu2019t cover food damage. nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in na hingh-risk food zone. nSTORE nVALUABLES Store valuables and important documents in waterproof or wanter-resistant containers above the BFE n(preferably on an upper foor). Make copies and store them online or offsite. For more ninformation on purchasing food insurance through the National Flood nInsurance Program, visit nFlonodSmart.gov or contact nyour agent to get coverage. ELnEVATE UTILITIES nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. nREPLACE CARPETING nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. nFLOODPROOF nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. nINSTALL FLOOD nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water nto fow through, drain out, annd lower the risk of nstructural damage. nUSE FLOOD-RESISTnANT nINSULATION & DRYWALL Flnood-resistant insulation and drywall will help minimize ndamage and can be easily cleaned and sanitized. nPREVENT SEWnAGE nBACK-UP Inn some areas, fooding can cause sewage to back nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? nWhen following National Flood nInsurance Program regulations, nvnents can also help lower ninsunrance rates. n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}That's done a great job, it surfaced information from our key articles in https://www.fema.gov/sites/default/files/2020-11/fema_protect-your-home_flooding.pdf, plus an article from https://www.fema.gov/sites/default/files/documents/fema_protect-your-property-storm-surge.pdf, the only other document in the set which mentions flooding.
Retrieval Prompt Length
At this point, it's worth noting that the final summarization prompt which turns document excerpts into a nice answer can be quite long depending on the size and number of excerpts. A continual battle with LLMs is to achieve our goals without breaching token limits. Though it hasn't had a significant effect for our use case, other scenarios might need to employ contextual compression methods.
Question Context Can Be Important
We intentionally mixed documents related to (i) planning for disasters, and (ii) reacting immediately to dangerous events. This can result in confusing responses that mix both contexts …
result = ask_question("I live next to a canal and see the water rising right now, what should I do?", pdf_qa, output_docs=True)Question:
I live next to a canal and see the water rising right now, what should I do?
Answer:
If you notice water levels rising immediately, it's important to prioritize your safety. Here are some steps you can take:
1. Evacuate: If the water levels are rising rapidly, it's crucial to evacuate to higher ground as soon as possible.
2. Move your car: If you have time, move your car to higher ground. According to the National Weather Service, just two feet of water can move a vehicle.
3. Secure your home: If you have time, secure your home by moving valuables and important documents to an upper floor or at least above the Base Flood Elevation (BFE). Place them in waterproof or water-resistant containers.
4. Communicate: Let your family, friends, and neighbors know about the situation. If you have a pre-established meeting point, head there or inform others of your evacuation plans.
5. Contact authorities: Inform local authorities about the situation. They can provide guidance and also alert others who might be at risk.
6. Check flood maps: If you have access to the internet, check flood maps for your area to understand the potential risk.
Remember, these steps are general guidelines. Always follow the advice of local authorities and emergency services during a flood situation.The answer is now a bit of a mixed bag, with some points relating to immediate action "Move your car to higher ground" and some to preparation "Purchase flood insurance". When people are stressed during an emergency, they probably aren't thinking of prompt engineering and so we can expect slightly ambiguous inputs.
We could of course solve this problem by using more document metadata to split into sub-groups, but this requires work if that metadata is not available. Another option is to provide more context to the question to indicate if the user is interested in disaster preparation, or needs help immediately. We could build a classifier for this, but in these days of powerful LLMs let's use zero-shot classification with GPT-4 …
def get_time_context(question):
template = """Does the following question relate to 'planning' or 'taking immediate action': {question}
Answer with one of the following: 'I am planning ahead:' or 'I need to take immediate action:' or 'ambiguous'"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm = OpenAI()
llm_chain = LLMChain(prompt=prompt, llm=llm)
answer = llm_chain.run(question)
return answer
questions = [
"I live next to a canal and the water is rising, what should I do?",
"Help my roof is blowing off!",
"How do I prevent my roof from blowing off in a hurricane",
"Dog",
"How do I prepare my home for floods?"
]
for q in questions:
print(f"Question: {q}")
answer = get_time_context(q)
print(answer.strip(), "n")Question: I live next to a canal and the water is rising, what should I do?
I need to take immediate action:
Question: Help my roof is blowing off!
I need to take immediate action:
Question: How do I prevent my roof from blowing off in a hurricane
I am planning ahead.
Question: Dog
Ambiguous
Question: How do I prepare my home for floods?
I am planning ahead. Nice! With very minimal effort, we can easily determine whether a question relates to planning or taking immediate action.
We can now prefix the user question with this …
def get_time_context(question: str) -> str:
template = """Does the following question relate to 'planning' or 'taking immediate action': {question}
Answer with one of the following: 'I am planning ahead:' or 'I need to take immediate action:' or 'ambiguous'"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm = OpenAI()
llm_chain = LLMChain(prompt=prompt, llm=llm)
answer = llm_chain.run(question)
return answer
def ask_question(
query: str,
qa: object,
output_docs: bool = True,
preprocess_time_context: bool = False
) -> dict:
# First get the time context
if preprocess_time_context:
time_context = get_time_context(query)
if 'planning' in time_context.lower():
query = f"I am planning ahead: {query}"
elif 'take immediate action' in time_context.lower():
query = f"I need to take immediate action: {query}"
print(f"nQuestion: n{query}")
result = qa({"question": query})
print(f"nAnswer:n{result['answer']}")
if output_docs:
for doc in result['source_documents']:
print('n')
print(json.dumps(vars(doc), indent=4))
return result
result = ask_question("I live next to a canal and see the water rising, what should I do?", pdf_qa, output_docs=False, preprocess_time_context=True)Which gives …
Question:
I need to take immediate action: I live next to a canal and see the water rising, what should I do?
Answer:
If you see the water rising next to your canal, you should immediately implement your emergency plan. This includes evacuating your home and moving your car to higher ground, as just two feet of water can move a vehicle. You should also ensure that your valuables and important documents are stored above the Base Flood Elevation (BFE), preferably on an upper floor, and placed in waterproof or water-resistant containers. If you have time, consider moving appliances and utilities such as water heaters, washers, dryers, and electric panels to higher floors to prevent them from getting damaged or ruined by flood water. Always prioritize your safety and the safety of your family over protecting property.Great, worked nicely and didn't try to sell us any insurance, it gave advice that could be acted upon immediately.
Let's test the converse …
result = ask_question("What can I do to make my house forest fire resistant?", pdf_qa, output_docs=False, preprocess_time_context=True)Question:
I am planning ahead: What can I do to make my house forest fire resistant?
Answer:
There are several steps you can take to make your house resistant to forest fires:
1. Install or replace your roof with a Class A-rated roof with noncombustible coverings. The roof is the most at-risk part of a house in a wildfire due to its size and orientation.
2. Install and replace exterior wall coverings with noncombustible or fire-resistant materials. A minimum fire-resistance rating of one hour for the wall assembly is recommended.
3. Create a 30 feet defensible space around your home by reducing or removing flammable vegetation and using noncombustible materials such as gravel, brick, or concrete.
4. Regularly clean and remove debris from the roof and gutters to reduce the likelihood of something catching on fire on top of your home.
5. Enclose your home's foundation to lower the chance of wind-blown embers getting underneath your home.
6. Plan for access to water by purchasing and installing external sprinkler systems with dedicated power sources or a water tank.
7. Protect large windows from radiant heat by installing multi-pane windows, tempered safety glass, or fireproof shutters.
8. Install highly visible street signs and property addresses to help firefighters and other emergency responders quickly find your property.
9. Seal gaps around openings in exterior walls and roofs with fire-resistant caulk, mortar, or fire-protective expanding foam.
10. Cover exterior attic vents and under-eave vents with metal wire mesh no larger than 1/8 inch to keep embers out.
11. Install a fire block in the gap between the top of framed walls and the foundation of the house to starve the fire of oxygen and prevent it from spreading.
Remember, these tips may work better together than others and need to be tailored to your property. Always consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.Perfect, it provided forest fire preparation information.
Obviously, this would need a lot of testing for anything high-risk that could be used in a real emergency, but it illustrates one way we can enhance performance by enriching the user's prompt.
Making Sure Answers Only Come from the Provided Documents
For an application where the information can affect safety, it's important that the information presented only comes from the documents provided. Hallucinations containing incorrect information could have some really dire effects.
Let's try asking something totally unrelated to disasters …
result = ask_question("How do I make a sponge cake?", pdf_qa, output_docs=False)Which gives …
Question:
How do I make a sponge cake?
Answer:
I'm sorry, but the provided context does not contain information on how to make a sponge cake.Though I am a bit sad not finding out how to make some yummy cake, LangChain took care of this scenario nicely and determines that the question does not relate to the information in the PDF guides provided.
That's good, no major hallucinations! The last thing I'd want in a disaster is to be told to make a sponge cake and fly a spaceship or something.
