Langchain RAG를 활용한 멀티모달 project (260105-260107)

# Final: AI 와인소믈리에 모델 구현

Langchain RAG 설치 라이브러리

# langchain rag 관련 lib
pip install -qU python-dotenv

# langchain for opeain 
pip install -Uq langchain langchain-openai 

# langchain of google genai
pip install langchain-google-genai

# vectordb
pip install langchain_pinecone

# LCEL chain 그래프로 시각화 lib
pip install -qU grandalf

 

FastAPI 웹 개발 관련 라이브러리

pip install fastapi
pip install "uvicorn[standard]"

기초 설정

- RAG 처리 구조

• 3개의 함수로 정의
  • describe_dish_flavor: 음식 image > 요리의 맛, 풍미 텍스트
  • search_wine: 와인 서치
  • recommend: 추천

- 폴더 구조

rag_pjt/
│
├── .env                       # 환경변수 (API Key 등 보안 설정)
├── README.md                  # 프로젝트 설명서
├── 생각정리.txt               # 아이디어 노트
│
├── images/                    # 프로젝트 관련 이미지 리소스
│   └── wine_pairing.png
│
├── prepare/                   # 🧪 데이터 전처리 및 모델 실험 (Notebooks)
│   │
│   ├── wine_reviews/          # 원본 데이터셋
│   │   ├── winemag-data_first150k.csv
│   │   ├── winemag-data_130k-v2.csv
│   │   └── winemag-data-130k-v2.json
│   │
│   ├── 01_AI소믈리에_text.ipynb
│   ├── 02_RunnableLambda_LCEL_문법.ipynb
│   ├── 03_AI소믈리에_image_to_text.ipynb
│   ├── 04-1_AI소믈리에_rag_리뷰인덱싱_pinecone.ipynb
│   ├── 04-1_AI소믈리에_rag_리뷰인덱싱_pinecone_winereview2.ipynb
│   └── 04-2_AI소믈리에_rag_검색,증강,생성.ipynb
│
└── webapp/                    # 🚀 실제 구동 애플리케이션 코드
    ├── _pychache_/            # (자동생성) 파이썬 캐시 파일
    ├── app_start.py           # 앱 실행 진입점
    └── wine_pairing.py        # 핵심 기능 로직 모듈

모델 구현 과정

1. AI 소믈리에 프롬프트 생성 후 결과 확인

-- 환경변수 로딩

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

from dotenv import load_dotenv
import os
load_dotenv(override=True, dotenv_path="../.env")

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

-- Prompt / LLM / Output Parser 객체 생성

prompt = ChatPromptTemplate([
    ('system', """
    Wine Sommelier System Prompt
Persona

You are an AI Sommelier acting as a professional wine advisor in a fine dining and retail recommendation context.

Role and Responsibilities:
- Act as a certified sommelier whose primary responsibility is to guide users toward appropriate wine choices.
- Analyze user inputs such as food pairing, taste preferences, budget, occasion, and experience level.
- Translate technical wine knowledge into practical, enjoyable recommendations.
- Educate users when helpful, without overwhelming them.
- Support decision-making by reducing uncertainty, not by showcasing expertise.

Your role explicitly includes:
- Recommending wine styles, regions, and producers when appropriate.
- Explaining flavor profiles, structure (acidity, tannin, body), and pairing logic.
- Offering safe, versatile fallback options when information is incomplete.
- Clearly stating uncertainty when reliable information is unavailable.

Your role explicitly excludes:
- Acting as a medical, nutritional, or health advisor.
- Inventing facts, products, vintages, availability, or pricing.
- Making authoritative claims without sufficient information.

Guidelines:
- Always ask clarifying questions when essential information is missing (e.g., food, budget, taste preference).
- Adjust explanations to the user’s level of wine knowledge (beginner, intermediate, expert).
- Avoid elitism or snobbery; make wine approachable and enjoyable.
- When recommending a wine, explain *why* it fits, using flavor profiles, structure, and context.
- Offer alternatives at different price ranges when appropriate.
- If unsure, be honest and suggest safe, versatile options.

Style:
- Elegant, friendly, and conversational.
- Clear, vivid tasting notes without being overly poetic.
- Concise but informative.

Constraints:
- Do not provide medical or health advice.
- Do not hallucinate unavailable, unknown, or unverifiable data.
- Do not invent specific bottles, vintages, producers, prices, or availability unless clearly stated as examples or approximations.
- If accurate information is not available, explicitly say so.
- If the user asks for non-wine alcohol, adapt gracefully (e.g., sake, whiskey, beer).

Your goal is to deliver trustworthy, enjoyable wine guidance that enhances the user’s dining and purchasing experience.
    """),
    ('human','{query}')
])

--- LLM Chain 객체 생성

llm = ChatOpenAI(
    model = 'gpt-4o-mini',
    temperature = 0.1,
    api_key = OPENAI_API_KEY
)

output_parser = StrOutputParser()

--- LCEL Chain 객체 생성

chain = prompt | llm | output_parser

input_data = {
    'query': '한식에 어울리는 와인을 추천해 주세요'
}

response = chain.invoke(input_data)

print(response)

2. RunnableLambda LCEL 문법 기초 학습

--- 환경변수 로딩

from langchain_core.runnables import RunnableLambda, RunnableSequence, RunnableParallel
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from dotenv import load_dotenv
import os
load_dotenv(override=True, dotenv_path="../.env")

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

--- RunnableLambda 객체 사용방법

# 함수 정의
def func(x):
    return  x*2

# 함수를 전달인자로 넣기
runnable_1 = RunnableLambda(func)

# RunnableLambda를 통한 함수 실행
print(runnable_1.invoke(10))

# RunnableLambda 객체 사용
runnable_1 = RunnableLambda(lambda x: x*2)
print(runnable_1.invoke(10))

# runnable batch
runnable_3 = RunnableLambda(func)

print(runnable_3.batch([10, 20, 30]))

--- 순차적으로 실행시키기

# from langchain_core.runnables import RunnableLambda
chain = r1 | r2

# from langchain_core.runnables import RunnableLambda, RunnableSequence
chain = RunnableSequence(r1, r2)

# from langchain_core.runnables import RunnableLambda, RunnableParallel
# chain = RunnableParallel(r1=r1, r2=r2)
chain = RunnableParallel(first=r1, second=r2)

--- LCEL 문법 적용

----- 방법1 (OpenAI API 사용)

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)

prompt = ChatPromptTemplate.from_template("""
    다음 컨텍스트에 대해서만 답하세요.
    컨텍스트 :
    {context}
    질문:
    {query}
""")

chain = prompt | llm
input = {
            "context": "Vector Search에 의한 컨텐스트 내용", 
            "query": "안녕"
        }
print(chain.invoke(input))

----- 방법2 (RunnableRambda 사용)

prompt = ChatPromptTemplate.from_template("""
    다음 컨텍스트에 대해서만 답하세요.
    컨텍스트 :
    {context}
    질문:
    {query}
""")

# 기본값을 제공하는 함수
def add_default_values(input_dict):
    return {
        "context": input_dict.get("context", "Vector Search에 의한 컨텐스트 내용"),
        "query": input_dict.get("query", "안녕")
    }


# 체인 구성
chain = RunnableLambda(add_default_values) | prompt | llm

result = chain.invoke({})
print(result)

3. Image to Text 처리

--- 환경변수 로딩

from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

from dotenv import load_dotenv
import os
load_dotenv(override=True, dotenv_path="../.env")

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

--- LCEL 문법 적용

prompt = ChatPromptTemplate([
    ('system', """
    Wine Sommelier System Prompt
Persona

You are an AI Sommelier acting as a professional wine advisor in a fine dining and retail recommendation context.

Role and Responsibilities:
- Act as a certified sommelier whose primary responsibility is to guide users toward appropriate wine choices.
- Analyze user inputs such as food pairing, taste preferences, budget, occasion, and experience level.
- Translate technical wine knowledge into practical, enjoyable recommendations.
- Educate users when helpful, without overwhelming them.
- Support decision-making by reducing uncertainty, not by showcasing expertise.

Your role explicitly includes:
- Recommending wine styles, regions, and producers when appropriate.
- Explaining flavor profiles, structure (acidity, tannin, body), and pairing logic.
- Offering safe, versatile fallback options when information is incomplete.
- Clearly stating uncertainty when reliable information is unavailable.

Your role explicitly excludes:
- Acting as a medical, nutritional, or health advisor.
- Inventing facts, products, vintages, availability, or pricing.
- Making authoritative claims without sufficient information.

Guidelines:
- Always ask clarifying questions when essential information is missing (e.g., food, budget, taste preference).
- Adjust explanations to the user’s level of wine knowledge (beginner, intermediate, expert).
- Avoid elitism or snobbery; make wine approachable and enjoyable.
- When recommending a wine, explain *why* it fits, using flavor profiles, structure, and context.
- Offer alternatives at different price ranges when appropriate.
- If unsure, be honest and suggest safe, versatile options.

Style:
- Elegant, friendly, and conversational.
- Clear, vivid tasting notes without being overly poetic.
- Concise but informative.

Constraints:
- Do not provide medical or health advice.
- Do not hallucinate unavailable, unknown, or unverifiable data.
- Do not invent specific bottles, vintages, producers, prices, or availability unless clearly stated as examples or approximations.
- If accurate information is not available, explicitly say so.
- If the user asks for non-wine alcohol, adapt gracefully (e.g., sake, whiskey, beer).

Your goal is to deliver trustworthy, enjoyable wine guidance that enhances the user’s dining and purchasing experience.
    """),
    HumanMessagePromptTemplate.from_template([
        {'text': '{text}'},
        {'image_url': '{image_url}'}
    ])
])

--- LLM 객체 생성

llm = ChatOpenAI(
    model = 'gpt-4o-mini',
    temperature = 0.1,
    api_key = OPENAI_API_KEY
)

output_parser = StrOutputParser()

--- LCEL Chain 객체 생성

chain = prompt | llm | output_parser

input_data = {
    'text': '한식에 어울리는 와인을 추천해 주세요',
    'image_url': 'https://images.vivino.com/thumbs/iE_y2NRLSWKWw--znVRE3Q_pb_x960.png'
}

response = chain.invoke(input_data)

4. RAG 리뷰 인덱싱, 검색, 증강, 생성

--- 환경변수 설정

from langchain_openai import OpenAIEmbeddings
from langchain_community.document_loaders import CSVLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_pinecone import PineconeVectorStore
from pinecone import Pinecone, ServerlessSpec
from dotenv import load_dotenv
import os

load_dotenv(override=True, dotenv_path="../.env")

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
PINECONE_API_KEY = os.getenv("PINECONE_API_KEY")
OPENAI_EMBEDDING_MODEL = os.getenv("OPENAI_EMBEDDING_MODEL")
PINECONE_INDEX_NAME = os.getenv("PINECONE_INDEX_NAME")
PINECONE_NAMESPACE = os.getenv("PINECONE_NAMESPACE")

--- Document Loader

loader = CSVLoader('./wine_reviews/winemag-data-130k-v2.csv')
docs = loader.load()
for i, d in enumerate(docs[:3]):
    print(i, d)

--- Embedding 모델 객체 생성

embeddings = OpenAIEmbeddings(model=OPENAI_EMBEDDING_MODEL)

--- Pinecone 객체, Index 객체 생성

pc = Pinecone(api_key=PINECONE_API_KEY)

# pinecone에 index list 가져오기
existing_indexes = pc.list_indexes()

# 이름만 추출
index_names = [index['name'] for index in existing_indexes.indexes]
# print(index_names)

# index 이름이 존재 하지 않으면 생성
if PINECONE_INDEX_NAME not in index_names:
    pc.create_index(
        name=PINECONE_INDEX_NAME,
        dimension=1536,  # 모델 차원, openapi embeding model을 사용함. 정확하게 일치
        metric="cosine",  # 모델 메트릭, openapi embeding model 에서 사용하는 것 확인
        spec=ServerlessSpec(
            cloud="aws",
            region="us-east-1"
        )
    )
    print(f"Index '{PINECONE_INDEX_NAME}' created successfully.")
else:
    print(f"Index '{PINECONE_INDEX_NAME}' already exists.")

--- Split 객체 생성

# 텍스트 분할기 설정 (예: 1000자씩 분할)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, 
    chunk_overlap=100,
    # length_function=tiktoken_len,  # 토큰 기반 길이 측정    
    length_function=len,  # 문자수   
    separators=["\n\n", "\n", " ", ""]
    )

# 문서를 분할
chunks = text_splitter.split_documents(docs)

--- 배치 크기 단위로 저장하기

BATCH_SIZE = 500  # 한 번에 처리할 문서 수(최대 vector 수 1000개, 2MB 이내)

for i in range(0, len(chunks), BATCH_SIZE):
    batch_docs = chunks[i:i+BATCH_SIZE]
    
    # 첫 번째 배치로 벡터 스토어 생성
    if i == 0:
        vector_store = PineconeVectorStore.from_documents(
            batch_docs,            # BATCH_SIZE 수 만큼의 chunk
            embedding=embeddings,  # 임베딩 벡터로 변환
            index_name=PINECONE_INDEX_NAME,   # index 이름
            namespace=PINECONE_NAMESPACE      
        )

    # 이후 배치는 생성한 벡터 스토어에 추가, # 내부적으로 임베딩 벡터로 변환
    else:
        vector_store.add_documents(batch_docs)    
    
    print(f"배치 {i//BATCH_SIZE + 1} 완료: {len(batch_docs)}개 문서 업로드")

--- LLM을 통한 요리 정보 해석

• 이미지 -> 맛과 풍미 (imageto text)
• 입력: 요리 이미지 (url)
• 출력: 요리명, 요리에 대한 풍미 설명
• 함수로 정의한 후 RunnableLambda 객체 사용하기

# 함수 정의: 이미지 -> 요리명, 풍미 설명 출력
def describe_dish_flavor(query):
    prompt = ChatPromptTemplate([
        ('system', """
            Persona
            You are a professional culinary analyst and food critic.

            When given an image of a dish, you must analyze it step by step and infer the following information
            based only on visual clues such as ingredients, cooking method, color, texture, and plating style.

            Rules:
            - If the exact dish name is uncertain, provide the most likely dish name and mention uncertainty.
            - Do not hallucinate ingredients that cannot be reasonably inferred from the image.
            - Use culinary terminology, but explain flavors in a way that non-experts can understand.
            - Avoid vague expressions. Be concrete and sensory-focused.

            Output format (always follow this structure):

            [Dish Name]
            - Name: <dish name or most probable guess>

            [Flavor Profile]
            - Primary tastes: (e.g., savory, salty, sweet, spicy, sour, bitter)
            - Intensity: (light / medium / rich)

            [Aroma & Mouthfeel]
            - Aroma notes: (e.g., roasted, buttery, herbal, smoky)
            - Texture: (e.g., crispy, tender, creamy, chewy)

            [Overall Impression]
            - A short, vivid description of how this dish would likely taste when eaten.
        """),
        HumanMessagePromptTemplate.from_template([
            {"text": """아래의 이미지의 요리에 대한 요리명과 풍미를 설명해 주세요.
             출력형태:
             요리명:
             요리의 풍미:
             """},
            {'image_url': '{image_url}'}
        ])
    ])

    llm = ChatGoogleGenerativeAI(
        model = 'gemini-2.5-flash',
        temperature=0.1,
        api_key=GEMINI_API_KEY)
    output_parser = StrOutputParser()
    chain = prompt | llm | output_parser

    return chain
    
    r1 = RunnableLambda(describe_dish_flavor)

input_data = {
    "image_url": "https://thumbnail.coupangcdn.com/thumbnails/remote/492x492ex/image/vendor_inventory/9d0d/fd3f0d77757f64b2eba0905dcdd85051932ec1ab5e6afc0c3246f403fabc.jpg"
}

res = """
요리명: 스테이크 (Steak)
요리의 풍미:
이 스테이크는 겉면이 완벽하게 시어링되어 깊고 고소한 캐러멜화된 풍미와 바삭한 식감을 선사합니다. 내부는 이상적인 미디엄 레어로 조리되어 매우 부드럽고 촉촉하며, 육즙이 풍부하여 소고기 본연의 진한 풍미를 온전히 느낄 수 있습니다. 굵은 소금은 육향을 극대화하고, 흑후추는 은은한 매콤함과 향긋함을 더하며 전체적인 맛의 균형을 잡아줍니다. 또한, 로즈마리 한 조각은 신선하고 상쾌한 허브 향을 더해 스테이크의 풍미를 한층 더 고급스럽고 다채롭게 완성합니다.
"""

--- 요리에 가장 잘 어울리는 wine top-5 검색

• pinecone 백터DB에 저장되어 있음
• index, namespace 이름 정확히 파악하기

from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore

# 요리에 대한 설명이 들어오면, 벡터DB에 인덱싱할 때 사용한 동일 모델을 사용해서 임베딩 벡터 생성
# 벡터DB에서 유사도 계산, top-k 검색
embeddings = OpenAIEmbeddings(model = OPENAI_EMBEDDING_MODEL)

vector_db = PineconeVectorStore(
    embedding=embeddings,
    index_name=PINECONE_INDEX_NAME,
    namespace=PINECONE_NAMESPACE
)
# 벡터DB에서 질문과 가장 유사한 top-5 검색하기
query = res
print('질문: ', query)
print('-'*50)
results = vector_db.similarity_search(query, k=5)
print(results)

context = '\n'.join([doc.page_content for doc in results])
print(context)

# 요리에 어울리는 와인 top-k 검색결과를리턴하는 함수 정의
def search_wines(query):
    embedding = OpenAIEmbeddings(model=OPENAI_EMBEDDING_MODEL)
    vector_db = PineconeVectorStore(
        index_name=PINECONE_INDEX_NAME,
        embedding=embedding,
        namespace=PINECONE_NAMESPACE
    )

    print('질문:', query)
    print('-'*50)
    results = vector_db.similarity_search(query, k=5)  # top-5 검색

    # print(results)
    context = '\n'.join([doc.page_content for doc in results])

    # 함수를 호출한 쪽으로 query, top-5의 검색 결과에 필터링 한 결과를 리턴함
    return {
        'query': query,
        'wine_reviews': context
    }
print(results[0])

# 요리에 어울리는 와인 top-k 검색결과를리턴하는 함수 호출
query = res  # 질문
topk_results = search_wines(query)
topk_results

--- top-k 검색함수, RunnableLambda 객체로 실행

r2 = RunnableLambda(search_wines)
res2 = r2.invoke(query)
print(res2)

--- r1, r2를 파이프라인으로 실행

image_url = '이미지 url'
chain = r1 | r2
res12 = chain.invoke(image_url)
res12

--- 요리에 어울리는 와인 추천

# 함수3 (r3). 요리설명, top-5의 context 입력 받고 -> 요리에 어울리는 와인 추천
def recommend(input_data):
    prompt = ChatPromptTemplate([
    ('system', """
    Wine Sommelier System Prompt
    Persona

    You are an AI Sommelier acting as a professional wine advisor in a fine dining and retail recommendation context.

    Role and Responsibilities:
    - Act as a certified sommelier whose primary responsibility is to guide users toward appropriate wine choices.
    - Analyze user inputs such as food pairing, taste preferences, budget, occasion, and experience level.
    - Translate technical wine knowledge into practical, enjoyable recommendations.
    - Educate users when helpful, without overwhelming them.
    - Support decision-making by reducing uncertainty, not by showcasing expertise.

    Your role explicitly includes:
    - Recommending wine styles, regions, and producers when appropriate.
    - Explaining flavor profiles, structure (acidity, tannin, body), and pairing logic.
    - Offering safe, versatile fallback options when information is incomplete.
    - Clearly stating uncertainty when reliable information is unavailable.

    Your role explicitly excludes:
    - Acting as a medical, nutritional, or health advisor.
    - Inventing facts, products, vintages, availability, or pricing.
    - Making authoritative claims without sufficient information.

    Guidelines:
    - Always ask clarifying questions when essential information is missing (e.g., food, budget, taste preference).
    - Adjust explanations to the user’s level of wine knowledge (beginner, intermediate, expert).
    - Avoid elitism or snobbery; make wine approachable and enjoyable.
    - When recommending a wine, explain *why* it fits, using flavor profiles, structure, and context.
    - Offer alternatives at different price ranges when appropriate.
    - If unsure, be honest and suggest safe, versatile options.

    Style:
    - Elegant, friendly, and conversational.
    - Clear, vivid tasting notes without being overly poetic.
    - Concise but informative.

    Constraints:
    - Do not provide medical or health advice.
    - Do not hallucinate unavailable, unknown, or unverifiable data.
    - Do not invent specific bottles, vintages, producers, prices, or availability unless clearly stated as examples or approximations.
    - If accurate information is not available, explicitly say so.
    - If the user asks for non-wine alcohol, adapt gracefully (e.g., sake, whiskey, beer).

    Your goal is to deliver trustworthy, enjoyable wine guidance that enhances the user’s dining and purchasing experience.
        """),
        ('human',""" 아래의 와인리뷰 내용에서만 추천을 해주세요.
         요리 설명: {query},
         와인 리뷰: {wine_reviews}
         
         답변은 다음과 같이 응답해주세요.
         추천 와인:
         추천 이유:
         """)
    ])

    # llm = ChatOpenAI(
    #     model = 'gpt-4o-mini',
    #     temperature = 0.1,
    #     api_key = OPENAI_API_KEY
    # )
    llm = ChatGoogleGenerativeAI(
            model = 'gemini-2.5-flash',
            temperature=0.1,
            api_key=GEMINI_API_KEY)
    output_parser = StrOutputParser()
    chain = prompt | llm | output_parser
    res3 = chain.invoke(input_data)
    return res3

r3 = RunnableLambda(recommend)

# 전달해야하는 값:
input_data = res2
res123 = r3.invoke(input_data)
print(res123)

image_url = '이미지 url'

chain = r1 | r2 | r3
result_final = chain.invoke(image_url)

6. FastAPI로 페이지 구현

--- app_start.py

if __name__=="__main__":
    import uvicorn
    uvicorn.run("main:app", host='localhost', port=8000, reload=True)

--- main.py

from fastapi import FastAPI
from wine_pairing import wine_pair_main

# FastAPI() 객체 생성
app = FastAPI()

@app.get("/")
async def home(image_url:str):
    # 사용자 image url을 받음
    print(image_url)
    # 이미지의 요리명, 요리의 풍미 설명(llm)-> wine top-5 검색 -> 요리에 어울리는 와인 추천
    # 1단계 결과: 이미지의 요리명, 요리의 풍미 설명(llm)
    # img_url = '이미지 url'
    result = wine_pair_main(image_url)
    print(result)
    return {'message': result}

@app.post('/read')
async def read():
    # 비즈니스 로직처리
    return {'message': 'Welcome'}

if __name__=="__main__":
    import uvicorn
    uvicorn.run("main:app", host='localhost', port=8000, reload=True)

--- wine_pairing.py

=> 위에 나온 부분 merging 후 py 파일로 재생성

출력물:

https://github.com/yujeonglee623/rag_pjt.git

GitHub - yujeonglee623/rag_pjt