Content Based Image Retrieval Analysis (a.k.a. COBRA)

A python library that illustrates how to do "Content Based image Retrieval Analysis", a technique for prompting vision-enabled LLMs to extract details from video and image content.

Getting Started

1. Clone the repository to your device. Navigate to the repository directory and install dependencies:

cd /path/to/local/repo/cobrapy
 pip install -e . #"." means install at the current location

2. Follow the official instructions to download and install FFmpeg: https://ffmpeg.org/download.html

3. Copy sample.env to .env at the repository root and populate it with your service credentials. The backend and UI components both consume environment variables from this shared file. Run python scripts/apply_database_url.py to materialize the local DATABASE_URL entry using the shared values in config/database_urls.json; the same configuration file provides the cloud connection string used by the container setup and deployment scripts.

4. Start the FastAPI backend on port 8000:

poetry install
poetry run uvicorn cobrapy.api.app:app --host 0.0.0.0 --port 8000

5. In a separate terminal, install the UI dependencies and start the Next.js frontend on port 3000:

cd src/ui
npm install
npm run dev

The UI automatically proxies requests to http://localhost:8000, so no additional environment variables are required to wire the services together.

Deploy to Azure with Azure Developer CLI (azd)

The fastest way to deploy VIPER to Azure is using the Azure Developer CLI (azd).

Prerequisites

• Azure Developer CLI
• Docker
• An Azure subscription

Quick Start

1. Copy sample.env to .env and populate it with your Azure service credentials:

   cp sample.env .env
   # Edit .env with your Azure OpenAI, Speech, Storage, and Search credentials

2. Initialize the azd environment (first time only):

   azd init

3. Provision infrastructure and deploy:

   azd up

   This command will:

   • Create or update Azure resources (Container Registry, Container Apps, Storage, Search, Cosmos DB)
   • Build and push Docker images for the backend and frontend
   • Deploy the containers with environment variables from your .env file

Environment Variables

The deployment inherits configuration from your .env file. Key variables include:

Variable	Description
AZURE_OPENAI_GPT_VISION_ENDPOINT	Azure OpenAI endpoint for vision analysis
AZURE_OPENAI_GPT_VISION_API_KEY	API key for Azure OpenAI
AZURE_SPEECH_REGION	Azure Speech Services region
AZURE_STORAGE_ACCOUNT_URL	Blob storage URL for videos (auto-generated if not provided)
AZURE_SEARCH_ENDPOINT	Azure AI Search endpoint (auto-generated if not provided)
DATABASE_URL	PostgreSQL connection string

See sample.env for the complete list of configuration options.

Manual Deployment

For more control over the deployment process, use the PowerShell script:

./scripts/Deploy-ViperToAzure.ps1 `
    -SubscriptionId "your-subscription-id" `
    -ResourceGroupName "viper-prod" `
    -Location "eastus"

See azure/README.md for detailed deployment options.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.