# Ads System Deployment Guide

## Overview
This guide covers the deployment of the improved ads ingestion system for Bama and Divar sources.

## Changes Made

### 1. Database Schema Updates
- Added `external_code` and `hash` columns to `ads_raw` table
- Added `scheduled_at` and `external_code` columns to `opportunities` table
- Created `user_search_sessions` table for tracking user searches
- Added proper indexes for performance

### 2. API Client Improvements
- Fixed Bama API parsing to handle direct array responses (not wrapped in `data`/`result`)
- Improved Divar scraping with stable selectors (`post-list__items-container-* .kt-post-card`)
- Added proper headers and rate limiting
- Implemented upsert logic to preserve data history

### 3. Normalization Enhancements
- Added support for Persian digits in prices and mileage
- Improved brand/model mapping
- Better handling of different source structures (Bama vs Divar)
- Enhanced city ID resolution

### 4. URL Strategy for Divar
- Implemented hierarchical URL structure as per spec:
  - Brand only: `/s/{city}/car/{brand}`
  - Brand + Model: `/s/{city}/car/{brand}/{model}`
  - Independent models: `/s/{city}/car/{model}` (for tiba, quick, saina, pride)
  - Fallback: `/s/{city}/car?q={query}`

## Deployment Steps

### 1. Run Migrations
```bash
php artisan migrate
```

### 2. Verify Database Schema
```bash
php artisan tinker
```
```php
// Check if new columns exist
Schema::hasColumn('ads_raw', 'external_code');
Schema::hasColumn('ads_raw', 'hash');
Schema::hasColumn('opportunities', 'scheduled_at');
Schema::hasColumn('opportunities', 'external_code');
```

### 3. Test the System
```bash
# Test with Bama
php artisan ads:test-system --source=bama --query=206

# Test with Divar
php artisan ads:test-system --source=divar --query=پژو

# Test full scraping
php artisan ads:scrape-cars 206 --city-id=6 --sources=bama,divar --session-id=test --user-id=1 --force
```

### 4. Run Tests
```bash
# Run unit tests
php artisan test tests/Unit/AdsNormalizerTest.php

# Run integration tests
php artisan test tests/Integration/AdsPipelineTest.php

# Run all ads-related tests
php artisan test --filter=Ads
```

### 5. Monitor Logs
```bash
# Watch for errors
tail -f storage/logs/laravel.log | grep -E "(ERROR|WARNING)"

# Check specific job logs
tail -f storage/logs/laravel.log | grep -E "(BamaSearchJob|DivarScrapeJob)"
```

## Configuration

### Environment Variables
Make sure these are set in `.env`:
```env
MAJIDAPI_BASE_URL=https://api.majidapi.ir
MAJIDAPI_KEY=your_api_key_here
```

### Ads Configuration
The system uses `config/ads.php` for:
- Scoring thresholds
- Brand mappings
- Fallback rules
- TTL settings

## Expected Results

### Bama Integration
- Should parse array responses directly (not wrapped in `data`)
- Should use `detail.code` as unique identifier
- Should preserve history with upsert logic
- Should handle pagination correctly

### Divar Integration
- Should use stable selectors for card extraction
- Should build hierarchical URLs correctly
- Should handle Persian digits in prices/mileage
- Should extract external codes from URLs

### Normalization
- Should convert Persian digits to English
- Should normalize prices to millions of Tomans
- Should map brands correctly
- Should handle missing fields gracefully

## Troubleshooting

### Common Issues

1. **No results from Divar**
   - Check if selectors are still valid
   - Verify URL structure
   - Check rate limiting

2. **Bama API errors**
   - Verify API key
   - Check response structure changes
   - Monitor rate limits

3. **Normalization issues**
   - Check Persian digit conversion
   - Verify brand mappings
   - Check city ID resolution

### Debug Commands
```bash
# Check raw data
php artisan tinker
DB::table('ads_raw')->latest()->take(5)->get();

# Check opportunities
DB::table('opportunities')->latest()->take(5)->get();

# Check specific source
DB::table('ads_raw')->where('source', 'bama')->latest()->take(5)->get();
```

## Performance Considerations

- Database indexes are added for common queries
- Upsert logic prevents duplicate data
- Rate limiting prevents API abuse
- Pagination is implemented for large datasets

## Security Notes

- API keys are stored in environment variables
- No sensitive data is logged
- Rate limiting prevents abuse
- Input validation is implemented

## Monitoring

Monitor these metrics:
- API response times
- Error rates
- Data quality (normalization success rate)
- Database performance
- Queue processing times
