← Back to Orders

Greyne Middleware - Order Processing Documentation

System Overview: This documentation covers the complete CSV order processing workflow for Home Depot, Lowe's, and Home Depot Special Orders through the Greyne Middleware system.
Access Control: Available only to developer-role accounts. Administrators should request updates or exports from a developer if documentation details are required.

1. System Overview

1,043
Mixed Retailer Batches Processed
3
Supported Retailers
9
Character Batch Identifiers
100%
Mixed Retailer Support

The Greyne Middleware system is a multi-retailer order processing platform that handles CSV imports from major home improvement retailers and converts them into warehouse-ready order data.

Supported Retailers

  • The Home Depot Inc - Standard retail orders
  • Lowe's - Standard retail orders
  • The Home Depot Special Orders - Custom/special order division
Key Feature: The system natively supports mixed retailer CSV files, meaning one upload can contain orders from all three retailers simultaneously.

1.1 User Roles & Access Control

The middleware now differentiates between Admin and Developer accounts to scope high-sensitivity tools and reference materials.

  • Developer: Full operational access plus documentation viewing. Required to access this page and manage technical configuration.
  • Admin: Administrative access across orders, UPCs, carriers, etc. Documentation is intentionally hidden to keep sensitive implementation notes within the engineering team.
Tip: Use the Users screen or local reset utility to assign roles. Newly created users default to admin; set developer explicitly for engineering team members.

UPC Usage & Soft Delete Plan

TODO: The UPCs table maps product codes to UPC numbers. Legacy exports already translate here; re-enable these lookups in the modern middleware.

Where UPCs are referenced

  • src/Model/Table/UpcsTable.php::GetUpc() – utility lookup by productcode.
  • TODO – Workflow timing: legacy CRS/JET exports fetch UPCs immediately after resolving Fishbowl part numbers and before writing batch CSV rows or FedEx label payloads; port this step to the new flow.
  • Legacy v1 flows (reference): v1/src/Controller/CrsController.php reads UPCNumber and calls Upcs->GetUpc() in multiple export paths.
  • TODO – Modern controllers: current Cake controllers (under src/Controller) do not yet use UPC translations; wire them into the new export pipeline.
  • UI Management: /upcs Tabulator grid for CRUD.

Soft delete considerations

  • Add one of: is_deleted TINYINT(1) (preferred) or deleted_at DATETIME.
  • Auto-filter enabled: UpcsTable::beforeFind() now excludes soft-deleted rows globally.
  • /upcs/api and server CSV export also exclude soft-deleted rows.
  • Keep historical rows for audit; hide from active lookups and exports.

Implementation Plan

  1. DB migration: add is_deleted to upcs with default 0.
  2. Table hook: beforeFind added in UpcsTable to auto-filter is_deleted=0 (or deleted_at IS NULL).
  3. Controller: soft-delete endpoint updates is_deleted=1 (already implemented with fallbacks).
  4. UI: /upcs delete button calls soft delete; grid reloads.
  5. Downstream: ensure GetUpc() and any legacy reads exclude soft-deleted UPCs.
  6. QA: verify exports omit soft-deleted UPCs; add tests for lookups.

2. Complete Workflow

CSV Upload
File Validation
Header Detection
Data Extraction


Retailer Processing
Batch Generation
Database Storage
Order Hierarchy

Step 1: File Upload & Validation

  • Location: webroot/uploads/
  • Format Check: UTF-8 encoded CSV
  • Error Handling: Invalid formats rejected with error message
  • Security: Files validated for proper CSV structure

Step 2: Header Detection & Cleanup

  • Pattern Search: Looks for "Status,Merchant" header pattern
  • Extra Row Removal: Strips export metadata above data
  • Flexibility: Handles both quoted and unquoted headers

Step 3: Data Extraction

16 Required Columns:
• Status
• Merchant
• PO Number
• Customer Name
• ShipTo Address1/2
• ShipTo Day Phone
• ShipTo City/State/Postal Code
• Vendor SKU
• Quantity
• Unit Cost
• Unit Cost Currency
• ShipTo First Name/Name

Step 4: Retailer-Specific Processing

Home Depot Orders:
  • PO Number padding (7→8 digits)
  • Customer name fallback logic
  • Phone number formatting
Lowe's Orders:
  • Standard PO format processing
  • Same customer name logic
  • Different carrier rules

Step 5: Batch Number Generation

Algorithm:
  1. Generate 8 random characters from 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
  2. Calculate checksum: Σ(character_value × position)
  3. Append checksum character (checksum % 36)
  4. Result: 9-character unique identifier (e.g., ABC123DEF9)

Step 6: Database Storage

Two-Phase Process:
  1. Temporary Storage: All rows saved with contentbody='Dummy'
  2. Order Hierarchy: Groups by PO Number, creates SO/Item structure
  3. Cleanup: Removes temporary records
  4. Result: Structured order data ready for fulfillment

System Modules

This section lists the main application areas for quick reference.

  • Orders:
  • Carriers & Routes:
    • Carriers — grouped by carrier with rule counts.
    • Routes — full list of routing rules (state → carrier).
  • UPCs: UPC Management — edit, search, export with Tabulator grid.
  • Users: User Management — manage application users.
  • Tags: Tag Management — create and assign tags.

EDI Scope Note (BOL & Advanced Routing)

Important: Some Bill of Lading (BOL) automation and advanced carrier routing features were originally scoped as part of an EDI integration effort. That EDI scope was refunded and these items are currently considered out‑of‑scope/paused. The application may contain references or placeholders, but any automated EDI hand‑offs (e.g., scheduled pushes, ASN/BOL EDI messages, advanced multi‑leg routing tied to trading‑partner specs) are disabled.

Examples that were tied to EDI scope:

  • Automated BOL generation & submission to trading partners via EDI/SFTP
  • Advanced routing logic driven by partner SLAs and EDI acknowledgements
  • Auto‑push of files (ASN/labels/manifests) with receipt checks, retries, and error routing

If you want to re‑enable these capabilities, we can re‑estimate and stage them as a separate milestone (keeping current CSV/label exports intact).

3. CSV File Format

Required Header Row

Status,"Merchant","PO Number","Customer Name","ShipTo Address1","ShipTo Address2","ShipTo Day Phone","ShipTo City","ShipTo State","ShipTo Postal Code","Vendor SKU","Quantity","Unit Cost","Unit Cost Currency","ShipTo First Name","ShipTo Name"

Sample Data Rows

"Open","The Home Depot Inc","8568387","John Smith Construction","1234 Main Street","Suite 200","7045551234","Charlotte","NC","28202","HD-BRK-001","2","45.99","USD","John","John Smith"
"Pending","Lowe's","PO242755788","Sarah Wilson Homes","2468 Oak Tree Lane","Apt 15B","9195553344","Raleigh","NC","27601","LW-DRL-156","4","67.88","USD","Sarah","Sarah Wilson"
"Processing","The Home Depot Special Orders","SO23535922","Premium Kitchen Designs","1111 Designer Way","Studio C","9045557890","Jacksonville","FL","32202","HD-SP-CAB-401","12","589.99","USD","Jennifer","Jennifer Adams"
Validation Rules:
  • PO Numbers cannot contain "=" character (triggers error)
  • Files must be UTF-8 encoded
  • All 16 columns must be present
  • Mixed retailer files are supported and encouraged

4. Batch Processing

Batch Information Display

After successful upload, the system displays:
Sales Order File Has Been Successfully Uploaded. Batch Number: ABC123DEF9 | Sequence ID: 12345 | Total Orders Processed: 87
Field Description Example Source
Batch Number 9-character unique identifier with checksum ABC123DEF9 Generated algorithm
Sequence ID Starting sequence number for this batch 12345 MAX(sequence) + 1
Orders Processed Count of records successfully saved 87 Save counter

Order Hierarchy Creation

For each unique PO Number:
  • First occurrence: Becomes "SO" (Sales Order) record
  • Additional items: Become "Item" records linked to SO
  • Result: Hierarchical structure for multi-item orders

5. Database Schema

Orders Table Structure

Field Type Purpose Example
batchnumber VARCHAR(9) Batch identifier ABC123DEF9
sequence INT Unique sequence ID 12345
vendorname VARCHAR Retailer identifier The Home Depot Inc
PoNumber VARCHAR Purchase order number 08568387
contentbody VARCHAR Record type SO, Item
orderstatus INT Processing status 0, 1
CustomerName VARCHAR Customer information "John Smith Construction"
VendorSKU VARCHAR Product identifier HD-BRK-001
Quantity INT Order quantity 2
UnitCost DECIMAL Product cost 45.99

Related Tables

  • tackingfiles: SFTP tracking file metadata
  • carrier_details: Shipping carrier assignments by state

6. Retailer-Specific Logic

The Home Depot Inc

  • PO Number: Padded to 8 digits (7→8)
  • Customer Name: Uses ShipTo Name if Customer Name is "Receiving Clerk" or "N/A"
  • Phone Formatting: Removes dashes and spaces
  • Carrier Logic: NC origin, state-based routing

Lowe's

  • PO Number: No padding, processed as-is
  • Customer Name: Same fallback logic as Home Depot
  • Carrier Logic: Different carrier preferences (FedEx vs UPS)

The Home Depot Special Orders

  • PO Number: Usually prefixed with "SO"
  • Processing: Same core logic as standard Home Depot
  • Higher Values: Typically larger order amounts

7. Export Functions

Available Exports

Export Type Function Purpose Output Format
Customer Data downloadjobs() Generate customer records Customers.csv
Sales Orders downloadorders() Create sales order data SalesOrder.csv
Tracking Files updownloadtrackingfile() Process SFTP tracking data ShipCartonTracking.csv
Batch Processing: Export functions process entire batches, including mixed retailer orders. This maintains unified workflow while preserving retailer-specific business rules.

Detailed Vendor Processing & Address Formatting

Overview: The export process handles vendor-specific business rules, address formatting, and carrier assignments automatically. This section details exactly what happens during the "downloadorders" export process.

Vendor Processing Steps

1. Vendor Detection & Categorization
The system identifies and processes three main vendor types:
  • The Home Depot Inc - Standard Home Depot orders
  • Lowe's - Lowe's company orders
  • The Home Depot Special Orders - Special order processing

Vendor-Specific Business Information

Vendor Business Name Business Address Customer Name Format
Home Depot Inc The Home Depot 2455 Paces Ferry Road
Atlanta, GA 30339
The Home Depot:Home Depot - PO [PO_NUMBER]
Lowe's Lowe's Companies Inc. 1000 Lowe's Blvd
Mooresville, NC 28117
Lowe's Companies Inc.:Lowe's - PO [PO_NUMBER]
Home Depot Special The Home Depot 2455 Paces Ferry Road
Atlanta, GA 30339
The Home Depot:Home Depot - PO [PO_NUMBER]

Address Formatting Process

Address Consolidation Logic:
The system builds a single merged address field from multiple components:
  1. ShipToAddress1: Primary shipping address
  2. ShipToAddress2: Secondary address (apartment, suite, etc.)
  3. Phone Number: Formatted as (XXX) XXX-XXXX
Example:
Input: Address1="123 Main St", Address2="Apt 2B", Phone="5551234567"
Output: "123 Main St Apt 2B (555) 123-4567"

Code Reference:
  • Implementation lives in src/Controller/OrdersController.phpdownloadorders( $batch), inside each vendor block (e.g., Home Depot Inc, Lowe's). The controller builds an $addressParts array, conditionally appends ShipToAddress1, ShipToAddress2 (if not "N/A"), and a formatted phone number, then joins and quotes it for the CSV.
// src/Controller/OrdersController.php (downloadorders)
$addressParts = [];
if (!empty($order->ShipToAddress1)) {
    $addressParts[] = $order->ShipToAddress1;
}
if ($order->ShipToAddress2 !== '"N/A"') {
    $addressParts[] = $order->ShipToAddress2;
}
if ($order->ShipToDayPhone !== 'N/A') {
    $phone = $order->ShipToDayPhone;
    $formatted = '(' . substr($phone,0,3) . ') ' . substr($phone,3,3) . '-' . substr($phone,6,4);
    $addressParts[] = $formatted;
}
$mergedAddress = implode(' ', $addressParts);
$mergedAddress = str_replace('"', '', $mergedAddress);
$mergedAddress = '"' . $mergedAddress . '"';

Carrier Assignment System

Database Table: Carriers are stored in the carrier_details table with state-based routing logic.
Carrier Assignment Process:
  1. State Pair Creation: System creates identifier like "NC_CA" (from NC to California)
  2. Database Lookup: Queries carrier_details table for matching route
  3. Carrier Mapping: Assigns appropriate carrier based on destination
  4. Fallback Logic: Uses UPS Ground for unmapped routes
Fallback Behavior: When no carrier is found in the carrier_details table for a specific state route, the system automatically assigns UPS Ground to ensure all orders receive a carrier assignment. This prevents shipment delays due to missing carrier configurations.

Supported Carriers

Carrier Code Carrier Name Coverage
ODFLOld Dominion Freight LineRegional LTL
CTIICon-way TransportationRegional
EXLAEstes Express LinesRegional LTL
SAIASaia LTL FreightRegional LTL
SEFLSoutheastern Freight LinesSoutheast Regional
PAAFPriority ExpressRegional
SEKWSoutheast ExpressSoutheast Regional
UPSUPS GroundNational (Fallback)

CSV Output Structure

Two-Row Header System:
Flag,SONum,Status,CustomerName,CustomerContact,BillToName...
Flag,SOItemTypeID,ProductNumber,ProductDescription,ProductQuantity...

Record Types:
  • "SO" Records: Sales Order headers (one per PO)
  • "Item" Records: Line items linked to each SO

Processing Logic Flow

Detect Vendor
Apply Business Rules
Format Addresses
Assign Carriers
Generate CSV
Update Status
Important: Once exported, orders are marked as salesstatus = 1 and cannot be exported again. The process is irreversible to prevent duplicate shipments.

8. Troubleshooting

Common Error Messages

Error Cause Solution
"PO Number contains '=' character" Excel formula in PO field Export as CSV UTF-8, not Excel format
"File Format is not correct" Invalid CSV encoding Save as CSV UTF-8 (Comma delimited)
"No new files available" SFTP files already processed Check file timestamps and export status
"Data has already been exported" Batch already processed Check batch status flags in database

File Upload Issues

Checklist:
  • ✓ File is CSV format (not Excel)
  • ✓ UTF-8 encoding is used
  • ✓ All 16 required columns present
  • ✓ No "=" characters in PO numbers
  • ✓ Headers match expected format exactly

8. Test Data & Templates

Overview: Sample CSV files and templates are available for testing uploads, validation, and understanding file formats. These files demonstrate proper formatting and edge cases.

Available Test Files

File Name Purpose Contents Download
mixed-retailer-orders.csv Mixed retailer batch testing Home Depot, Lowe's, and Special Orders combined Download
home-depot-test-orders.csv Home Depot specific testing Standard Home Depot orders with various statuses Download
lowes-test-orders.csv Lowe's specific testing Lowe's orders with proper formatting Download
home-depot-special-orders.csv Special orders testing Home Depot Special Orders with custom SKUs Download
edge-case-validation-test.csv Error handling testing Tests validation rules and edge cases Download

Jet Warehouse Integration Templates

Jet.com Integration: The system supports integration with Jet's warehouse management system through specialized templates and import processes.
Template Description Format Download
Order Import Template Standard Jet order import format Excel (.xlsx) Download
Flex Header Requirements Flexible header field specifications Excel (.xlsx) Download

Jet Implementation Status

Implementation Phase: Jet warehouse integration is being developed as a separate workflow parallel to the existing Home Depot/Lowe's processing system.
Jet Processing Workflow (In Development):
  1. Template Validation: Validates against Jet's flexible header requirements
  2. Data Mapping: Maps Jet fields to internal order structure
  3. Warehouse Routing: Routes to appropriate Jet warehouse locations
  4. Status Synchronization: Maintains sync with Jet's order status system
  5. Export Integration: Generates Jet-compatible output formats
Development Note: Jet implementation is handled separately from the main retailer processing pipeline to maintain system stability and allow for specialized business logic requirements.

Testing Workflow

Recommended Testing Steps:
  1. Start Small: Upload single-retailer test files first
  2. Mixed Testing: Test mixed-retailer-orders.csv for complex scenarios
  3. Edge Cases: Use edge-case-validation-test.csv to verify error handling
  4. Export Testing: Test CSV exports with pending orders
  5. Status Verification: Confirm orders change from pending to exported
File Validation Checklist:
  • ✓ UTF-8 encoding
  • ✓ All 16 required columns present
  • ✓ No "=" characters in PO numbers
  • ✓ Proper quote escaping
  • ✓ Valid phone number formats

9. ❓ Frequently Asked Questions

Q: Can I upload mixed retailer files?

A: Yes! Mixed retailer uploads are not only supported but are the standard practice. The system has processed over 1,000 mixed retailer batches successfully.

Q: What happens to the original CSV file?

A: The CSV file is processed and the data is extracted into the database. The original file is not permanently stored - only the structured data remains.

Q: How are batch numbers generated?

A: Batch numbers use a 9-character format with 8 random alphanumeric characters plus a checksum character for validation. Example: ABC123DEF9

Q: Can I reprocess a batch?

A: Each batch can only be exported once (controlled by status flags). This prevents duplicate processing and maintains data integrity.

Q: Where are carrier assignments determined?

A: Carrier selection is based on the carrier_details table using origin state (NC), destination state, and retailer. The system automatically assigns the optimal carrier for each order.

Q: What's the difference between sequence ID and batch number?

A:
  • Sequence ID: Auto-incrementing database identifier for individual orders
  • Batch Number: Generated identifier that groups all orders from a single CSV upload

Additional Documentation

Fishbowl Integration Analysis

Complete analysis of Fishbowl database usage, including current status, legacy integration, and security improvements.

  • Investigation findings and current usage
  • Legacy v1 vs current system comparison
  • Security improvements implemented
  • Future integration readiness
→ View Fishbowl Analysis

Security Configuration Guide

Comprehensive security guide covering environment variables, credential management, and best practices.

  • Environment variable setup
  • Security improvements made
  • Usage examples and best practices
  • Deployment and testing checklist
→ View Security Guide

Project Workflow Diagrams

End-to-end flows for authentication, CSV import/export, tracking ingestion, and UPC management (Mermaid diagrams).

  • High-level architecture
  • Orders import and exports
  • Tracking SFTP intake and consolidation
  • UPC CRUD
Support: For additional questions or issues not covered in this documentation, please contact the development team or check the system logs for detailed error information.