← Back to Orders
Greyne Middleware - Order Processing Documentation
System Overview: This documentation covers the complete CSV order processing workflow for Home Depot, Lowe's, and Home Depot Special Orders through the Greyne Middleware system.
Access Control: Available only to developer-role accounts. Administrators should request updates or exports from a developer if documentation details are required.
1. System Overview
1,043
Mixed Retailer Batches Processed
9
Character Batch Identifiers
100%
Mixed Retailer Support
The Greyne Middleware system is a multi-retailer order processing platform that handles CSV imports from major home improvement retailers and converts them into warehouse-ready order data.
Supported Retailers
- The Home Depot Inc - Standard retail orders
- Lowe's - Standard retail orders
- The Home Depot Special Orders - Custom/special order division
Key Feature: The system natively supports mixed retailer CSV files, meaning one upload can contain orders from all three retailers simultaneously.
1.1 User Roles & Access Control
The middleware now differentiates between Admin and Developer accounts to scope high-sensitivity tools and reference materials.
- Developer: Full operational access plus documentation viewing. Required to access this page and manage technical configuration.
- Admin: Administrative access across orders, UPCs, carriers, etc. Documentation is intentionally hidden to keep sensitive implementation notes within the engineering team.
Tip: Use the Users screen or local reset utility to assign roles. Newly created users default to admin; set developer explicitly for engineering team members.
UPC Usage & Soft Delete Plan
TODO: The UPCs table maps product codes to UPC numbers. Legacy exports already translate here; re-enable these lookups in the modern middleware.
Where UPCs are referenced
src/Model/Table/UpcsTable.php::GetUpc() – utility lookup by productcode.
- TODO – Workflow timing: legacy CRS/JET exports fetch UPCs immediately after resolving Fishbowl part numbers and before writing batch CSV rows or FedEx label payloads; port this step to the new flow.
- Legacy v1 flows (reference):
v1/src/Controller/CrsController.php reads UPCNumber and calls Upcs->GetUpc() in multiple export paths.
- TODO – Modern controllers: current Cake controllers (under
src/Controller) do not yet use UPC translations; wire them into the new export pipeline.
- UI Management:
/upcs Tabulator grid for CRUD.
Soft delete considerations
- Add one of:
is_deleted TINYINT(1) (preferred) or deleted_at DATETIME.
- Auto-filter enabled:
UpcsTable::beforeFind() now excludes soft-deleted rows globally.
- /upcs/api and server CSV export also exclude soft-deleted rows.
- Keep historical rows for audit; hide from active lookups and exports.
Implementation Plan
- DB migration: add
is_deleted to upcs with default 0.
- Table hook:
beforeFind added in UpcsTable to auto-filter is_deleted=0 (or deleted_at IS NULL).
- Controller: soft-delete endpoint updates
is_deleted=1 (already implemented with fallbacks).
- UI: /upcs delete button calls soft delete; grid reloads.
- Downstream: ensure
GetUpc() and any legacy reads exclude soft-deleted UPCs.
- QA: verify exports omit soft-deleted UPCs; add tests for lookups.
2. Complete Workflow
CSV Upload
→
File Validation
→
Header Detection
→
Data Extraction
Retailer Processing
→
Batch Generation
→
Database Storage
→
Order Hierarchy
Step 1: File Upload & Validation
- Location:
webroot/uploads/
- Format Check: UTF-8 encoded CSV
- Error Handling: Invalid formats rejected with error message
- Security: Files validated for proper CSV structure
Step 2: Header Detection & Cleanup
- Pattern Search: Looks for "Status,Merchant" header pattern
- Extra Row Removal: Strips export metadata above data
- Flexibility: Handles both quoted and unquoted headers
Step 3: Data Extraction
16 Required Columns:
• Status
• Merchant
• PO Number
• Customer Name
• ShipTo Address1/2
• ShipTo Day Phone
• ShipTo City/State/Postal Code
• Vendor SKU
• Quantity
• Unit Cost
• Unit Cost Currency
• ShipTo First Name/Name
Step 4: Retailer-Specific Processing
Home Depot Orders:
- PO Number padding (7→8 digits)
- Customer name fallback logic
- Phone number formatting
Lowe's Orders:
- Standard PO format processing
- Same customer name logic
- Different carrier rules
Step 5: Batch Number Generation
Algorithm:
- Generate 8 random characters from
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
- Calculate checksum: Σ(character_value × position)
- Append checksum character (checksum % 36)
- Result: 9-character unique identifier (e.g.,
ABC123DEF9)
Step 6: Database Storage
Two-Phase Process:
- Temporary Storage: All rows saved with
contentbody='Dummy'
- Order Hierarchy: Groups by PO Number, creates SO/Item structure
- Cleanup: Removes temporary records
- Result: Structured order data ready for fulfillment
System Modules
This section lists the main application areas for quick reference.
- Orders:
- Carriers & Routes:
- Carriers — grouped by carrier with rule counts.
- Routes — full list of routing rules (state → carrier).
- UPCs: UPC Management — edit, search, export with Tabulator grid.
- Users: User Management — manage application users.
- Tags: Tag Management — create and assign tags.
EDI Scope Note (BOL & Advanced Routing)
Important: Some Bill of Lading (BOL) automation and advanced carrier routing features were
originally scoped as part of an EDI integration effort. That EDI scope was refunded and these items are
currently considered out‑of‑scope/paused. The application may contain references or placeholders, but any
automated EDI hand‑offs (e.g., scheduled pushes, ASN/BOL EDI messages, advanced multi‑leg routing tied to
trading‑partner specs) are disabled.
Examples that were tied to EDI scope:
- Automated BOL generation & submission to trading partners via EDI/SFTP
- Advanced routing logic driven by partner SLAs and EDI acknowledgements
- Auto‑push of files (ASN/labels/manifests) with receipt checks, retries, and error routing
If you want to re‑enable these capabilities, we can re‑estimate and stage them as a separate milestone
(keeping current CSV/label exports intact).
3. CSV File Format
Required Header Row
Status,"Merchant","PO Number","Customer Name","ShipTo Address1","ShipTo Address2","ShipTo Day Phone","ShipTo City","ShipTo State","ShipTo Postal Code","Vendor SKU","Quantity","Unit Cost","Unit Cost Currency","ShipTo First Name","ShipTo Name"
Sample Data Rows
"Open","The Home Depot Inc","8568387","John Smith Construction","1234 Main Street","Suite 200","7045551234","Charlotte","NC","28202","HD-BRK-001","2","45.99","USD","John","John Smith"
"Pending","Lowe's","PO242755788","Sarah Wilson Homes","2468 Oak Tree Lane","Apt 15B","9195553344","Raleigh","NC","27601","LW-DRL-156","4","67.88","USD","Sarah","Sarah Wilson"
"Processing","The Home Depot Special Orders","SO23535922","Premium Kitchen Designs","1111 Designer Way","Studio C","9045557890","Jacksonville","FL","32202","HD-SP-CAB-401","12","589.99","USD","Jennifer","Jennifer Adams"
Validation Rules:
- PO Numbers cannot contain "=" character (triggers error)
- Files must be UTF-8 encoded
- All 16 columns must be present
- Mixed retailer files are supported and encouraged
4. Batch Processing
Batch Information Display
After successful upload, the system displays:
Sales Order File Has Been Successfully Uploaded.
Batch Number: ABC123DEF9 | Sequence ID: 12345 | Total Orders Processed: 87
| Field |
Description |
Example |
Source |
| Batch Number |
9-character unique identifier with checksum |
ABC123DEF9 |
Generated algorithm |
| Sequence ID |
Starting sequence number for this batch |
12345 |
MAX(sequence) + 1 |
| Orders Processed |
Count of records successfully saved |
87 |
Save counter |
Order Hierarchy Creation
For each unique PO Number:
- First occurrence: Becomes "SO" (Sales Order) record
- Additional items: Become "Item" records linked to SO
- Result: Hierarchical structure for multi-item orders
5. Database Schema
Orders Table Structure
| Field |
Type |
Purpose |
Example |
| batchnumber |
VARCHAR(9) |
Batch identifier |
ABC123DEF9 |
| sequence |
INT |
Unique sequence ID |
12345 |
| vendorname |
VARCHAR |
Retailer identifier |
The Home Depot Inc |
| PoNumber |
VARCHAR |
Purchase order number |
08568387 |
| contentbody |
VARCHAR |
Record type |
SO, Item |
| orderstatus |
INT |
Processing status |
0, 1 |
| CustomerName |
VARCHAR |
Customer information |
"John Smith Construction" |
| VendorSKU |
VARCHAR |
Product identifier |
HD-BRK-001 |
| Quantity |
INT |
Order quantity |
2 |
| UnitCost |
DECIMAL |
Product cost |
45.99 |
Related Tables
- tackingfiles: SFTP tracking file metadata
- carrier_details: Shipping carrier assignments by state
6. Retailer-Specific Logic
The Home Depot Inc
- PO Number: Padded to 8 digits (7→8)
- Customer Name: Uses ShipTo Name if Customer Name is "Receiving Clerk" or "N/A"
- Phone Formatting: Removes dashes and spaces
- Carrier Logic: NC origin, state-based routing
Lowe's
- PO Number: No padding, processed as-is
- Customer Name: Same fallback logic as Home Depot
- Carrier Logic: Different carrier preferences (FedEx vs UPS)
The Home Depot Special Orders
- PO Number: Usually prefixed with "SO"
- Processing: Same core logic as standard Home Depot
- Higher Values: Typically larger order amounts
7. Export Functions
Available Exports
| Export Type |
Function |
Purpose |
Output Format |
| Customer Data |
downloadjobs() |
Generate customer records |
Customers.csv |
| Sales Orders |
downloadorders() |
Create sales order data |
SalesOrder.csv |
| Tracking Files |
updownloadtrackingfile() |
Process SFTP tracking data |
ShipCartonTracking.csv |
Batch Processing: Export functions process entire batches, including mixed retailer orders. This maintains unified workflow while preserving retailer-specific business rules.
Detailed Vendor Processing & Address Formatting
Overview: The export process handles vendor-specific business rules, address formatting, and carrier assignments automatically. This section details exactly what happens during the "downloadorders" export process.
Vendor Processing Steps
1. Vendor Detection & Categorization
The system identifies and processes three main vendor types:
- The Home Depot Inc - Standard Home Depot orders
- Lowe's - Lowe's company orders
- The Home Depot Special Orders - Special order processing
Vendor-Specific Business Information
| Vendor |
Business Name |
Business Address |
Customer Name Format |
| Home Depot Inc |
The Home Depot |
2455 Paces Ferry Road Atlanta, GA 30339 |
The Home Depot:Home Depot - PO [PO_NUMBER] |
| Lowe's |
Lowe's Companies Inc. |
1000 Lowe's Blvd Mooresville, NC 28117 |
Lowe's Companies Inc.:Lowe's - PO [PO_NUMBER] |
| Home Depot Special |
The Home Depot |
2455 Paces Ferry Road Atlanta, GA 30339 |
The Home Depot:Home Depot - PO [PO_NUMBER] |
Address Formatting Process
Address Consolidation Logic:
The system builds a single merged address field from multiple components:
- ShipToAddress1: Primary shipping address
- ShipToAddress2: Secondary address (apartment, suite, etc.)
- Phone Number: Formatted as (XXX) XXX-XXXX
Example:
Input: Address1="123 Main St", Address2="Apt 2B", Phone="5551234567"
Output:
"123 Main St Apt 2B (555) 123-4567"
Code Reference:
- Implementation lives in
src/Controller/OrdersController.php → downloadorders(
$batch), inside each vendor block (e.g., Home Depot Inc, Lowe's).
The controller builds an $addressParts array, conditionally appends
ShipToAddress1, ShipToAddress2 (if not "N/A"), and a
formatted phone number, then joins and quotes it for the CSV.
// src/Controller/OrdersController.php (downloadorders)
$addressParts = [];
if (!empty($order->ShipToAddress1)) {
$addressParts[] = $order->ShipToAddress1;
}
if ($order->ShipToAddress2 !== '"N/A"') {
$addressParts[] = $order->ShipToAddress2;
}
if ($order->ShipToDayPhone !== 'N/A') {
$phone = $order->ShipToDayPhone;
$formatted = '(' . substr($phone,0,3) . ') ' . substr($phone,3,3) . '-' . substr($phone,6,4);
$addressParts[] = $formatted;
}
$mergedAddress = implode(' ', $addressParts);
$mergedAddress = str_replace('"', '', $mergedAddress);
$mergedAddress = '"' . $mergedAddress . '"';
Carrier Assignment System
Database Table: Carriers are stored in the carrier_details table with state-based routing logic.
Carrier Assignment Process:
- State Pair Creation: System creates identifier like "NC_CA" (from NC to California)
- Database Lookup: Queries carrier_details table for matching route
- Carrier Mapping: Assigns appropriate carrier based on destination
- Fallback Logic: Uses UPS Ground for unmapped routes
Fallback Behavior: When no carrier is found in the carrier_details table for a specific state route, the system automatically assigns UPS Ground to ensure all orders receive a carrier assignment. This prevents shipment delays due to missing carrier configurations.
Supported Carriers
| Carrier Code |
Carrier Name |
Coverage |
| ODFL | Old Dominion Freight Line | Regional LTL |
| CTII | Con-way Transportation | Regional |
| EXLA | Estes Express Lines | Regional LTL |
| SAIA | Saia LTL Freight | Regional LTL |
| SEFL | Southeastern Freight Lines | Southeast Regional |
| PAAF | Priority Express | Regional |
| SEKW | Southeast Express | Southeast Regional |
| UPS | UPS Ground | National (Fallback) |
CSV Output Structure
Two-Row Header System:
Flag,SONum,Status,CustomerName,CustomerContact,BillToName...
Flag,SOItemTypeID,ProductNumber,ProductDescription,ProductQuantity...
Record Types:
- "SO" Records: Sales Order headers (one per PO)
- "Item" Records: Line items linked to each SO
Processing Logic Flow
Detect Vendor
→
Apply Business Rules
→
Format Addresses
→
Assign Carriers
→
Generate CSV
→
Update Status
Important: Once exported, orders are marked as salesstatus = 1 and cannot be exported again. The process is irreversible to prevent duplicate shipments.
8. Troubleshooting
Common Error Messages
| Error |
Cause |
Solution |
| "PO Number contains '=' character" |
Excel formula in PO field |
Export as CSV UTF-8, not Excel format |
| "File Format is not correct" |
Invalid CSV encoding |
Save as CSV UTF-8 (Comma delimited) |
| "No new files available" |
SFTP files already processed |
Check file timestamps and export status |
| "Data has already been exported" |
Batch already processed |
Check batch status flags in database |
File Upload Issues
Checklist:
- ✓ File is CSV format (not Excel)
- ✓ UTF-8 encoding is used
- ✓ All 16 required columns present
- ✓ No "=" characters in PO numbers
- ✓ Headers match expected format exactly
8. Test Data & Templates
Overview: Sample CSV files and templates are available for testing uploads, validation, and understanding file formats. These files demonstrate proper formatting and edge cases.
Available Test Files
| File Name |
Purpose |
Contents |
Download |
| mixed-retailer-orders.csv |
Mixed retailer batch testing |
Home Depot, Lowe's, and Special Orders combined |
Download |
| home-depot-test-orders.csv |
Home Depot specific testing |
Standard Home Depot orders with various statuses |
Download |
| lowes-test-orders.csv |
Lowe's specific testing |
Lowe's orders with proper formatting |
Download |
| home-depot-special-orders.csv |
Special orders testing |
Home Depot Special Orders with custom SKUs |
Download |
| edge-case-validation-test.csv |
Error handling testing |
Tests validation rules and edge cases |
Download |
Jet Warehouse Integration Templates
Jet.com Integration: The system supports integration with Jet's warehouse management system through specialized templates and import processes.
| Template |
Description |
Format |
Download |
| Order Import Template |
Standard Jet order import format |
Excel (.xlsx) |
Download |
| Flex Header Requirements |
Flexible header field specifications |
Excel (.xlsx) |
Download |
Jet Implementation Status
Implementation Phase: Jet warehouse integration is being developed as a separate workflow parallel to the existing Home Depot/Lowe's processing system.
Jet Processing Workflow (In Development):
- Template Validation: Validates against Jet's flexible header requirements
- Data Mapping: Maps Jet fields to internal order structure
- Warehouse Routing: Routes to appropriate Jet warehouse locations
- Status Synchronization: Maintains sync with Jet's order status system
- Export Integration: Generates Jet-compatible output formats
Development Note: Jet implementation is handled separately from the main retailer processing pipeline to maintain system stability and allow for specialized business logic requirements.
Testing Workflow
Recommended Testing Steps:
- Start Small: Upload single-retailer test files first
- Mixed Testing: Test mixed-retailer-orders.csv for complex scenarios
- Edge Cases: Use edge-case-validation-test.csv to verify error handling
- Export Testing: Test CSV exports with pending orders
- Status Verification: Confirm orders change from pending to exported
File Validation Checklist:
- ✓ UTF-8 encoding
- ✓ All 16 required columns present
- ✓ No "=" characters in PO numbers
- ✓ Proper quote escaping
- ✓ Valid phone number formats
9. ❓ Frequently Asked Questions
Q: Can I upload mixed retailer files?
A: Yes! Mixed retailer uploads are not only supported but are the standard practice. The system has processed over 1,000 mixed retailer batches successfully.
Q: What happens to the original CSV file?
A: The CSV file is processed and the data is extracted into the database. The original file is not permanently stored - only the structured data remains.
Q: How are batch numbers generated?
A: Batch numbers use a 9-character format with 8 random alphanumeric characters plus a checksum character for validation. Example: ABC123DEF9
Q: Can I reprocess a batch?
A: Each batch can only be exported once (controlled by status flags). This prevents duplicate processing and maintains data integrity.
Q: Where are carrier assignments determined?
A: Carrier selection is based on the carrier_details table using origin state (NC), destination state, and retailer. The system automatically assigns the optimal carrier for each order.
Q: What's the difference between sequence ID and batch number?
A:
- Sequence ID: Auto-incrementing database identifier for individual orders
- Batch Number: Generated identifier that groups all orders from a single CSV upload
Additional Documentation
Fishbowl Integration Analysis
Complete analysis of Fishbowl database usage, including current status, legacy integration, and security improvements.
- Investigation findings and current usage
- Legacy v1 vs current system comparison
- Security improvements implemented
- Future integration readiness
→ View Fishbowl Analysis
Security Configuration Guide
Comprehensive security guide covering environment variables, credential management, and best practices.
- Environment variable setup
- Security improvements made
- Usage examples and best practices
- Deployment and testing checklist
→ View Security Guide
Project Workflow Diagrams
End-to-end flows for authentication, CSV import/export, tracking ingestion, and UPC management (Mermaid diagrams).
- High-level architecture
- Orders import and exports
- Tracking SFTP intake and consolidation
- UPC CRUD
Support: For additional questions or issues not covered in this documentation, please contact the development team or check the system logs for detailed error information.