模型推理移至管道末尾降舵机延迟60→16ms + 视觉Mat复用 + 端到端延迟计时 + OpenCV 4.13 + WASD遥控
This commit is contained in:
spdis
@@ -3,7 +3,7 @@
|
||||
## Project overview
|
||||
|
||||
龙芯 2K0300 嵌入式自动驾驶智能车，OpenCV + Mild v12 魔改 YOLO 模型。
|
||||
板端运行，x86 Linux 交叉编译。
|
||||
板端运行，x86 Linux 交叉编译。No unit tests, no CI, no linting — tested on-device only.
|
||||
|
||||
## Build
|
||||
|
||||
@@ -17,30 +17,33 @@ make -j$(nproc)
|
||||
- **Cross-compilation target**: LoongArch64 (`-march=loongarch64 -mtune=loongarch64`)
|
||||
- **Toolchain**: `/opt/loongson-gnu-toolchain-8.3-x86_64-loongarch64-linux-gnu-rc1.6`
|
||||
- **C++ standard**: 17
|
||||
- **OpenCV path** (device-side): `D:\PPPProgram\smartcar\opencv_device\` (mounted as `/mnt/d/...` in WSL)
|
||||
- **OpenCV path** (WSL): `/home/ilikara/loongson/opencv-4.13.0/loongson` (v4.13.0, 无 SIMD)
|
||||
- **OpenCV 设备库**: `opencv_libs/` 目录下 `.so.413` 文件，部署到设备 `/home/root/opencv/lib/` 并更新 `.so` 符号链接
|
||||
- `cross.cmake` is included **before** `cmake_minimum_required` — intentional, to set compiler before project().
|
||||
- Set `CROSS_COMPILE` to `0` in `cross.cmake` for native x86 builds.
|
||||
|
||||
## Project structure
|
||||
|
||||
| Directory | Purpose |
|
||||
|-----------|---------|
|
||||
| `src/` | Source → compiled into `common_lib` (static lib) |
|
||||
| `lib/` | Public headers only (no .cpp) |
|
||||
| `main/` | Entry point → executable `smartcar_demo1` |
|
||||
| `docs/` | Architecture docs (`ARCHITECTURE.md` is the design reference) |
|
||||
| `src/` | Source (.cpp + .hpp) → compiled into `common_lib` (static lib) |
|
||||
| `lib/` | Public headers (.h only) + `vl53l0x/` C API sources |
|
||||
| `main/` | Entry points → `smartcar_demo1` (main) + `lidar_test` (standalone test) |
|
||||
| `docs/` | Design docs (ARCHITECTURE, CONE_DESIGN, ENCODER_BRAKE, LIDAR_AVOID, TRAFFIC_LIGHT) |
|
||||
| `build/` | CMake build output (gitignored) |
|
||||
|
||||
**Executable name**: `smartcar_demo1` (CMake project name `smartcar_demo2` ≠ executable name).
|
||||
**Executable name**: `smartcar_demo1` (CMake project name is `smartcar_demo2` — mismatch is intentional).
|
||||
|
||||
## Excluded modules
|
||||
|
||||
These source files exist but are **excluded from build**:
|
||||
These source files exist in `src/` but are **excluded from build** via CMakeLists.txt `FILTER EXCLUDE`:
|
||||
|
||||
- `vl53l0x.cpp` — laser ranging module, hardware not connected
|
||||
- `zebra_detect.cpp` — classic zebra-crossing detection (replaced by Mild model)
|
||||
- `PIDController.cpp` — unused (motor is open-loop direct drive)
|
||||
- `serial.cpp` — Vofa/serial image streaming, unused
|
||||
|
||||
Note: `vl53l0x.cpp` IS compiled (lidar ranging is active).
|
||||
|
||||
## Device-side operation
|
||||
|
||||
Controlled via `ctl.sh` (writes to text files, no CLI args):
|
||||
@@ -48,106 +51,69 @@ Controlled via `ctl.sh` (writes to text files, no CLI args):
|
||||
```sh
|
||||
# On device:
|
||||
sh ctl.sh init # initialize GPIO/PWM pins + write default config files
|
||||
sh ctl.sh start # start the demo in background
|
||||
sh ctl.sh start # init_pins + start the demo in background
|
||||
sh ctl.sh stop # stop and zero PWM
|
||||
```
|
||||
|
||||
`ctl.sh` sets default config values by writing to files like `./speed`, `./kp`, `./steer_gain`, etc. The running binary reads these files at startup and optionally re-reads them every 7 frames (when `./debug` = 1).
|
||||
`ctl.sh` writes default config values to text files (`./speed`, `./steer_gain`, etc.). The binary reads them at startup via `cfg_load_all()` and re-reads every 30 frames when `./debug` = 1.
|
||||
|
||||
`start.sh` is an alternative launcher using `LD_PRELOAD=./gpio_fix_final.so` for the GPIO 74 workaround. **Note**: `start.sh` references the binary as `smartcar_demo` but the actual executable is `smartcar_demo1` — this is a known bug, use `ctl.sh` instead.
|
||||
**Bug**: `start.sh` references binary as `smartcar_demo` but actual executable is `smartcar_demo1` — use `ctl.sh` instead.
|
||||
|
||||
## Configuration system
|
||||
|
||||
All config is via single-value text files in the working directory:
|
||||
All config is via single-value text files in the working directory. The full list of parameters and defaults is in `lib/global.h` (struct `CfgCache`). Config is loaded by `cfg_load_all()` in `src/global.cpp`.
|
||||
|
||||
### Steering
|
||||
- `./speed` (double) — target speed (duty cycle %)
|
||||
- `./deadband` — steering deadband (pixels)
|
||||
- `./steer_gain` — steering gain multiplier
|
||||
- `./center_bias` — midline offset correction
|
||||
- `./foresee` — look-ahead row for steering
|
||||
|
||||
### Debug / Display
|
||||
- `./start` (0/1) — motor enable switch
|
||||
- `./showImg` (0/1) — LCD display toggle
|
||||
- `./debug` (0/1) — when 1, reloads all config files every 7 frames
|
||||
- `./destfps` — target framerate
|
||||
- `./saveImg` — when written as 1 (in debug mode), saves current frame
|
||||
|
||||
### Zebra crossing
|
||||
- `./zebrasee` — zebra crossing near-threshold (cy > this = near)
|
||||
|
||||
### Cone avoidance
|
||||
- `./cone_avoid_gain` — midline deformation push amount (normalized)
|
||||
- `./cone_avoid_range` — deformation ramp steepness
|
||||
- `./cone_speed` — speed multiplier when cone triggered
|
||||
- `./cone_min_frames` — consecutive confirmation frames
|
||||
- `./cone_margin` — 0=center point, 1=box edge
|
||||
- `./cone_thresh` — confidence threshold
|
||||
- `./cone_hold_frames` — hold deformation frames after cone disappears
|
||||
|
||||
### Encoder brake
|
||||
- `./brake_scale` — speed (pps) → brake duty cycle (ns) scaling factor
|
||||
- `./brake_max` — maximum brake duty cycle (ns)
|
||||
|
||||
### PID gains (reserved, motor is open-loop)
|
||||
- `./kp`, `./ki`, `./kd` — steering PID (currently not used)
|
||||
- `./mortor_kp`, `./mortor_ki`, `./mortor_kd` — motor encoder PID (note: spelling `mortor` is intentional)
|
||||
Key groups:
|
||||
- **Steering**: `speed`, `deadband`, `steer_gain`, `center_bias`, `foresee`
|
||||
- **Debug**: `start` (0/1), `showImg` (0/1), `debug` (0/1), `destfps`, `saveImg`
|
||||
- **Zebra**: `zebrasee`
|
||||
- **Cone avoidance**: `cone_avoid_gain`, `cone_avoid_range`, `cone_speed`, `cone_min_frames`, `cone_margin`, `cone_thresh`, `cone_hold_frames`, `cone_return_gain`, `cone_return_frames`
|
||||
- **Curve slowdown**: `curve_slope` (default 0.4), `curve_min` (default 0.8)
|
||||
- **Encoder brake**: `brake_scale`, `brake_max`
|
||||
- **Lidar avoidance**: `lidar_enable`, `lidar_thresh`, `lidar_pre`, `lidar_near`, `lidar_far`, `lidar_near_start`, `lidar_near_end`, `lidar_far_span`, `lidar_min_frames`, `lidar_avoid_gain`, `lidar_avoid_range`, `lidar_speed`, `lidar_hold_frames`
|
||||
|
||||
## Architecture
|
||||
|
||||
### Per-frame pipeline in `CameraHandler()` (`src/camera.cpp:578`)
|
||||
### Per-frame pipeline in `CameraHandler()` (`src/camera.cpp:755`)
|
||||
|
||||
CameraHandler 现已拆分为多个函数，主函数 ~40 行仅做编排：
|
||||
|
||||
```cpp
|
||||
```
|
||||
CameraHandler()
|
||||
├── capture_frame() // 1. MJPG 取帧 → 1/4解码 320×240 BGR
|
||||
├── capture_frame() // 1. MJPG 取帧 → 1/4解码 320×240 BGR
|
||||
├── save_image_if_requested() // 2. 保存帧 (debug)
|
||||
├── image_main() // 3. 视觉巡线 (80×60 HSV-Otsu-FloodFill)
|
||||
├── run_model_inference() // 4. Mild v12 每2帧推理 (160×120, 4类)
|
||||
├── zebra_process() // 5a. 斑马线去抖 + Z_STOP(4s)/Z_COOLDOWN(5s)
|
||||
├── traffic_light_process() // 5b. 红绿灯 TL_NORMAL→TL_STOP→TL_WAIT_GREEN
|
||||
├── cone_detect_and_deform() // 5c. 锥桶检测 + 中线变形 + 保持衰减
|
||||
├── steering_update() // 6. 舵机比例控制 (deadband过滤)
|
||||
├── motor_update() // 7. 电机开环PWM + 编码器刹车 + 弯道减速
|
||||
├── lcd_render() // 8. LCD RGB565渲染 (边界线+检测框+状态)
|
||||
└── fps_log() // 9. 每15帧打印分步耗时
|
||||
├── image_main() // 3. 视觉巡线 (80×60 HSV-Otsu-FloodFill)
|
||||
├── lidar_avoid_process() // 4. VL53L0X 激光避障 (每2帧) + 中线变形
|
||||
├── run_model_inference() // 5. Mild v12 每2帧推理 (160×120, 4类)
|
||||
├── zebra_process() // 6a. 斑马线去抖 + Z_STOP/Z_COOLDOWN
|
||||
├── traffic_light_process() // 6b. 红绿灯 TL_NORMAL→TL_STOP→TL_WAIT_GREEN
|
||||
├── cone_detect_and_deform() // 6c. 锥桶检测 + 中线变形 + 保持衰减
|
||||
├── steering_update() // 7. 舵机比例控制 (deadband过滤)
|
||||
├── motor_update() // 8. 电机开环PWM + 编码器刹车 + 弯道减速
|
||||
├── lcd_render() // 9. LCD RGB565渲染 (边界线+检测框+状态)
|
||||
└── fps_log() // 10. 分步耗时日志
|
||||
```
|
||||
|
||||
### Motor control details
|
||||
Also: `malloc_trim(0)` every 150 frames to return freed memory to OS (embedded optimization).
|
||||
|
||||
- **Open-loop**: `MotorController::updateduty(spd)` — direct PWM duty cycle, no encoder feedback in normal driving.
|
||||
- **Encoder brake**: When `zebra_block=true` (red light or zebra stop), reads `g_enc_speed` from the encoder thread and applies reverse braking proportional to current speed.
|
||||
- **Encoder thread** (`control.cpp:32`): polls GPIO67 (LSB pulse) + GPIO72 (direction) at ~2kHz, 100ms window speed calculation → `g_enc_speed` (pulses/sec). Now includes 500µs sleep to prevent CPU starvation.
|
||||
- **Curve slowdown**: `speed *= (1.0 - |deviation| × 0.4)`, clamped to ≥ 60%.
|
||||
### Motor control
|
||||
|
||||
- **Open-loop**: `MotorController::updateduty(spd)` — direct PWM, no encoder feedback in normal driving.
|
||||
- **Encoder brake**: When `zebra_block || tl_block`, reads `g_enc_speed` from encoder thread and applies reverse braking.
|
||||
- **Encoder thread** (`control.cpp`): polls GPIO67 (pulse) + GPIO72 (direction), 100ms window → `g_enc_speed` (pulses/sec). 500µs sleep to prevent CPU starvation.
|
||||
- **Curve slowdown**: `speed *= (1.0 - |deviation| × curve_slope)`, clamped to ≥ `curve_min`.
|
||||
- **Cone/lidar speed reduction**: Code exists but is currently **commented out** in `motor_update()`.
|
||||
|
||||
### Model: Mild v12
|
||||
|
||||
- **Architecture**: 3→9→9→13→13→19→19→19→44→44→64→64 → head(64→64,dw) → out(64→9)
|
||||
- **Input**: 160×120 BGR
|
||||
- **Output**: 4 classes (cone / red light / green light / zebra) + background
|
||||
- **Classes used in decision logic**:
|
||||
- cls=0 (cone) → midline deformation avoidance
|
||||
- cls=1 (red light) → traffic light state machine stop
|
||||
- cls=2 (green light) → traffic light state machine resume
|
||||
- cls=3 (zebra) → zebra crossing state machine stop
|
||||
- **Per-class thresholds**: `{0.90, 0.75, 0.80, 0.90}` (cone/red/green/zebra)
|
||||
- **Input**: 160×120 BGR, **Output**: 4 classes + background
|
||||
- **Class indices**: 0=cone, 1=red light, 2=green light, 3=zebra
|
||||
- **Per-class thresholds**: `{0.65, 0.75, 0.80, 0.82}` (cone/red/green/zebra)
|
||||
- **Weight file**: `mild_v12.bin` (105KB custom binary format)
|
||||
- **Inference engine**: `src/model_v10.{hpp,cpp}` (names kept as `model_v10_*` for compatibility, internally Mild v3)
|
||||
|
||||
### Vision pipeline details
|
||||
|
||||
- Processing resolution: 80×60 (optimal balance for 2K0300 CPU)
|
||||
- HSV dual-channel Otsu: H channel (blue hue) + S channel (saturation) → bitwise AND → track region
|
||||
- FloodFill: seed at bottom center (40, 50), 8-connected, extracts connected track
|
||||
- Per-row longest-run search on floodFill result → left/right boundaries
|
||||
- Missing-line recovery: propagate downward midline + virtual boundaries
|
||||
- **Inference engine**: `src/model_v10.{hpp,cpp}` (file names say v10, internally Mild v3 — historical naming)
|
||||
|
||||
## Style and conventions
|
||||
|
||||
- C++ files have **no copyright headers** — comments are ascii-box-style block comments when present
|
||||
- Header guards use `#ifndef FILENAME_H_` / `#define FILENAME_H_` format (with some exceptions: `#pragma once` in model files)
|
||||
- Global state uses `extern` globals (e.g., `g_cfg`, `g_steer_deviation`, `g_boxes`)
|
||||
- `lib/` contains .h files only; `src/` contains .cpp files only
|
||||
- No unit tests, no CI, no linting — this is embedded code tested on-device
|
||||
- No copyright headers — comments use ascii-box-style block dividers
|
||||
- Header guards: `#ifndef FILENAME_H_` / `#define FILENAME_H_` (some model files use `#pragma once`)
|
||||
- Global state via `extern` globals (e.g., `g_cfg`, `g_steer_deviation`, `g_boxes`)
|
||||
- `lib/` has `.h` headers; `src/` has `.cpp` implementations + `.hpp` model headers
|
||||
- No unit tests, no CI, no linting — do not add comments to code unless asked
|
||||
|
||||
Reference in New Issue
Block a user