Validation and Reasonableness Checks

references: Model Validation and Reasonableness Checking Manual and Calibrating and Adjustment of System Planning Models

In the following sections, the validation and calibration procedures recommended in Calibration and Adjustment of System Planning Models are compared to those found in the Model Validation and Reasonableness Checking Manual. The first section contains a short description of validation procedures and reasonableness checks in the Model Validation and Reasonableness Checking Manual that are not described in the Calibration and Adjustment of System Planning Models report. The second section shows how some validation procedures and reasonableness checks have small changes between the two reports.

In the Calibration and Adjustment of System Planning Models report, the author assumes that there is little or no good OD data available; therefore, most of his model checks are against regional values. In the Model Validation and Reasonableness Checking Manual, the authors assumed that household survey data is available; therefore, many of their recommendations involve checking the estimated values (i.e., calculated P&As) to the observed values (i.e., survey results).

New Validation Procedures and Reasonableness Checks

Model Inputs

Land Use and Socioeconomic Data

  • Summarize socioeconomic data at the city/county/regional level and compare with regional values.
  • (Employed Residents + External Residents Working in Region - Residents Working Outside the Region)/(Total Employment in Region) ~ 1.0
  • Use a GIS to calculate the population and employment densities and then display with a thematic map. Can also use GISs to plot the differences in the existing and forecasted totals for each zone or district.

Household Trip Rate comparisons

Daily Person Trips per Household by Household Size

Site

1 person HH

 2 person HH

 3 person HH

 4+ person HH

 

 

 

 

 

Amarillo

4.40

8.80

11.56

18.51

Baltimore

3.19

6.30

8.59

12.44

Eugene

4.99

9.55

14.12

22.91

Los Angeles

4.04

6.62

8.91

10.83

Madison

2.90

5.87

6.06

7.79

Minn. / St. Paul

3.80

8.40

12.20

15.9

Phoenix

4.11

6.92

9.46

27.11

Seattle

4.30

7.80

10.30

12.40

 

Source: Surveys of MPOs.

 

In looking at this table, what factors might account for the large variation in daily person trip rates for household Size 4 plus?

Transportation Network

Summarize route

  • miles or lane miles by functional class, capacity, or speed.

  • Calculate average speed or per-lane capacity by facility type and area type.
  • Adjusting the capacities and speeds in future networks to reflect advanced signal coordination or ITS strategies included in the long-range plan.

Trip Generation

Socioeconomic Disaggregation Submodels

  • Compare observed and estimated households by socioeconomic subgroups (look at auto ownership, HH size, and HH incomes).

  • Calculate the coefficient of determination (R2) of shares (i.e., percentages) of observed and estimated households by subgroups.

  • Calculate the coefficient of determination (R2) between the observed and estimated households by subgroups.

Trip Productions

  • Compare the observed and estimated trips produced at the regional level (i.e., by purpose, by area, by income, or by auto ownership).

 

 

  • Calculate the coefficient of determination (R2) between the observed and estimated trips (or trip rate) by district or zone.

Trip Attractions

  • Check the HBW person trip attractions per total employment, the HBSc trips per school enrollment, and the HBSh trip per retail employment.

Trip Distribution

  • For each area type or trip purpose, compare average trip length for productions to the average trip length for attractions or to regional values.

  • Compare the trip length distributions of the different trip purposes.

  • Using income classes, summarize the trip lengths and/or trip interchanges.

Assignment Procedures

Urban Area VMT by Facility Type

Facility Type

Urban Area Population

Small
(50-200K)

Medium
(200K-1M)

Large
(>1M)

Freeways/Expressways

18-23%

33-38%

40%

Principal Arterials

37-43%

27-33%

27%

Minor Arterials

25-28%

18-22%

18-22%

Collectors

12-15%

8-12%

8-12%

   

 

 

 

 

 

 

 

 

 

 

 From BARTON-Aschman report

 

Assignment / count error targets by Functional class

Note: The expectations are not RMSE percentages, but are average differences

Table 7-7
Percent Difference Targets for Daily Traffic Volumes by Facility Type

Facility Type

FHWA Targets

MDOT Targets

Freeway

+/- 7%

+/- 6%

Major Arterial

10%

7%

Minor Arterial

15%

10%

Collector

25%

20%

Sources: FHWA, Calibration and Adjustment of System Planning Models, 1990; Michigan Department of Transportation (MDOT), Urban Model Calibration Targets, June 10, 1993

 

Adjust the BPR coefficients for different design speeds of different facility types to improve equilibrium traffic assignments:

t =  tLOS (1 + 0.15( V/C)4 )

 

Figure 7-1

 

 

“New” factors recommended in Barton-Aschmann report are shown below, but these have not been accepted as the final answer.   TRANSCAD still starts with the original coefficients.

Table 7-2
Updated BPR Parameters Using HCM Procedures

Coefficient

Freeways

Multilane

70 mph

60 mph

50 mph

70 mph

60 mph

50 mph

Alpha

0.88

0.83

0.56

1.00

0.83

0.71

Beta

9.80

5.50

3.60

5.40

2.70

2.10

 

  • Calculate the percent RMSE.

  • Check the routing of through trips by assigning the EE trip table separately.

  • Check shortest paths using the assigned travel times.
  • Use a select link analysis to track trip patterns at key points (i.e., river or RR crossings)

Changes in Validation Procedures and Reasonableness Checks

Model Inputs

In building the transportation network, recommendations in Calibrating and Adjustment of System Planning Models focus more on the microscopic details (i.e., intersection penalities, defining TAZs to limit intrazonal trips, and correct placement of centroid connectors) where the Model Validation and Reasonableness Checking Manual takes more of a macroscopic approach (i.e., the number of route/lane miles for a functional class or the average speed in a subarea). The Model Validation and Reasonableness Checking Manual does address the problems associated with the microscopic details, but not to the degree that the Calibrating and Adjustment of System Planning Models does.

Trip Generation

In the Model Validation and Reasonableness Checking Manual, it is recommended to calculate the number of person trips per capita and then compare to other regional values. This method helps to avoid problems that may arise from variations in household sizes. The author of Calibrating and Adjustment of System Planning Models instead recommends calculating the number of person trips per dwelling unit and then comparing to regional values. Although the latter does not explicitly discuss evaluating person trips per capita, the appendix of the Calibrating and Adjustment of System Planning Models does included tabled values for this measurement.

Trip Distribution

Trip Length

The Model Validation and Reasonableness Checking Manual recommends that estimated trip lengths for each purpose should be within five percent of observed trip lengths (again the authors are assuming that travel survey data is available). The Calibration and Adjustment of System Planning Models report provides an equation for the average trip length of each purpose using the area's population. The author recommends that these values be compared to the trip distribution results.

Employment Distribution

A common problem for trip distribution in urban areas is when people living near the CBD (low income households) are matched with jobs within the area (generally high-income jobs). Both reports recommend that K factors or stratification of P&As into income classes can help account for this problem. In addition, the Model Validation and Reasonableness Checking Manual recommends the stratifying P&A data by auto ownership can also be done to account for the economic discrepancy. (NOTE: There may be issues with stratifying P&A data by auto ownership in rural areas)

Assignment Procedures

The Calibrating and Adjustment of System Planning Models report discusses the issues concerning specific assignment procedures where as the Model Validation and Reasonableness Checking Manual doesn't look at any specific assignment routine except for suggesting alternative BPR parameters for equilibrium assignment. Most traffic assignment checks in the Model Validation and Reasonableness Checking Manual are statistical measurements (i.e., VMT, VHT, and R2) which are also discussed in the Calibrating and Adjustment of System Planning Models.