What is TTS Adjuster?
TTS Adjuster is a program intended to provide a user-friendly and efficient
means to compile and correct raw data files collected using the Turbidity
Threshold Sampling program on a Campbell data logger. Turbidity
Threshold Sampling (TTS) is an automated procedure for measuring turbidity
and sampling suspended sediment. The system was designed by the U.S. Forest
Service Redwood Sciences
Laboratory in Arcata, California.
The program creates yearly files of corrected stage, discharge, and turbidity
data and allows the user to graphically view, adjust and correct problems
in the data. Zoomable plots show raw and corrected stage and turbidity, observer
staff plate readings, and pumped samples. In order to better interpret the
data, the user may view electronic field forms and scatterplots of turbidity
versus concentration if these are available. Adjustment methods include interpolation,
variable and constant shifts, and reconstruction from reference gaging stations.
The program supports unlimited undo and redo of both edits and zooms. As the
stage and turbidity data are accepted or corrected, they are coded to reflect
the types of change that were made. Users may supply up to 10 stage-discharge
rating equations for unstable rated sections.
Getting Started
- Choose a TTS Home Directory in which all your TTS data will be stored.
Under the TTS Home Directory, create a station
information file.
- Under the TTS Home Directory, create a station directory for each gaging
station.
- Under each station directory, if you do not have field form files created
by Paradox, create a file of observer records.
- Rating equations for weirs and flumes can be hard-coded into the program
by special request. Otherwise, create a stage-discharge rating file under
each station directory.
- Under each station directory, create raw data directories for each water
year for which you have TTS data. If a file straddles two water years,
place it in the directory for the first year.
- Under each raw data directory, place your original Campbell data logger
files for the appropriate station and water year.
- If you have electronic field forms, put them in the same raw data directory
as the corresponding data files. When you open a data file for adjusting,
the program will automatically create a file of observer records from any
electronic field forms that are in the output text format of the Paradox
field form application.
Directory Structure and File Naming
Requirements
- TTS Home directory: any valid system filename
- Station information file, named stationinfo.txt
- Station directories, named stn, where stn is a unique
3-letter name for each gaging station
- Appended files: stnhy.flo, where stn
is the unique 3-letter station name and hy is the 2-digit water
year. These are the annual files that are created and modified by the
Adjuster program.
- Observer record files (optional): stnhy.or,
where stn is the unique 3-letter station name and hy
is the 2-digit water year.
- Stage-discharge rating file (optional):
stn.sdr, where stn is the unique 3-letter station
name.
- Raw data directories: rawhy, where hy
is the 2-digit water year
- Raw data files: stnyymmdd.dat, where stn
is the unique 3-letter station name, and yymmdd is the 6-digit
date of the first line in the file. The program will also recognize
data filenames beginning with 2-letter station names and file formats
created by both mixed-array (e.g. CR510/10X) and CR-Basic data loggers
(e.g. CR800/1000).
- Electronic field forms (optional, created by Paradox field form
used at Caspar Creek): stnyymmdd.nnf,
where stn is the unique 3-letter station name, yymmdd
is the 6-digit date of the oldest data in the file, and nn
is the data dump number. (The program will also recognize field form
filenames beginning with 2-letter station names)
- Laboratory data files (optional): stnhy.isc,
where stn is the unique 3-letter station name and hy
is the 2-digit water year.
Example directory structure (blue directories, red files):
File Formats
All files are comma-delimited text files. They can be created and edited
by any text editor such as NotePad in Windows. If they are created by exporting
an Excel spreadsheet as a .csv file, the extension will need to be changed
to conform to the naming requirements
described above.
Station Information File
Each line of the file is a list of comma-separated values:
- unique 3-letter station name
- watershed area in hectares (needed only for reconstructions of discharge
that use the time-linear difference method )
- units of stage measurement, either ft or m
- names of staff plate readings, if any, that are routinely read by observers
at the site. These will be shown in the plot legend. This list specifies
the order of staff plate readings in the observer
record file. For sites using Paradox field forms, the observer record
file will be automatically created with stilling well, right staff, left
staff, and average (of right and left) staff, so the list must have
four names corresponding to these staff plate readings.
- if there is more than one staff plate listed in item 4 above, the default
that is to be displayed on plots must be repeated at the end of the list.
Example station information file:
sfc,424,ft,stilling well,right staff,left staff,average,left staff
zie,25.3,ft,stilling well,right staff,left staff,average,stilling well
hyp,2750,m,right staff,left staff,right staff
ftr,3445,ft,right staff
ujc,1500,m,left staff
Observer Record File
Each line of the file is a list of comma-separated values. The staff plate
readings must correspond to the list of names starting at item 4 in the station
information file. However, the default reading is only listed once in the
observer record file. For example, an observer record file for station hyp
as defined in the example station information file, would contain the following
values:
- 4-digit year
- 2-digit month
- 2-digit day
- 4-digit military time of the staff plate reading(s)
- dump (data file) number of the electronic data recorded at the time of
the staff plate reading(s)
- one or more staff plate reading(s)
Example observer record file for station hyp, containing
a right and left staff plate reading as defined in the example station information file above:
2003,12,01,1240,3,0.06,0.06
2003,12,08,1210,4,0.05,0.04
2003,12,10,1500,4,0.11,0.10
Stage-Discharge Rating File
Each line of the file is a list of comma-separated values indicating the
starting date for each stage-discharge rating and the corresponding expression.
Use the special expression "DO NOT CALCULATE" if you have inserted
discharge values in the FLO file that you do not want to be modified.
- 4-digit year
- 2-digit month
- 2-digit day
- 4-digit military time of the staff plate reading(s)
- A mathematical expression to compute discharge from stage. Expressions
must contain a single variable, either s or x. These are not
equations, so they should not begin with "q =". Spaces
are ignored. Expressions may include the arithmetic operators + , - ,
* , / , and ^; and the mathematical functions exp, ln, log10,
log2, abs, sqrt, sin, cos, tan, cot, sec, csc, arcsin, arccos, arctan, and
round. In addition, expressions may include the one-character relational
operators =, <, and >. Relational expressions evaluate
to 0 if false or 1 if true, and are included to permit different expressions
to be applied for different stage ranges (see example below). The absence
of two-character relational operators requires a little extra work (see
the last 3 expressions below, which are all equivalent if stage is recorded
to 3 decimals).
Example stage-discharge rating equation file:
2000,08,01,0000, DO NOT CALCULATE
2001,08,01,0000, 4*s^2.5
2002,08,02,0000, 81.56*s^(1.4174 - 0.2927*ln(s))
2003,02,05,1200, (s<2)*4*s^2.5 + (s=2)*4*s^2.5 + (s>2)*3.5*s^2.1
2003,08,01,0000, 4*s^2.5 + (s>2)*(3.5*s^2.1 - 4*s^2.5)
2004,08,01,0000, (s<2.0005)*4*s^2.5 + (s>2.0005)*3.5*s^2.1
Lab Data File
Each line of the file is a list of comma-separated values:
- dump number
- bottle number
- suspended sediment concentration (mg/l), use NA for missing values
- lab quality code or other additional columns (optional)
Example lab data file:
10,002, 1.499E+02,00
10,003, 1.379E+02,00
11,001, NA,50
11,002, 6.745E+00,07
11,003, 3.083E+00,00
Appended/Corrected Output (FLO file)
Each line of the file is a list of comma-separated values:
- 4-digit year
- 2-digit month
- 2-digit day of month
- 4-digit military time
- dump number
- bottle number
- threshold and sampling codes
- threshold code: R=rising, F=falling, B=baseflow (below minimum stage)
- sampling code: X=none, T=threshold, A=auxiliary, D=aux w/DIS, S=startup,
E=equal time-interval sample above turbidity sensor limit
- raw stage (units defined in station information file)
- corrected stage
- stage code
- discharge (units determined by stage-discharge rating)
- raw turbidity
- corrected turbidity
- turbidity code
- optional fields such as water and air temperature, rainfall, etc.
Example FLO file with optional water temperature:
2003,11,12,0510,6,0,BX,0.270,0.262,4,1.514E-1,0007,0007,0,9.23
2003,11,12,0520,6,0,BX,0.270,0.262,4,1.514E-1,0008,0008,0,9.23
2003,11,12,0530,6,0,BX,0.270,0.262,4,1.514E-1,0007,0007,0,9.23
2003,11,12,0540,6,0,BX,0.270,0.262,4,1.514E-1,0020,0007,5,9.23
2003,11,12,0550,6,0,BX,0.270,0.262,4,1.514E-1,0010,0007,5,9.22
Running the Program: Overview
- To start the program under Windows or MacIntosh, simply execute adjuster.jar
or freeadjuster.jar, which may be located anywhere on your system.
adjuster.jar runs everything in one window, which is convenient, but
graphical operations are faster in freeadjuster.jar.
- Starting the program under Unix requires invoking the name of a shell script
or typing a long command.
- Identify the correct TTS Home Directory (by browsing to it or typing its
name) if it is not displayed by default in the File Dialog.
- The first time you run the program you will be prompted for the starting
month of the water year. Enter a number between 1 and 12.
- Choose a station from the Station Dropdown List.
- Starting a new .flo file
- If it will be the first .flo file for the selected station, click
on the Append button.
- If it will not be the first .flo file for the selected station,
choose last year's file from the File Name Dropdown List
and click on the Append button. If last year's file is
up to date and raw data are found in the raw data directory for the new
year, you will be prompted to confirm that you want to start a new .flo
file. If there are any data in last year's last raw file that belong to
the new water year, these data will start the new .flo file as
dump 0.
- Working with an existing .flo file
- In the File Name Dropdown List, select a file name
that you want to work on if it is not already selected by default.
- Click on the Append button if you want to add more
raw data to the selected file.
- Enter a date range and click on Read Dates, OR enter a
dump range and click on Read Dumps.
- Choose Save As from the File menu if you wish to save a
backup of the current file. In this dialog, you provide your own file name.
Give the file an extension that will not be recognized by the program. If,
at a later time, you want to restore this version, you must manually (e.g.
in Windows Explorer) replace the edited file with this backup.
- Make corrections and adjustments as described in this document.
- Choose Save from the File menu at any time to save changes
back to the original file.
- Choose Open from the File menu at any time to select a
new dump range, date range, or a new file to work on.
Changing the Appearance of the Plot
Choose Edit Format from the Edit Menu to remove
datasets or add datasets to the plot, and to
change the plot title, axis
labels, bottle labels, grids,
observer records, axis
ranges, and plotting symbols. By default, the
datasets displayed are
- Corrected turbidity
- Corrected stage
- Raw turbidity
- Raw stage
- Bottle numbers of pumped samples are plotted at the corresponding corrected
turbidity. However, when turbidity is adjusted, the bottle plotting positions
are not updated until the file is reopened.
- The default observer record type as specified in the station information
file and the differences between each observed staff plate reading and the
corresponding raw stage.
To remove a dataset, check the none
radio button next the name of the dataset and, if the data set is stage or turbidity,
uncheck the Line radio button.
To add a dataset or change its plotting symbol,
simply check a radio button for one of the symbols next to the name of the dataset
and/or check its Line radio button.
To change the plot title, stage axis label or turbidity
axis label, type the label you want in the corresponding text box.
To change the date range, stage range, or turbidity
range, type the minimum and maximum values in the corresponding text box. The
two values must be separated by commas and dates must conform strictly to the
format ddMONyyyy hh:mm.
To change the bottle label, make a selection
from the drop-down box next to the bottle symbol. This will change the bottle
label on the scatterplot as well, if it is currently displayed. Displaying a
bottle label forces a bottle symbol to be displayed as well. If the selected
bottle label is SSC, bottles with no lab data will be labelled NA.
To toggle the display of observer/raw stage
differences, click on the Show Diff radio button next to the
corresponding observer record type. Displaying the difference forces a symbol
to be displayed for the corresponding observer stage.
To toggle the display of stage or turbidity grids,
click on the corresponding radio button. Choosing a dataset to edit automatically
turns on the grid for that dataset and turns off the grid for the unselected
dataset.
Display a Field Form
Right-click on an observer record and the field form file containing that OR
will appear in a popup window
Displaying a Scatterplot
To display a scatterplot of suspended sediment concentration versus turbidity,
click on the Scatterplot button in the button bar. The display will include
only data that are within the time range shown in the main plot. Points will
be displayed either as symbols, dump numbers, or bottle numbers, depending on
the bottle label selection in Edit
Format under the Edit Menu. The scatterplot will be updated each time
the turbidity is adjusted or the time range in the main plot is altered.
Outliers in the scatterplots are useful, in conjunction with the main turbidity
plot, for identifying problems with the turbidity or concentration:
- Outlier high turbidity values associated with upward spikes in the main
turbidity plot commonly indicate fouling of the turbidity sensor.
- Poor correlation between SSC and turbidity during a noisy turbidity record
with both high and low turbidities can indicate interference from air bubbles
in a very turbulent flow.
- Outlier low turbidity values associated with a downward displacement in
the main turbidity plot may indicate the turbidity sensor was not submerged.
- Outlier high SSC values not associated with a downward displacement in the
main turbidity plot may indicate that the pumping sampler intake is very close
to or partly buried in the streambed.
Zooming In: Drag downwards with left mouse button (either right to
left or left to right)
Zooming Out: Drag upwards with left mouse button (either right to
left or left to right). Two rectangles will appear as you are dragging. These
represent the relative size of the old and new views.
Undo/Redo Zoom: The Undo Zoom(Ctrl-U) and Redo Zoom(Ctrl-V)
buttons provide unlimited undo and redo capabilities for all zooming operations.
Resetting to the Original View: From the Special
menu, choose Reset axes.
Selecting a dataset to adjust
Before you can change the data, you must first select either turbidity or
stage using Choose a Dataset on either the button bar or the Edit
menu.
Defining a time segment for adjustment
A time segment must be defined in order to use all correction methods except
freehand drawing.
Two mouse clicks anywhere on the plot define the boundaries of the segment
to be corrected.
After a segment has been defined, a subsequent mouse click will replace the
boundary that was defined first, which is always drawn in yellow.
Alt-click replaces the boundary that was defined most recently, which
is always drawn in green.
The green boundary can be shifted by one interval using the Nudge Forward
(Ctrl-F) and Nudge Backward (Ctrl-B) buttons. Nudging is necessary
when editing two adjacent segments to avoid shifting the common boundary twice.
All edits other than freehand are restricted to the defined segment, including
both endpoints.
Press Clear Segment Boundaries (Ctrl-X) to erase the current boundaries
before printing or exporting a plot.
Adjustment Methods
Control-click the mouse on one of the boundaries to define a new y-coordinate
at that boundary. The entire defined segment will be shifted by a constant
to pass through the new y-coordinate at that boundary. Use this method or
a time-variable shift when an adjustment is necessary because of an incorrect
sensor calibration intercept. (Sets the stage or turbidity code to 3)
Shift-click the mouse on one of the boundaries to
define a new y-coordinate at that boundary. A time-variable shift will occur.
The magnitude of the shift will be a function of time, varying linearly to
zero at the opposite boundary. Use this method or a constant shift when an
adjustment is necessary because of an incorrect sensor calibration intercept.
(Sets the stage or turbidity code to 4). To apply a non-zero shift at both
boundaries requires two variable shift operations.
Control-shift-click the mouse on one of the boundaries
to define a new y-coordinate at that boundary. A y-proportional shift will
occur. The magnitude of the shift will be a function of y, varying linearly
to zero for the y-value at the opposite boundary. (Sets the stage or turbidity
code to 8). To apply a non-zero shift at both boundaries requires two y-proportional
shift operations. Use this method when an adjustment is necessary because
of an incorrect sensor calibration slope.
Press the Interpolate (Ctrl-I) button to connect the
endpoints of the defined segment with a straight line. (Sets the stage or turbidity
code to 5)
Press the Reconstruct (Ctrl-K) button to reconstruct
the segment from another gaging station. (Sets the stage or turbidity code to
6). There are 4 methods implemented:
- Linear relation. This method assumes a linear relationship between the
responses at the two stations. This method works best when all missing
stage data falls within the range of the stages at start and end of correction
segment.
- Quadratic relation. This method treats the ratio of responses as a linear
function of the response at the reference station. It is equivalent to
a quadratic relationship (with no intercept) between the responses at
the two stations.
- Time-linear ratio. This method treats the ratio of the responses as
a linear function of time. This method or the next may be needed when
some of the missing stage data falls outside the range of the stages at
start and end of correction segment.
- Time-linear difference. This method treats the difference in turbidity
or unit-area discharge as a linear function of time.
The lag time is usually left at zero for stations within
Caspar Creek, although values between +/- 30 minutes are reasonable. If the
reference station is downstream, the lag should be a positive number.
Freehand Drawing
Freehand drawing is accomplished by holding the shift-key
while dragging with the right mouse button. No edit boundaries need be set for
this operation. All y values in the time period traced by the mouse will be
replaced. (Sets the stage or turbidity code to 7)
Press the Replace with Raw Data (Ctrl-R) button to
replace all data in the defined segment with raw data. (Sets the stage or turbidity
code to 0)
Press the Code Raw Data as Good (Ctrl-G) button to
recode any raw data in the segment as good. (Sets the stage or turbidity code
to 0). Codes for edited data (code > 1) in the segment will not be affected.
Code Raw Data as Questionable
Press the Recode Raw Data as Questionable (Ctrl-Q)
button to recode any raw data in the defined segment as questionable. (Sets
the stage or turbidity code to 1). Codes for edited data (code > 1) in the
segment will not be affected.
Replace Bad Data with Missing Values
Press the Replace Bad with Missing (Ctrl-M) button
to replace all data in the defined segment with missing value codes. (Sets the
stage or turbidity code to 2)
Round to Output Precision
Press the Round to Output Precision (Ctrl-T) button
to round the data and display it to the precision at which the data is stored
(3 decimals for stage and nearest whole number for turbidity). Has no effect
on stage or turbidity code.
The Undo Edit (Ctrl-Z) and Redo Edit (Ctrl-Y)
buttons provide unlimited undo and redo capabilities for all editing operations.
Stage and Turbidity Codes inserted by
program
| -1 |
Unedited,
unapproved |
| 0 |
Raw data, accepted as good |
| 1 |
Raw data, accepted but questionable |
| 2 |
Bad data, replaced with NA |
| 3 |
Constant shift was applied |
| 4 |
Time-variable (linear) shift was applied |
| 5 |
Interpolated (linearly) |
| 6 |
Reconstructed from another site |
| 7 |
Free-hand reconstruction |
| 8 |
Y-proportional
shift was applied |