The most common data file format used with the IDV is NetCDF. NetCDF data can be used for mosty any kind of geophysics data, such as 2D and 3D grids in latitude-longitude-depth (or altitude) and time such as from seismic tomography, scattered point data such as earthquake focal parameters and GPS observations, and even seismic raypaths, geological basin shapes, and seismic anisotropy.
NetCDF is used by more than 5800 scientific institutions. NetCDF was created by Unidata, part of the National Center for Atmospheric Research. Unidata software engineers continue to maintain, support and improve NetCDF. There is a complete Unidata NetCDF web site. NetCDF is a software package with data format specifications, file creation and editing software, and online support. The NetCDF format includes user-defined metadata describing the data, in particular its mapping on the Earth, and units and names for variables. This allows data in NetCDF to be used in many display and computation programs.
NetCDF data can be displayed and manipulated by dozens of independent and commercial software packages. If you use GMT you already have NetCDF software on your system. NetCDF files are machine-independent. If you make a NetCDF file on Windows and provide it to others running the IDV on Linux, Mac, various forms of UNIX, and SunOS, that data file is going to work on all those systems.
The metadata in NetCDF files described here provides complete attribution about the creators of the data, geospatial location, and other information valuable when sharing data.
The IDV can use NetCDF metadata from two or more files of differing sources to make automatic unit conversions and remappings of grids, when making computations and displays. The IDV and NetCDF can do a lot of work for you, automatically.
Put your data in NetCDF
To create NetCDF data files for use in the IDV, you can write NetCDF (binary) files directly using the NetCDF software libaries for FORTRAN, C, C++, or Java. Add FORTRAN, C, or C++ code to your programs to create NetCDF binary files.
If you already have data files in ASCII, an easier way is to write an ASCII equivalent of a NetCDF file, called a CDL file, and then convert that CDL file to the binary NetCDF file using a NetCDF utiltiy program called ncgen. Ncgen is available for many platforms. CDL files are easily read and understood by humans, and can be revised with a text editor. (Why not just use the CDL file in NetCDf applications? - they are not machine independent, and they require much more disk space than the equivalent binary NetCDF file.)
You can also invert a binary NetCDF file to the equivalent ASCII CDL form so it is readable, using the NetCDF "ncdump" utility. This lets you examine and reuse the format in any working NetCDF binary file. The command to convert a file "sample.nc" to ascii cdl form is "ncdump sample.nc > sample.cdl" on Linux. To just see the header (metadata), do ncdump -h sample.nc > sampleheader.cdl"
NetCDF files have dimensions, variables, and attributes. "Dimensions" are named integers such as latitude=1200, giving the size of a related coordinate variable array. "Variables" have one-dimensional arrays of observed data, and of location data - the dimension values - where the observed data occur. For the IDV, Every data value must have a latitude, longitude, and depth value, and in some cases, a time value. (There is also a way to put an x-y-z grid of data into NetCDF (not lat-long-altitude), if you know the mapping to a lat-long-altitude.) A variable with the same name as a dimension(number of coordinates) is a "coordinate," a location variable. It defines a physical coordinate (such as latitude) with that dimension. Typical coordinate variables are latitude, longitude, depth, and time. "Attributes" are metadata about variables, such as a variable's unit name such as kilometer/s. "Global attributes" are metadata about the entire data set, such as information about the origin of the data, campaign or station remarks, etc.
CDL
The NetCDF package includes an ascii format, called CDL, to make ascii files that are equivalent to NetCDF files. CDL files are complete inverses of the equivalent NetCDF, and vice versa. CDL is convenient since you can create, read and edit an ascii file, and then convert it to a binary NetCDF file. For the NetCDF documentation about CDL, see Section 2.1.2 Network Common Data Form Language (CDL), CDL Syntax, and CDL data types. Other parts of the complete NetCDF documentation may be useful to you.
CDL files end with extension ".cdl". NetCDF files (binary) end with extension ".nc".
NetCDF Conventions
There are several different "conventions" for NetCDF files. Conventions are best practices for metadata and structure in a NetCDF file to represent certain kinds of data. Conventions are created by users' working groups, with Unidata advice in some cases.
The IDV works best with the NetCDF CF conventions, designed for 3D and 4D grids with earth locations. You can also use the COARDS convention, a simple subset of CF 1.0.
The CF web site says the CF conventions "are designed to promote the processing and sharing of files created with the NetCDF API. The CF conventions are increasingly gaining acceptance and have been adopted by a number of projects and groups as a primary standard. The conventions define metadata that provide a definitive description of what the data in each variable represents, and the spatial and temporal properties of the data. This enables users of data from different sources to decide which quantities are comparable, and facilitates building applications with powerful extraction, regridding, and display capabilities." There is an online CF compliance checker. This is not a Unidata service but may be useful.
NetCDF files of gridded data should adher to the CF conventions ( or for simple grids COARDS conventions). CF is a superset of COARDS and is very complete; COARDS is suitable for simple regular grids.
Standards for "Point data" files with data at independent random locations
For "point data," data at independent locations on or in the earth, such as GPS measurements, earthquakes, data along seismic ray paths, and vertical soundings, Unidata provides the Unidata Observation Dataset Conventions.
This Conventions is deprecated in favor of the CF Conventions for Point Observations. We recommend that you use that as soon as it is officially adopted.
In the IDV, to recognize that a NetCDF data file is for point data, in the Data Source selector window, make the menu choice Data Type -> NetCDF point data.
Older IDV point data files did not use " Unidata Observation Dataset Conventions v 1.0." These files use an older technique to get point data into the IDV: 1. the binary NetCDF file must end in "_pd.nc" or some other special characters; 2. the depth variable must be named "Depth;" the time variable must be named "timeObs." If you have ahy of this old data it should still work but don't use this approach in future.
Units
For best use of the IDV, all variables need unit names. For recognized names of units use the UD units and see Supported Units. Check the units in sample CDL files below for data like yours.
For distances in geophysics, CDL file units of "kilometer" or "meter" covers most cases. Avoid using the names "km" or "m" as ambiguous.
CDL Time unit names
For data with time in calendar years (only) use integer time:units "years since 1900-1-1" with integer values such as 99 for 1999.
For time values to the second use a value like time:units = "seconds since 1995-01-01 00 UTC" with values such as 473385600 for a set of values all based from that time. Other examples are (hours since 1999-09-12 18:00:00), (days since 2002-12-01 00:00:00 +6:00); and (seconds since 1956-1-30 9:12:12 -3:00) (3 hrs west of UTC). See also Formatting time in NASA Ames files.
For geological times in the long past use time:units = "megayear" and negative values such as -250.0 (for "250 million years ago"), or time:units="year" and values such as -13500 (ice age period). In these cases the UDUnits package used by the IDV will interpret the time to the nearest second and you may see some very ugly time values in IDV display labels. Perhaps there is a way to indicate precision of time in UDUnits?
Global Attributes for Crediting the Data Creator and Supporting Online Data Discovery
A special set of global attributes, "the Unidata Dataset Discovery" convention attributes, should be included in your NetCDF CDL files. (These are not needed if you will not share your NetCDF data files with others).
Here is a template of the CDL for the Unidata Dataset Discovery attributes. Include this text in your CDL, and fill in the values of the attributes as needed.
These attributes are part of the NetCDF convention Unidata Dataset Discovery. For public use of NetCDF files, these attributes are strongly promoted by Unidata and UNAVCO. These attributes tell who created the data, contact information, and references to publications and web sites. These attributes also provide the data's time coverage and spatial coverage. They is not part of the CF convention.
In the IDV you can display all of any NetCDF file's metadata with a right click in the Field Selector window. In the menu that pops up, chose Properties->Metadata. If the NetCDF file does not have these attributes that menu choise may not appear.
Sample File for Gridded data (time-depth-latitude-longitude)
Here are sample CF-compliant CDL files for gridded data, with the CF convention's requirements and recommendations, and the Unidata dataset discovery global attributes. Model your ASCII CDL files on this sample CDL file. The file format, and names for coordinates, and attributes will be all or mostly the same; the variable data values and metadata values will differ.
sample 3D grid without time (AK135 earth model, with 3 variables: Vp, Vs, and density)
Sample File for a X-Y-X Grid mapped to Latitude-Longitude
Some data is on regular x-y-z grids (in kilometers for example) with a known mapping onto latitude and longitude. NetCDF provides a way to include all this information. It is pretty tricky, so check this
sample CF-compliant CDL file for x-y-z grids.
Be sure to include the global attributes as well.
Sample Files for "Point data" (independent locations)
In place of data on grids with dimensions (latitude, longtitude, depth, time), data at independent points are a list, each with a record number. Each "record" in the NetCDF file has its own latitude, longtitude, depth(optional), and time(optional), and one or several parameters at each point. Note also that if all the lat and lon and time values are the same, and depth varies, you have a vertical sounding.
To connect point data sources to the IDV, in the Data Source Chooser, use "Data Source Type" of "NetCDF point data for geophysics."
Point data files conform to the new "Unidata Observation Dataset Conventions v 1.0." Use these as a template for your point data; the lat-lon-depth-time variables and global attributes should be the same with your own data values.
Sample GPS vectors and error ellipses CDL file
Sample earthquake locations with magnitudes CDL file
Sample CDL Files for elevation data
Use special variable "Altitude" for the IDV if you want to make 3d relief plots.
CDL file of topography data
Ncgen and ncdump
Ncgen and ncdump are executables that are inverses of each other. Ncgen makes a NetCDF .nc binary file from an ASCII .cdl file; ncdump does the reverse.
You use ncgen to make a binary NetCDF file from your ascii CDL file. Make an ASCII CDL file of your data using any program or text editor you care to use. You can get ncgen by downloading and installing NetCDf on your system from here; as of October 2005 use verison 3.5.1 or 3.6.0. You will find ncgen and ncdump in for example a Linux directory like /home/user22/netcdf/netcdf-3.5.0/bin/ or on Windows in My Documents\netcdf\netcdf-3.5.0\bin\.
You make a NetCDF file from a CDL file with the command
ncgen -o outputfilename.nc inputfile.cdl
Note that the cdl file has extension .cdl and the output NetCDF file has extension .nc.
You can even use ncgen to create C or FORTRAN code to write the cdl file you desire. Ncgen is described in the online NetCDF documentation here. The ncgen "Man" page is here.
Ncdump is
a utility program to examine the contents of a NetCDF binary file by creating the CDL equivalent.
The ncdump tool generates the CDL text representation of a NetCDF dataset on standard output, optionally excluding some or all of the variable data in the output.
The output from ncdump is intended to be acceptable as input to ncgen. Thus ncdump and ncgen can be used as inverses to transform data representation between binary and text representations.
To get a new CDL file from a NetCDF file, do
ncdump datafile.nc > datafile.cdl
Often ncdump is used to reveal only the header in CDL,
so you can see how it works, or why it does not work.
Add "-h" after "ncdump " to see only the material before the "data:" line; the header.
Special NetCDF Variable Names for the IDV
In general NetCDF conventions stipulate the names of the coordiante variables, but you are free to create your own data variable names and their long_names.
In addition to CF-required and CF-recommended items in NetCDF files, the IDV recognizes special NetCDF file variable names used for some displays. These variable names must be used to create the special displays associated with each data type. If you create a valid NetCDF file with earthquake focal mechanism data, but name the NetCDF variables for strike and dip of the fault plane "var1" and "var2" the IDV can read and display the data in some simplistic ways (such as point values colored by dip), but cannot make focal mechanism symbols from it. Somehow the IDV must be told which of your data items are the strike and dip. We choose to require self-descrining NetCDF files with complete metadata and the special variable names.
For GPS motion vectors and error ellipses:
float ve(recNum) ; ve:long_name = "eastward component of vector " ; ve:units = "mm yr-1" ; float vn(recNum) ; vn:long_name = "northward component of vector" ; vn:units = "mm yr-1" ; float a(recNum) ; a:long_name = "uncertainty 1-sigma major axis length" ; a:units = "mm yr-1" ; float b(recNum) ; b:long_name = "uncertainty 1-sigma minor axis length " ; b:units = "mm yr-1" ; float corr(recNum) ; corr:long_name = "error correlation" ; corr:_FillValue = 0.0f ; corr:reference = " " ; corr:units = "" ;For earthquake locations and magnitudes use
dimensions: recNum = UNLIMITED ; variables: float lat(recNum) ; lat:long_name = "latitude" ; lat:units = "degrees_north" ; lat:standard_name = "latitude"; lat:_FillValue = 0.0f ; float lon(recNum) ; lon:long_name = "longitude" ; lon:units = "degrees_east" ; lon:standard_name = "longitude"; lon:_FillValue = 0.0f ; float depth(recNum) ; depth:long_name = "depth of epicenter below surface; negative downward"; depth:units = "kilometers" ; depth:standard_name = "depth"; depth:_FillValue = 0.0f ; int time(recNum) ; time:units = "seconds since 1970-01-01 00 UTC" ; time:long_name = "time in secs since 1970-1-1"; time:standard_name="time" ; float magnitude(recNum) ; magnitude:long_name="magnitude (Ms, mb, ...)"; magnitude:_FillValue = 0.0f;For earthquake focal mechanisms use these variable names, units, and directions.
float strike(recNum) ; // degrees CW from North float dip(recNum) ; // degrees downwards from horizon level float rake(recNum) ; // degrees down from horizon, in plane of mechanism, of motionFor earth strain observations:
float eps2_az(recNum) ; eps2_az:long_name = "azimuth of eps2, degrees clockwise from true north" ; eps2_az:units = "degrees" ; float eps2(recNum) ; eps2:long_name = "earth strain rate; most compressional eigenvalue of strain rate tensor; extension positive" ; eps2:units = "10-9 per year" ; float eps1(recNum) ; eps1:long_name = "earth strain rate; most extensional eigenvalue of strain rate tensor; extension positive" ; eps1:units = "10-9 per year" ;For grids of elevation (topography) values as data (not height coordinates) to make contour plots and 3D relief images:
float Altitude(lat, lon) ; Altitude:units = "meters" ;For seismic ray paths and other data along a track, use the variable name "altitude" in place of depth. This reserved word keys the IDV that you have track data. Also note that you can use the time variable as the dimension along the track. See also the sample raypath CDL file.
float altitude(time) ; altitude:units = "km" ; altitude:long_name = "depth below surface (negative down)" ;
Coordinate Systems and Transforms for Gridded Data
You may have gridded data in row-column or x-y-z form, with uniform spacings in kilometers for example. The IDV needs earth-located coordinates. If you know a transform of your grid points to a latitude-longitude cooordinate systme, that information can be included in a NetCDF file which the IDV can read and use to plot the data on the earth. This is part of the CF 1.0 convention, but not COARDS. It is tricky to get it to work so copy the example below exactly.