README for sg3_utils
                        ====================
Introduction
------------
sg3_utils is a package of utilities originally written to send individual
SCSI commands to storage devices that used one of the SCSI command sets.
These utilities can be divided into three groups:
   - sg_raw: the user supplies the CDB (Command Descriptor Block) and
     optionally the size of the data-in and data-out buffers
   - one command utilities: the majority of the utilities in this package
     send one SCSI command. Their names start with "sg_" while the
     remaining part of their name alludes to the name of the command which
     is sent. For example, "sg_inq" sends the SCSI INQUIRY command. Some
     utilities in this group send one of a selection of commands, typically
     those commands have a lot it common (e.g. sg_write_x).
   - copy type utilities: sg_dd, sgp_dd and sgm_dd use the Unix dd command
     functionality and command line syntax as a template. sg_dd may also be
     used to verify that two storage devices (or part thereof) contain the
     same user data, stopping at the first "miscompare". sg_xcopy sends the
     SCSI EXTENDED COPY command which in some cases can do offloaded copies.

Platforms
---------
These utilities were written on Linux and should work from Linux kernel
(lk) 2.4 through to the current series 6. The third group ("copy type")
are only implemented on Linux, but a separate portable package/utility
called ddpt implements similar functionality. The first two groups are
implemented (i.e. ported) to Android, FreeBSD, NetBSD, Solaris and
Windows.  The Windows port uses either a Cygwin or MinGW (plus Msys)
build environment (rather than Visual Studio).

Library
-------
Many of these utilities share a lot of code (e.g. SCSI error messages)
so a lot of repetition (potentially error prone) is saved by having a
library called libsgutils or some variation on that name. Distributions
(especially of Linux) have differing policies on how a library (and a
package) should be named. For that reason this package is sometimes
known as "sg3-utils" (i.e. the underscore is turned into a hyphen).
Various other packages use libsgutils. The library interface is not
altered from one package release, to the next, but the library interface
may be expanded. If a utility from one release is used with a libsgutils
from an earlier release, then the runtime linking may fail. Typically
package managers take care of these details so that runtime linking
errors should be rare.

Command Sets
------------
SCSI command sets are not the only storage command sets in wide use, there
are also ATA and NVMe command sets. There is a SCSI command set to
translate SCSI commands to ATA commands (called SAT: SCSI to ATA
Translation). SAT includes an ATA PASS-THROUGH SCSI command and sg_sat_*
utilities (there are four) are examples of using SAT. The SAS transport
(Serial Attached SCSI) can convey ATA commands through a SCSI/SAS domain
via its Serial ATA Tunnelled Protocol (STP).

NVMe command sets (e.g. Admin, NVM and MI) are relatively new. There was an
early paper on a SCSI to NVMe Translation Layer (SNTL) but it hasn't been
standardized. The sg_inq utility will send (and decode the response of) a
SCSI INQUIRY command if the underlying device is a SCSI device. If the
underlying device is a NVMe controller or namespace, then sg_inq will send
a NVMe Admin Identify command and decode the response. The sg_ses utility
(for SCSI Enclosure Services) also checks whether its underlying device is
SCSI or NVME. In the NVMe case, sg_ses translates the SCSI SEND DIAGNOSTIC
and READ DIAGNOSTIC RESULTS commands to the NVMe Management Interface (MI)
SES Send and SES Receive commands respectively. The output of the sg_ses
utility should be similar, irrespective of whether the "SES" device is
SCSI or NVMe.

The sg_raw utility may send NVMe Admin or NVM commands (as well as SCSI
commands). One difficulty with a command-line utility invoking NVME
commands is that those commands contain memory addresses for data-in (i.e.
data from the storage device) or data-out (i.e. data sent toward the
storage device) transfers. See the sg_raw manpage for how this difficulty
is addressed.

JSON
----
Taking a lead from smartmontools project, JSON is slowly being added to
those utilities that decode a significant amount of (meta-)data supplied
by storage devices. Human readable output is still the default with the
'--json' option changing the output to JSON. The '--json' option itself
takes an optional argument which is a string where each character turns
on (or off, if preceded by a '-') some attribute of the JSON output
(e.g. '--json=h-e').

One difficulty with JSON (or the json_builder library) is that its integer
representation is signed, 64 bit in size, and rendered in decimal, with a
leading negative sign if needed. However storage metadata tends to use
hexadecimal for large values (e.g. Logical Block Addresses (LBAs)). This
issue is addressed with the 'h' argument (e.g. '--json=h') which changes
decimal rendered numbers into a JSON subobject with two name-value pairs:
one name is 'i' whose value is the same decimal rendered integer, and the
other named 'hex' whose value is a JSON string with that integer rendered
in (unsigned) hexadecimal in that string.

To simplify name decoding in JSON, the "snake notation" convention is used.
This restricts names to the 26 lower case, alphabetical characters (from
English), the 10 digits and underscore. Further an underscore is never the
first or last character of the name and is never repeated. So underscore
can be used as a name separator (e.g. "device_id"). Other JSON producing
utilities follow different naming conventions. One advantage of the "snake
notation" is that it is compatible with various Unix pseudo file system
naming conventions (e.g. procfs and sysfs in Linux).

There is a manual page called "sg3_utils_json" that contains more
information.

Documentation
-------------
Manual pages ("manpages") are the primary method of utility documentation.
All utilities and scripts that are installed by this package have a
manpage. There are utilities in the examples, testing and utils
directories that are not installed and do not have manpages. Nearly
all utilities have runtime help, usually invoked with either the '-h'
short option or the '--help' long option. There is also an overarching
manpage called "sg3_utils". All manpages are placed in chapter 8 which
is for system administration commands/utilities.

The sg3_utils package and some more complex utilities have html pages:
   sg3_utils: https://sg.danny.cz/sg/sg3_utils.html
   sg_ses:    https://sg.danny.cz/sg/sg_ses.html
   sg_dd:     https://sg.danny.cz/sg/sg_dd.html

A tarball (and zip) of all the manpages from the previous release are
here:
   https://sg.danny.cz/sg/p/sg3_utils_man_html.tgz
   https://sg.danny.cz/sg/p/sg3_utils_man_html.zip

There is a html rendering of the sg3_utils manpage in the same directory
as this README file called sg3_utils.man8.html .

The previous README file is now called README.details plus there are
these OS specific files: README.freebsd , README.solaris , README.tru64
and README.win32 . To know the current state of the package the ChangeLog
file is the good reference.

The author's primary source code repository uses subversion and is on
the author's equipment (a RPi). One advantage of subversion is its
revision numbers which are simply integers starting at 1 and ascending.
For this package the current revision is 928 . The subversion repository
is mirrored in git (using "git svn" tools) here:
    https://github.com/doug-gilbert/sg3_utils


Douglas Gilbert
26th April 2023