NFS Concepts and Troubleshooting

Resolving The Problem

This document will highlight the major areas of NFS, and should assist you in understanding and troubleshooting common NFS problems.

Topics covered:

1. NFS General Concepts
2. Client/Server Function and Jobs
3. Security - GID/UID and Concepts
4. NFS Mount/Export
5. etc/exports Directory and Concepts
6. Limitations of NFS
7. NFS Communications Trace Example

1. NFS General Concepts

NFS

The networked file system (known as NFS and defined in RFC 1094) is used to allow hosts to share files across a network. It was first described by Sun Microsystems Inc in 1989 and has been part of their standard product offerings since that date. It has been widely implemented on other platforms. Version 3 of the protocol defined in RFC 1813 was published in 1995.

The networked file system operates on a client-server basis using network messages to transfer requests and data between clients and servers. Data values are encoded using a method known as XDR (external data representation) described in RFC 1014. Traffic between the client and server uses the remote procedure call mechanism described in RFC 1057. NFS normally uses the UDP protocol (at R610 and above it can also use TCP) and servers listen on port 2049.

2. Client/Server Functions and Jobs

The server in NFS is the system that is doing the EXPORT. The client then mounts over this exported file system.

The QNFSBIOD (Block I/O Daemon) handles caching and routing to the server when the iSeries is the client. The QNFSNFSD is the server side job on the iSeries. It handles all of the data transactions from the clients. There can be multiple instances of both of these jobs to handle data transfers in parallel.

There are four other jobs that are used in both roles:

QNFSRPCD (RPC Binder Daemon) is the port mapper job and determines the port of a specified RPC service; in this case, NFS.

QNFSMNTD (Mount Daemon) listens for client requests involving mounting. It is used with every mount or unmount. This job checks to see if the client is allowed to mount over the file system they are requesting. If they are allowed to mount, it sends back a file handle that uniquely describes the file structure.

QNFSNLMD (Network Lock Manager Daemon) locks a file system while it is in use.

QNFSNSMD (Network Status Monitor Daemon) monitors the status of all clients connected and notifies all interested parties when the state of a client changes.

3. Security - GID/UID and Concepts

NFS uses UIDs and GIDs for all file permissions rather than a user ID and password. The QNFSANON user ID is used when a client passes a UID/GID that is unknown to our server. The easiest and least secure method is to use *PUBLIC authority. The public authority given to the directory structure depends on what type of operations must be performed on these files. For example, if the users require RWX, that is required for *PUBLIC on both ends of the connection. GID and UID are used in a similar fashion. If a group of people needs to have access to this file structure, a GID is recommended. To apply a GID to a USRPRF, a user profile must be created with a number in the GID parameter.

User options . . . . . . . . . .   *NONE         *NONE, *CLKWD, *EXPERT...
                + for more values                                          
 User ID number . . . . . . . . .   *GEN          1-4294967294, *GEN        
 Group ID number  . . . . . . . .   5000          1-4294967294, *NONE, *GEN
 Home directory . . . . . . . . .   *USRPRF    

This USRPRF then becomes a group profile and can be assigned to all of the users that participate in this NFS operation through the Group profile parameter or through the supplemental groups parameter:

Job description  . . . . . . . .   QDFTJOBD      Name                  
   Library  . . . . . . . . . . .     *LIBL       Name, *LIBL, *CURLIB  
 Group profile  . . . . . . . . .   NFSGROUP      Name, *NONE          
 Owner  . . . . . . . . . . . . .   *USRPRF       *USRPRF, *GRPPRF     

Or, if there already is a group profile, then in the supplemental groups parameter:

Group authority  . . . . . . . .   *NONE         *NONE, *ALL, *CHANGE, *USE...
Group authority type . . . . . .   *PRIVATE      *PRIVATE, *PGP              
Supplemental groups  . . . . . .   NFSGROUP      Name, *NONE                  

The UID is a unique value on the system. Therefore, if *PUBLIC is exclude only user IDs that have explicit authority to the file structure will be able to access it with this method. With this implementation, each additional user that needs to access the NFS share must have authority to the structure. If a large number of users need to have access to this share, using the GID method is the preferred implementation.

4a. NFS Mount

IBM OS/400 can mount over the following file systems:

-- Integrated File System (can not mount over the ROOT)
-- NFS
-- UDFS (user-defined file system)

When mounting over the local file system, whatever is beneath it is not accessible as long as the mount is active. The only objects that will be in this part of the file system are what is in the exported file system on the server. To issue the mount command, a user must have *IOSYSCFG special authority and read/write authority to the file system being mounted.

Following is an example of a mount command that will mount over /nfstest in the Integrated File System on the local machine. In most cases, the default mount options can be used.

Type of file system . . . . . . > *NFS *NFS, *UDFS, *NETWARE
File system to mount . . . . . . > 'rchasslh:/nfstest'
Directory to mount over . . . . > '/nfstest'        

RCHASSLH is the remote system, and /qsys.lib/nfstest.lib is the exported directory on that system.

The unmount command can be used to end an existing mount.


Type of file system . . . . . . > *NFS *NFS, *UDFS, *NETWARE, *ALL
Directory mounted over . . . . . > 'rchasslh:/nfstest'

To view all mounts, run the STATFS command:

                               Display Mounted FS Information (STATFS)
                                                           
Type choices, press Enter.                                  
                                                           
Object . . . . . . . . . . . . .   '/sreding2'              
                                                           
Output . . . . . . . . . . . . .   *             *, *PRINT  
                                                           

This is the output:

                         Display Mounted FS Information      
                                                             
 Object . . . . . . . . . . . . :   /sreding2                
                                                             
 File system type . . . . . . . :   Network File System (NFS)
                                                             
 Block size . . . . . . . . . . :   8192                      
 Total blocks . . . . . . . . . :   173223936                
 Blocks free  . . . . . . . . . :   55531980                  
 Object link maximum  . . . . . :   32767                    
 Directory link maximum . . . . :   32767                    
 Pathname component maximum . . :   255                      
 Path name maximum  . . . . . . :   No maximum                
 Change owner restricted  . . . :   Yes                      
 No truncation  . . . . . . . . :   Yes                      
 Case Sensitivity . . . . . . . :   No                        
                                                             


Another option is to view all mounts a CALL QP0FPTOS *DUMP, search for NFS, and page down to STATFS.

************************************************************************  
**********************   STATFS                 ***************************    
************************************************************************    
type          4           bsize          4096        blocks 0x0000000000000000
link_max_obj  32767       link_max_dir   32767       name_max       510        
no_trunc      1           pipe_buf       -1          max_canon      -1        
Mounted Over Directory Name:   /nfstest                                        
File System Name:                        /nfstest                                        
Mount Options:                                                                
mnt_opt_exist_flag   0x00019FFE                                                
 QP0LMNT_SETUID                     

4b. NFS Export

OS/400 can export the following file systems:

-- Integrated File System
-- QDLS
-- QSYS.LIB
-- UDFS

The EXPORTFS command requires a user profile with *ALLOBJ and *IOSYSCFG special authority. Below is an example of the EXPORTFS command:

NFS export options . . . . . . . *DFT
Directory . . . . . . . . . . . > '/nfstest'

To unexport a directory, the EXPORTFS can be used with the –U option as follows:

NFS export options . . . . . . . -U
Directory . . . . . . . . . . . > '/nfstest'
All exported directories can be unexported in the same manner using the –A option.

NFS export options . . . . . . . > '-U -A'
Directory . . . . . . . . . . .

All directories that are exported can be viewed through Operations Navigator by right-clicking on the NFS server and going to Exports. They can also be added and removed in this manner.




For a complete list of mount/export options, refer to the IBM Redbooks, Exploring NFS on AS/400, SG24-2158.Exploring NFS.sg24-5128.pdf

5. etc/exports Directory and Concepts

This file stores information about the file systems to be exported upon startup of the server. This can be edited through EDTF, DFU, or through the above Operations Navigator interface by choosing New and typing the appropriate information:



6. Limitations of NFS

Specific commands do not work well with NFS because of the nature of NFS and the differences in file structures. Problems have been found when using the CPYTOIMF, CPYTOSTMF, and CPYTOPCD commands. Because of how the CPYTOIMPF and CPYTOSTMF commands use the STMFCODPAG parameter for NFS files, it is necessary that the target stream file exist prior to calling the commands. This is a limitation of how the commands work with NFS. If greater flexibility is desired, the user could specify a target path in the local Integrated File System. The output file could then be copied/moved to the NFS directory to avoid issues that arise when using these commands.

7. NFS Communications Trace Example

This is a communications trace of a NFS write error where we are the Export (server).

COMMUNICATIONS TRACE       Title: NFS FAILURE                       10/09/02  14:11:47                           Page:       3
 Record       Data    Record           Controller  Destination   Source        Frame            Number  Number    Poll/
 Number  S/R  Length  Timer            Name        MAC Address   MAC Address   Format  Command  Sent    Received  Final  DSAP  SSAP
 ------  ---  ------  ---------------  ----------  ------------  ------------  ------  -------  ------  --------  -----  ----  ----
    274   R      169  13:54:20.15672               40007A037AE6  C00071019400   LLC     UI                         OFF    AA    AA
         Routing Info . :  0270
                      SNAP Header:  0000000800
                     Frame Type :  IP            DSCP: 0         Length:   164        Protocol: UDP           Datagram ID: 0E9C
                                   Src Addr: 9.5.186.14          Dest Addr: 9.5.186.23          Fragment Flags: MAY  ,LAST
                     IP Header  :  450000A40E9C00004011E57D0905BA0E0905BA17
                     IP Options :  NONE
                     UDP  . . . :  Src Port:   560,Unassigned    Dest Port:  2049,Unassigned    Message Length:        144
                     UDP Header :  023008010090121C

Data . . . . . :  000025BF 00000000 00000002 000186A3  00000003 00000003 00000001 0000002C   *..%*..........**...............,*
                  000003E8 00000018 52434841 53313934  2E524348 4C414E44 2E49424D 2E434F4D   *...*....RCHAS194.RCHLAND.IBM.COM*
                  00000DC7 00000000 00000000 00000000  00000000 00000020 07221998 80400001   *...*................... .".**@..*
                  000A8218 F5E70000 94CA4040 000A8218  F5E70000 94CA4040 00000009 6B617468   *..*.**..**@@..*.**..**@@....kath*
                  792E7478 74000000 *y.txt...  
                                                   

NFS Expanded:

000025BF =  Transaction ID (Unique to each RPC request/reply sequence)
00000000 =  Request/Reply Field  00000000= Request, 00000001= Reply
00000002 =  RPC Version, in this case, Version 2.
000186A3 =  RPC Program being requested (NFS is one RPC registered program, there may be others)  
    000186A0 = RPC binder/portmapper
000186A3 = NFS server
000186A5 = Mount daemon
000186B4 = Local lock mgr
000186B5 = Network lock mgr
000186B8 = Network status mon
00000003 =  Version of Protocol (could be 2 or 3, NFS ver. 1 was never released)
00000003 =  NFS Procedure Requested (NFSPROC3_LOOKUP)

Figures 1 and 1.1 (below) are for NFS version 2 and 3, since this is Version 3, figure 1.2 was used.  
Procedure NFSPROC3_LOOKUP is a common operation seen when attempting to write an object to a file.

00000001 =  Authentication type.  1= Unix (pretty much standard), could be 0 or 3 as well (very rare)
0000002C =  Authentication Length.  This defines the following amount of data that is being used to identify the client.
            Since it can vary in size, the length needs to be declared here.  In this case, the length is 2C, or, 44 bytes.

The next 44 Bytes:
000003E80000001852434841533139342E5243484C414E442E49424D2E434F4D 00000DC7 00000000     ...*....RCHAS194.RCHLAND.IBM.COM UID      GID

00000000 =  Padded Data to allow larger Authentication Length
00000000 =  Padded Data to allow larger Authentication Length
00000000 =  Verifier Auth Flavor
00000020 =  Verifier Auth Length (32 bytes, in this case)

The next 32 Bytes:
0722199880400001000A8218F5E7000094CA4040000A8218F5E7000094CA4040 file handles/file system information

00000009 =  File name length

The next 9 Bytes:
6B617468792E747874
  kathy.txt
 
Now, let's look at the response to this user's attempt to write kathy.txt to the mounted directory:



   275   S      149  13:54:20.17125               400071019400  C0007A037AE6   LLC     UI                         OFF    AA    AA
         Routing Info . :  0270
                      SNAP Header:  0000000800
                      Frame Type :  IP            DSCP: 0         Length:   144        Protocol: UDP           Datagram ID: 7F1A
                                    Src Addr: 9.5.186.222         Dest Addr: 9.5.186.14          Fragment Flags: MAY  ,LAST
                      IP Header  :  450000907F1A00004011744C0905BADE0905BA0E
                      IP Options :  NONE
                      UDP  . . . :  Src Port:  2049,Unassigned    Dest Port:   560,Unassigned    Message Length:        124
                      UDP Header :  08010230007C0F19
Data . . . . . :  000025BF 00000001 00000000 00000000  00000000 00000000 0000000D 00000001   *..%*............................*
                  00000002 000005F8 00000002 000000C4  00000000 00000000 00010000 00000000   *.......*.......*................*
                  00010000 00000000 00000000 00008040  00000001 00000000 000094CA 3DA47A34   *..............*@..........**=*z4*
                  1373FB00 3DA47A44 042D5600 3DA47AD5  193C7300                            *.s*.=*zD.-V.=*z*.<s.            *

000025BF =  Transaction ID (matches the above)
00000001 =  Reply
00000000 =  Reply Status (accepted = 0)
00000000 =  Authentication Flavor
00000000 =  Authenticaiton Length
00000000 =  Message Status (success = 0)
0000000D =  NFS V3 Error, see Figure 1.4
            ACCES (permission denied)


Figure 1.1

NFS Version 2
Decimal Hex Name
0 00 NFSPROC_NULL
1 01 NFSPROC_GETATTR
2 02 NFSPROC_SETATTR
4 04 NFSPROC_LOOKUP
5 05 NFSPROC_READLINK
6 06 NFSPROC_READ
8 08 NFSPROC_WRITE
9 09 NFSPROC_CREATE
10 0A NFSPROC_REMOVE
11 0B NFSPROC_RENAME
12 0C NFSPROC_LINK
13 0D NFSPROC_SYMLINK
14 0E NFSPROC_MKDIR
15 0F NFSPROC_RMDIR
16 10 NFSPROC_READDIR
17 11 NFSPROC_STATFS


Figure 1.2

NFS Version 3
Decimal Hex Name
0 00 NFSPROC3_NULL
1 01 NFSPROC3_GETATTR
2 02 NFSPROC3_SETATTR
3 03 NFSPROC3_LOOKUP
4 04 NFSPROC3_ACCESS
5 05 NFSPROC3_READLINK
6 06 NFSPROC3_READ
7 07 NFSPROC3_WRITE
8 08 NFSPROC3_CREATE
9 09 NFSPROC3_MKDIR
10 0A NFSPROC3_SYMLINK
11 0B NFSPROC3_MKNOD
12 0C NFSPROC3_REMOVE
13 0D NFSPROC3_RMDIR
14 0E NFSPROC3_RENAME
15 0F NFSPROC3_LINK
16 10 NFSPROC3_READDIR
17 11 NFSPROC3_READDIRPLUS
18 12 NFSPROC3_FSSTAT
19 13 NFSPROC3_FSINFO
20 14 NFSPROC3_PATHCONF
21 15 NFSPROC3_COMMIT

Figure 1.3 NFS V2 Errors

Dec Hex Error
0 0000 OK
1 0001 PERM (not owner)
2 0002 NOENT
5 0005 IO
6 0006 NXIO (no such device or addr)
13 000D ACCES (permission denied)
17 0011 EXIST
19 0013 NODEV
20 0014 NOTDIR
21 0015 ISDIR
27 001B FBIG
28 001C NOSPC
30 001E ROFS (read-only file system)
63 003F NAMETOOLONG
66 0042 NOTEMPTY
69 0045 DQUOT (disk quota exceeded)
70 0046 STALE (file handle no longer valid)

Figure 1.4 NFS V3 Errors

Dec Hex Error
0 0000 OK
1 0001 PERM (not owner)
2 0002 NOENT
5 0005 IO
6 0006 NXIO (no such device or addr)
13 000D ACCES (permission denied)
17 0011 EXIST
18 0012 XDEV (cross device hard link)
19 0013 NODEV
20 0014 NOTDIR
21 0015 ISDIR
22 0016 INVAL
27 001B FBIG
28 001C NOSPC
30 001E ROFS
31 001F MLINK
63 003F NAMETOOLONG
66 0042 NOTEMPTY
69 0045 DQUOT (disk quota exceeded)
70 0046 STALE (file handle no longer valid)
71 0047 REMOTE (fh refers to non-local)
10001 2711 BADHANDLE
10002 2712 NOT_SYNC
10003 2713 BAD_COOKIE
10004 2714 NOTSUPP
10005 2715 TOOSMALL
10006 2716 SERVERFAULT
10007 2717 BADTYPE
10008 2718 JUKEBOX (server busy)