First of all, i would like to thanks Suraj Verma, Venkateswarlu Tella, Venkateswara Chennuru, and the great opensolaris community.

If you do not have the AVS setup already, take a look at part I of this howto. Ok, let’s go..

First, we will need to create a custom agent (Resource type), because the HA.StoragePlus resource needs global devices for HA, and this howto is for HA using local devices.
So, let’s create the resource named: MRSL.NONsharedDevice.
We will use the directory /usr/scnonshared:

   # mkdir /usr/scnonshared

In that directory let’s create a file named MRSL.NONsharedDevice.rt with the following lines:

##############################################
# Resource Type: NONsharedDevice             #
# byLeal                                     #
# 17/ago/2007                                #
##############################################
RESOURCE_TYPE   = NONsharedDevice;
RT_DESCRIPTION  = "NON-Shared Device Service";
VENDOR_ID       = MRSL;
RT_VERSION      = "1.0";
RT_BASEDIR      = "/usr/scnonshared/bin";
START           = nsd_svc_start;
STOP            = nsd_svc_stop;

{
        PROPERTY = Zpools;
        EXTENSION;
        STRINGARRAY;
        TUNABLE = AT_CREATION;
        DESCRIPTION = "The Pool name";
}
{
        PROPERTY = Mount_Point;
        EXTENSION;
        STRINGARRAY;
        TUNABLE = AT_CREATION;
        DESCRIPTION = "The mount point";
}

Let’s go through some lines of that file:
p.s: Lines starting with “#” are just comments…
1) RESOURCE_TYPE: Here we have the name of our brand new Resource Type (agent).
2) RT_BASEDIR: That’s the directory base of the Resource type, where “must” be the start/stop methods.
3,4) START/STOP: The two methods that we need to implement for mount/unmount our pool/fs. The filenames will be “nsd_svc_start” and “nsd_svc_stop“.
Here you can download the two perl scripts: nsd_svc_start, nsd_svc_stop.
Ok, ok, i’m not a “real” perl hacker, but i think that two scripts do the job… the two scripts are basically the same, i don’t know if we can have just one script to start and stop methods. But, at least we can put the functions in an “include” file.. feel free to enhance it!
p.s: You will need to change the string “primaryhostname” in both scripts to the name of your server (master of the sndr volumes).
The sections between “{}“, are extensions properties that we need for customize each resource that we will be creating in this Resource Type:
1) Zpools: Like in SUNW.HAStoragePlus RT, we need this extension to associate ZFS pool with the MRSL.NONsharedDevice resource.
2) Mount_Point: This extension we need to mount the ZFS pool/fs. Because we must set the ZFS pool mount property to “legacy_mount“, for each ZFS pool that we want to use in a HA solution with “local” devices .
Both extensions must be provided “AT_CREATION” time, and there is no default.
If you change the perl scripts, keep in mind that the important point is the control of the AVS’s synchronization/replication. Like Jim Dunham have said to me:
“Golden Rule of AVS and ZFS:”

When replicating data between Solaris hosts, never
allow ZFS to have a storage pool imported on both nodes
when replication is active, or becomes active. ZFS does not
support shared writer, and thus if ZFS on one node, sees a
replicated data block from ZFS on another node, it will
panic Solaris.

and more..

Now in a failback or switchback scenario, you do have a
decision to make. Do you keep the changes made to the
SNDR secondary volume (most likely), or do you discard
any changes, and just continue to reuse the data on the
SNDR primary volume (least likely). The first thing that
needs to be done to switchback, which is automatically
provided by the NFS resource group, is to ZFS legacy
unmount the ZFS storage pool on the SNDR secondary node.

– If you want to retain the changes made on the SNDR secondary, in a script perform a “sndradm -n -m -r “
– If you want to dispose of the changes made on the SNDR secondary, do nothing.

Next allow the NFS resource group to ZFS legacy mount
the ZFS storage pool on the SNDR primary node, and now
you are done.

…

In this HOWTO we will “retain the changes made on the SNDR secondary“, so the stop/start scripts must handle that. If you are implementing a solution that does not need to retain the changes, you can edit the scripts and remove the sndradm lines. Remember that you need put that files in the directory “/usr/scnonshared/“, rename them to “nsd_svc_start” and “nsd_svc_stop“, and give them execution permissions.
You can use the start/stop scripts on command line to test them in your system. You will need to create two directories in both nodes:

 # mkdir /var/log/scnonshared
 # mkdir /var/scnonshared/run

Be sure that the ZFS pool is unmounted and exported, also, keep in mind the “AVS/ZFS golden rule”… You can try the scripts running a command similar to:

/usr/scnonshared/bin/nsd_svc_start -R poolname-nonshareddevice-rs \\
-T MRSL.NONsharedDevice -G poolname-rg

and

/usr/scnonshared/bin/nsd_svc_stop -R poolname-nonshareddevice-rs \\
-T MRSL.NONsharedDevice -G poolname-rg

The SC subsystem will call the scripts with the options we have used above…
p.s: I think this should be a information that i could find out without have to make a script to print “$@“. Would be nice find it in the scha_* man pages…

The scripts will log everything on screen, and in the log files named: nsd_svc_stop.`date`.log and nsd_svc_start.`date`.log…
Ok, after `cd /usr/scnonshared`,we can register our new RT:

 # clresourcetype register -f MRSL.NONsharedDevice.rt MRSL.NONsharedDevice

…and configure the whole resource group:

 # clresourcegroup create -p \\
PathPrefix=/dir1/dir2/POOLNAME/HA poolname-rg
 # clreslogicalhostname create -g poolname-rg -h \\
servernfs servernfs-lh-rs
 # clresource create -g poolname-rg -t \\
MRSL.NONsharedDevice -p Zpools=POOLNAME \\
-p Mount_Point=/dir1/dir2/POOLNAME poolname-nonshareddevice-rs
 # clresourcegroup online -M poolname-rg

Now, the resource group associated with the ZFS pool (POOLNAME) is online, hence the ZFS pool too. Before configure the resource for the NFS services, you will need to create a file named: dfstab.poolname-nfs-rs, in the directory: /dir1/dir2/POOLNAME/HA/SUNW.nfs/.
p.s: As you know, the dfstab is the file containing commands for sharing resources across a network (NFS shares – “man dfstab” for informations about the sintaxe of that file).

To avoid the SC “validate” check, to configure the SUNW.nfs resource, we will need to have the ZFS pool mounted on both nodes. So, let’s put the AVS software in logging mode (on primary node):

   #sndradm -C local -g POOLNAME -n -l

After that, we can import the pool (on secondary node):

  #zpool import -f POOLNAME

Now we can proceed with the cluster configuration:

   # clresource create -g poolname-rg -t SUNW.nfs -p \\
Resource_dependencies=poolname-nonshareddevice-rs \\
poolname-nfs-rs

So, now we can unmount/export the ZFS pool on secondary node, put the AVS software in replication mode, and go to the last step.. bring all the resources online.
On the secondary node:

   # unmount POOLNAME
   # zpool export POOLNAME

On primary node:

   # sndradm -C local -g POOLNAME -n -u
   # sndradm -C local -g POOLNAME -n -w
   # clresourcegroup online -M poolname-rg

That’s it, you can test the failover/switchback scenarios with the commands:

 # clnode evacuate < nodename >

The command above will take all the resources from the nodename, and will bring them online on other cluster node.. or you can use the clresourcegroup command to switch the resource group to one specific host (nodename):

clresourcegroup switch -n < nodename > poolname-rg

WARNING: There is a “time” (60 seconds by default), to keep resource groups from switching back onto a node, after the resources have been evacuated from a node (man clresourcegroup). Look here..

If you need undo all the configurations (if something goes wrong), here is the step-by-step procedure for understanding:

   # clresourcegroup offline poolname-rg
   # clresource disable poolname-nonshareddevice-rs
   # clresource disable poolname-nfs-rs
   # clresource delete poolname-nfs-rs
   # clresource delete poolname-nonshareddevice-rs
   # clresource disable servernfs-lh-rs
   # clresource delete servernfs-lh-rs
   # clresourcegroup delete poolname-rg

I think we can enhance this procedure using a point-in-time copy of the data, to avoid “inconsistency” issues during the synchronization task… but it is something i will let you comment! That’s all..

Edited by MSL (09/24/2007):
This “Agent” is changing over time (for better, i guess :), and i will use the comments sections like a “Changelog“. So, if you want to implement that solution, i recommend you to read the comments section, and see if there is some changes in the above procedure. The “stop” and “start” scripts are always permanent links, and the updated RT file can be downloaded here.