Secrets of an AIX Administrator
Christian PruettBreaking into the Business
To kick off this series, it’s best to start at the beginning. How does someone get into the position of being an AIX systems administrator? Do you have to have some accreditation from IBM? Is a computer science degree needed? Does a secret handshake or shadowy ritual grant you the powers of root access?I’ve found that no two administrators took the exact same path to get where they are. On the one hand, I’ve met folks such as electrical engineers who have extensive programming knowledge, possess master’s degrees and become admins. On the other hand, there are plenty of people—like me—who come from non-technical focuses and have minimal formal training. (I majored in history and minored in political science, and took only one computer science class in college, which was primarily an introduction to Microsoft Windows and Office.) But there are some commonalities and steps that any admin will recommend for those interested in becoming an AIX systems administrator.
Check your motives. Ask yourself why you want to be an AIX systems administrator. There are plenty of rewards to becoming an admin, such as decent compensation, telecommuting, and flexible schedules; finding the right company or institution can make the job even better. But being an admin comes with plenty of pain, such as on-call rotations, long nights, off-hours work, and the perception that because you can telecommute, you can be contacted any time. So don’t charge lightly into this kind of commitment—you must first determine what’s important to you in your life.
Get some exposure. If you decide you want to become an AIX admin, start by getting some exposure to an AIX server. Ideally, you should find a simple system and have your own user ID for experimentation purposes. But servers aren’t typically available for just anyone to pick up; you’ll have to hunt one down. Your company might have a server available and some flexibility in setting up an account for you, but I’ve found that the easiest place to find access is through a university or a local community college. In most cases, the university will give you a bare-bones account with a few megabytes of storage for free just for taking some classes. You might have to contact the department that supports the servers to find out if they have an AIX server or other forms of UNIX, and you’ll need to be specific about what kind of access you’ll want.
Read all you can. There’s a tremendous wealth of free information out there on AIX systems administration. Publications such as POWER IT Pro keep people up-to-date with the latest news and information about AIX technology with practical tips and articles. IBM’s Redbooks on AIX are available on the web for free download in PDF format. And most admins will have a small library of books from publishers such as O’Reilly that they will lend out or recommend that you purchase. You can beef up your book knowledge quickly and easily, but real-world experience will always trump theory and handbooks.
Install Linux or play with other flavors of UNIX. Since its inception in the early 1990s, Linux has become hugely prolific. And with the advent of AIX 5L, there’s integration between AIX and Linux. Even if you have access to an AIX server, by building your own Linux box at home, you’ll be able to play around and experiment with commands as the root user—blowing a few things up in a safe environment—before you lay hands on an AIX production server that’s worth millions of dollars.
Find a mentor. Try to locate a skilled AIX admin who’s generous with sharing information and has a patient teacher’s spirit. Don’t ever think you know more than him or her; approach your admin with a kind, inquisitive attitude and learn as much as you can from seeing how he or she reacts to various situations over time. Take copious notes and ask questions whenever you can. And buy your mentor a meal or drink after work to show your appreciation for his or her knowledge and wisdom.
Apply and prepare for rejection. When trying to find a position, there’s a great paradox that happens to nearly everyone: You can’t get a job as an admin without experience, and you can’t get experience without a job. This is why you’ll need to beef up your resume by demonstrating all the time and energy you’ve invested into learning AIX. You might need to quantify this by passing a certification exam, getting references from qualified people, or finding a low-level entry position and working your way up from there. Be ready to get rejected, but don’t give up hope; stay determined and watchful.
If you follow these steps, you’ll be on the right path for a solid career in this field. And from there, you’ll understand more of the secrets of being an AIX systems administrator.
And what are the secrets, you ask? Let’s start with the all-important user ID.
User ID Utopia
One of the first things that AIX systems administrators get tasked with in order to familiarize themselves with their environments is user ID management. The theory is that this is one of the best ways for admins to get to know their user communities and the function and purpose of their servers. However, more often than not, user ID management is seen almost as a janitorial task—a sort of necessary evil that everyone has to take part in from time to time. That’s because it’s no fun working through endless password resets, changing huge groups of users who weren’t set up properly initially, or manually managing dozens of user IDs on many servers.There are some good rules for managing user IDs that aren’t typically found in manuals or training classes. I’ve acquired these guidelines after years of seeing things go amiss, taking drastic measures to fix problems, and being kept awake at all hours of the night. In general, if you use these secrets, your life as an AIX admin will go much more smoothly.
Protect the root password. The more people who know the root password, the greater the likelihood of someone running the wrong command. Guard it as though it’s a bank account number.
Homogenize user IDs. Try to get a spreadsheet or system going to keep all of the user ID numbers for all of the user IDs across your servers in sync. This way, you won’t need to be brought in to set permissions or ownership should someone need to transfer files from one server to another or use an NFS mount across systems. If you’re administering more than just a few servers, consider using a utility such as NIS or LDAP to handle things from a single location.
Make important user IDs local. Make sure that any user IDs and home directories, such as those used by applications, are defined locally to the server. I’ve seen some environments in which network outages have crippled mission-critical servers because all of their resources were located externally to the systems. If your business won’t work without some specific user IDs being present, keep them local.
Establish all groups up front. One of the strangest problems an admin can encounter is getting the message, “Cannot set process credentials,” when someone attempts to log in. This message happens when the user’s primary group doesn’t exist on the server. To solve this problem and make management easier in the long run, give all the user IDs all of the possible groups they need up front.
Use proper home directory management. If you have user IDs that consume a good deal of space in their home directories, make the home directories into their own file systems, and then set permissions and ownership as you create the user IDs. Put the home directories in places in which they make sense; I’ve seen home directories placed in all sorts of locations such as \etc and \tmp, where they could potentially harm the server or cause general confusion. Keep them in \home or another file system structure that stands out for the applications on the server.
Set password expiration warnings. If you’re using best computer security practices or have to conform to federal laws or regulations, your user IDs will have regularly scheduled password expiration dates. Do yourself a favor and set a reminder email to go to your users a week or so before the change so that they can take action before they get locked out.
Install sudo. Lastly, I believe that there’s no greater application for managing user IDs than sudo. Sudo provides a safe and secure way to give access to privileged commands while minimizing risk to the server. The sudo configuration file can be ported across multiple servers and even onto different versions of AIX, UNIX, or Linux systems. I believe in using it on any server I administer, if for no other reason than providing another audit trail to becoming the root user.
On the list of the most common activities that AIX systems administrators perform, file system maintenance comes right after user ID maintenance and password resets. I can’t count the number of times I’ve created, grown, shrunk, or deleted file systems in the course of my professional career. When you think about it, this makes sense: Almost every user needs space to store his or her data.
AIX’s flexibility for managing data comes with a downside: There are many ways to make an error in file system maintenance that will set the stage for future problems on the server. It takes only one or two commands to make a server practically impossible to grow, so that you can’t keep up with demands for additional places for data. Growing a file system the wrong way—for example, growing an IBM PowerHA file system on a server without using the PowerHA command set—can also create more work in the long run. To help you avoid file system maintenance mistakes that could result in inefficiencies on your server, this article offers some pearls of file system wisdom that I’ve acquired over the years.
Use the Root Volume Group for Rootvg Data Only
The most common mistake administrators—especially novice admins—make is to put a file system inside the root volume group (rootvg) or to use space that’s within the rootvg for application-related work. Here are the three big reasons you shouldn’t do this:- If you’re using internal disks, you’ll be more limited as to how large you can make your file systems.
- In the event that the file systems need to be exported to another system, you’ll wind up exporting your entire rootvg.
- Putting a file system within rootvg can make backups performed using mksysb or other AIX backup utilities exceedingly large.
Break Down Volume Groups
When you begin to plan a Power Systems server environment, think about how to use your disk space efficiently and purposefully. Separate the data into volume groups according to guidelines such as how dynamically VGs will change, types of data and access levels, and types of storage. For example, AIX 6.1 and 7.1 provide a new flag, -X, for the mkvg and chvg commands, which can restrict VGs to using a certain type of storage, such as solid-state drive (SSD) disks, thereby preventing a mixed-storage environment that could cause a hit on the server’s I/O. Use these tools with foresight to design a maintainable system.Use JFS2 and Scalable VGs
The more modern versions of AIX include two features of the Logical Volume Manager (LVM) that I recommend be used almost everywhere. The first is scalable volume groups. In the past, AIX had limitations as to how many physical volumes could be in a volume group, how large the volume group could get, and how large the physical partition size could be. Scalable volume groups eliminate almost every reasonable constraint previously associated with managing volume groups.The second feature is the enhanced journaled file system structure, better known as JFS2. Just as scalable volume groups improved management of larger disks, JFS2 took the limitations of JFS and rendered them obsolete by permitting file systems to grow larger than the 2TB limit. JFS2 also gave us better tools, such as the ability to shrink the size of a file system. If you create your volume groups and file systems using these two features, you’ll eliminate 90 percent of the growth and sustainability problems that existed just a decade ago.
Keep Physical Volumes the Same Size Whenever Possible
Aesthetically speaking, there’s nothing worse than pulling up information on the physical volumes in a volume group and seeing all sorts of random disk sizes. I’ve seen servers with disks ranging from 20GB to 500GB—all in the same volume group. This discrepancy in sizes can be caused by disks being mixed and matched across local and SAN storage, people responding to emergency file system growth requests, and sloppy SAN administrators. However, this kind of non-uniform disk layout can have other implications, such as limiting growth (e.g., a logical volume’s inter-disk policy is reached) and impacting I/O (e.g., a large, hot disk takes 80 percent or more of the I/O for a volume group).I recommend working with SAN administrators and those in charge of storage to come up with a uniform disk-size policy for volume groups. Set the physical volume size to something appropriate—for example, 50GB—and stick to that size policy. If you set the policies for your logical volumes to be striped across all the disks (assuming there’s some type of RAID configuration on the back end, as is the case in most SAN environments), you’ll avoid creating any hot spots. This configuration has the added benefit of making the root disk easily visible when you’re booting through a System Management Services (SMS) menu if you pick a unique size, such as 48GB, for its disks.
Simple Steps to Better File Management
By following my advice, you’ll find that once-troublesome file-system requests will become one of the easiest tasks you to perform. The tips I provided will also help you in the long run with patching your servers.Nowhere is the truism "the only constant thing is change" more relevant than in the IT world. We have to deal with the constant presence of external threats such as exploits and vulnerabilities that must be removed to prevent servers from being compromised. Then, too, software is continually updated as new tools are released and features are added to existing software and OSs. So as technology constantly evolves, keeping AIX servers up to date with the latest versions of the OS is crucial. In this article, we’ll look at how to patch AIX servers and maintain software on them.
Breaking Down the OS
Before we delve into the actual topic of patching, I want to back up a bit and discuss AIX's versioning structure. AIX has one of the easiest structures for keeping track of and understanding the levels of software on a server. Unlike Microsoft Windows, with its non-intuitive version numbers—for example, Windows 98 was released long before Windows 7—or an OS whose patch levels aren’t easily discerned when you look at the individual pieces of the OS at a glance (say, when watching a Solaris patch cluster as it runs), AIX has a simple way of iterating each version of its OS and components.The IBM AIX OS comprises four main levels: version, release, technology level (TL), and service pack (SP). The version and release numbers denote the main levels of AIX and typically change as major features or hardware enhancements are released. The next level under Release is TL; TLs are groupings of large quantities of OS filesets that contain new functions or features. Finally, under TLs are SPs, which are more minor revisions (i.e., fixes) within each TL. You can view the levels installed on your system by running the oslevel -s command.
Likewise, AIX filesets generally reflect the OS level. You can obtain the levels for a particular fileset by running the lslpp -l command against the fileset; the four numbers returned in the command output generally reflect the version, release, TL, and SP pretty well. However, other pieces of software (i.e., DB2) often don’t have any relationship to the level of the AIX OS.
Commit-Apply-Commit
The first step to maintaining the AIX OS on servers is to understand the difference between a committed and an applied fileset. When a fileset is committed on a server, there’s no way to roll back to a previous version of the software, short of a reinstall. The committed fileset is locked in place. When a fileset is applied, however, it can be rejected and removed from the server in the event that applying the fileset causes something to go wrong on the server.When it’s time to apply some patches or upgrade a server, especially when dealing with TLs and SPs, I recommend starting with a mksysb backup of the server first. Then, after the backup is done, commit any applicable filesets to cement the state of the OS. Following this procedure will provide a solid foundation and restore point for the rest of the process.
Install the software in the applied state so that in the event that something is off-kilter with the patches or upgrade, it can be promptly rejected. If you can afford to do so (even if the software doesn’t require it), I advise that you also reboot the server afterward to make sure that the system periodically gets tested and any latent problems can be shaken out. After a few weeks of the server running well and going through a vetting period, commit the software to complete the process and set the new baseline for the server.
Stick with the Big Groupings
Sometimes an application will require an individual OS fileset to be upgraded in order to work properly. For example, the pre-check for installing Oracle will often call out several filesets to be patched before the database software can be installed. However, I never advise upgrading filesets on a one-off basis; I always stick with the practice of upgrading through TLs or SPs.When admins deviate from this practice and place a number of incongruous updates on their servers, it makes administration more difficult in the long run. Commands like oslevel or instfix will return inconsistent numbers for the software levels that are running. Future patches may balk at having some components already at higher levels than they should be. Plus, if you have to contact IBM Support for help, it will make explaining your server much more complicated than simply saying, "I'm running AIX 6.1, TL 06, SP 05."
The only cases in which I would endorse deviating from this practice are if something is seriously broken on the server or if a high-profile vulnerability could impact the server. In such cases, it’s appropriate to add an IBM eFix and resume the recommended patching practice after IBM integrates the fix into a future TL or SP.
Use NIM or a Software Repository
One big headache that plagues AIX systems admins is working in an environment where servers are all across the board with their OSs. I’ve been in some shops that ran every version of AIX, from 5.1 to 7.1, with multiple TLs and SPs in each version and release. The variation of OS versions made keeping track of servers a nightmare: finding which systems needed to be patched, which ones had vulnerabilities, and picking and choosing which needed immediate patching, especially when IBM emailed warnings about high-impact alerts.The Network Installation Manager (NIM) utility makes it a breeze to install the same level of OS onto multiple servers at once. Find a good level of the AIX OS that works for your servers, install NIM onto a small LPAR, and turn that system into a bastion host that serves as a software fix repository. By doing this, you’ll only need to be concerned with one or two iterations of the OS, and you’ll find it easier to mass-deploy a homogenous OS configuration.
Watch for RPMs Versus Filesets
With the advent of Linux affinity in the AIX OS about a decade ago, it became possible to compile and install Red Hat packages onto Power Systems. While most OS patches pertain to AIX filesets, there are a fair number of applications that work closely with the OS that use RPM packages. These applications include utilities like sudo, wget, and gcc, which all have a role in administration.When you go to patch your servers, be sure to also check the RPM packages installed on your system to determine whether they should also be updated. Use both the rpm and lslpp commands to check all those other pieces of software.
Use nimadm_migrate and alt_disk_install Strategies
One of the best features of AIX that has come out in the past few years is a combination of two technologies: nimadm_migrate, which is the AIX System Management Interface Tool (SMIT) fastpath to migrating a client box to a new version of the OS, and alt_disk_install. These two commands allow admins to take the currently running OS, clone it onto a blank hdisk, and then even apply patches onto that drive. This way, the server stays completely up and there’s no risk to the currently running root volume group (rootvg) as the server is migrated to a new SP, TL, or even a completely new version and release of the OS. All it takes is a reboot to switch over to the fresh OS. And in the event something goes wrong, the server just needs to be rebooted onto the original disk to put things back the way they were.Stay Current with Patching
Keeping your servers up to date with the AIX OS and patching them on a routine schedule can stave off unnecessary downtime, give your users more features, and periodically "shake the tree" to see if anything falls out. We'll continue our exploration of AIX management in the next article, when we look at a few techniques to make networking more understandable and manageable.Networking.
In AIX, I’ve found that networking is probably one of the most complicated subjects around. Some components of networking are addressed by making changes to flat files, while others are handled through the Object Data Manager (ODM). There are typically at least three commands or ways of doing something, from adding a route to turning off an interface. And what seems like a subtle or innocuous action can wind up causing headaches on the server in due time—for example, a reboot that brings out duplicate default routes.
I have extensive experience in both setting up and messing up networking on AIX servers worldwide. Along the way, I’ve taken copious notes on how to manage networking, so that others will hopefully avoid the same plights I encountered. The following is a condensed version of the knowledge I've gleaned about AIX networking through hard-won experience.
Understand Physical and Logical Relationships
Although most forms of UNIX and Linux will have only one device under the /dev directory to describe each networking interface, AIX is different in that it will create both a device for the physical interface (e.g., ent0) and a device for the logical interface (e.g., en0). Each device serves a unique purpose; for example, media speeds are set on the physical interface, whereas IP addresses are defined on the logical interface.Knowing this about the physical and logical devices, your one key take-away is that if you need to do something to the physical interface, you’ll need to shut down, disable, or remove the logical interface before AIX will accept any commands to manipulate the physical interface.
Establish EtherChannel Devices First
Before you define any IPs on your servers, to create redundancy and get higher performance, I recommend setting up devices that use Cisco EtherChannel technology wherever possible. Much like bonding in Linux or IP network multipathing (IPMP) in Solaris, EtherChannel lets you gather multiple physical interfaces into one connection. The interfaces can be aggregated so that traffic goes down both pipes simultaneously, or they can work in an active-passive mode conditional upon failover. The "smitty etherchannel" fastpath is the easiest way I’ve found to define EtherChannel devices.One thing worth noting is that the EtherChannel device, although logical in nature, still has a physical and a logical component. For example, if you choose to use ent0 and ent1 to create an EtherChannel adapter, the physical device will be created as ent2 with a logical device of en2. The EtherChannel adapter behaves almost identically to any other standard network adapter as far as setting network tunable parameters or establishing IP aliases, but the physical interface will show some additional information when viewed using the lsdev or lsattr command.
Use SMIT for Your Initial IPs
I’ve always subscribed to the philosophy that the proficiency of an AIX administrator can generally be determined by how often or seldom the admin relies upon the System Management Interface Tool (SMIT) to do his or her work. But there are a few tasks where, rather than memorize long commands with dozens of flags, I will turn to SMIT to accomplish my work. Setting up initial IP addresses is one of those tasks.The "smitty tcpip" fastpath is the quickest route to navigating to the menus for setting up everything network related. From the SMIT Minimum Configuration & Startup screen, define your first IP address and default gateway through the menu. Then, as you need to add IPs to other interfaces, use the "smitty chinet" fastpath for those other devices. Using the Minimum Configuration & Startup screen to define additional interfaces could adversely affect the routing on the server.
Use the ODM Wisely
Most networking information in AIX is stored within the ODM. Only a few flat files that pertain to networking can be modified manually, such as /etc/hosts or the /etc/rc.tcpip and /etc/rc.nfs files. This can make it difficult to manage, change, or remove items like network routing from the server quickly and easily.Aside from the initial plumbing-up of IPs on interfaces, the one other thing I’ll do through the ODM is to assign any IP aliases onto devices. I do so using the following command:
chdev -l enX -a alias4=$IP,$NETMASK
This way, following a reboot, all IPs on that specific adapter will come up automatically. However, beyond this point, I stick with modifying other flat files.
Case in point: I’ve found the task of adding additional routes to a server by means of the ODM to be a beast. I've had to hunt through obscure corners of the CuAt database when trying to remove or adjust a single route. Instead of trying to jump through hoops in adding or removing routes through the ODM, I typically modify the /etc/rc.net file under "Part II - Traditional Configuration" and add all my routes to this area using the route command. This way, I can delete active routes off the server using a route delete command and strip them from the file without having to do a reboot or worrying about the ODM.
Smoother Networking
The tips I've provided here should help make your AIX networking routine go a little more smoothly. In my next article, I'll pull together the information I've provided in this article and the other "Secrets of an AIX Administrator" articles (see the Learning Path for an articles list and links), and I’ll conclude this series by giving you some tips on what to do following your first reboot after building a new server.What to Do After the First Server Reboot
If you’ve been an AIX systems administrator for a while, you know that when you've finished building a server and installing an OS, you haven't really finished building the server. Developers or application owners might think that because a system is up and pingable they can go ahead and load all their stuff onto it, but there’s still a lot of other work that must be done to make the system usable. In this fifth and final article in my "Secrets of an AIX Administrator" series, we’ll walk through the steps admins should take immediately after building an AIX server from scratch for the first time—the tasks you need to perform to complete the server-building process.
Step 1: Lock Down Root
The very first step you should take after your server reboots following the installation of AIX is to change the root file system password. This is when the server is most vulnerable to problems, and the key to the kingdom must be secured.
Step 2: Mirror the OS
If you’re using internal drives, I recommend that at this time you mirror the root volume group (rootvg) because there will be next to no I/O going on with the server. Although mirroring should typically fall later on the priority list, it’s worth your while to take 20 minutes now to have the system mirror the rootvg, instead of later on, when you'll be contending against users logging in and doing their initial configuration.
Step 3: Patch the Server
Similar to mirroring the OS, if you patch your server with the proper technology level (TL) and service pack (SP) at this point in the process, you’ll also avoid the hassle later on of trying to find a window of time in which to reboot the server. Patching the server now provides the additional benefit of letting you establish an expectation for the OS deployment, which will prevent users from complaining about future patching interfering with their installations and software.
Step 4: Set the Date and Time Zone
Although your server will likely have the correct time by this point, check the TZ variable in the /etc/environment file. Make sure that the system's time zone is in the correct locale, or set the time zone if your project requires doing so. You want to avoid the painful future scenario of discovering that a database or web page has been logging time in GMT when the system should have been in PST, requiring records to be backlogged or skipped forward by hours to correct the discrepancy. Also, by this point in the process, you shouldn’t have to reboot your server from now on for any further initial OS work.
Step 5: Disable Unnecessary Network Services; Install and Use SSH
The /etc/inetd.conf file is chock-full of various network services, from telnet and ftp to rsh and fingerd. Pretty much every service running either poses a security risk in today's world or is unnecessary. Thus, you should comment out the entire file and refresh the inetd daemon and instead use more secure communication protocols such as Secure Shell (SSH) for your communications to and from the server.
Step 6: Set Up IPs
Now that your system is secure from both a login and network perspective, this is the time to set up the IP addresses that you’ll need for communicating with the world.
Step 7: Grow and Develop rootvg File Systems
Depending on physical partition sizes, when an AIX server is built from scratch, the process will typically create the file systems on the server with the minimum amount of necessary space possible. This space limitation will become a hazard as soon as the server is put into use, so as a starting point, I recommend growing the file systems to at least the sizes listed in Table 1.
I’ll also do two other things at this time. First, I’ll break out /var/tmp into its own file system, sized at about 1GB. I’ll do this because many applications will use this space as a temporary area for files, and separating /var/tmp in this way will offset some of the risk of a user maxing out /var by doing something like using vi on a mammoth-sized file. The second thing I'll do is grow the paging space and dump devices to sizes that will work for application owners and enable the capture the data from any crashes that might happen.
Step 8: Define User IDs and Groups
Usually, I would recommend hooking users up with access (including sudo) as one of the last things to do on a server, because the users could prematurely hop onto the box and start doing work before everything is completed. But there’s one good reason I put this action here in the list of things to do: Defining user IDs makes it easier to perform the next step.
Step 9: Establish External Volume Groups
Rather than coming back around after disk space has been appropriated, logical volumes are defined, and file systems have been mounted to set ownership and permissions on all applicable directories and files, I typically prefer that user IDs (or at least the local application-specific user IDs) already exist, to make establishing external volume groups a job that can be done in one fell swoop.
Step 10: Create a mksysb image
If you haven't had a chance to do so, create a mksysb image of the server and store it away. There’s nothing more frustrating than having to redo all your work, and a mksysb image will save you time and energy in case you have to go to backups for a restore. And speaking of mksysb images…
Step 11: Create a "Golden" Image Using NIM
Ideally, you’ll only ever have to set up a server from scratch once. After you do this, the mksysb image that you created can be used as a "gold standard" from which all future AIX builds can be done, facilitated by Network Installation Manager (NIM). This golden image can be rapidly distributed to new servers, which reduces your time investment in creating AIX builds. Plus, if there are any components such as tunables, additional user IDs, or system configurations that you need to mass-distribute in the future, it's much easier to do so using a single administrative source rather than having to go out to each and every server individually.
Shorten Your AIX Learning Curve
Becoming a skilled AIX systems admin is no easy task. It will take years to develop and hone your skills and to become familiar with your servers and customers. Most professional admins I know will agree that it typically takes at least six months to become proficient with an individual component of new technology, such as when LPARs and Virtual I/O Servers came on the scene, and at least nine months of understanding your employer's business before your learning curve takes you from being dangerous to knowing the basics of how their business works.In this five-article series, I’ve shared a number of tips and tricks I've learned along the way—sometimes painfully—that have helped me whenever I go into new environments and lay hands on keys for the first time in different industries. Integrating these ideas and concepts into your own skill set will better your chances of avoiding mistakes, improve your work, and help you develop as an AIX systems admin.
No comments:
Post a Comment