[Dshield] Certifications: Not worth the paper they are printed on?

Jon Kibler Jon.Kibler at aset.com
Sun Oct 5 18:20:38 GMT 2008

Hash: SHA1


Yesterday I was reading a blog where someone with no security experience
whatsoever was grousing that they flunked the Security+ exam. The
blogger also claimed to have over 100 certifications. In my opinion,
that many certifications undoubtedly qualifies this blogger to be the
Poster Boy for everything that is wrong with the certification process.

I do not know of anyone who has the real world experience to pass 100+
certification exams based only upon their experience. The fact that
someone can pass a certification exam WITHOUT ANY EXPERIENCE clearly
illustrates something is critically wrong with our industry's
certification process. (MCSE: Must Call Someone Experienced!)

The certification process today is utterly and completely broken.  The
single biggest problem that I see with the certification industry is the
scarcity of "real world" certifications -- those certifications that
cannot be passed by book knowledge alone -- certifications that require
hands-on real-world experience to pass, such as the RHCE, CCIE, or any
of the GIAC Gold certifications. All certifications should be as
rigorous as these and similar certifications that reflect one's ability
to do real work in the area in which they are certified.

In my humble opinion, most certifications today are not worth the paper
they are printed on. Certifications were originally conceived as a means
to help weed out fictitious resumes, or to verify that someone claiming
to have "10 years of experience" is not someone who really has "the
equivalent of one year of experience, times ten."

However, the fact that so many certifications are so lame that anyone
can buy a book, memorize it, and take and pass an exam, shows how
critically broken is the certifications process. Most certifications
today do not show that you are capable of DOING anything except
memorizing mostly useless and dated facts.

Certifications have gone from something potentially useful and
meaningful to being the equivalent of Country Club Dues. It has become
the price of admission to join a certain group of people in the
workplace. Just like your ability to pay your country club dues does not
say anything about your ability to play golf, certifications say nothing
about your ability to do the work associated with the certification. We
need to change certifications from being country club dues to being more
like PGA tour qualifications.

The entire certification process needs to change. Certifications must
once again reflect an individual's ability to DO something, verses their
ability to memorize. When someone presents a certification, an employer
needs to have some confidence that the prospective employee can actually
do the job in the real world. What needs to change? At least four things
immediately come to mind:

   1) Before taking a certification exam, you must be able to
demonstrate an auditable degree of associated work experience. For
example, the new Security+ certification calls for a minimum of 2 years
of day-to-day security experience as a recommended prerequisite. Well,
it should be made a REQUIREMENT that you MUST HAVE at least 2 years of
experience doing day-to-day security work before you are allowed to sit
for the exam.

   2) Exams must be changed from being fact-based to become
experience-based. It should not be possible to simply read books and
pass an exam. For example, the Security+ exam should include questions
that only a security practitioner would be able to answer. It should
include packet captures and ask for an interpretation. It should require
you to be able to verify a digital signature. It should present log
files and ask you to identify how the system was compromised. Etc. Real
world experience-based questions should be an integral part of each
exam's questions. It should not be possible to pass the exam without the
required hands-on experience.

   3) Certifications must have an expiration date. Knowledge in every
area of technology is transient in nature. Certifications must reflect
that they are based on the qualifications to do a job at a particular
point in time, and that those qualifications will change over time. As I
stated previously, the initial certification should require auditable
work experience. Recertification should require not only demonstrated
continued work experience, it should also require CEUs/CPEs to maintain
the certification. In fact, continuing education should be made an
annual requirement to maintain certifications between recertifications.

   4) Instructors teaching certification courses *MUST* have
demonstrable real world work experience before being deemed qualified to
teach the certification course. Probably the two certifications with the
greatest "Instructor Qualification Laugh Factor" are the EC-Council's
CEH and CHFI courses. The majority of instructors that I have met that
teach either of these two courses have NEVER done ANY real work in
either associated profession.
   -- How can an instructor properly convey to students the real thought
processes of a hacker, if they themselves have not performed dozens of
successful real world penetration tests?
   -- How can an instructor properly convey to students all that they
need to know about forensics, if they themselves have never performed a
real world forensics examination, and prepared and presented evidence in
   -- It is simply not possible to study, get a certification, and teach
these (and similar) courses without the instructor and ed center doing
an extreme disservice to their students. Instructors should be required
to not only have the certification, but they must have real world work
experience actually doing what they are teaching.
   -- Instructors should also be required to maintain additional
CEUs/CPEs beyond those required to maintain certification. Attending two
relevant conferences a year should be mandatory. (I would bet that most
CEH instructors have never even been to Defcon! How many CHFI
instructors have ever attended TechnoForensics? I bet almost none have!)
Similar qualifications and continuing education needs to be mandated of
all instructors teaching in any area of technology.

Perhaps another analogy would help clarify my concerns. Would you hire a
pilot for your corporate jet that only has a certificate saying that
they had passed flight school ground training? Someone that had no
actual experience as a pilot? Would you want this same person teaching
other wannabe pilots? I would hope not!

However, that is the situation we find ourselves in with technology
certifications. We are getting hordes of people that simply "pass ground
school" and now claim to be "capable of flying a 747." Still worse, the
majority of our instructors for technology certifications have only
"passed ground school", but are using that as the basis to hang out
their shingle claiming that they can teach others to fly, when they
themselves have never even seen the inside of the cockpit of an
airplane, not less ever actually having piloted a real aircraft.

Until certifications can become a meaningful means of verifying a
claimed level of experience and expertise, they shall remain not worth
the paper they are printed on.

In the meantime, we in the industry need to educate our managers, and
our training and HR departments as to what certifications are meaningful
and which ones are not. At the same time, we need to be teaching them
what certifications are appropriate for a given job skill. For example,
 I see CISSP mandated for numerous jobs (such as penetration tester)
where other more appropriate certifications should be used instead. But,
because CISSP is thought to be the ultimate certification in security,
they think that "one size fits all" security positions. We need help
change that thought process!

Jon Kibler
- --
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC  USA
o: 843-849-8214
c: 843-224-2494
s: 843-564-4224

My PGP Fingerprint is:
BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253

Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


Filtered by: TRUSTEM.COM's Email Filtering Service
No Spam. No Viruses. Just Good Clean Email.

More information about the Dshield mailing list