Pass or Fail? Researchers Grade Progress of Long-Term Cybersecurity Goals

In November 2003, fifty of the nation’s top computer scientists met in the Virginia countryside to create a plan tackling the biggest problems facing the growing field of computer security and privacy, known then as trustworthy computing. 

The meeting borrowed elements from the Gordon Research Conferences, meaning the discussions and attendees were never made public. It was the second in a series of highly nontraditional conferences meant to define important questions rather than present current research. 

The grand challenges established by this group of academics drove the cybersecurity research agenda for over a decade.  

Twenty-one years later, conference leaders Rich DeMillo, Georgia Tech professor and Charlotte B. and Roger C. Warren Chair in Computing, and Georgia Tech alumnus Eugene Spafford have collected feedback from the original participants and created a report on the progress made, and where they fell short.

“When we started the retrospective, we were all convinced that the whole exercise had been a failure,” said DeMillo.  “But after some reflection, that judgment seemed too harsh.” 

The rapid evolution of technology made it difficult to give each challenge a simple pass/fail grade, but the group was still able to highlight successes in their report. For example, DeMillo points out that global scale denial of service attacks never materialized because scientists figured out the right combination of policy, governance, and technology to make them ineffective at that scale.

“Context matters for when these challenges were issued,” said DeMillo. “We could not predict the new technologies and methods that sprang up over the years.” 

The Official Report Card

In May 2023, twenty years after the first Grand Challenges meeting, DeMillo and Spafford reconvened the original participants for a retrospective at Purdue University’s Center for Education and Research in Information Assurance and Security. The meeting was to see how well the community had done in predicting the course of the field. In effect, they wanted to grade the work of the original Computing Research Association (CRA) conference. 

Here’s the complete list and how today’s researchers grade the community’s progress:

Grand Challenge 1: Within the decade, eradicate widespread viral, spam, and denial of service attacks. 

Grade B-: Although global viral attacks have largely been avoided, ransomware, supply chain attacks, and malware that cripple important systems were not foreseen.

Grand Challenge 2: Develop the scientific principles, tools, and development methods for building large-scale systems to operate critical infrastructure, support democratic institutions, and further significant societal goals, ensuring their trustworthiness even though they are appealing targets. 

Grade F: Infrastructure protection has not received the same level of attention as IT, and as a result, critical systems, from electrical power grids to electronic voting systems, remain vulnerable to foreign and domestic threats.

Grand Challenge 3: For the coming dynamic, ubiquitous computing systems and applications, create an overall framework to provide end users with comprehensible security and privacy that they can manage. 

Grade D: Usable security is still an elusive goal, and a unified approach to privacy protection in the U.S. lags most developed countries.

Grand Challenge 4: In the next ten years, aim to create and implement quantitative models, methods, and tools for managing information systems risks that are on par with quantitative financial risk management techniques. 

Grade Incomplete: The economics of cybersecurity remains unexplored. From board rooms to kitchen tables, cybersecurity customers still do not know how much protection they get for every dollar spent on cybersecurity products.

Highlighting the Successes

Despite advancing technology creating a moving target for researchers, the framework laid out by DeMillo and the CRA committee has been a baseline for security research for the past twenty years. 

“The National Science Foundation, DARPA, the Department of Homeland Security, and others reflected on these challenges when they considered new research proposals,” said DeMillo. “The 2003 conference laid an important foundation for scientific growth.”

This growth created a ripple effect across “generations” of academic researchers. When the students of DeMillo and his colleagues graduated, they began advising students of their own or guiding Fortune 500 companies through the pitfalls of an ever-changing cyber landscape. Either way, these graduates confronted the challenges defined by their mentors while adapting to new ones. 

“The growth of cybersecurity academic programs like the ones we offer at SCP are directly traceable to the skills gaps that the grand challenges exposed,” he added. “And new fields like the security of engineered systems being invented here and elsewhere, are novel ways to approach the problem of systems that society can trust.”

What are the Grand Challenges to Cybersecurity Now?

According to DeMillo, those questions need to be defined by the researcher leaders of today. 

“The 2003 report was a milestone, but I hope there will be a cohort of young scientists who will lay out new grand challenges and how to confront them,” he said. 

As he points out in a report published on the CRA website, topics like AI, side-channel attacks, blockchain, and quantum computing are just a few of the emerging subfields with the potential to define the next 20 years of cybersecurity research. 

More Information on the Original Conference

In 2002, the CRA sponsored its first Grand Research Challenges in Computer Science and Engineering. This was the first in a series of highly non-traditional conferences where the goal was to define important questions rather than expose current research. Grand challenges meetings sought out-of-the-box thinking to expose some of the exciting, deep challenges yet to be met in computing research. 

Due to the importance and pressing needs for information security and assurance, CRA's second Grand Research Challenges Conference was devoted to defining technical and social challenges in information security and assurance.

The CRA and National Science Foundation tasked the conference- led by DeMillo, then dean of Georgia Tech's College of Computing, and Spafford- to define the biggest security problems facing the growing computing and communications infrastructure of the early 2000’s.

The resulting report Four Grand Challenges in Trustworthy Computing was released to the public in a ceremony at the National Press Club. It has become one of the pillars for research planners and policy-makers.