Last Day to Save up to $400 on Cyber Security Training at SANSFIRE 2018 in Washington DC!

SEC487: Open-Source Intelligence Gathering and Analysis Beta

Immeasurable amounts of personal, potentially incriminating data is currently stored in the websites, apps, and social media platforms that people access and update via their devices daily. That data can become evidence for citizens, governments, and businesses to use in solving real financial, employment, and criminal issues with the help of a professional information gatherer.

SEC487 will teach students legitimate and effective ways to find, gather, and analyze this data from the Internet. You'll learn about reliable places to harvest data using manual and automated methods and tools. Once you have the data, we'll show you how to ensure that it is analyzed, sound, and useful to your investigations.

This is a foundational course in open-source intelligence (OSINT) gathering and, as such, will move quickly through many areas of the field. The course will teach you current, real-world skills, techniques, and tools that law enforcement, private investigators, cyber attackers, and defenders use to scour the massive amount of information across the Internet, analyze the results, and pivot on interesting pieces of data to find other areas for investigation. Our goal is to provide the OSINT knowledge base for students to be successful in their fields whether they are cyber defenders, threat intelligence analysts, private investigators, insurance claims investigators, intelligence analysts, law enforcement personnel, or just someone curious about OSINT.

Throughout the course week, students will participate in numerous hands-on labs using the tools and techniques that are the basis for gathering free data from the Internet. The 20 labs in this course use the live Internet and dark web to help students gain real-world confidence. You'll leave the course knowing not just how to use search features on a website, but all of the scenario-based requirements and OSINT techniques needed to gather truly important OSINT data.

Course Syllabus

Overview

We begin with the basics and answer the questions "what is OSINT" and "how do people use it." This first day is about level-setting and ensuring that all students understand the background behind what we do in the OSINT field. We also establish the foundation for the rest of the week by learning how to document findings and set up an OSINT platform, and we discuss effective research habits for OSINT analysts. This day is a key component for the success of an OSINT analyst because without these concepts and processes in place, researchers can get themselves into serious trouble during assessments by inadvertently alerting their targets or improperly collecting data, making it less useful when delivered to the customer.

During the first half of the day we work through the pieces of the OSINT cycle to understand what our process might look like. Then we move into how law-abiding people use OSINT to get the data they need, be it parents trying to figure out if the person they want to hire as a child care worker is trustworthy, a person "googling" someone they are going out with on a date, businesses looking for information about a rival company, or law enforcement using social media and OSINT to capture alleged criminals. We then move into how criminals use OSINT to target victims, via "SWATting," identity theft, and other attacks against people and their electronic data. We finish the morning with a review of documentation tools for creating MindMaps, compiling notes, creating timelines, and analyzing relationships within data.

The day continues by jumping into understanding threat profiles so that we can protect ourselves and infiltrate the places we need to gather the data our customers want. Recognizing that some students will be creating their own OSINT collection platforms, we move into what that may look like, covering topics such as the platform, operating system, tools, and plugins. At the end of the day, students learn about the use of sock puppet (false accounts) and will set up their own account for the labs in the course.

Exercises
  • Setting up the course virtual machine and configuring the VPN that is used to secure all web traffic
  • Using a MindMap tool to document OSINT data and then analyzing relationships between people using a data visualization application
  • Setting up a password manager to securely store all the passwords that we will need for our sock puppets and other accounts
  • Creating a sock puppet account with realistic user-attributes, which will be key to succeeding in some of the other labs later in the course
  • Joining a class Slack group to discuss OSINT and the class by way of a lab that walks you through the setup and use of the application

CPE/CMU Credits: 6

Topics
  • Understanding OSINT
    • The OSINT cycle
  • Goals of OSINT Collection
    • Law enforcement
    • Parents
    • Spouses
    • Businesses
    • Media
    • Intelligence agencies
    • Criminals
  • Diving into Collecting
    • How just "diving" into an assessment can cause problems
    • Case study
  • Taking Excellent Notes
    • Why good note-taking is important
    • How to document
    • Tools to document
      • Visualization
      • Note-taking applications
      • Documentation application/Hunchly
      • Word processing
    • Taking screenshots
    • Timelines
  • Determining Your Threat Profile
    • How "covert" do you need to be?
    • Methods that may reveal what are doing OSINT to a target
  • Setting Up an OSINT Platform
    • Types of platforms
      • Virtual machine (VM)
      • USB media
      • Cloud server
    • Web browsers
      • Useful extensions and add-ons
    • Data at rest
    • Mobile OSINT
    • Password management
  • Effective Research Habits
    • Engaging your target
    • Sensitive data
    • Sanitizing your platform
    • Managing your time
    • Being disciplined
  • Creating Sock Puppets
    • What is a sock puppet?
    • Why do we use them?
Overview

OSINT data collection begins on day two after we get a glimpse of some of the fallacies that could influence our conclusions and recommendations. From this point in the class forward, we examine distinct categories of data and think about what it could mean for our investigations. Retrieving data from the Internet could mean using a web browser to view a page or, as we learn in this section, command line tools, scripts, and helper applications can also be used.

At first, our focus turns to grabbing data from and about websites. Tools such as SSL/TLS certificates and Google analytics IDs can be important to our assessments. We then examine different methods for finding and validating basic data about people such as home addresses, phone numbers, and email addresses. These pieces of data become "pivot points" for our investigations, as we can often perform additional searches using these key data points to discover additional data that may be useful in our work. Students learn how to harvest user names and avatars and how they tie a single user to multiple user profiles across sites.

With user avatar images fresh in our minds, we pivot and consider how to perform reverse image searches. Day two concludes with a deep look at how we can execute advanced search engine queries to increase our chances of getting meaningful results.

Exercises
  • Harvesting web data such as Google Analytics IDs and content from HTTP certificates
  • Tracing a home address and phone number to their owners
  • Gathering email addresses for a company and then determining whether they have been compromised
  • Using a reconnaissance framework to rapidly scan websites looking for specific user accounts
  • Searching reverse images to find the identity of the person and other places that image was used

CPE/CMU Credits: 6

Topics
  • Data Analysis Challenges
    • Inaccurate data
    • Bias
    • Analysis fallacies
  • Creating Your OSINT Process
    • Start with a clean system
    • Gathering requirements
    • Decide on TTPs
    • Gathering data
    • Analyzing data
    • Creating output for customer
  • Harvesting Web Data
    • Proxy web applications
    • Command line tools for harvesting data
    • Scripting with python
    • APIs
    • Cached content
    • Google Analytics
    • Encryption certificates
  • OSINT Frameworks
    • PTES
    • Advanced Recon Framework (ARF)
  • Basic Data: Street Addresses
    • Why gather street addresses?
    • Places you can find street addresses
  • Basic Data: Phone Numbers
    • Reverse phone look-ups
    • Places you can find phone numbers
  • Basic Data: Email Addresses
    • Places you can gather email addresses
    • Email formats
    • Email validation
    • Gathering emails in bulk
  • User Names
    • Understanding why we collect user names
    • Websites and tools that can be used to harvest user names
  • Avatars and Reverse Image Searches
    • Why avatars are interesting to OSINT
    • Image search engines
  • Leveraging Search Engines
    • Detailed examination of several search engines
    • Advanced queries and directives
Overview

Finding data on people, especially basic content such as email addresses, home address, and phone numbers, can be made easier using online people search engines. This is how day three kicks off, examining free and paid choices in this data aggregator area and understanding how to use the data we receive from them. Some of these engines provide social media content in their results. This makes a terrific transition for us to move into social media data.

The first social media site we look at from an OSINT perspective is Facebook, as it is one of the largest in the world. Students explore Facebook profiles, groups, events, and other Facebook objects using graph searches and Facebook query techniques. We then move to detailed examinations of LinkedIn, Twitter, and Instagram, and what OSINT data can offer for each of them.

Focusing on the "social" aspect of social media, we dive into the content on dating and adult websites. A natural progression from dating is sometimes a wedding, so we inspect wedding websites and registries for OSINT data. Next, we see how we can use web and traffic cameras for remote reconnaissance. We finish the day examining document and image metadata to glean interesting data points from different document types.

Exercises
  • Executing queries on search engines to find information about someone
  • Conducting Facebook queries to retrieve surface and deep data
  • Analyzing tweets to determine sentiment and to discover where the tweets are geolocated
  • Scraping metadata and mapping GPS coordinates

CPE/CMU Credits: 6

Topics
  • People Search Engines
    • Free people search engines
    • Paid consumer-level search engines
    • Commercial aggregators
    • Family trees
  • Facebook Analysis
    • Facebook primer
    • Intro to graph search
    • Websites that make searching Facebook easier
    • Crafting custom graph searches
    • Finding intersections and relationships
    • Gathering business data
  • LinkedIn Data
    • LinkedIn's value for OSINT
    • Data that can be gleaned from LinkedIn
  • Twitter Data
    • OSINT value of Twitter
    • Searching Twitter
    • Tweet content analysis
    • Twitter geolocation
    • Deleted tweets
    • Gathering data from protected accounts
  • Instagram
    • OSINT value of Instagram
    • Retrieving data from Instagram with custom URLs
    • Instagram API
  • Dating and Adult Websites
    • Harvesting data from dating and adult site user profiles
  • Registries and Wish Lists
    • OSINT value of registries and wish lists
    • Finding registries and wish lists
  • Web and Traffic Cameras
    • Reliable web and traffic camera sites
    • Leveraging these cameras in assessments
  • File Metadata Analysis
    • What is metadata?
    • How do we use it?
    • How do we retrieve it?
Overview

Day four focuses on many different but related OSINT issues. We begin by looking at how various mapping sites can assist our assessments with aerial data, distance-measuring, and "street view" imagery. Moving beyond just using one vendor's mapping system, students will learn about the breadth of free mapping resources available for OSINT.

We then shift from OSINT about people and locations to OSINT about networks and computers, as researching IP addresses, domain names, and related content can be important aspects of our investigations. Starting with the basics, we get comfortable retrieving information about IP addresses and network blocks, and using the whois protocol. Students then move to making advanced queries to the domain name system (DNS) to grab subdomains and other domain data. To complete our work looking at computers, we examine how we use wireless network data in our work.

The second portion of the day has three modules. The first covers OSINT framework tool suites. These tools can accelerate our OSINT research by very rapidly acquiring data about people, networks, hosts, and more. We examine three frameworks in-depth during class. The next module covers harvesting information from federal, state, and local government web pages. The public data on these sites can help us research people and businesses. Completing the day, we look at the methods that can be used to gather data about businesses.

Exercises
  • Using online mapping sites to recon an area
  • Searching for wireless network data and using it to verify an alibi
  • Running an OSINT framework to discover what information can be found about a domain
  • Examining various government websites to answer trivia questions
  • Gathering data points about the CEO and the systems used at a business

CPE/CMU Credits: 6

Topics
  • Remote Location Recon
    • Satellite and aerial imagery
    • Ground-based imagery (commercial and consumer)
    • Using mapping tools to measure data and markup maps
    • Using historical ground-based imagery
  • Geolocation
    • OSINT value of geolocation
    • Faking GPS locations
    • Analysis of social media that geolocates
    • Geolocation discovery tools
  • IP Address and Whois
    • Basic introduction of computer networking concepts
    • Whois protocol
  • IP Address Geolocation
    • Accuracy of IP address geolocation
    • How to geolocate from an IP address
  • Domain Name System (DNS)
    • Description of DNS
    • DNS data
  • Wireless Networks
    • Searching wireless network data
  • Recon Tool Suites and Frameworks
    • Detailed comparison of three reconnaissance frameworks
      • SpiderFoot
      • Intrigue
      • Recon-ng
  • U.S. Government Data
    • Federal resources
    • State and local government resources
  • Researching Companies
    • Retrieving basic data about businesses
    • Business profile sites
    • Non-profit and charity organization OSINT
    • Business filings/EDGAR
    • International business OSINT
    • Management analysis
    • Business systems analysis
      • Censys.io
      • Shodan.io
Overview

The entire morning of day five focuses on understanding and using three of the most popular dark web networks for OSINT purposes. Students will learn why people (good and bad) use Freenet, I2P, and Tor. Each network is discussed at length so that students don't just know how and why to use it - they will also gain an understanding of how those networks work. With the Tor network being such a big player in the dark web, the course spends extra time diving into its resources.

The first module in the afternoon examines how we scrape content from paste sites. These websites sometimes contain content such as user names and passwords of compromised user accounts, detailed network information about our target's systems, or just data that our customers need to know. We then turn our focus to international issues by performing OSINT activities on websites outside of the United States. Considering that a big barrier to using non-English websites can be the language, students learn how to use techniques to translate content and search locally for relevant information. We also will examine how to discover popular websites and applications used in foreign countries. Since we talk about international data and traveling around the world, our courseware finishes up with an examination of how we track transportation (planes, boats, cars, etc.).

We leave some time at the end of the day for a massive lab (the "Grand Exercise") that helps students put together all that they have learned in a semi-guided walk-through that touches on many of the concepts taught throughout the week. Setting aside time to work through our OSINT process in an organized manner will reinforce key concepts and allow students to practice both executing OSINT procedures and techniques.

Exercises
  • Diving into the deep web by using Tor to visit Internet sites and hidden services, and even setting up our own hidden service and then visiting it
  • Using translation sites to practice transforming languages into other languages
  • Discovering the popular websites and mobile apps used in several countries
  • Undertaking the Grand Exercise that brings together many of the previous labs and helps students practice process

CPE/CMU Credits: 6

Topics
  • The Surface, Deep, and Dark Webs
    • Levels of the Internet
    • Understanding of what data is at what layer and how to access
  • The Dark Web
    • Risks in using the dark web
    • Overview of top three dark web networks
  • Freenet
    • Modes of Freenet
    • Accessing Freenet
    • Services and resources in the Freenet
  • I2P - Invisible Internet Project
    • What data is in I2P?
    • I2P tunnels
    • Using I2P
    • Eepsites
  • Tor
    • Who uses Tor and why?
    • How Tor works
    • Dangers of using Tor
    • Accessing Tor
    • Tor hidden services
    • Sharing files in Tor
  • Searching Data Dump Sites
    • What do people use paste sites for?
    • Harvesting content from paste sites
  • International Issues
    • Language tools
    • Popular websites
    • Popular mobile applications
    • Searching regionally
  • Vehicle Searches
    • License plates
    • Vehicle ID Numbers (VINs)
    • Plane registrations
    • Plane tracking
    • Ships and watercraft
    • Putting It All Together
Overview

The capstone for the course is a group event that brings together everything that students learned throughout the week. This is not a "canned" Capture the Flag event where specific flags are planted and your team must find them. It is a competition where each team will collect specific OSINT data about a certain group of people. The output from this work will be turned in as a "deliverable" to the "client" (the instructor), and then the three teams with the most-complete work will present their research to the class for voting.

This four-hour, hands-on event will reinforce what the students practiced in the Grand Exercise the day before and add the complexity of performing OSINT assessments under pressure and in a group.

CPE/CMU Credits: 6

Topics
  • Capstone Capture the Flag Event

Additional Information

!! IMPORTANT - BRING YOUR OWN LAPTOP CONFIGURED USING THESE DIRECTIONS!!

A properly configured system is required for each student participating in this course. Before coming to class, carefully read and follow these instructions exactly.

You can use any 64-bit version of Windows, MacOS, or Linux as your core operating system that also can install and run VMware virtualization products. You also must have 8 GB of RAM or higher for the VM to function properly in the class.

It is critical that your CPU and operating system support 64-bit so that our 64-bit guest virtual machine will run on your laptop.

In addition to having 64-bit capable hardware, AMD-V, Intel VT-x, or the equivalent must be enabled in BIOS/UEFI.

Please download and install VMware Workstation 11, VMware Fusion 7, or VMware Workstation Player 7 or higher versions on your system prior to the start of the class. If you do not own a licensed copy of VMware Workstation or Fusion, you can download a free 30-day trial copy from VMware. VMware will send you a time-limited serial number if you register for the trial on its website.

MANDATORY SEC487 SYSTEM REQUIREMENTS:

  • CPU: 64-bit 2.0+ GHz processor or higher-based system is mandatory for this course (Important - Please Read: a 64-bit system processor is mandatory)
  • BIOS/UEFI: VT-x, AMD-V, or the equivalent must be enabled in the BIOS/UEFI
  • RAM: 8 GB (Gigabytes) of RAM or higher is mandatory for this course (Important - Please Read: 8 GB of RAM or higher is mandatory)
  • Wireless Ethernet 802.11 G/N/AC
  • USB 3.0 ports highly recommended
  • Disk: 25 gigabytes of free disk space
  • VMware Workstation 11, Workstation Player 7, or Fusion 7 (or newer)
  • Privileged access to the host operating system with the ability to disable security tools
  • A Linux virtual machine will be provided in class

If you have additional questions about the laptop specifications, please contact laptop_prep@sans.org.

A USB storage device with a custom Linux virtual machine containing software ready to conduct your own investigations

  • Understand the data collection life cycle
  • Create a secure platform for data collection
  • Analyze customer collection requirements
  • Capture and record data
  • Create sock puppet accounts
  • Create your own OSINT process
  • Harvesting web data
  • Perform searches for people
  • Access social media data
  • Assess a remote location using online cameras and maps
  • Examine geolocated social media
  • Research businesses
  • Use government-provided data
  • Collect data from the Dark Web
  • Leverage international sites and tools

"Great intro to OSINT" - Jason Adamson, CrowdStrike

"The application of OSINT is broad. This course provides opportunities to apply those to my day-to-day work" - Timothy DeBlock, Premise Health

Author Statement

"I have always been intrigued by the types and amount of data that are available on the Internet. From researching the best restaurants in a foreign town to watching people via video cameras, it all fascinates me. As the Internet evolved, more high-quality, real-time resources became available and every day was like a holiday, with new and wonderous tools and sites coming online and freely accessible.

"At a certain point, I was no longer in awe of the great resources on the web and, instead, transitioned to surprise that people would post images of themselves in illegal or compromising positions or that a user profile contained such explicit, detailed content. My wonder shifted to concern for these people. Didn't they know that their [profiles, images, videos, comments, etc.] were publicly accessible? Didn't they care about it? What I found was that, if you looked in the right places, you could find almost anything about a person, a network, or a company. Piecing together seemingly random pieces of data into meaningful stories became my passion and, ultimately, the reason for this course.

"I recognized that the barrier to performing excellent OSINT was not that there was no free data on the Internet. It was that there was too much data on the Internet. The challenge transitioned from 'how do I find something' to 'how do I find only what I need?' This course was born from this need to help others learn the tools and techniques to effectively gather and analyze OSINT data from the Internet."

- Micah Hoffman

*CPE/CMU credits not offered for the SelfStudy delivery method


1 Training Result
Type Topic Course / Location / Instructor Date Register

Training Event
Security
New
SANS London November 2018
London, United Kingdom
Nov 5, 2018 -
Nov 10, 2018
 

*Course contents may vary depending upon location, see specific event description for details.