Basic Open Source Intelligence Analysis

Importance of Search Techniques

• Understanding effective search techniques is crucial for efficient open-source intelligence (OSINT). Knowing how to filter and search alone is not sufficient. The amount of information available online is immeasurable, and search results are often personalized to your interests rather than tailored to your organization's needs or the relevance of your research.

• Analysts can use several techniques to conduct searches efficiently and produce accurate results. This is important because the questions you are helping to answer likely have an expiration date. At some point, the decision point your intelligence informs may pass, and that intelligence will no longer be valuable.

• Search operators, such as quotes for exact phrases, asterisks for wildcard expansions, and Boolean operators like AND, OR, and NOT, can significantly refine your search and reduce the time spent sifting through web results. So, let's go through some common search operators you can use to speed up information gathering.

Quotation Marks

• When you enclose a phrase in quotation marks, you are instructing the search engine to find that exact phrase. This significantly narrows down your search results to more relevant information.

• If you are searching for a specific report on data breaches in 2023, simply type "data breach report 2023". This focuses results on documents or pages containing the exact phrase "data breach report 2023". This is especially helpful when looking for specific titles, names, or quotes.

• Try searching for a phrase relevant to your work and note the difference in results when you use quotation marks versus a broad search.

Minus Sign

• The minus sign operator excludes specific words from your search results, helping to filter out irrelevant information.

• If you are looking for cybersecurity news but want to exclude results that require a subscription, you would type: cybersecurity news -subscription. This will remove any results that mention "subscription," giving you a cleaner set of results to work with.

• Perform a search using the minus sign to exclude unwanted terms and observe how it refines your results.

Asterisk

• The asterisk operator (*) serves as a wildcard, replacing one or more words in your search query. This is particularly useful when you are still determining the exact phrase.

• To find variations of phrases like "data breach * 2023," simply type: "data breach * 2023". This will display results such as "data breach incidents 2023" or "data breach statistics 2023."

• Try using an asterisk in a search query related to your field and observe the variety of results it returns.

Site

• The site operator limits your search to a specific website or domain, which helps you find information within a known, trusted source.

• For example, to find annual security reports on a UK government site, you would type site: gov.uk "annual security report". This ensures all results come from the .gov.uk domain, providing credible, official information.

• Choose a reputable website in your field and utilize the site: operator to locate specific information within that site.

Intitle

• The intitle operator allows you to search for specific words in the title of web pages or documents, making it easier to find relevant content.

• To find articles with "insider threat" in the title, you would type: intitle: "insider threat". This will show results where "insider threat" is prominently featured in the title, indicating a primary focus on that topic.

• Use the intitle: operator to find documents or pages containing your chosen keywords in the title.

Inurl

• The inurl operator allows you to search for specific words in the URL, which can help locate documents or sections within a website.

• To find reports related to financial fraud, use the search query inurl: report "financial fraud". This will return pages with "report" in the URL, likely detailed documents on financial fraud.

• Now, it's your turn to try using the inurl operator. See if you can find specific document types within a website using this tool.

Filetype

• The filetype: operator lets you search for specific file types, such as PDFs, Word documents, or Excel sheets.

• To find PDFs on incident response plans, you would type: filetype:pdf "incident response plan". This filters results to show only PDF documents, which are often more detailed and structured.

• Use the filetype: operator to find documents in your preferred format related to your field.

Related

• The related: operator helps you find websites related to a specified site, useful for discovering new sources of information.

• For example, to find sites related to BBC, you would type: related:bbc.com. This will show websites similar to BBC, potentially providing additional news sources or perspectives.

• Use the related: operator to discover new websites related to your key information sources.

Combining Operators

• Now that you are familiar with individual search operators, let's combine them for more powerful searches.

• To find PDF documents on UK government sites related to cyber security, you would type: site:gov.uk intitle:"cyber security" filetype: pdf -inurl: archive. This complex search query limits results to UK government sites, focuses on titles with "cyber security," filters for PDFs, and excludes archived pages.

• Create a complex search query relevant to your work using multiple operators and observe how effectively it narrows down your search results.

Leveraging Social Media Search Tools

• Social media platforms are rich sources of real-time information and relationship mapping. Mastering the use of different Social Media tools can significantly enhance your intelligence-gathering capabilities.

• One thing to note about social media regarding the ethical use of it as an OSINT tool is that it is not a good practice, or in some cases, a legal practice, to use fake social media profiles to collect OSINT. This is almost always against the platform's policies and, in some countries, an illegal invasion of privacy. Only use Social Media as a resource if you are collecting the information ethically and if the information is for public use.

X (formerly Twitter), Facebook, Instagram, and LinkedIn Basic Search

• Most social media platforms have a search bar that allows you to find posts containing specific words, hashtags, or phrases.

• Suppose you want to find posts related to a data breach. You would simply type: data breach. This search will return posts containing the words "data breach." You can refine your search by adding more keywords or hashtags.

Advanced Social Media Search Operators

• Most social media platforms support advanced search operators for more precise searches.

• If you're looking for a specific phrase, use quotation marks, as we learned in the last session. For example, “data breach".

• To search hashtags, use the hashtag symbol and the word. If you are searching for a phrase, do not add spaces between the words, for example, “#databreach".

• If you want to search posts from a specific user, search the user's handle, starting with the @symbol. For example, “@username”.

Specialized search tools for X and LinkedIn

• In X (formerly Twitter), you have powerful tools for targeted searches. You can conduct advanced searches through the “advanced search” link or by typing the search operator directly into the search bar.

• There are numerous search operators available in X, like the to:, from:, date range, geolocation, and filter:, operators. Let's go through some of the more common advanced search operators you might use in OSINT. You can also go to the link provided in the notes to see a full list from X.

• When looking for tweets from specific users, simply start with "to: or from:" followed by the @ symbol and their handle, like this: to:@elonmusk or from:@elonmusk.

• To refine your search in X, exclude specific words using a minus sign, such as data breach -report, especially when dealing with large data sets.

• Searching for specific date ranges in X is straightforward. You can use the since: and until: commands, like data breach since:2023-01-01 until:2023-12-31. This lets you pinpoint the exact time frame you need, making your search more precise and efficient.

• To search for tweets near a specific location, which can help identify whether a potential threat is nearby, enter the geocode: latitude, longitude, and the radius of your search (with no spaces between).

• X Pro subscribers have access to Tweetdeck, a useful platform for creating multiple search feeds. It includes the ability to create boolean and location-based searches. For more information on using Tweetdeck, please refer to additional training specifically on these platforms from Morton Executive Decisions.

• LinkedIn offers specialized tools like Sales Navigator, which allow users to narrow searches by user or company market. This information is readily available but may not be considered OSINT. Privacy laws vary by country. Make sure you use the information within the parameters of the platform's policies.

• Now is your chance to use advanced social media search operators to find specific information relevant to your field.

Mastering these techniques will streamline your searches and improve your intelligence gathering. Remember, this is an introductory course, and we are just scratching the surface of what is possible, especially with extensive data sets. We can cover numerous platforms in future courses.

Introduction to OSINT Platforms

• OSINT platforms provide tools to collect and analyze publicly available information. They help you uncover hidden connections, monitor emerging threats, and understand the broader context of security incidents.

• Many free tools are available that you may already know, like Google Earth, Maltego, The Harvester, and more. You can also subscribe to paid platforms like Palantir, Dataminr, or Factal.

Introduction to Entity Relationship Mapping

• One core function of many OSINT platforms is entity relationship mapping. This helps you visualize and analyze relationships between entities such as people, companies, and websites.

• Entity relationship mapping helps uncover hidden connections and understand complex networks.

• To investigate a specific individual, create an entity for that person, then explore connections to other entities such as associates, organizations, and online profiles.

• Numerous paid and free software tools are available for relationship mapping. Simple free tools like Canva or even Microsoft Word can depict relationships visually.

Advanced Relationship Analysis

• Advanced Relationship Analysis involves using sophisticated methods and tools to uncover and analyze the intricate connections between various entities. These entities can include individuals, organizations, domains, email addresses, phone numbers, social media profiles, and more.

• Key Features of Advanced Relationship Analysis include;

  • Transforms and Queries are pre-configured searches that extract specific types of information, such as email addresses from a domain, social media profiles related to a person, or associations between organizations.

  • Graphical Link Analysis visualizes connections between entities in a graphical format, like Excel. This helps us understand the network and identify critical nodes and relationships. It helps see how entities are interconnected and spot hidden connections.

  • Multi-source data Integration combines data from various sources to provide a comprehensive view. Sources include public records, social media, websites, and databases, enhancing the depth and accuracy of the analysis.

  • Pattern Recognition identifies recurring patterns and trends within the data. Patterns can indicate underlying structures or common behaviors among entities, which helps predict future activities or identify anomalous behavior.

  • Temporal Analysis analyzes the timing and sequence of events and how relationships evolve. This is crucial for identifying the chronology of interactions and key moments in the network.

• This level of analysis helps you identify key connections and potential security risks. To learn more about these advanced relationship features, sign up for our advanced analytical training courses.

• Check out the entity mapping exercise to test your ability to connect entities using OSINT.

Introduction to Real-Time Data Collection

• Another essential feature of OSINT platforms is real-time data collection. This allows you to monitor live information from various sources, such as news sites, social media, and public records.

• Real-time data collection helps you to stay informed about emerging threats and incidents as they occur. For instance, you can set up alerts for specific keywords or topics in programs like Feedly or Brand24 to receive real-time updates on relevant developments.

• You can also track hashtags on most social media platforms to see what is trending around a specific topic or follow other vetted social media users who are already sifting through and verifying data around the issues that are relevant to you.

  
  

  Analyzing Real-Time Data

• Analyzing real-time data involves filtering, categorizing, and interpreting the information collected to make informed decisions.

• For instance, you can utilize filters to organize data based on relevance, date, or source and classify it to recognize patterns or trends. In X, you can track a particular user account, filter it for keywords such as "attack" or "hate," and then filter that stream by the most recent posts, alerting you when someone from that account adds a post with the words attack or hate.

• Google Alerts allows you to monitor keywords, phrases, or data patterns across various open-source. You can limit how often you receive alerts and even create your own Really Simple Syndication or RSS feeds. RSS feeds can also be integrated into your websites and dashboards.

• Companies like Feedly allow you to categorize, monitor, and analyze articles and social media posts. They even allow you to create your own Boolean Logic filters and crawlers, ensuring the most relevant information pops up in your feeds.

• This analysis helps you understand the broader context of an incident and anticipate potential risks.

• Analyze a set of real-time data collected from your alerts and identify key insights.

Introduction to Data Aggregation

• OSINT platforms often offer tools for data aggregation, letting you compile information from multiple sources into a single, comprehensive view.

• Aggregating data helps you see the bigger picture and identify connections that might not be apparent from individual sources.

• For example, you can compile data on a specific individual from their social media accounts, mentions on news sites, and public records to create a unified profile.

• Let's dive a little deeper into the Techniques used to visualize data.

Data Visualization Techniques

• Data visualization is a powerful feature of OSINT platforms that helps you interpret complex data through graphical representations such as charts, graphs, and maps. Visualizing data makes it easier to identify patterns, trends, and anomalies. It also makes it easier to communicate your intelligence to decision-makers.

• A strong illustration of data visualization's role in OSINT is creating a heatmap of specific criminal actions in the United States. This tool visualizes crime-prone areas, giving decision-makers the information they need to mitigate risk effectively.

• Another use case is collecting and categorizing contacts from potential criminals' social media accounts to determine their center of gravity: the user connected to everyone. By downloading the names of all the friends from each account into a spreadsheet, you can sort and filter to determine the user with the most connections, giving you a lead on who might be running criminal operations for that group.

• Use the data visualization tools in the following exercise to create a graphical representation of your aggregated data.

Using these tools efficiently will amplify your OSINT efforts. Technology's speed is improving our ability to analyze data and find intelligence. It is a good practice to search for new tools often to ensure you are as effective as possible. Next, we will address the analytical skills you need to analyze the information you have collected.

Evaluating Source Reliability

• In open-source intelligence, the credibility of your information can significantly affect your analysis. Verification is a crucial step in intelligence, mainly when working with open-source information. It entails confirming the information's authenticity, accuracy, and reliability before utilizing it for decision-making.

• The access to unlimited information we enjoy today is great, but it can also cause issues when searching for answers to specific questions. Sifting through the data is only part of an analyst's job. Validating sources is also critical to ensuring we understand the information's accuracy.

Cross-Verification with Multiple Sources

• The most reliable method is cross-verification. Always confirm information with multiple independent sources. Check if other reputable sources report the same facts without relying on each other’s reports. This approach helps to avoid misinformation that might be widely circulated.

• When using news or journalists as sources, it is vital to look for quotes from their sources. Many news organizations repeat reporting from the Associated Press, for example, changing very little information. You might find fifteen separate articles or videos documenting the same story, all originating from the same source.

• Look for sources that validate each other without having collaborated. The stories should vary slightly due to differing perspectives. If the accounts are the same, the sources likely collaborated on the story.

Understanding Source Bias

• Evaluate the source's bias and reliability. Every source has its own perspective, and understanding these biases can help you assess the information's objectivity. Consider the source's history, ownership, political leanings, and past accuracy. Generally, sources with a track record of reliability are more trustworthy.

• Check your own biases multiple times throughout your analysis. The best way to do this is by asking for perspective from a trusted colleague you know does not always agree with you. Their analysis may differ significantly from yours, but you may see something you missed before and find answers somewhere in the middle.

• You often encounter biased information from news, media, and professionals who present extreme viewpoints to cater to their specific audience. Recognizing the target audience helps you understand the spin on the information. For example, there is a noticeable difference between Fox News and CNN. You might naturally reach for one or the other as a source. We all have biases. I make it a practice to seek out sources that I may not trust. Nonetheless, I can always find key points in the information that confirm my source information. I then choose to leave the biased information alone.

Expert Opinion and Peer Reviews

• Seeking expert opinions or peer reviews is another effective strategy. Experts in a particular field can provide insights into the accuracy of technical data and facts. Similarly, peer-reviewed information or data corroborated by academic or professional communities often carries greater credibility.

• Remember that just because someone claims to be an expert doesn't mean they are right. Validate your sources and the information they provide. This process is often more straightforward than you think. Most intelligence professionals conduct predictive analysis to identify potential hazards that could hinder a mission. You can assess a professional's track record to see how often they are correct.

Technical Verification Tools

• Remember to utilize technical tools designed explicitly for verification. These tools may include reverse image search tools to authenticate images, metadata analysis tools to examine the source and history of digital files, and fact-checking websites that can help verify or debunk claims and news stories.

• Fact-checking websites are often operated by people who also write articles for news sources. Many fact-checking resources have had to issue retractions or revise their "facts" later.

• Fact-checking requires thorough investigation, which can take time. If you're fact-checking new information, verify how it was fact-checked and examine the fact-checker's sources and evidence. Information fact-checked before an investigation is complete is not typically reliable.

• With the rise of AI and the deepfake industry, identifying information's authenticity will become increasingly difficult. As validating information becomes more complex, new tools for verifying information will hopefully be created.

Adequate verification is crucial for ensuring the reliability of your OSINT. It protects your projects from misinformation and improves the quality of your intelligence. With these analytical skills, you can turn raw data into valuable intelligence and answer critical questions. If you want more comprehensive training in source identification and validation, consider enrolling in our advanced analysis courses.

Introduction to Critical Thinking

• Critical thinking involves several vital skills: Analysis, Evaluation, Inference, and Explanation. Analysis means examining information in detail by breaking it down into smaller parts. Evaluation involves assessing the credibility and usefulness of information. Inference is about drawing conclusions based on evidence and reasoning, and Explanation entails communicating your findings.

• Critical thinking is vital in OSINT for several reasons. It helps us avoid bias, ensures data accuracy, and supports practical interpretation. Without it, we risk making decisions based on flawed or incomplete information, which can seriously affect security contexts.

Recognizing and Mitigating Biases

• Let's examine biases, which are systematic errors in thinking that affect judgments and decisions. We don't want mistakes in our analysis; recognizing these biases is crucial in OSINT work.

• Common biases include Confirmation Bias, where we favor information that confirms our existing beliefs; the Availability Heuristic, where we overemphasize readily available or recent information; and Anchoring Bias, where we rely too heavily on the first piece of information we see.

• To mitigate these biases, practice challenging your assumptions. Seek out contradictory information and consider multiple perspectives before forming conclusions. Use structured analytical techniques to ensure a balanced approach to information gathering and analysis.

Questioning Techniques

• Effective questioning is a powerful OSINT tool. It guides how we gather data and interpret it. After all, we are trying to answer questions.

• Use Open-ended Questions to explore data more deeply. These questions encourage detailed responses and insights. Closed Questions help confirm specific details or yes/no answers, but they leave very little room to expound on.

• Socratic questioning challenges assumptions and explores the implications of the information. Questions like 'What is the evidence for this claim?' or 'What might be an alternative explanation?' help deepen understanding and prevent superficial analysis.

Synthesizing Information

• Synthesizing information is a critical step in OSINT Analysis. This involves integrating various pieces of information to form a coherent understanding. Use logical reasoning to connect disparate data points and look for patterns and trends that help build a solid narrative or conclusion.

• The most critical part of analysis is delivering your conclusions in a way the audience can digest. You must understand your audience and how they prefer to receive information. For example, if you asked an associate at Amazon how their executives like to receive information, they might tell you to present it as a six-page white paper in a specific format, with no PowerPoint allowed.

• On the other hand, if you were presenting to a general in the US Army, they might expect you to use PowerPoint formatted in a specific way, with their unit logo in the corner of each slide. A busy executive might require you to condense your research from 150 pages (not including graphics) to a single page.

• Analysts' work often results in masterpieces. However, being a good analyst requires humility and an understanding that we are not the ones making tough decisions all day. While our work is essential, it may not always be at the top of a decision-maker's priority list.

• To ensure your hard work pays off, make it easy to understand and straight to the point. Put the bottom line up front, with your research easily accessible for the decision-maker's questions and answers. Your success is measured in organizational growth and the safety of your colleagues and customers.

Analytical Techniques for Critical Thinking

• Understanding Deductive and Inductive Reasoning is key. Deductive reasoning starts with a general statement and reaches a specific conclusion, while inductive reasoning takes specific data and extrapolates a general rule.

• An example of Deductive Reasoning might be: All birds have feathers. A robin is a bird, so a robin has feathers.

• An example of Inductive Reasoning might sound like this: You observe that a swan on a lake is white. You see another swan on a different lake, and it is also white. A friend reports seeing only white swans at a nearby park. Therefore, based on that specific data, your general conclusion is that all swans are white.

• Critical reading and listening involve evaluating the information’s source, context, and content. A good analyst must distinguish between facts (verifiable), opinions (personal views), and assumptions (taken for granted without proof).

• These critical thinking skills strengthen your OSINT capabilities and help ensure your conclusions are well-founded and reliable.

• There is a ton of information available on critical thinking. This is a basic overview, but we will include some of our favorite resources in the notes section for you to review at your own pace.

• If you have any questions about critical thinking, please let us know in the chat, and we will do our best to answer them promptly.

Critical thinking and the ability to effectively disseminate analytical products enhance good intelligence. They help us navigate misinformation and develop insights that are not only informative but also actionable. These crucial skills will ensure that your hard work is implemented and your reputation as a trustworthy analyst grows.

Understanding the legal landscape of OSINT is essential to ensure compliance and avoid legal pitfalls. This session will cover privacy laws, intellectual property rights, and specific regulations that affect OSINT practices.

Overview of Relevant Laws

• Different countries and even states maintain specific Privacy Laws. A few high-profile examples include;

  • General Data Protection Regulation (GDPR): Regulates the processing of individuals' personal data within the EU.

  • California Consumer Privacy Act (CCPA): Provides privacy rights to consumers in California.

  • Health Insurance Portability and Accountability Act (HIPAA): Protects sensitive patient information in the U.S.

• There are also numerous Intellectual Property Rights policies to navigate when using OSINT:

  • Copyright Laws: Protect the use of creative works such as articles, images, and videos.

  • Trademark Laws: Protect brand names, logos, and slogans from unauthorized use.

• There are also Specific Regulations concerning OSINT:

  • Computer Fraud and Abuse Act (CFAA): U.S. legislation governing unauthorized computer and network access.

  • Electronic Communications Privacy Act (ECPA): This act protects wire, oral, and electronic communications while they are being made, in transit, and stored on computers.

• These are just a few examples of the many regulations and policies to consider when collecting and utilizing OSINT for business and government purposes. We have provided links to each of these in the notes so you can dive in on your own.

Types of Legally Assessable OSINT

• Numerous sources of information are available for OSINT use, but which are legally permitted? How do you know when you might be crossing a line?

  • Publicly Available information is legally accessible and does not require special permission. Typically, there is no expectation of privacy. You can determine this by checking source data, citations, policies on open-source databases, and tools that list licensing requirements. Be aware that just because it's on the open net does not mean it was shared with permission.

  • Public Records are government records that are legally available to the public, such as court records, offender registries, and property deeds. Typically, government databases list requirements for using the information in their terms of service.

  • Publicly accessible Social Media posts and profiles that do not require a login, connection, or special access are typically legal to use in OSINT. Keep in mind that just because someone makes you a friend on Facebook and gives you access, they are not giving up their right to privacy.

Potential Legal Violations

• Accessing private information or systems without permission or unauthorized access is unacceptable and generally illegal. This falls under hacking and can result in criminal charges.

• Data harvesting and scraping, which involve automated data collection from websites, may violate terms of service or laws. If you use scraping software, you must know where the data you collect comes from. Review the website's terms of service and the privacy laws of the country where the organization that owns the website is based.

• Information gathered in a way that violates an individual's reasonable expectation of privacy is considered an invasion of privacy. Collecting large data sets this way can lead to numerous violations, and verifying sources can be challenging. Therefore, narrowing your search as much as possible and identifying the specific source for that information is crucial to minimize violations.

Staying within legal boundaries is crucial for ethical OSINT practice. Next, we'll delve deeper into the ethical aspects and how to handle ethical dilemmas.

Developing an Ethical Framework

• Ethics in OSINT are not just about legality. It's about integrity and responsibility. Establishing an ethical framework ensures your intelligence gathering is respectful, responsible, and trustworthy. This is a good practice for you as an intelligence professional and your organization.

• Establishing components for an ethical framework is a good idea to ensure that you operate in a manner that promotes continuity and compliance for your organization and sets you up for long-term success as a professional.

Principles of an Ethical Framework for OSINT

• Embracing professionalism is crucial. This means upholding professional standards and codes of conduct, committing to continuous learning, and following OSINT best practices.

• Here are some examples of principles that you may consider integrating into your Ethical Framework.

  • Conducting your work with integrity means always acting honestly and transparently in your OSINT activities. Ensure that your collection and analysis methods, as well as your sources, are reliable and verifiable.

  • Confidentiality means protecting the privacy of individuals and organizations. Avoid intruding on individuals’ privacy unless absolutely necessary and legally permissible. This shows that you respect others' privacy, promotes professionalism, and strengthens your reputation as a trustworthy analyst.

  • Embrace accountability for your decisions and conduct. Maintain thorough documentation of your processes and be confident in articulating your methods.

Maintaining your Ethical Framework for OSINT

• Developing a framework is great, but you must implement and maintain it. Here are some ways that you can do just that.

• Develop clear guidelines for ethical OSINT practices and attach key performance indicators to ensure compliance.

• Seek regular training and update your guidelines to ensure you understand and follow the ethical framework.

• Continuously monitor and evaluate your practices to ensure compliance with ethical standards. Check yourself regularly.

• As an individual, you are ultimately responsible for collecting and analyzing OSINT. Your organization may share some accountability, but generally, the analyst bears the brunt of any policy or legal violations and potential consequences.

We’ve covered a lot in this course, from OSINT basics to advanced tools and ethical considerations. Remember, the open-source intelligence landscape is constantly evolving, and continuous learning is critical. We plan on updating this course as OSINT evolves and recommend you check back occasionally for changes. In the meantime, feel free to submit any questions about this course or sign up for our virtual and in-person training through Morton Executive Decisions.

Thank you for all you do to keep people safe and ensure organizations continue to grow and prosper.

Adhering to ethical standards is vital for maintaining the trustworthiness and reliability of your intelligence outputs. Ethical OSINT practices not only protect the rights and privacy of individuals but also ensure the integrity and credibility of your work.