Discover new courses and celebrate learning with us today. 🎓

Development

Web Development

Information Retrieval and Mining Massive Data Sets

Master the techniques to build a Google-scale Information Retrieval System and explore data mining algorithms for big data challenges.

4.9

315,475 rating
11 Lessons
229 Students
Last updated 4 months ago

By Mentors Net

By Omkar Deshpande

via Udemy

Go To Course

Brief Summary

This course dives into the fun world of building an Information Retrieval System, much like what Google uses! You'll pick up essential techniques for tackling big data and learn to analyze massive data sets with cool mining algorithms. Awesome, right?

Key Points

Learn how to build a Google-like Information Retrieval System.
Explore techniques for handling big data problems.
Evaluate different solutions and their trade-offs.
Understand data mining algorithms for big data.
The course is broken down into 6 parts for easier learning.

Learning Outcomes

Gain practical skills in building an IR System.
Identify effective techniques for big data solutions.
Evaluate the trade-offs of different approaches.
Understand and apply data mining algorithms.
Work through real-world examples to boost your learning.

About This Course

Learn various techniques to build a Google scale Information Retrieval System.

The goal is to introduce various techniques required to build an IR System. In this course we will explore various methods to solve big data problem. We will evaluate alternative solutions and trade offs. In the later part of the course we will discuss various data mining algorithms to make sense of massive data sets.

The course is primarily divided into 6 parts.
Part 1: Building an Information Retrieval System
Part 2: Mining Frequent Patterns and Associations

Course Curriculum

Introduction To a Boolean Search Engine

15 Lectures

What is Data Mining

Structured Data, Unstructured data and Information Retrieval

Term-Document Incidence Matrix (1)

Term-Document Incidence Matrix (2)

Inverted Index

Tradeoffs in implementing an Inverted Index

Processing AND, OR, NOT queries

Overview of Index Construction Pipeline

Query optimization using Document Frequency (1)

Query Optimization Using Document Frequency (2)

Boolean Retrieval Model

Example of a Boolean Retrieval Model

Limitations of Boolean Retrieval Model

How to evaluate performance of an IR System

Google zeitgeist

Dictionary Data Structure. Tolerant retrieval

8 Lectures

Parsing Documents and Issues Associated with it

Tokenization Process in an IR System

Normalization to Terms

Faster Postings Merges With Skip Pointers

How to Handle Phrase Query

Phrase Query Using Positional Index

How to handle proximity query

Discussion on Positional Index Size

Index construction. Postings size estimation, sort-based indexing, dynamic index

15 Lectures

Dictionary Data Structure Implementation

Wild card queries

Questions on Wild Card Queries

Wild Card Query Handling Using Permuterm Index

Wild Card Query Handling Using K-Gram Index

Soundex Algorithm

Spelling Correction Techniques in an IR System

Question On Soundex Algorithm

Spelling Correction (Part 2)

Introduction To Dynamic Programming

How To Calculate Edit Distance Between Two Strings

Spelling Correction Using Weighted Edit Distance

Spelling Correction Using Ngram Overlap Technique

Calculating Jaccard Coefficient (An Example)

Context Sensitive Spell Correction

Dictionary Compression, Posting Compression

11 Lectures

Introduction to Index Construction

Index Construction Using InMemory Sorting

Index Construction Using BSBI Algorithm

Index Construction Using SPIMI Algorithm

Introduction To Distributed Indexing

How To build distributed indexes

Q & A on Distributed Index

Map Reduce

Dynamic indexing using naive approach

Dynamic indexing using logarithimic merge

Issues With Multiple Indexes

Scoring, term weighting, and the vector space model

5 Lectures

Why do we compress indexes

Important Statistics about RCV Collection

Various Dictionary Compression Techniques

Various Dictionary Compression Techniques Part 2

Various Posting Compression Techniques

Efficient vector space scoring. Nearest neighbor techniques

10 Lectures

Ranked Retrieval Model

Jaccard Score

Term Frequency Weighing And Bag Of Words Model

Inverse Document Frequency

TF-IDF Score

Documents AS TF-IDF Vectors

Length Normalization

Cosine Similarity Example

Computing Cosine Scores On Index

Variants of TF IDF Weights

Evaluating search engines. User happiness, precision, recall, F-measure

12 Lectures

Term at a Time Scoring

Efficient Cosine Ranking

Generic Approach For Speeding up Cosine Similarity

Index Elimination

Champion Lists

Static Quality Score

High And Low Lists

Impact Ordered Posting

Cluster Pruning

Parametric Zone Tired Index

Query Term Proximity And Query Parsing

How A Search Engine Works

Advertisement Systen. Google AdSense. Search Engine Optimization

5 Lectures

Performance of a Search Engine Part 1

Performance of a Search Engine Part 2

Performance of a Search Engine Part 3

Performance of a Search Engine Part 4

Performance of a Search Engine Part 5

Supervised Learning. Text Classification. Naive-Bayes Text Classification

4 Lectures

ECommerce Vs. Traditional Businesses

Pricing Models For Online Advertisement

AdWords and AdSense

SEM And SEO

Link analysis. Web as a graph. PageRank

9 Lectures

Classification System

Document Classification

Manual Classification Methods

Naive Bayes Classifiers

Bayes Rules Of Text Classification

Various Classification Methods

Example of Multivariate Bernouli Model

Second Version of Naive Bayes

Example of Second Version of Naive Bayes

Clustering. Introduction to the problem. Partitioning methods: k-means clusterin

4 Lectures

Reputation System

Examples of Reputation System

Limitations of Reputation System

Page Rank Calculation

Web Crawler

11 Lectures

What is Clustering

Applications of Clustering in IR Systems

Issues For Clustering

Introduction to Clustering Algorithms

K-Means Clustering Algorithms

Rocchio Algorithms

K Nearest Neighbor Algorithms

Discussion on K Nearest Neighbor

Proof of Rocchio's Algorithm as linear classifier

Worked out Example On Rocchio Algorithms

Examples On Bigram Index

Association Rules. Market Basket Model and Frequent Item Sets. A Priori Algorith

10 Lectures

How a Web Crawler Works

Complications in Crawling

Advance Crawler Architecture

URL Frontier

Association Rule Introduction

Market Basket Model and Frequent Item Sets

A formal approach to Association Rules

How to find association Rules

Storage Considerations for Market Basket

Memory Bottleneck in Storage of Market Basket

A Naive Algorithm to discover Association Rules Part1

A Naive Algorithm to discover Association Rules Part2

A Priori Algorithm

Extension of A Priori Algorithm

Instructors

Mentors Net

4.9

315,475 Reviews
345 Students
34 Course

We bring highly qualified network of mentors who are passionate about developing next generation of engineers. Our professionals are from top US universities like Stanford UC Berkeley and with significant industry experience with companies like IBM, Google, Cisco and VMware. They bring to the table invaluable real time experience in bite-sized video nuggets. Their services make our focused, role based...

Instructors

Omkar Deshpande

4.9

315,475 Reviews
345 Students
34 Course

I got my Bachelors in Computer Science and Engineering from the Indian Institute of Technology, Delhi in 2002, followed by Masters (2005) and Ph.D (2008) in Computer Science from Stanford University, where I was a Stanford Graduate Fellow. I was a part of the Batzoglou lab, and I worked with Luca Cavalli-Sforza and Marc Feldman on using simulations to build...

More Courses By Mentors Net

HP Quick Test Professional

4.9

(230)

Web Application Performance Testing

4.9

(230)

Load Testing Using HP LoadRunner 12

4.9

(230)

Review

4.9 course rating

4K ratings

Gaurav M.

4.5

9 months ago

Yes, great match indeed.

Helpful
Not helpful

D I.

5.0

1 year ago

this is a very important and easy way to enhance our knowledge. expected to update causes with real time. specially in AI and data science area

Helpful
Not helpful

Lakshay A.

1.0

1 year ago

Too much theory

Helpful
Not helpful

Miqdad

5.0

2 years ago

Course is quite lengthy but worth spending time and money. This course is all about theory of Information Retrieval, machine learning algorithm; supervised and unsupervised. Each concept is explained with easy to understand examples.

Helpful
Not helpful

Jaime A. G. L.

3.0

2 years ago

Very slowly its development

Helpful
Not helpful

Savanna R. E.

4.5

3 years ago

So Far the Information has Been Useful

Helpful
Not helpful

Kunja M.

4.5

3 years ago

Pros:

- Good knowledge and communication skills
- Good Presentation of facts
- Deep Dive into Concepts

Cons :

- Recorded Skype Call
- Missing Video in between for Page Rank
- Section to Video mapping is wrong.

Overall worth it and great course ! Cheers !

Helpful
Not helpful

Kannan S.

5.0

4 years ago

very nice course.

Helpful
Not helpful

Farzad S.

2.0

4 years ago

Difficult to follow. The teacher does the job by using slides. He draws messy diagrams which make it really difficult to follow.

Helpful
Not helpful

Sanjeev B.

4.0

5 years ago

The pace of explanation could have been faster

Helpful
Not helpful

Ratings

This course includes:

54.5 hours on-demand video
3 articles
249 downloadable resources
Access on mobile and TV
Full lifetime access
Certificate of completion

Courses You May Like

Lorem ipsum dolor sit amet elit

Show More Courses

Become a Certified Web Developer: HTML, CSS and JavaScript

4.9

(230)

By: Carolyn Welborn

Information Retrieval and Mining Massive Data Sets

Brief Summary

Key Points

Learning Outcomes

About This Course

Course Curriculum