Text Data Processing and Analysis Using Regular Expressions

The project focuses on harnessing the power of Regex to extract, preprocess, and analyze valuable information from raw data. Develop a comprehensive text data processing pipeline that utilizes regular expressions to achieve the following goals: Clean and preprocess text data. Extract structured information and pattern recognition. The project will involve Python programming and the use of a library as 're' for regular expression.

Regular Expression

Importing regex library for Python

import re

Find all numbers in the file

Use regular expression to find all numbers contain in the text.


                                numbers = re.findall(r'\d+', contents)

Output:

['3036', '7209', '4497', '6702', '8454', '7449', '3665', '7936', '9772', '7114', '956', '2564', '8003', '1704', '3816', '6662', '5858', '7777', '6482', '9634', '8805', '7123', '9703', '4676', '6373', '2834', '7221', '2981', '5415', '6872', '4806', '5460', '8533', '3538', '9663', '8001', '9795', '8752', '1117', '5349', '4509', '1004', '9258', '4183', '4034', '3342', '3482', '8567', '1052', '8135', '5561', '517', '1218', '4', '8877', '8062', '1720', '279', '2054', '801', '918', '8687', '7073', '1865', '7084', '2923', '63', '8824', '1079', '5801', '5047', '5', '2572', '5616', '171', '3062', '9552', '7655', '829', '6096', '2312', '6015', '7100', '9548', '2727', '1792', '8402', '4', '3', '42']

The line only contains a number

For each line use a regular expression to find if this line is a number.

re.match(r'^\d+$', line.strip()):

Output:

['7114', '9634', '2834', '4806', '5', '42']

Searching ULR with regex.

Searching in text-only that contains URL, the word should begin with www. / https or http.

url_pattern = r'\b(?:https?://|www\.)\S+\b'
matches = re.finditer(url_pattern, contents)

Output:

['www.py4e.com', 'www.python.org', 'http://www.py4e.com/code3']

Starts with function

For each line, find if starts with a pattern, using rstrip and startswith methods.

line.rstrip()
if line.startswith("Terminology:"):

Output:

['Terminology: interpreter and compiler\n']

Show Code

views.py Code

def regex(request):
    search_line = "def regex(request):"
    pattern = "def file_analytic(request):"

    try:
        with open(__file__, 'r') as file:   # este módulo se documenta a sí mismo
            content = file.readlines()
            index = None

            for i, line in enumerate(content):
                if search_line in line and index is None:
                    index = i

                if line.startswith(pattern) and index is not None:
                    pos = i

        with open(os.path.join(static_dir, 'json/data-science/content.json'), 'r') as file:
            content_json = file.read()

        content_raw = json.loads(content_json)
        content_filter = {key: value for key, value in content_raw.items() if key.startswith("regex")}
    except FileNotFoundError:
        result = "The file could not be found."

    views = content[index: pos]
    try:
        file_path = os.path.join(static_dir, 'datasets/Regex sample file.txt')
        # Open and read the text file
        with open(file_path, 'r') as file:
            contents = file.read()

        with open(file_path, 'r') as file:
            contents_by_line = file.readlines()

        # Use regular expression to find all numbers
        numbers = re.findall(r'\d+', contents)
        # Use regular expression to find all line to contain only a number.
        line_with_digits = []
        for line in contents_by_line:
            # Use regular expression to check if the line is a number
            if re.match(r'^\d+$', line.strip()):
                line_with_digits.append(line.strip())

        # Define Regex expresion
        url_pattern = r'\b(?:https?://|www\.)\S+\b'

        # Search in text to find all the URL that begin with www. / https or http.
        found_urls = []
        found_urls_with_startwish = []

        matches = re.finditer(url_pattern, contents)
        for match in matches:
            found_urls.append(match.group(0))

        # Find line that begin with some text
        for line in contents_by_line:
            line.rstrip()
            if line.startswith("Terminology:"):
                found_urls_with_startwish.append(line)

        context = {'views': views, 'content': content_filter, "e1": numbers,
                   'e2': line_with_digits, 'e3': found_urls, 'e4': found_urls_with_startwish}

        return render(request, 'data-science/regex/index.html', context)
    except FileNotFoundError:
        context = {'description': 'File not found. Please check the file path or create the file if it does not',
                   'reference': "Books Store connection from XML File"}
        return render(request, 'data-science/retrieve_error.html', context)
    except TemplateDoesNotExist:
        context = {'description': 'Template not found. Please check the file path or create the file if it does not',
                   'reference': "Books Store connection from XML File"}
        return render(request, 'data-science/retrieve_error.html', context)